chore: import upstream snapshot with attribution
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,82 @@
|
||||
---
|
||||
title: "AstraDocumentStore"
|
||||
id: astradocumentstore
|
||||
slug: "/astradocumentstore"
|
||||
---
|
||||
|
||||
# AstraDocumentStore
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [Astra](/reference/integrations-astra) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/astra |
|
||||
|
||||
</div>
|
||||
|
||||
DataStax Astra DB is a serverless vector database built on Apache Cassandra, and it supports vector-based search and auto-scaling. You can deploy it on AWS, GCP, or Azure and easily expand to one or more regions within those clouds for multi-region availability, low latency data access, data sovereignty, and to avoid cloud vendor lock-in. For more information, see the [DataStax documentation](https://docs.datastax.com/en/home/docs/index.html).
|
||||
|
||||
### Initialization
|
||||
|
||||
Once you have an AstraDB account and have created a database, install the `astra-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install astra-haystack
|
||||
```
|
||||
|
||||
From the configuration in AstraDB’s web UI, you need the database ID and a generated token.
|
||||
|
||||
You will additionally need a collection name and a namespace. When you create the collection name, you also need to set the embedding dimensions and the similarity metric. The namespace organizes data in a database and is called a keyspace in Apache Cassandra.
|
||||
|
||||
Then, in Haystack, initialize an `AstraDocumentStore` object that’s connected to the AstraDB instance, and write documents to it.
|
||||
|
||||
We strongly encourage passing authentication data through environment variables: make sure to populate the environment variables `ASTRA_DB_API_ENDPOINT` and `ASTRA_DB_APPLICATION_TOKEN` before running the following example.
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.document_stores.astra import AstraDocumentStore
|
||||
|
||||
document_store = AstraDocumentStore()
|
||||
|
||||
document_store.write_documents(
|
||||
[Document(content="This is first"), Document(content="This is second")],
|
||||
)
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
### Supported Retrievers
|
||||
|
||||
[AstraEmbeddingRetriever](../pipeline-components/retrievers/astraretriever.mdx): An embedding-based Retriever that fetches documents from the Document Store based on a query embedding provided to the Retriever.
|
||||
|
||||
### Indexing Warnings
|
||||
|
||||
When you create an Astra DB Document Store, you might see one of these warnings:
|
||||
|
||||
> Astra DB collection `...` is detected as having indexing turned on for all fields (either created manually or by older versions of this plugin). This implies stricter limitations on the amount of text each string in a document can store. Consider indexing anew on a fresh collection to be able to store longer texts.
|
||||
|
||||
Or:
|
||||
|
||||
> Astra DB collection `...` is detected as having the following indexing policy: `{...}`. This does not match the requested indexing policy for this object: `{...}`. In particular, there may be stricter limitations on the amount of text each string in a document can store. Consider indexing anew on a fresh collection to be able to store longer texts.
|
||||
|
||||
#### Why You See This Warning
|
||||
|
||||
The collection already exists and is configured to [index all fields for search](https://docs.datastax.com/en/astra-db-serverless/api-reference/collections.html#the-indexing-option), possibly because you created it earlier or an older plugin did. When Haystack tries to create the collection, it applies an indexing policy optimized for your intended use. This policy lets you store longer texts and avoids indexing fields you won’t filter on, which also reduces write overhead.
|
||||
|
||||
#### Common Causes
|
||||
|
||||
1. You created the collection outside Haystack (for example, in the Astra UI or with AstraPy’s `Database.create_collection()`).
|
||||
2. You created the collection with an older version of the plugin.
|
||||
|
||||
#### Impact
|
||||
|
||||
This is only a warning. Your application keeps running unless you try to store very long text fields. If you do, Astra DB returns an indexing error.
|
||||
|
||||
#### Solutions
|
||||
|
||||
- **Recommended:** _Drop and recreate the collection_ if you can repopulate it. Then rerun your Haystack application so it creates the collection with the optimized indexing policy.
|
||||
- _Ignore the warning_ if you’re sure you won’t store very long text fields.
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Using AstraDB as a data store in your Haystack pipelines](https://haystack.deepset.ai/cookbook/astradb_haystack_integration)
|
||||
+70
@@ -0,0 +1,70 @@
|
||||
---
|
||||
title: "AzureAISearchDocumentStore"
|
||||
id: azureaisearchdocumentstore
|
||||
slug: "/azureaisearchdocumentstore"
|
||||
description: "A Document Store for storing and retrieval from Azure AI Search Index."
|
||||
---
|
||||
|
||||
# AzureAISearchDocumentStore
|
||||
|
||||
A Document Store for storing and retrieval from Azure AI Search Index.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **API reference** | [Azure AI Search](/reference/integrations-azure_ai_search) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/azure_ai_search |
|
||||
|
||||
</div>
|
||||
|
||||
[Azure AI Search](https://learn.microsoft.com/en-us/azure/search/search-what-is-azure-search) is an enterprise-ready search and retrieval system to build RAG-based applications on Azure, with native LLM integrations.
|
||||
|
||||
`AzureAISearchDocumentStore` supports semantic reranking and metadata/content filtering. The Document Store is useful for various tasks such as generating knowledge base insights (catalog or document search), information discovery (data exploration), RAG, and automation.
|
||||
|
||||
### Initialization
|
||||
|
||||
This integration requires you to have an active Azure subscription with a deployed [Azure AI Search](https://azure.microsoft.com/en-us/products/ai-services/ai-search) service.
|
||||
|
||||
Once you have the subscription, install the `azure-ai-search-haystack` integration:
|
||||
|
||||
```python
|
||||
pip install azure-ai-search-haystack
|
||||
```
|
||||
|
||||
To use the `AzureAISearchDocumentStore`, you need to provide a search service endpoint as an `AZURE_AI_SEARCH_ENDPOINT` and an API key as `AZURE_AI_SEARCH_API_KEY` for authentication. If the API key is not provided, the `DefaultAzureCredential` will attempt to authenticate you through the browser.
|
||||
|
||||
During initialization the Document Store will either retrieve the existing search index for the given `index_name` or create a new one if it doesn't already exist. Note that one of the limitations of `AzureAISearchDocumentStore` is that the fields of the Azure search index cannot be modified through the API after creation. Therefore, any additional fields beyond the default ones must be provided as `metadata_fields` during the Document Store's initialization. However, if needed, [Azure AI portal](https://azure.microsoft.com/) can be used to modify the fields without deleting the index.
|
||||
|
||||
It is recommended to pass authentication data through `AZURE_AI_SEARCH_API_KEY` and `AZURE_AI_SEARCH_ENDPOINT` before running the following example.
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.azure_ai_search import (
|
||||
AzureAISearchDocumentStore,
|
||||
)
|
||||
from haystack import Document
|
||||
|
||||
document_store = AzureAISearchDocumentStore(index_name="haystack-docs")
|
||||
document_store.write_documents(
|
||||
[
|
||||
Document(content="This is the first document."),
|
||||
Document(content="This is the second document."),
|
||||
],
|
||||
)
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
:::info[Latency Notice]
|
||||
|
||||
Due to Azure search index latency, the document count returned in the example might be zero if executed immediately. To ensure accurate results, be mindful of this latency when retrieving documents from the search index.
|
||||
:::
|
||||
|
||||
You can enable semantic reranking in `AzureAISearchDocumentStore` by providing [SemanticSearch](https://learn.microsoft.com/en-us/python/api/azure-search-documents/azure.search.documents.indexes.models.semanticsearch?view=azure-python) configuration in `index_creation_kwargs` during initialization and calling it from one of the Retrievers. For more information, refer to the [Azure AI tutorial](https://learn.microsoft.com/en-us/azure/search/search-get-started-semantic) on this feature.
|
||||
|
||||
### Supported Retrievers
|
||||
|
||||
The Haystack Azure AI Search integration includes three Retriever components. Each Retriever leverages the Azure AI Search API and you can select the one that best suits your pipeline:
|
||||
|
||||
- [`AzureAISearchEmbeddingRetriever`](../pipeline-components/retrievers/azureaisearchembeddingretriever.mdx): This Retriever accepts the embeddings of a single query as input and returns a list of matching documents. The query must be embedded beforehand, which can be done using an [Embedder](../pipeline-components/embedders.mdx) component.
|
||||
- [`AzureAISearchBM25Retriever`](../pipeline-components/retrievers/azureaisearchbm25retriever.mdx): A keyword-based Retriever that retrieves documents matching a query from the Azure AI Search index.
|
||||
- [`AzureAISearchHybridRetriever`](../pipeline-components/retrievers/azureaisearchhybridretriever.mdx): This Retriever combines embedding-based retrieval and keyword search to find matching documents in the search index to get more relevant results.
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
title: "ChromaDocumentStore"
|
||||
id: chromadocumentstore
|
||||
slug: "/chromadocumentstore"
|
||||
---
|
||||
|
||||
# ChromaDocumentStore
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [Chroma](/reference/integrations-chroma) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/chroma |
|
||||
|
||||
</div>
|
||||
|
||||
[Chroma](https://docs.trychroma.com/) is an open source vector database capable of storing collections of documents along with their metadata, creating embeddings for documents and queries, and searching the collections filtering by document metadata or content. Additionally, Chroma supports multi-modal embedding functions.
|
||||
|
||||
Chroma can be used in-memory, as an embedded database, or in a client-server fashion. When running in-memory, Chroma can still keep its contents on disk across different sessions. This allows users to quickly put together prototypes using the in-memory version and later move to production, where the client-server version is deployed.
|
||||
|
||||
## Initialization
|
||||
|
||||
First, install the Chroma integration, which will install Haystack and Chroma if they are not already present. The following command is all you need to start:
|
||||
|
||||
```shell
|
||||
pip install chroma-haystack
|
||||
```
|
||||
|
||||
To store data in Chroma, create a `ChromaDocumentStore` instance and write documents with:
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.chroma import ChromaDocumentStore
|
||||
from haystack import Document
|
||||
|
||||
document_store = ChromaDocumentStore()
|
||||
document_store.write_documents(
|
||||
[
|
||||
Document(content="This is the first document."),
|
||||
Document(content="This is the second document."),
|
||||
],
|
||||
)
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
In this case, since we didn’t pass any embeddings along with our documents, Chroma will create them for us using its [default embedding function](https://docs.trychroma.com/embeddings#default-all-minilm-l6-v2).
|
||||
|
||||
### Connection Options
|
||||
|
||||
1. **In-Memory Mode (Local)**: Chroma can be set up as a local Document Store for fast and lightweight usage. You can use this option during development or small-scale experiments. Set up a local in-memory instance of `ChromaDocumentStore` like this:
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.chroma import ChromaDocumentStore
|
||||
|
||||
document_store = ChromaDocumentStore()
|
||||
```
|
||||
2. **Persistent Storage**: If you need to retain the documents between sessions, Chroma supports persistent storage by specifying a path to store data on disk:
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.chroma import ChromaDocumentStore
|
||||
|
||||
document_store = ChromaDocumentStore(persist_path="your_directory_path")
|
||||
```
|
||||
3. **Remote Connection**: You can connect to a remote Chroma database through HTTP. This is suitable for distributed setups where multiple clients might interact with the same remote Chroma instance.
|
||||
|
||||
Note that this option is incompatible with in-memory or persistent storage modes.
|
||||
|
||||
First, start a Chroma server:
|
||||
|
||||
```shell
|
||||
chroma run --path /db_path
|
||||
```
|
||||
|
||||
Or using docker:
|
||||
|
||||
```shell
|
||||
docker run -p 8000:8000 chromadb/chroma
|
||||
```
|
||||
|
||||
Then, initialize the Document Store with `host` and `port` parameters:
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.chroma import ChromaDocumentStore
|
||||
|
||||
document_store = ChromaDocumentStore(host="localhost", port="8000")
|
||||
```
|
||||
|
||||
## Supported Retrievers
|
||||
|
||||
The Haystack Chroma integration comes with three Retriever components. They all rely on the Chroma [query API](https://docs.trychroma.com/reference/Collection#query), but they have different inputs and outputs so that you can pick the one that best fits your pipeline:
|
||||
|
||||
- [`ChromaQueryTextRetriever`](../pipeline-components/retrievers/chromaqueryretriever.mdx): This Retriever takes a plain-text query string in input and returns a list of matching documents. Chroma will create the embeddings for the query using its [default embedding function](https://docs.trychroma.com/embeddings#default-all-minilm-l6-v2).
|
||||
- [`ChromaEmbeddingRetriever`](../pipeline-components/retrievers/chromaembeddingretriever.mdx): This Retriever takes the embeddings of a single query in input and returns a list of matching documents. The query needs to be embedded before being passed to this component. For example, you can use an [embedder](../pipeline-components/embedders.mdx) component.
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Use Chroma for RAG and Indexing](https://haystack.deepset.ai/cookbook/chroma-indexing-and-rag-examples)
|
||||
+67
@@ -0,0 +1,67 @@
|
||||
---
|
||||
title: "ElasticsearchDocumentStore"
|
||||
id: elasticsearch-document-store
|
||||
slug: "/elasticsearch-document-store"
|
||||
description: "Use an Elasticsearch database with Haystack."
|
||||
---
|
||||
|
||||
# ElasticsearchDocumentStore
|
||||
|
||||
Use an Elasticsearch database with Haystack.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [Elasticsearch](/reference/integrations-elasticsearch) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/elasticsearch |
|
||||
|
||||
</div>
|
||||
|
||||
ElasticsearchDocumentStore is excellent if you want to evaluate the performance of different retrieval options (dense vs. sparse) and aim for a smooth transition from PoC to production.
|
||||
|
||||
It features the approximate nearest neighbours (ANN) search.
|
||||
|
||||
### Initialization
|
||||
|
||||
[Install](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html) Elasticsearch and then [start](https://www.elastic.co/guide/en/elasticsearch/reference/current/starting-elasticsearch.html) an instance. Haystack supports Elasticsearch 8.
|
||||
|
||||
If you have Docker set up, we recommend pulling the Docker image and running it.
|
||||
|
||||
```shell
|
||||
docker pull docker.elastic.co/elasticsearch/elasticsearch:8.11.1
|
||||
docker run -p 9200:9200 -e "discovery.type=single-node" -e "ES_JAVA_OPTS=-Xms1024m -Xmx1024m" -e "xpack.security.enabled=false" elasticsearch:8.11.1
|
||||
```
|
||||
|
||||
As an alternative, you can go to [Elasticsearch integration GitHub](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/elasticsearch) and start a Docker container running Elasticsearch using the provided `docker-compose.yml`:
|
||||
|
||||
```shell
|
||||
docker compose up
|
||||
```
|
||||
|
||||
Once you have a running Elasticsearch instance, install the `elasticsearch-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install elasticsearch-haystack
|
||||
```
|
||||
|
||||
Then, initialize an `ElasticsearchDocumentStore` object that’s connected to the Elasticsearch instance and writes documents to it:
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.elasticsearch import (
|
||||
ElasticsearchDocumentStore,
|
||||
)
|
||||
from haystack import Document
|
||||
|
||||
document_store = ElasticsearchDocumentStore(hosts="http://localhost:9200")
|
||||
document_store.write_documents(
|
||||
[Document(content="This is first"), Document(content="This is second")],
|
||||
)
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
### Supported Retrievers
|
||||
|
||||
[`ElasticsearchBM25Retriever`](../pipeline-components/retrievers/elasticsearchbm25retriever.mdx): A keyword-based Retriever that fetches documents matching a query from the Document Store.
|
||||
|
||||
[`ElasticsearchEmbeddingRetriever`](../pipeline-components/retrievers/elasticsearchembeddingretriever.mdx): Compares the query and document embeddings and fetches the documents most relevant to the query.
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
title: "InMemoryDocumentStore"
|
||||
id: inmemorydocumentstore
|
||||
slug: "/inmemorydocumentstore"
|
||||
---
|
||||
|
||||
# InMemoryDocumentStore
|
||||
|
||||
The `InMemoryDocumentStore` is a very simple document store with no extra services or dependencies.
|
||||
|
||||
It is great for experimenting with Haystack, however we do not recommend using it for production.
|
||||
|
||||
### Initialization
|
||||
|
||||
`InMemoryDocumentStore` requires no external setup. Simply use this code:
|
||||
|
||||
```python
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
```
|
||||
|
||||
### Supported Retrievers
|
||||
|
||||
[`InMemoryBM25Retriever`](../pipeline-components/retrievers/inmemorybm25retriever.mdx): A keyword-based Retriever that fetches documents matching a query from a temporary in-memory database.
|
||||
|
||||
[`InMemoryEmbeddingRetriever`](../pipeline-components/retrievers/inmemoryembeddingretriever.mdx): Compares the query and document embeddings and fetches the documents most relevant to the query.
|
||||
+59
@@ -0,0 +1,59 @@
|
||||
---
|
||||
title: "MongoDBAtlasDocumentStore"
|
||||
id: mongodbatlasdocumentstore
|
||||
slug: "/mongodbatlasdocumentstore"
|
||||
---
|
||||
|
||||
# MongoDBAtlasDocumentStore
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [MongoDB Atlas](/reference/integrations-mongodb-atlas) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/mongodb_atlas |
|
||||
|
||||
</div>
|
||||
|
||||
`MongoDBAtlasDocumentStore` can be used to manage documents using [MongoDB Atlas](https://www.mongodb.com/atlas), a multi-cloud database service by the same people who build MongoDB. Atlas simplifies deploying and managing your databases while offering the versatility you need to build resilient and performant global applications on the cloud providers of your choice. You can use MongoDB Atlas on cloud providers such as AWS, Azure, or Google Cloud, all without leaving Atlas' web UI.
|
||||
|
||||
MongoDB Atlas supports embeddings and can therefore be used for embedding retrieval.
|
||||
|
||||
## Installation
|
||||
|
||||
To use MongoDB Atlas with Haystack, install the integration first:
|
||||
|
||||
```shell
|
||||
pip install mongodb-atlas-haystack
|
||||
```
|
||||
|
||||
## Initialization
|
||||
|
||||
To use MongoDB Atlas with Haystack, you will need to create your MongoDB Atlas account: check the [MongoDB Atlas documentation](https://www.mongodb.com/docs/atlas/getting-started/) for help. You also need to [create a vector search index](https://www.mongodb.com/docs/atlas/atlas-vector-search/create-index/#std-label-avs-create-index) and [a full-text search index](https://www.mongodb.com/docs/atlas/atlas-search/manage-indexes/#create-an-atlas-search-index) for the collection you plan to use.
|
||||
|
||||
Once you have your connection string, you should export it in an environment variable called `MONGO_CONNECTION_STRING`. It should look something like this:
|
||||
|
||||
```python
|
||||
export MONGO_CONNECTION_STRING="mongodb+srv://<username>:<password>@<cluster_name>.gwkckbk.mongodb.net/?retryWrites=true&w=majority"
|
||||
```
|
||||
|
||||
At this point, you’re ready to initialize the store:
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.mongodb_atlas import (
|
||||
MongoDBAtlasDocumentStore,
|
||||
)
|
||||
|
||||
## Initialize the document store
|
||||
document_store = MongoDBAtlasDocumentStore(
|
||||
database_name="haystack_test",
|
||||
collection_name="test_collection",
|
||||
vector_search_index="embedding_index",
|
||||
full_text_search_index="search_index",
|
||||
)
|
||||
```
|
||||
|
||||
## Supported Retrievers
|
||||
|
||||
- [`MongoDBAtlasEmbeddingRetriever`](../pipeline-components/retrievers/mongodbatlasembeddingretriever.mdx): Compares the query and document embeddings and fetches the documents most relevant to the query.
|
||||
- [`MongoDBAtlasFullTextRetriever`](../pipeline-components/retrievers/mongodbatlasfulltextretriever.mdx): A full-text search Retriever.
|
||||
+76
@@ -0,0 +1,76 @@
|
||||
---
|
||||
title: "OpenSearchDocumentStore"
|
||||
id: opensearch-document-store
|
||||
slug: "/opensearch-document-store"
|
||||
description: "A Document Store for storing and retrieval from OpenSearch."
|
||||
---
|
||||
|
||||
# OpenSearchDocumentStore
|
||||
|
||||
A Document Store for storing and retrieval from OpenSearch.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [OpenSearch](/reference/integrations-opensearch) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opensearch |
|
||||
|
||||
</div>
|
||||
|
||||
OpenSearch is a fully open source search and analytics engine for use cases such as log analytics, real-time application monitoring, and clickstream analysis. For more information, see the [OpenSearch documentation](https://opensearch.org/docs/).
|
||||
|
||||
This Document Store is great if you want to evaluate the performance of different retrieval options (dense vs. sparse). It’s compatible with the Amazon OpenSearch Service.
|
||||
|
||||
OpenSearch provides support for vector similarity comparisons and approximate nearest neighbors algorithms.
|
||||
|
||||
### Initialization
|
||||
|
||||
[Install](https://opensearch.org/docs/latest/install-and-configure/install-opensearch/index/) and run an OpenSearch instance.
|
||||
|
||||
If you have Docker set up, we recommend pulling the Docker image and running it.
|
||||
|
||||
```shell
|
||||
docker pull opensearchproject/opensearch:2.11.0
|
||||
docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -e "ES_JAVA_OPTS=-Xms1024m -Xmx1024m" opensearchproject/opensearch:2.11.0
|
||||
```
|
||||
|
||||
As an alternative, you can go to [OpenSearch integration GitHub](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opensearch) and start a Docker container running OpenSearch using the provided `docker-compose.yml`:
|
||||
|
||||
```shell
|
||||
docker compose up
|
||||
```
|
||||
|
||||
Once you have a running OpenSearch instance, install the `opensearch-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install opensearch-haystack
|
||||
```
|
||||
|
||||
Then, initialize an `OpenSearchDocumentStore` object that’s connected to the OpenSearch instance and writes documents to it:
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.opensearch import OpenSearchDocumentStore
|
||||
from haystack import Document
|
||||
|
||||
document_store = OpenSearchDocumentStore(
|
||||
hosts="http://localhost:9200",
|
||||
use_ssl=True,
|
||||
verify_certs=False,
|
||||
http_auth=("admin", "admin"),
|
||||
)
|
||||
document_store.write_documents(
|
||||
[Document(content="This is first"), Document(content="This is second")],
|
||||
)
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
### Supported Retrievers
|
||||
|
||||
[`OpenSearchBM25Retriever`](../pipeline-components/retrievers/opensearchbm25retriever.mdx): A keyword-based Retriever that fetches documents matching a query from the Document Store.
|
||||
|
||||
[`OpenSearchEmbeddingRetriever`](../pipeline-components/retrievers/opensearchembeddingretriever.mdx): Compares the query and document embeddings and fetches the documents most relevant to the query.
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [PDF-Based Question Answering with Amazon Bedrock and Haystack](https://haystack.deepset.ai/cookbook/amazon_bedrock_for_documentation_qa)
|
||||
@@ -0,0 +1,109 @@
|
||||
---
|
||||
title: "PgvectorDocumentStore"
|
||||
id: pgvectordocumentstore
|
||||
slug: "/pgvectordocumentstore"
|
||||
---
|
||||
|
||||
# PgvectorDocumentStore
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [Pgvector](/reference/integrations-pgvector) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/pgvector/ |
|
||||
|
||||
</div>
|
||||
|
||||
Pgvector is an extension for PostgreSQL that enhances its capabilities with vector similarity search. It builds upon the classic features of PostgreSQL, such as ACID compliance and point-in-time recovery, and introduces the ability to perform exact and approximate nearest neighbor search using vectors.
|
||||
|
||||
For more information, see the [pgvector repository](https://github.com/pgvector/pgvector).
|
||||
|
||||
Pgvector Document Store supports embedding retrieval and metadata filtering.
|
||||
|
||||
## Installation
|
||||
|
||||
To quickly set up a PostgreSQL database with pgvector, you can use Docker:
|
||||
|
||||
```shell
|
||||
docker run -d -p 5432:5432 -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=postgres ankane/pgvector
|
||||
```
|
||||
|
||||
For more information on installing pgvector, visit the [pgvector GitHub repository](https://github.com/pgvector/pgvector).
|
||||
|
||||
To use pgvector with Haystack, install the `pgvector-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install pgvector-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Connection String
|
||||
|
||||
Define the connection string to your PostgreSQL database in the `PG_CONN_STR` environment variable. Two formats are supported:
|
||||
|
||||
**URI format:**
|
||||
|
||||
```shell
|
||||
export PG_CONN_STR="postgresql://USER:PASSWORD@HOST:PORT/DB_NAME"
|
||||
```
|
||||
|
||||
**Keyword/value format:**
|
||||
|
||||
```shell
|
||||
export PG_CONN_STR="host=HOST port=PORT dbname=DB_NAME user=USER password=PASSWORD"
|
||||
```
|
||||
|
||||
:::caution[Special Characters in Connection URIs]
|
||||
|
||||
When using the URI format, special characters in the password must be [percent-encoded](https://en.wikipedia.org/wiki/Percent-encoding). Otherwise, connection errors may occur. A password like `p=ssword` would cause the error `psycopg.OperationalError: [Errno -2] Name or service not known`.
|
||||
|
||||
For example, if your password is `p=ssword`, the connection string should be:
|
||||
|
||||
```shell
|
||||
export PG_CONN_STR="postgresql://postgres:p%3Dssword@localhost:5432/postgres"
|
||||
```
|
||||
|
||||
Alternatively, use the keyword/value format, which does not require percent-encoding:
|
||||
|
||||
```shell
|
||||
export PG_CONN_STR="host=localhost port=5432 dbname=postgres user=postgres password=p=ssword"
|
||||
```
|
||||
|
||||
:::
|
||||
|
||||
For more details, see the [PostgreSQL connection string documentation](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING).
|
||||
|
||||
## Initialization
|
||||
|
||||
Initialize a `PgvectorDocumentStore` object that’s connected to the PostgreSQL database and writes documents to it:
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.pgvector import PgvectorDocumentStore
|
||||
from haystack import Document
|
||||
|
||||
document_store = PgvectorDocumentStore(
|
||||
embedding_dimension=768,
|
||||
vector_function="cosine_similarity",
|
||||
recreate_table=True,
|
||||
search_strategy="hnsw",
|
||||
)
|
||||
|
||||
document_store.write_documents(
|
||||
[
|
||||
Document(content="This is first", embedding=[0.1] * 768),
|
||||
Document(content="This is second", embedding=[0.3] * 768),
|
||||
],
|
||||
)
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
To learn more about the initialization parameters, see our [API docs](/reference/integrations-pgvector#pgvectordocumentstore).
|
||||
|
||||
To properly compute embeddings for your documents, you can use a Document Embedder (for instance, the [`SentenceTransformersDocumentEmbedder`](../pipeline-components/embedders/sentencetransformersdocumentembedder.mdx)).
|
||||
|
||||
### Supported Retrievers
|
||||
|
||||
- [`PgvectorEmbeddingRetriever`](../pipeline-components/retrievers/pgvectorembeddingretriever.mdx): An embedding-based Retriever that fetches documents from the Document Store based on a query embedding provided to the Retriever.
|
||||
- [`PgvectorKeywordRetriever`](../pipeline-components/retrievers/pgvectorembeddingretriever.mdx): A keyword-based Retriever that fetches documents matching a query from the Pgvector Document Store.
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
title: "PineconeDocumentStore"
|
||||
id: pinecone-document-store
|
||||
slug: "/pinecone-document-store"
|
||||
description: "Use a Pinecone vector database with Haystack."
|
||||
---
|
||||
|
||||
# PineconeDocumentStore
|
||||
|
||||
Use a Pinecone vector database with Haystack.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [Pinecone](/reference/integrations-pinecone) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/pinecone |
|
||||
|
||||
</div>
|
||||
|
||||
[Pinecone](https://www.pinecone.io/) is a cloud-based vector database. It is fast and easy to use.
|
||||
Unlike other solutions (such as Qdrant and Weaviate), it can’t run locally on the user's machine but provides a generous free tier.
|
||||
|
||||
### Installation
|
||||
|
||||
You can simply install the Pinecone Haystack integration with:
|
||||
|
||||
```shell
|
||||
pip install pinecone-haystack
|
||||
```
|
||||
|
||||
### Initialization
|
||||
|
||||
- To use Pinecone as a Document Store in Haystack, sign up for a free Pinecone [account](https://app.pinecone.io/) and get your API key.
|
||||
The Pinecone API key can be explicitly provided or automatically read from the environment variable `PINECONE_API_KEY` (recommended).
|
||||
- In Haystack, each `PineconeDocumentStore` operates in a specific namespace of an index. If not provided, both index and namespace are `default`.
|
||||
If the index already exists, the Document Store connects to it. Otherwise, it creates a new index.
|
||||
- When creating a new index, you can provide a `spec` in the form of a dictionary. This allows choosing between serverless and pod deployment options and setting additional parameters. Refer to the [Pinecone documentation](https://docs.pinecone.io/reference/api/control-plane/create_index) for more details. If not provided, a default spec with serverless deployment in the `us-east-1` region will be used (compatible with the free tier).
|
||||
- You can provide `dimension` and `metric`, but they are only taken into account if the Pinecone index does not already exist.
|
||||
|
||||
Then, you can use the Document Store like this:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.document_stores.pinecone import PineconeDocumentStore
|
||||
|
||||
## Make sure you have the PINECONE_API_KEY environment variable set
|
||||
document_store = PineconeDocumentStore(
|
||||
index="default",
|
||||
namespace="default",
|
||||
dimension=5,
|
||||
metric="cosine",
|
||||
spec={"serverless": {"region": "us-east-1", "cloud": "aws"}},
|
||||
)
|
||||
|
||||
document_store.write_documents(
|
||||
[
|
||||
Document(content="This is first", embedding=[0.1] * 5),
|
||||
Document(content="This is second", embedding=[0.1, 0.2, 0.3, 0.4, 0.5]),
|
||||
],
|
||||
)
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
### Supported Retrievers
|
||||
|
||||
[`PineconeEmbeddingRetriever`](../pipeline-components/retrievers/pineconedenseretriever.mdx): Retrieves documents from the `PineconeDocumentStore` based on their dense embeddings (vectors).
|
||||
@@ -0,0 +1,103 @@
|
||||
---
|
||||
title: "QdrantDocumentStore"
|
||||
id: qdrant-document-store
|
||||
slug: "/qdrant-document-store"
|
||||
description: "Use the Qdrant vector database with Haystack."
|
||||
---
|
||||
|
||||
# QdrantDocumentStore
|
||||
|
||||
Use the Qdrant vector database with Haystack.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [Qdrant](/reference/integrations-qdrant) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/qdrant |
|
||||
|
||||
</div>
|
||||
|
||||
Qdrant is a powerful high-performance, massive-scale vector database. The `QdrantDocumentStore` can be used with any Qdrant instance, in-memory, locally persisted, hosted, and the official Qdrant Cloud.
|
||||
|
||||
### Installation
|
||||
|
||||
You can simply install the Qdrant Haystack integration with:
|
||||
|
||||
```shell
|
||||
pip install qdrant-haystack
|
||||
```
|
||||
|
||||
### Initialization
|
||||
|
||||
The quickest way to use `QdrantDocumentStore` is to create an in-memory instance of it:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses.document import Document
|
||||
from haystack_integrations.document_stores.qdrant import QdrantDocumentStore
|
||||
|
||||
document_store = QdrantDocumentStore(
|
||||
":memory:",
|
||||
recreate_index=True,
|
||||
return_embedding=True,
|
||||
wait_result_from_api=True,
|
||||
)
|
||||
document_store.write_documents(
|
||||
[
|
||||
Document(content="This is first", embedding=[0.0] * 768),
|
||||
Document(content="This is second", embedding=[0.1] * 768),
|
||||
],
|
||||
)
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
:::warning[Collections Created Outside Haystack]
|
||||
|
||||
When you create a `QdrantDocumentStore` instance, Haystack takes care of setting up the collection. In general, you cannot use a Qdrant collection created without Haystack with Haystack. If you want to migrate your existing collection, see the sample script at https://github.com/deepset-ai/haystack-core-integrations/blob/main/integrations/qdrant/src/haystack_integrations/document_stores/qdrant/migrate_to_sparse.py.
|
||||
:::
|
||||
|
||||
You can also connect directly to [Qdrant Cloud](https://cloud.qdrant.io/login) directly. Once you have your API key and your cluster URL from the Qdrant dashboard, you can connect like this:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses.document import Document
|
||||
from haystack_integrations.document_stores.qdrant import QdrantDocumentStore
|
||||
from haystack.utils import Secret
|
||||
|
||||
document_store = QdrantDocumentStore(
|
||||
url="https://XXXXXXXXX.us-east4-0.gcp.cloud.qdrant.io:6333",
|
||||
index="your_index_name",
|
||||
embedding_dim=1024, # based on the embedding model
|
||||
recreate_index=True, # enable only to recreate the index and not connect to the existing one
|
||||
api_key=Secret.from_token("YOUR_TOKEN"),
|
||||
)
|
||||
|
||||
document_store.write_documents(
|
||||
[
|
||||
Document(content="This is first", embedding=[0.0] * 5),
|
||||
Document(content="This is second", embedding=[0.1, 0.2, 0.3, 0.4, 0.5]),
|
||||
],
|
||||
)
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
:::tip[More information]
|
||||
|
||||
You can find more ways to initialize and use QdrantDocumentStore on our [integration page](https://haystack.deepset.ai/integrations/qdrant-document-store).
|
||||
:::
|
||||
|
||||
### Supported Retrievers
|
||||
|
||||
- [`QdrantEmbeddingRetriever`](../pipeline-components/retrievers/qdrantembeddingretriever.mdx): Retrieves documents from the `QdrantDocumentStore` based on their dense embeddings (vectors).
|
||||
- [`QdrantSparseEmbeddingRetriever`](../pipeline-components/retrievers/qdrantsparseembeddingretriever.mdx): Retrieves documents from the `QdrantDocumentStore` based on their sparse embeddings.
|
||||
- [`QdrantHybridRetriever`](../pipeline-components/retrievers/qdranthybridretriever.mdx): Retrieves documents from the `QdrantDocumentStore` based on both dense and sparse embeddings.
|
||||
|
||||
:::note[Sparse Embedding Support]
|
||||
|
||||
To use Sparse Embedding support, you need to initialize the `QdrantDocumentStore` with `use_sparse_embeddings=True`, which is `False` by default.
|
||||
|
||||
If you want to use Document Store or collection previously created with this feature disabled, you must migrate the existing data. You can do this by taking advantage of the `migrate_to_sparse_embeddings_support` utility function.
|
||||
:::
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Sparse Embedding Retrieval with Qdrant and FastEmbed](https://haystack.deepset.ai/cookbook/sparse_embedding_retrieval)
|
||||
@@ -0,0 +1,155 @@
|
||||
---
|
||||
title: "WeaviateDocumentStore"
|
||||
id: weaviatedocumentstore
|
||||
slug: "/weaviatedocumentstore"
|
||||
---
|
||||
|
||||
# WeaviateDocumentStore
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| API reference | [Weaviate](/reference/integrations-weaviate) |
|
||||
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/weaviate |
|
||||
|
||||
</div>
|
||||
|
||||
Weaviate is a multi-purpose vector DB that can store both embeddings and data objects, making it a good choice for multi-modality.
|
||||
|
||||
The `WeaviateDocumentStore` can connect to any Weaviate instance, whether it's running on Weaviate Cloud Services, Kubernetes, or a local Docker container.
|
||||
|
||||
## Installation
|
||||
|
||||
You can simply install the Weaviate Haystack integration with:
|
||||
|
||||
```shell
|
||||
pip install weaviate-haystack
|
||||
```
|
||||
|
||||
## Initialization
|
||||
|
||||
### Weaviate Embedded
|
||||
|
||||
To use `WeaviateDocumentStore` as a temporary instance, initialize it as ["Embedded"](https://weaviate.io/developers/weaviate/installation/embedded):
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.weaviate import WeaviateDocumentStore
|
||||
from weaviate.embedded import EmbeddedOptions
|
||||
|
||||
document_store = WeaviateDocumentStore(embedded_options=EmbeddedOptions())
|
||||
```
|
||||
|
||||
### Docker
|
||||
|
||||
You can use `WeaviateDocumentStore` in a local Docker container. This is what a minimal `docker-compose.yml` could look like:
|
||||
|
||||
```yaml
|
||||
---
|
||||
version: '3.4'
|
||||
services:
|
||||
weaviate:
|
||||
command:
|
||||
- --host
|
||||
- 0.0.0.0
|
||||
- --port
|
||||
- '8080'
|
||||
- --scheme
|
||||
- http
|
||||
image: semitechnologies/weaviate:1.30.17
|
||||
ports:
|
||||
- 8080:8080
|
||||
- 50051:50051
|
||||
volumes:
|
||||
- weaviate_data:/var/lib/weaviate
|
||||
restart: 'no'
|
||||
environment:
|
||||
QUERY_DEFAULTS_LIMIT: 25
|
||||
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
|
||||
PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
|
||||
DEFAULT_VECTORIZER_MODULE: 'none'
|
||||
ENABLE_MODULES: ''
|
||||
CLUSTER_HOSTNAME: 'node1'
|
||||
volumes:
|
||||
weaviate_data:
|
||||
...
|
||||
```
|
||||
|
||||
:::warning
|
||||
With this example, we explicitly enable access without authentication, so you don't need to set any username, password, or API key to connect to our local instance. That is strongly discouraged for production use. See the [authorization](#authorization) section for detailed information.
|
||||
|
||||
:::
|
||||
|
||||
Start your container with `docker compose up -d` and then initialize the Document Store with:
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.weaviate.document_store import (
|
||||
WeaviateDocumentStore,
|
||||
)
|
||||
from haystack import Document
|
||||
|
||||
document_store = WeaviateDocumentStore(url="http://localhost:8080")
|
||||
document_store.write_documents(
|
||||
[Document(content="This is first"), Document(content="This is second")],
|
||||
)
|
||||
print(document_store.count_documents())
|
||||
```
|
||||
|
||||
### Weaviate Cloud Service
|
||||
|
||||
To use the [Weaviate managed cloud service](https://weaviate.io/developers/wcs), first, create your Weaviate cluster.
|
||||
|
||||
Then, initialize the `WeaviateDocumentStore` using the API Key and URL found in your [Weaviate account](https://console.weaviate.cloud/):
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.weaviate import (
|
||||
WeaviateDocumentStore,
|
||||
AuthApiKey,
|
||||
)
|
||||
from haystack import Document
|
||||
|
||||
import os
|
||||
|
||||
os.environ["WEAVIATE_API_KEY"] = "YOUR-API-KEY"
|
||||
|
||||
auth_client_secret = AuthApiKey()
|
||||
|
||||
document_store = WeaviateDocumentStore(
|
||||
url="YOUR-WEAVIATE-URL",
|
||||
auth_client_secret=auth_client_secret,
|
||||
)
|
||||
```
|
||||
|
||||
## Authorization
|
||||
|
||||
We provide some utility classes in the `auth` package to handle authorization using different credentials. Every class stores distinct [secrets](../concepts/secret-management.mdx) and retrieves them from the environment variables when required.
|
||||
|
||||
The default environment variables for the classes are:
|
||||
|
||||
- **`AuthApiKey`**
|
||||
- `WEAVIATE_API_KEY`
|
||||
- **`AuthBearerToken`**
|
||||
- `WEAVIATE_ACCESS_TOKEN`
|
||||
- `WEAVIATE_REFRESH_TOKEN`
|
||||
- **`AuthClientCredentials`**
|
||||
- `WEAVIATE_CLIENT_SECRET`
|
||||
- `WEAVIATE_SCOPE`
|
||||
- **`AuthClientPassword`**
|
||||
- `WEAVIATE_USERNAME`
|
||||
- `WEAVIATE_PASSWORD`
|
||||
- `WEAVIATE_SCOPE`
|
||||
|
||||
You can easily change environment variables if needed. In the following snippet, we instruct `AuthApiKey` to look for `MY_ENV_VAR`.
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.weaviate.auth import AuthApiKey
|
||||
from haystack.utils.auth import Secret
|
||||
|
||||
AuthApiKey(api_key=Secret.from_env_var("MY_ENV_VAR"))
|
||||
```
|
||||
|
||||
## Supported Retrievers
|
||||
|
||||
[`WeaviateBM25Retriever`](../pipeline-components/retrievers/weaviatebm25retriever.mdx): A keyword-based Retriever that fetches documents matching a query from the Document Store.
|
||||
|
||||
[`WeaviateEmbeddingRetriever`](../pipeline-components/retrievers/weaviateembeddingretriever.mdx): Compares the query and document embeddings and fetches the documents most relevant to the query.
|
||||
Reference in New Issue
Block a user