chore: import upstream snapshot with attribution
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,125 @@
|
||||
---
|
||||
title: "AlloyDBEmbeddingRetriever"
|
||||
id: alloydbembeddingretriever
|
||||
slug: "/alloydbembeddingretriever"
|
||||
description: "An embedding-based Retriever compatible with the AlloyDB Document Store."
|
||||
---
|
||||
|
||||
# AlloyDBEmbeddingRetriever
|
||||
|
||||
An embedding-based Retriever compatible with the AlloyDB Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the semantic search pipeline 3. After a Text Embedder and before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of an [AlloyDBDocumentStore](../../document-stores/alloydbdocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query_embedding`: A vector representing the query (a list of floats) |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [AlloyDB](/reference/integrations-alloydb) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/alloydb |
|
||||
| **Package name** | `alloydb-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `AlloyDBEmbeddingRetriever` is an embedding-based Retriever compatible with the `AlloyDBDocumentStore`. It compares the query and Document embeddings and fetches the Documents most relevant to the query from the `AlloyDBDocumentStore` based on the outcome.
|
||||
|
||||
When using the `AlloyDBEmbeddingRetriever` in your Pipeline, make sure it has the query and Document embeddings available. You can do so by adding a Document Embedder to your indexing Pipeline and a Text Embedder to your query Pipeline.
|
||||
|
||||
In addition to the `query_embedding`, the `AlloyDBEmbeddingRetriever` accepts other optional parameters, including `top_k` (the maximum number of Documents to retrieve), `filters` to narrow down the search space, and `vector_function` to override the similarity function set on the Document Store.
|
||||
|
||||
Some relevant parameters that impact embedding retrieval must be defined when the corresponding `AlloyDBDocumentStore` is initialized: these include `embedding_dimension`, `vector_function`, and the search strategy (`"exact_nearest_neighbor"` or `"hnsw"`).
|
||||
|
||||
## Installation
|
||||
|
||||
Install the `alloydb-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install alloydb-haystack
|
||||
```
|
||||
|
||||
To set up an AlloyDB cluster and instance, follow the [AlloyDB quickstart](https://cloud.google.com/alloydb/docs/quickstart).
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
This Retriever needs the `AlloyDBDocumentStore` and indexed Documents to run.
|
||||
|
||||
Set the `ALLOYDB_INSTANCE_URI`, `ALLOYDB_USER`, and `ALLOYDB_PASSWORD` environment variables to connect to your AlloyDB instance.
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.alloydb import AlloyDBDocumentStore
|
||||
from haystack_integrations.components.retrievers.alloydb import (
|
||||
AlloyDBEmbeddingRetriever,
|
||||
)
|
||||
|
||||
document_store = AlloyDBDocumentStore()
|
||||
retriever = AlloyDBEmbeddingRetriever(document_store=document_store)
|
||||
|
||||
## using a fake vector to keep the example simple
|
||||
retriever.run(query_embedding=[0.1] * 768)
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersTextEmbedder,
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
)
|
||||
|
||||
from haystack_integrations.document_stores.alloydb import AlloyDBDocumentStore
|
||||
from haystack_integrations.components.retrievers.alloydb import (
|
||||
AlloyDBEmbeddingRetriever,
|
||||
)
|
||||
|
||||
document_store = AlloyDBDocumentStore(
|
||||
embedding_dimension=768,
|
||||
vector_function="cosine_similarity",
|
||||
recreate_table=True,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
document_embedder = SentenceTransformersDocumentEmbedder()
|
||||
documents_with_embeddings = document_embedder.run(documents)
|
||||
|
||||
document_store.write_documents(
|
||||
documents_with_embeddings.get("documents"),
|
||||
policy=DuplicatePolicy.OVERWRITE,
|
||||
)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", SentenceTransformersTextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
AlloyDBEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "How many languages are there?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
```
|
||||
@@ -0,0 +1,145 @@
|
||||
---
|
||||
title: "AlloyDBKeywordRetriever"
|
||||
id: alloydbkeywordretriever
|
||||
slug: "/alloydbkeywordretriever"
|
||||
description: "A keyword-based Retriever that fetches documents matching a query from the AlloyDB Document Store."
|
||||
---
|
||||
|
||||
# AlloyDBKeywordRetriever
|
||||
|
||||
A keyword-based Retriever that fetches documents matching a query from the AlloyDB Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. Before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the semantic search pipeline 3. Before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of an [AlloyDBDocumentStore](../../document-stores/alloydbdocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query`: A string |
|
||||
| **Output variables** | `documents`: A list of documents (matching the query) |
|
||||
| **API reference** | [AlloyDB](/reference/integrations-alloydb) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/alloydb |
|
||||
| **Package name** | `alloydb-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `AlloyDBKeywordRetriever` is a keyword-based Retriever compatible with the `AlloyDBDocumentStore`.
|
||||
|
||||
It uses PostgreSQL full-text search (`to_tsvector` / `plainto_tsquery`) to find Documents and ranks them with `ts_rank_cd`. The ranking considers how often the query terms appear in the Document, how close together the terms are, and how important the part of the Document is where they occur. For more details, see the [PostgreSQL documentation](https://www.postgresql.org/docs/current/textsearch-controls.html#TEXTSEARCH-RANKING).
|
||||
|
||||
Keep in mind that, unlike similar components such as `ElasticsearchBM25Retriever`, this Retriever does not apply fuzzy search out of the box, so it’s necessary to carefully formulate the query in order to avoid getting zero results.
|
||||
|
||||
The language used to parse query and Document content for keyword retrieval is set via the `language` parameter on the `AlloyDBDocumentStore` (defaults to `"english"`). To list the supported languages on your database, run:
|
||||
|
||||
```sql
|
||||
SELECT cfgname FROM pg_ts_config;
|
||||
```
|
||||
|
||||
In addition to the `query`, the `AlloyDBKeywordRetriever` accepts other optional parameters, including `top_k` (the maximum number of Documents to retrieve) and `filters` to narrow the search space.
|
||||
|
||||
## Installation
|
||||
|
||||
Install the `alloydb-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install alloydb-haystack
|
||||
```
|
||||
|
||||
To set up an AlloyDB cluster and instance, follow the [AlloyDB quickstart](https://cloud.google.com/alloydb/docs/quickstart).
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
This Retriever needs the `AlloyDBDocumentStore` and indexed Documents to run.
|
||||
|
||||
Set the `ALLOYDB_INSTANCE_URI`, `ALLOYDB_USER`, and `ALLOYDB_PASSWORD` environment variables to connect to your AlloyDB instance.
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.alloydb import AlloyDBDocumentStore
|
||||
from haystack_integrations.components.retrievers.alloydb import (
|
||||
AlloyDBKeywordRetriever,
|
||||
)
|
||||
|
||||
document_store = AlloyDBDocumentStore()
|
||||
retriever = AlloyDBKeywordRetriever(document_store=document_store)
|
||||
|
||||
retriever.run(query="my nice query")
|
||||
```
|
||||
|
||||
### In a RAG pipeline
|
||||
|
||||
The prerequisites necessary for running this code are:
|
||||
|
||||
- Set an environment variable `OPENAI_API_KEY` with your OpenAI API key.
|
||||
- Set the `ALLOYDB_INSTANCE_URI`, `ALLOYDB_USER`, and `ALLOYDB_PASSWORD` environment variables to connect to your AlloyDB instance.
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.builders.answer_builder import AnswerBuilder
|
||||
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
|
||||
from haystack_integrations.document_stores.alloydb import AlloyDBDocumentStore
|
||||
from haystack_integrations.components.retrievers.alloydb import (
|
||||
AlloyDBKeywordRetriever,
|
||||
)
|
||||
|
||||
## Create a RAG query pipeline
|
||||
prompt_template = [
|
||||
ChatMessage.from_system("You are a helpful assistant."),
|
||||
ChatMessage.from_user(
|
||||
"Given these documents, answer the question.\nDocuments:\n"
|
||||
"{% for doc in documents %}{{ doc.content }}{% endfor %}\n"
|
||||
"Question: {{question}}\nAnswer:",
|
||||
),
|
||||
]
|
||||
|
||||
document_store = AlloyDBDocumentStore(
|
||||
language="english", # this parameter influences text parsing for keyword retrieval
|
||||
recreate_table=True,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
document_store.write_documents(documents=documents, policy=DuplicatePolicy.SKIP)
|
||||
|
||||
retriever = AlloyDBKeywordRetriever(document_store=document_store)
|
||||
rag_pipeline = Pipeline()
|
||||
rag_pipeline.add_component(name="retriever", instance=retriever)
|
||||
rag_pipeline.add_component(
|
||||
instance=ChatPromptBuilder(
|
||||
template=prompt_template,
|
||||
required_variables={"question", "documents"},
|
||||
),
|
||||
name="prompt_builder",
|
||||
)
|
||||
rag_pipeline.add_component(instance=OpenAIChatGenerator(), name="llm")
|
||||
rag_pipeline.add_component(instance=AnswerBuilder(), name="answer_builder")
|
||||
rag_pipeline.connect("retriever", "prompt_builder.documents")
|
||||
rag_pipeline.connect("prompt_builder.prompt", "llm.messages")
|
||||
rag_pipeline.connect("llm.replies", "answer_builder.replies")
|
||||
rag_pipeline.connect("retriever", "answer_builder.documents")
|
||||
|
||||
question = "languages spoken around the world today"
|
||||
result = rag_pipeline.run(
|
||||
{
|
||||
"retriever": {"query": question},
|
||||
"prompt_builder": {"question": question},
|
||||
"answer_builder": {"query": question},
|
||||
},
|
||||
)
|
||||
print(result["answer_builder"])
|
||||
```
|
||||
@@ -0,0 +1,143 @@
|
||||
---
|
||||
title: "ArangoEmbeddingRetriever"
|
||||
id: arangoembeddingretriever
|
||||
slug: "/arangoembeddingretriever"
|
||||
description: "An embedding-based Retriever compatible with the ArangoDB Document Store."
|
||||
---
|
||||
|
||||
# ArangoEmbeddingRetriever
|
||||
|
||||
An embedding-based Retriever compatible with the ArangoDB Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline <br /><br /> 2. The last component in a semantic search pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of an [ArangoDocumentStore](../../document-stores/arangodocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query_embedding`: A vector representing the query (a list of floats) |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [ArangoDB](/reference/integrations-arangodb) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/arangodb |
|
||||
| **Package name** | `arangodb-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `ArangoEmbeddingRetriever` retrieves documents from an `ArangoDocumentStore` using ArangoDB's AQL vector functions. It compares the query embedding with document embeddings and returns the most similar documents.
|
||||
|
||||
In addition to `query_embedding`, the retriever accepts optional `filters` to narrow the search space and `top_k` to limit the number of results. Both can be set at initialization and overridden per call to `run()`.
|
||||
|
||||
The embedding dimension and similarity function (`cosine`, `dot_product`, or `l2`) are configured on the `ArangoDocumentStore` at initialization time.
|
||||
|
||||
## Installation
|
||||
|
||||
```shell
|
||||
pip install arangodb-haystack
|
||||
```
|
||||
|
||||
Ensure ArangoDB 3.12+ is running with the vector index enabled, for example via Docker:
|
||||
|
||||
```shell
|
||||
docker run -d -p 8529:8529 \
|
||||
-e ARANGO_ROOT_PASSWORD=test-password \
|
||||
arangodb:3.12 arangod --vector-index
|
||||
```
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.document_stores.arangodb import ArangoDocumentStore
|
||||
from haystack_integrations.components.retrievers.arangodb import (
|
||||
ArangoEmbeddingRetriever,
|
||||
)
|
||||
|
||||
document_store = ArangoDocumentStore(
|
||||
host="http://localhost:8529",
|
||||
embedding_dimension=3,
|
||||
recreate_collection=True,
|
||||
)
|
||||
document_store.write_documents(
|
||||
[
|
||||
Document(
|
||||
content="There are over 7,000 languages spoken around the world today.",
|
||||
embedding=[0.1, 0.2, 0.3],
|
||||
),
|
||||
Document(
|
||||
content="Elephants have been observed to recognize themselves in mirrors.",
|
||||
embedding=[0.8, 0.1, 0.5],
|
||||
),
|
||||
],
|
||||
)
|
||||
|
||||
retriever = ArangoEmbeddingRetriever(document_store=document_store, top_k=1)
|
||||
result = retriever.run(query_embedding=[0.1, 0.2, 0.3])
|
||||
print(result["documents"][0].content)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
SentenceTransformersTextEmbedder,
|
||||
)
|
||||
from haystack_integrations.document_stores.arangodb import ArangoDocumentStore
|
||||
from haystack_integrations.components.retrievers.arangodb import (
|
||||
ArangoEmbeddingRetriever,
|
||||
)
|
||||
|
||||
document_store = ArangoDocumentStore(
|
||||
host="http://localhost:8529",
|
||||
embedding_dimension=384,
|
||||
recreate_collection=True,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to recognize themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="Bioluminescent waves can be seen in the Maldives and Puerto Rico.",
|
||||
),
|
||||
]
|
||||
|
||||
document_embedder = SentenceTransformersDocumentEmbedder(
|
||||
model="sentence-transformers/all-MiniLM-L6-v2",
|
||||
)
|
||||
documents_with_embeddings = document_embedder.run(documents)
|
||||
|
||||
document_store.write_documents(
|
||||
documents_with_embeddings["documents"],
|
||||
policy=DuplicatePolicy.OVERWRITE,
|
||||
)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component(
|
||||
"text_embedder",
|
||||
SentenceTransformersTextEmbedder(model="sentence-transformers/all-MiniLM-L6-v2"),
|
||||
)
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
ArangoEmbeddingRetriever(document_store=document_store, top_k=3),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
result = query_pipeline.run(
|
||||
{"text_embedder": {"text": "How many languages are there?"}},
|
||||
)
|
||||
print(result["retriever"]["documents"][0].content)
|
||||
```
|
||||
@@ -0,0 +1,117 @@
|
||||
---
|
||||
title: "ArcadeDBEmbeddingRetriever"
|
||||
id: arcadedbembeddingretriever
|
||||
slug: "/arcadedbembeddingretriever"
|
||||
description: "An embedding-based Retriever compatible with the ArcadeDB Document Store."
|
||||
---
|
||||
|
||||
# ArcadeDBEmbeddingRetriever
|
||||
|
||||
An embedding-based Retriever compatible with the ArcadeDB Document Store. It uses ArcadeDB's LSM_VECTOR (HNSW) index for vector similarity search.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. After a Text Embedder and before a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) in a RAG pipeline 2. The last component in a semantic search pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of [ArcadeDBDocumentStore](../../document-stores/arcadedbdocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query_embedding`: A vector representing the query (a list of floats) |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [ArcadeDB](/reference/integrations-arcadedb) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/arcadedb |
|
||||
| **Package name** | `arcadedb-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `ArcadeDBEmbeddingRetriever` retrieves documents from `ArcadeDBDocumentStore` by comparing the query embedding with document embeddings using the store's HNSW index. It accepts optional `filters` for metadata filtering and `top_k` to limit the number of results. Use a Document Embedder in your indexing pipeline and a Text Embedder in your query pipeline so embeddings are available.
|
||||
|
||||
## Installation
|
||||
|
||||
```shell
|
||||
pip install arcadedb-haystack
|
||||
```
|
||||
|
||||
Ensure ArcadeDB is running, for example via Docker, and credentials are set (`ARCADEDB_USERNAME`, `ARCADEDB_PASSWORD`).
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.arcadedb import ArcadeDBDocumentStore
|
||||
from haystack_integrations.components.retrievers.arcadedb import (
|
||||
ArcadeDBEmbeddingRetriever,
|
||||
)
|
||||
|
||||
document_store = ArcadeDBDocumentStore(
|
||||
url="http://localhost:2480",
|
||||
database="haystack",
|
||||
embedding_dimension=768,
|
||||
)
|
||||
retriever = ArcadeDBEmbeddingRetriever(document_store=document_store, top_k=5)
|
||||
|
||||
# Example: run with a query embedding (e.g. from an embedder)
|
||||
result = retriever.run(query_embedding=[0.1] * 768)
|
||||
for doc in result["documents"]:
|
||||
print(doc.content)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersTextEmbedder,
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
)
|
||||
from haystack_integrations.document_stores.arcadedb import ArcadeDBDocumentStore
|
||||
from haystack_integrations.components.retrievers.arcadedb import (
|
||||
ArcadeDBEmbeddingRetriever,
|
||||
)
|
||||
|
||||
document_store = ArcadeDBDocumentStore(
|
||||
url="http://localhost:2480",
|
||||
database="haystack",
|
||||
embedding_dimension=768,
|
||||
recreate_type=True,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to recognize themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="Bioluminescent waves can be seen in the Maldives and Puerto Rico.",
|
||||
),
|
||||
]
|
||||
|
||||
document_embedder = SentenceTransformersDocumentEmbedder()
|
||||
documents_with_embeddings = document_embedder.run(documents)
|
||||
document_store.write_documents(
|
||||
documents_with_embeddings["documents"],
|
||||
policy=DuplicatePolicy.OVERWRITE,
|
||||
)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", SentenceTransformersTextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
ArcadeDBEmbeddingRetriever(document_store=document_store, top_k=3),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
result = query_pipeline.run(
|
||||
{"text_embedder": {"text": "How many languages are there?"}},
|
||||
)
|
||||
print(result["retriever"]["documents"][0])
|
||||
```
|
||||
@@ -0,0 +1,117 @@
|
||||
---
|
||||
title: "AstraEmbeddingRetriever"
|
||||
id: astraretriever
|
||||
slug: "/astraretriever"
|
||||
description: "This is an embedding-based Retriever compatible with the Astra Document Store."
|
||||
---
|
||||
|
||||
# AstraEmbeddingRetriever
|
||||
|
||||
This is an embedding-based Retriever compatible with the Astra Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline <br /> 2. The last component in the semantic search pipeline <br /> 3. After a Text Embedder and before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of [AstraDocumentStore](../../document-stores/astradocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query_embedding`: A list of floats |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Astra](/reference/integrations-astra) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/astra |
|
||||
| **Package name** | `astra-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`AstraEmbeddingRetriever` compares the query and document embeddings and fetches the documents most relevant to the query from the [`AstraDocumentStore`](../../document-stores/astradocumentstore.mdx) based on the outcome.
|
||||
|
||||
When using the `AstraEmbeddingRetriever` in your NLP system, make sure it has the query and document embeddings available. You can do so by adding a Document Embedder to your indexing pipeline and a Text Embedder to your query pipeline.
|
||||
|
||||
In addition to the `query_embedding`, the `AstraEmbeddingRetriever` accepts other optional parameters, including `top_k` (the maximum number of documents to retrieve) and `filters` to narrow down the search space.
|
||||
|
||||
### Setup and installation
|
||||
|
||||
Once you have an AstraDB account and have created a database, install the `astra-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install astra-haystack
|
||||
```
|
||||
|
||||
From the configuration in AstraDB’s web UI, you need the database ID and a generated token.
|
||||
|
||||
You will additionally need a collection name and a namespace. When you create the collection name, you also need to set the embedding dimensions and the similarity metric. The namespace organizes data in a database and is called a keyspace in Apache Cassandra.
|
||||
|
||||
Then, optionally, install the `sentence-transformers-haystack` package as well to run the example below:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
We strongly encourage passing authentication data through environment variables: make sure to populate the environment variables `ASTRA_DB_API_ENDPOINT` and `ASTRA_DB_APPLICATION_TOKEN` before running the following example.
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Use this Retriever in a query pipeline like this:
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersTextEmbedder,
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
)
|
||||
from haystack_integrations.components.retrievers.astra import AstraEmbeddingRetriever
|
||||
from haystack_integrations.document_stores.astra import AstraDocumentStore
|
||||
|
||||
document_store = AstraDocumentStore()
|
||||
|
||||
model = "sentence-transformers/all-mpnet-base-v2"
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
document_embedder = SentenceTransformersDocumentEmbedder(model=model)
|
||||
documents_with_embeddings = document_embedder.run(documents)
|
||||
|
||||
document_store.write_documents(
|
||||
documents_with_embeddings.get("documents"),
|
||||
policy=DuplicatePolicy.SKIP,
|
||||
)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component(
|
||||
"text_embedder",
|
||||
SentenceTransformersTextEmbedder(model=model),
|
||||
)
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
AstraEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "How many languages are there?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
```
|
||||
|
||||
The example output would be:
|
||||
|
||||
```python
|
||||
Document(id=cfe93bc1c274908801e6670440bf2bbba54fad792770d57421f85ffa2a4fcc94, content: 'There are over 7,000 languages spoken around the world today.', score: 0.8929937, embedding: vector of size 768)
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Using AstraDB as a data store in your Haystack pipelines](https://haystack.deepset.ai/cookbook/astradb_haystack_integration)
|
||||
@@ -0,0 +1,173 @@
|
||||
---
|
||||
title: "AutoMergingRetriever"
|
||||
id: automergingretriever
|
||||
slug: "/automergingretriever"
|
||||
description: "Use AutoMergingRetriever to improve search results by returning complete parent documents instead of fragmented chunks when multiple related pieces match a query."
|
||||
---
|
||||
|
||||
# AutoMergingRetriever
|
||||
|
||||
Use AutoMergingRetriever to improve search results by returning complete parent documents instead of fragmented chunks when multiple related pieces match a query.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Used after the main Retriever component that returns hierarchical documents. |
|
||||
| **Mandatory init variables** | `document_store`: Document Store from which to retrieve the parent documents |
|
||||
| **Mandatory run variables** | `documents`: A list of leaf documents that were matched by a Retriever |
|
||||
| **Output variables** | `documents`: A list resulting documents |
|
||||
| **API reference** | [Retrievers](/reference/retrievers-api) |
|
||||
| **GitHub link** | [https://github.com/deepset-ai/haystack/blob/dae8c7babaf28d2ffab4f2a8dedecd63e2394fb4/haystack/components/retrievers/auto_merging_retriever.py](https://github.com/deepset-ai/haystack/blob/dae8c7babaf28d2ffab4f2a8dedecd63e2394fb4/haystack/components/retrievers/auto_merging_retriever.py#L116) |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `AutoMergingRetriever` is a component that works with a hierarchical document structure. It returns the parent documents instead of individual leaf documents when a certain threshold is met.
|
||||
|
||||
This can be particularly useful when working with paragraphs split into multiple chunks. When several chunks from the same paragraph match your query, the complete paragraph often provides more context and value than the individual pieces alone.
|
||||
|
||||
Here is how this Retriever works:
|
||||
|
||||
1. It requires documents to be organized in a tree structure, with leaf nodes stored in a document index - see [`HierarchicalDocumentSplitter`](../preprocessors/hierarchicaldocumentsplitter.mdx) documentation.
|
||||
2. When searching, it counts how many leaf documents under the same parent match your query.
|
||||
3. If this count exceeds your defined threshold, it returns the parent document instead of the individual leaves.
|
||||
|
||||
The `AutoMergingRetriever` can currently be used by the following Document Stores:
|
||||
|
||||
- [AstraDocumentStore](../../document-stores/astradocumentstore.mdx)
|
||||
- [ElasticsearchDocumentStore](../../document-stores/elasticsearch-document-store.mdx)
|
||||
- [OpenSearchDocumentStore](../../document-stores/opensearch-document-store.mdx)
|
||||
- [PgvectorDocumentStore](../../document-stores/pgvectordocumentstore.mdx)
|
||||
- [QdrantDocumentStore](../../document-stores/qdrant-document-store.mdx)
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.preprocessors import HierarchicalDocumentSplitter
|
||||
from haystack.components.retrievers.auto_merging_retriever import AutoMergingRetriever
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
# create a hierarchical document structure with 3 levels, where the parent document has 3 children
|
||||
text = "The sun rose early in the morning. It cast a warm glow over the trees. Birds began to sing."
|
||||
original_document = Document(content=text)
|
||||
builder = HierarchicalDocumentSplitter(block_sizes=[10, 3], split_overlap=0, split_by="word")
|
||||
docs = builder.run([original_document])["documents"]
|
||||
|
||||
# store level-1 parent documents and initialize the retriever
|
||||
doc_store_parents = InMemoryDocumentStore()
|
||||
for doc in docs["documents"]:
|
||||
if doc.meta["children_ids"] and doc.meta["level"] == 1:
|
||||
doc_store_parents.write_documents([doc])
|
||||
retriever = AutoMergingRetriever(doc_store_parents, threshold=0.5)
|
||||
|
||||
# assume we retrieved 2 leaf docs from the same parent, the parent document should be returned,
|
||||
# since it has 3 children and the threshold=0.5, and we retrieved 2 children (2/3 > 0.66(6))
|
||||
leaf_docs = [doc for doc in docs["documents"] if not doc.meta["children_ids"]]
|
||||
docs = retriever.run(leaf_docs[4:6])
|
||||
>> {'documents': [Document(id=538..),
|
||||
>> content: 'warm glow over the trees. Birds began to sing.',
|
||||
>> meta: {'block_size': 10, 'parent_id': '835..', 'children_ids': ['c17...', '3ff...', '352...'], 'level': 1, 'source_id': '835...',
|
||||
>> 'page_number': 1, 'split_id': 1, 'split_idx_start': 45})]}
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
This is an example of a RAG Haystack pipeline. It first retrieves leaf-level document chunks using BM25, merges them into higher-level parent documents with `AutoMergingRetriever`, constructs a prompt, and generates an answer using OpenAI's chat model.
|
||||
|
||||
```python
|
||||
from typing import List, Tuple
|
||||
from haystack import Document, Pipeline
|
||||
from haystack_experimental.components.splitters import HierarchicalDocumentSplitter
|
||||
from haystack.components.builders.answer_builder import AnswerBuilder
|
||||
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.components.retrievers import AutoMergingRetriever
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
|
||||
def indexing(
|
||||
documents: List[Document],
|
||||
) -> Tuple[InMemoryDocumentStore, InMemoryDocumentStore]:
|
||||
splitter = HierarchicalDocumentSplitter(
|
||||
block_sizes={10, 3},
|
||||
split_overlap=0,
|
||||
split_by="word",
|
||||
)
|
||||
docs = splitter.run(documents)
|
||||
|
||||
leaf_documents = [doc for doc in docs["documents"] if doc.meta["__level"] == 1]
|
||||
leaf_doc_store = InMemoryDocumentStore()
|
||||
leaf_doc_store.write_documents(leaf_documents, policy=DuplicatePolicy.OVERWRITE)
|
||||
|
||||
parent_documents = [doc for doc in docs["documents"] if doc.meta["__level"] == 0]
|
||||
parent_doc_store = InMemoryDocumentStore()
|
||||
parent_doc_store.write_documents(parent_documents, policy=DuplicatePolicy.OVERWRITE)
|
||||
|
||||
return leaf_doc_store, parent_doc_store
|
||||
|
||||
|
||||
# Add documents
|
||||
docs = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
leaf_docs, parent_docs = indexing(docs)
|
||||
|
||||
prompt_template = [
|
||||
ChatMessage.from_system("You are a helpful assistant."),
|
||||
ChatMessage.from_user(
|
||||
"Given these documents, answer the question.\nDocuments:\n"
|
||||
"{% for doc in documents %}{{ doc.content }}{% endfor %}\n"
|
||||
"Question: {{question}}\nAnswer:",
|
||||
),
|
||||
]
|
||||
|
||||
rag_pipeline = Pipeline()
|
||||
rag_pipeline.add_component(
|
||||
instance=InMemoryBM25Retriever(document_store=leaf_docs),
|
||||
name="bm25_retriever",
|
||||
)
|
||||
rag_pipeline.add_component(
|
||||
instance=AutoMergingRetriever(parent_docs, threshold=0.6),
|
||||
name="retriever",
|
||||
)
|
||||
rag_pipeline.add_component(
|
||||
instance=ChatPromptBuilder(
|
||||
template=prompt_template,
|
||||
required_variables={"question", "documents"},
|
||||
),
|
||||
name="prompt_builder",
|
||||
)
|
||||
rag_pipeline.add_component(instance=OpenAIChatGenerator(), name="llm")
|
||||
rag_pipeline.add_component(instance=AnswerBuilder(), name="answer_builder")
|
||||
|
||||
rag_pipeline.connect("bm25_retriever.documents", "retriever.documents")
|
||||
rag_pipeline.connect("retriever", "prompt_builder.documents")
|
||||
rag_pipeline.connect("prompt_builder.prompt", "llm.messages")
|
||||
rag_pipeline.connect("llm.replies", "answer_builder.replies")
|
||||
rag_pipeline.connect("retriever", "answer_builder.documents")
|
||||
|
||||
question = "How many languages are there?"
|
||||
result = rag_pipeline.run(
|
||||
{
|
||||
"bm25_retriever": {"query": question},
|
||||
"prompt_builder": {"question": question},
|
||||
"answer_builder": {"query": question},
|
||||
},
|
||||
)
|
||||
```
|
||||
@@ -0,0 +1,158 @@
|
||||
---
|
||||
title: "AzureAISearchBM25Retriever"
|
||||
id: azureaisearchbm25retriever
|
||||
slug: "/azureaisearchbm25retriever"
|
||||
description: "A keyword-based Retriever that fetches Documents matching a query from the Azure AI Search Document Store."
|
||||
---
|
||||
|
||||
# AzureAISearchBM25Retriever
|
||||
|
||||
A keyword-based Retriever that fetches Documents matching a query from the Azure AI Search Document Store.
|
||||
|
||||
A keyword-based Retriever that fetches documents matching a query from the Azure AI Search Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. Before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the semantic search pipeline 3. Before an [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of [`AzureAISearchDocumentStore`](../../document-stores/azureaisearchdocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query`: A string |
|
||||
| **Output variables** | `documents`: A list of documents (matching the query) |
|
||||
| **API reference** | [Azure AI Search](/reference/integrations-azure_ai_search) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/azure_ai_search |
|
||||
| **Package name** | `azure-ai-search-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `AzureAISearchBM25Retriever` is a keyword-based Retriever designed to fetch documents that match a query from an `AzureAISearchDocumentStore`. It uses the BM25 algorithm which calculates a weighted word overlap between the query and the documents to determine their similarity. The Retriever accepts textual query but you can also provide a combination of terms with boolean operators. Some examples of valid queries could be `"pool"`, `"pool spa"`, and `"pool spa +airport"`.
|
||||
|
||||
In addition to the `query`, the `AzureAISearchBM25Retriever` accepts other optional parameters, including `top_k` (the maximum number of documents to retrieve) and `filters` to narrow down the search space.
|
||||
|
||||
If your search index includes a [semantic configuration](https://learn.microsoft.com/en-us/azure/search/semantic-how-to-query-request), you can enable semantic ranking to apply it to the Retriever's results. For more details, refer to the [Azure AI documentation](https://learn.microsoft.com/en-us/azure/search/hybrid-search-how-to-query#semantic-hybrid-search).
|
||||
|
||||
If you want a combination of BM25 and vector retrieval, use the `AzureAISearchHybridRetriever`, which uses both vector search and BM25 search to match documents and query.
|
||||
|
||||
## Usage
|
||||
|
||||
### Installation
|
||||
|
||||
This integration requires you to have an active Azure subscription with a deployed [Azure AI Search](https://azure.microsoft.com/en-us/products/ai-services/ai-search) service.
|
||||
|
||||
To start using Azure AI search with Haystack, install the package with:
|
||||
|
||||
```shell
|
||||
pip install azure-ai-search-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
This Retriever needs `AzureAISearchDocumentStore` and indexed documents to run.
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.retrievers.azure_ai_search import (
|
||||
AzureAISearchBM25Retriever,
|
||||
)
|
||||
from haystack_integrations.document_stores.azure_ai_search import (
|
||||
AzureAISearchDocumentStore,
|
||||
)
|
||||
|
||||
document_store = AzureAISearchDocumentStore(index_name="haystack_docs")
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
document_store.write_documents(documents=documents)
|
||||
|
||||
retriever = AzureAISearchBM25Retriever(document_store=document_store)
|
||||
retriever.run(query="How many languages are spoken around the world today?")
|
||||
```
|
||||
|
||||
### In a RAG pipeline
|
||||
|
||||
The below example shows how to use the `AzureAISearchBM25Retriever` in a RAG pipeline. Set your `OPENAI_API_KEY` as an environment variable and then run the following code:
|
||||
|
||||
```python
|
||||
|
||||
from haystack_integrations.components.retrievers.azure_ai_search import (
|
||||
AzureAISearchBM25Retriever,
|
||||
)
|
||||
from haystack_integrations.document_stores.azure_ai_search import (
|
||||
AzureAISearchDocumentStore,
|
||||
)
|
||||
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders.answer_builder import AnswerBuilder
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
|
||||
import os
|
||||
|
||||
api_key = os.environ["OPENAI_API_KEY"]
|
||||
|
||||
# Create a RAG query pipeline
|
||||
prompt_template = [
|
||||
ChatMessage.from_user(
|
||||
"""
|
||||
Given these documents, answer the question.\nDocuments:
|
||||
{% for doc in documents %}
|
||||
{{ doc.content }}
|
||||
{% endfor %}
|
||||
|
||||
\nQuestion: {{question}}
|
||||
\nAnswer:
|
||||
""",
|
||||
),
|
||||
]
|
||||
|
||||
document_store = AzureAISearchDocumentStore(index_name="haystack-docs")
|
||||
|
||||
# Add Documents
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
# policy param is optional, as AzureAISearchDocumentStore has a default policy of DuplicatePolicy.OVERWRITE
|
||||
document_store.write_documents(documents=documents, policy=DuplicatePolicy.OVERWRITE)
|
||||
|
||||
retriever = AzureAISearchBM25Retriever(document_store=document_store)
|
||||
rag_pipeline = Pipeline()
|
||||
rag_pipeline.add_component(name="retriever", instance=retriever)
|
||||
rag_pipeline.add_component(
|
||||
instance=ChatPromptBuilder(template=prompt_template, required_variables="*"),
|
||||
name="prompt_builder",
|
||||
)
|
||||
rag_pipeline.add_component(instance=OpenAIChatGenerator(), name="llm")
|
||||
rag_pipeline.add_component(instance=AnswerBuilder(), name="answer_builder")
|
||||
rag_pipeline.connect("retriever", "prompt_builder.documents")
|
||||
rag_pipeline.connect("prompt_builder.prompt", "llm.messages")
|
||||
rag_pipeline.connect("llm.replies", "answer_builder.replies")
|
||||
rag_pipeline.connect("retriever", "answer_builder.documents")
|
||||
|
||||
question = "Tell me something about languages?"
|
||||
result = rag_pipeline.run(
|
||||
{
|
||||
"retriever": {"query": question},
|
||||
"prompt_builder": {"question": question},
|
||||
"answer_builder": {"query": question},
|
||||
},
|
||||
)
|
||||
print(result["answer_builder"]["answers"][0])
|
||||
```
|
||||
@@ -0,0 +1,150 @@
|
||||
---
|
||||
title: "AzureAISearchEmbeddingRetriever"
|
||||
id: azureaisearchembeddingretriever
|
||||
slug: "/azureaisearchembeddingretriever"
|
||||
description: "An embedding Retriever compatible with the Azure AI Search Document Store."
|
||||
---
|
||||
|
||||
# AzureAISearchEmbeddingRetriever
|
||||
|
||||
An embedding Retriever compatible with the Azure AI Search Document Store.
|
||||
|
||||
This Retriever accepts the embeddings of a single query as input and returns a list of matching documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the embedding retrieval pipeline 3. After a Text Embedder and before an [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of [`AzureAISearchDocumentStore`](../../document-stores/azureaisearchdocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query_embedding`: A list of floats |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Azure AI Search](/reference/integrations-azure_ai_search) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/azure_ai_search |
|
||||
| **Package name** | `azure-ai-search-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `AzureAISearchEmbeddingRetriever` is an embedding-based Retriever compatible with the `AzureAISearchDocumentStore`. It compares the query and document embeddings and fetches the most relevant documents from the `AzureAISearchDocumentStore` based on the outcome.
|
||||
|
||||
The query needs to be embedded before being passed to this component. For example, you could use a Text [Embedder](../embedders.mdx) component.
|
||||
|
||||
By default, the `AzureAISearchDocumentStore` uses the [HNSW algorithm](https://learn.microsoft.com/en-us/azure/search/vector-search-overview#nearest-neighbors-search) with cosine similarity to handle vector searches. The vector configuration is set during the initialization of the document store and can be customized by providing the `vector_search_configuration` parameter.
|
||||
|
||||
In addition to the `query_embedding`, the `AzureAISearchEmbeddingRetriever` accepts other optional parameters, including `top_k` (the maximum number of documents to retrieve) and `filters` to narrow down the search space.
|
||||
|
||||
:::info[Semantic Ranking]
|
||||
|
||||
The semantic ranking capability of Azure AI Search is not available for vector retrieval. To include semantic ranking in your retrieval process, use the [`AzureAISearchBM25Retriever`](azureaisearchbm25retriever.mdx) or [`AzureAISearchHybridRetriever`](azureaisearchhybridretriever.mdx). For more details, see [Azure AI documentation](https://learn.microsoft.com/en-us/azure/search/semantic-how-to-query-request?tabs=portal-query#set-up-the-query).
|
||||
:::
|
||||
|
||||
## Usage
|
||||
|
||||
### Installation
|
||||
|
||||
This integration requires you to have an active Azure subscription with a deployed [Azure AI Search](https://azure.microsoft.com/en-us/products/ai-services/ai-search) service.
|
||||
|
||||
To start using Azure AI search with Haystack, install the package with:
|
||||
|
||||
```shell
|
||||
pip install azure-ai-search-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
This Retriever needs `AzureAISearchDocumentStore` and indexed documents to run.
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.azure_ai_search import (
|
||||
AzureAISearchDocumentStore,
|
||||
)
|
||||
from haystack_integrations.components.retrievers.azure_ai_search import (
|
||||
AzureAISearchEmbeddingRetriever,
|
||||
)
|
||||
|
||||
document_store = AzureAISearchDocumentStore()
|
||||
|
||||
retriever = AzureAISearchEmbeddingRetriever(document_store=document_store)
|
||||
|
||||
# example run query
|
||||
retriever.run(query_embedding=[0.1] * 384)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Here is how you could use the `AzureAISearchEmbeddingRetriever` in a pipeline. In this example, you would create two pipelines: an indexing one and a querying one.
|
||||
|
||||
In the indexing pipeline, the documents are passed to the Document Embedder and then written into the Document Store.
|
||||
|
||||
Then, in the querying pipeline, we use a Text Embedder to get the vector representation of the input query that will be then passed to the `AzureAISearchEmbeddingRetriever` to get the results.
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
SentenceTransformersTextEmbedder,
|
||||
)
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
from haystack_integrations.components.retrievers.azure_ai_search import (
|
||||
AzureAISearchEmbeddingRetriever,
|
||||
)
|
||||
from haystack_integrations.document_stores.azure_ai_search import (
|
||||
AzureAISearchDocumentStore,
|
||||
)
|
||||
|
||||
document_store = AzureAISearchDocumentStore(index_name="retrieval-example")
|
||||
|
||||
model = "sentence-transformers/all-mpnet-base-v2"
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="""Elephants have been observed to behave in a way that indicates a
|
||||
high level of self-awareness, such as recognizing themselves in mirrors.""",
|
||||
),
|
||||
Document(
|
||||
content="""In certain parts of the world, like the Maldives, Puerto Rico, and
|
||||
San Diego, you can witness the phenomenon of bioluminescent waves.""",
|
||||
),
|
||||
]
|
||||
|
||||
document_embedder = SentenceTransformersDocumentEmbedder(model=model)
|
||||
|
||||
# Indexing Pipeline
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component(instance=document_embedder, name="doc_embedder")
|
||||
indexing_pipeline.add_component(
|
||||
instance=DocumentWriter(document_store=document_store),
|
||||
name="doc_writer",
|
||||
)
|
||||
indexing_pipeline.connect("doc_embedder", "doc_writer")
|
||||
|
||||
indexing_pipeline.run({"doc_embedder": {"documents": documents}})
|
||||
|
||||
# Query Pipeline
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component(
|
||||
"text_embedder",
|
||||
SentenceTransformersTextEmbedder(model=model),
|
||||
)
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
AzureAISearchEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "How many languages are there?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
```
|
||||
@@ -0,0 +1,156 @@
|
||||
---
|
||||
title: "AzureAISearchHybridRetriever"
|
||||
id: azureaisearchhybridretriever
|
||||
slug: "/azureaisearchhybridretriever"
|
||||
description: "A Retriever based both on dense and sparse embeddings, compatible with the Azure AI Search Document Store."
|
||||
---
|
||||
|
||||
# AzureAISearchHybridRetriever
|
||||
|
||||
A Retriever based both on dense and sparse embeddings, compatible with the Azure AI Search Document Store.
|
||||
|
||||
This Retriever combines embedding-based retrieval and BM25 text search search to find matching documents in the search index to get more relevant results.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. After a TextEmbedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in a hybrid search pipeline 3. After a TextEmbedder and before an [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of [`AzureAISearchDocumentStore`](../../document-stores/azureaisearchdocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query`: A string <br /> <br />`query_embedding`: A list of floats |
|
||||
| **Output variables** | `documents`: A list of documents (matching the query) |
|
||||
| **API reference** | [Azure AI Search](/reference/integrations-azure_ai_search) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/azure_ai_search |
|
||||
| **Package name** | `azure-ai-search-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `AzureAISearchHybridRetriever` combines vector retrieval and BM25 text search to fetch relevant documents from the `AzureAISearchDocumentStore`. It processes both textual (keyword) queries and query embeddings in a single request, executing all subqueries in parallel. The results are merged and reordered using [Reciprocal Rank Fusion (RRF)](https://learn.microsoft.com/en-us/azure/search/hybrid-search-ranking) to create a unified result set.
|
||||
|
||||
Besides the `query` and `query_embedding`, the `AzureAISearchHybridRetriever` accepts optional parameters such as `top_k` (the maximum number of documents to retrieve) and `filters` to refine the search. Additional keyword arguments can also be passed during initialization for further customization.
|
||||
|
||||
If your search index includes a [semantic configuration](https://learn.microsoft.com/en-us/azure/search/semantic-how-to-query-request), you can enable semantic ranking to apply it to the Retriever's results. For more details, refer to the [Azure AI documentation](https://learn.microsoft.com/en-us/azure/search/hybrid-search-how-to-query#semantic-hybrid-search).
|
||||
|
||||
For purely keyword-based retrieval, you can use `AzureAISearchBM25Retriever`, and for embedding-based retrieval, `AzureAISearchEmbeddingRetriever` is available.
|
||||
|
||||
## Usage
|
||||
|
||||
### Installation
|
||||
|
||||
This integration requires you to have an active Azure subscription with a deployed [Azure AI Search](https://azure.microsoft.com/en-us/products/ai-services/ai-search) service.
|
||||
|
||||
To start using Azure AI search with Haystack, install the package with:
|
||||
|
||||
```shell
|
||||
pip install azure-ai-search-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
This Retriever needs `AzureAISearchDocumentStore` and indexed documents to run.
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.retrievers.azure_ai_search import (
|
||||
AzureAISearchHybridRetriever,
|
||||
)
|
||||
from haystack_integrations.document_stores.azure_ai_search import (
|
||||
AzureAISearchDocumentStore,
|
||||
)
|
||||
|
||||
document_store = AzureAISearchDocumentStore(index_name="haystack_docs")
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
document_store.write_documents(documents=documents)
|
||||
|
||||
retriever = AzureAISearchHybridRetriever(document_store=document_store)
|
||||
# fake embeddings to keep the example simple
|
||||
retriever.run(
|
||||
query="How many languages are spoken around the world today?",
|
||||
query_embedding=[0.1] * 384,
|
||||
)
|
||||
```
|
||||
|
||||
### In a RAG pipeline
|
||||
|
||||
The following example demonstrates using the `AzureAISearchHybridRetriever` in a pipeline. An indexing pipeline is responsible for indexing and storing documents with embeddings in the `AzureAISearchDocumentStore`, while the query pipeline uses hybrid retrieval to fetch relevant documents based on a given query.
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
SentenceTransformersTextEmbedder,
|
||||
)
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
from haystack_integrations.components.retrievers.azure_ai_search import (
|
||||
AzureAISearchHybridRetriever,
|
||||
)
|
||||
from haystack_integrations.document_stores.azure_ai_search import (
|
||||
AzureAISearchDocumentStore,
|
||||
)
|
||||
|
||||
document_store = AzureAISearchDocumentStore(index_name="hybrid-retrieval-example")
|
||||
|
||||
model = "sentence-transformers/all-mpnet-base-v2"
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="""Elephants have been observed to behave in a way that indicates a
|
||||
high level of self-awareness, such as recognizing themselves in mirrors.""",
|
||||
),
|
||||
Document(
|
||||
content="""In certain parts of the world, like the Maldives, Puerto Rico, and
|
||||
San Diego, you can witness the phenomenon of bioluminescent waves.""",
|
||||
),
|
||||
]
|
||||
|
||||
document_embedder = SentenceTransformersDocumentEmbedder(model=model)
|
||||
|
||||
# Indexing Pipeline
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component(instance=document_embedder, name="doc_embedder")
|
||||
indexing_pipeline.add_component(
|
||||
instance=DocumentWriter(document_store=document_store),
|
||||
name="doc_writer",
|
||||
)
|
||||
indexing_pipeline.connect("doc_embedder", "doc_writer")
|
||||
|
||||
indexing_pipeline.run({"doc_embedder": {"documents": documents}})
|
||||
|
||||
# Query Pipeline
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component(
|
||||
"text_embedder",
|
||||
SentenceTransformersTextEmbedder(model=model),
|
||||
)
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
AzureAISearchHybridRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "How many languages are there?"
|
||||
|
||||
result = query_pipeline.run(
|
||||
{"text_embedder": {"text": query}, "retriever": {"query": query}},
|
||||
)
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
```
|
||||
@@ -0,0 +1,112 @@
|
||||
---
|
||||
title: "ChromaEmbeddingRetriever"
|
||||
id: chromaembeddingretriever
|
||||
slug: "/chromaembeddingretriever"
|
||||
description: "This is an embedding Retriever compatible with the Chroma Document Store."
|
||||
---
|
||||
|
||||
# ChromaEmbeddingRetriever
|
||||
|
||||
This is an embedding Retriever compatible with the Chroma Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the semantic search pipeline 3. After a Text Embedder and before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a [ChromaDocumentStore](../../document-stores/chromadocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query_embedding`: A list of floats |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Chroma](/reference/integrations-chroma) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/chroma |
|
||||
| **Package name** | `chroma-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `ChromaEmbeddingRetriever` is an embedding-based Retriever compatible with the `ChromaDocumentStore`. It compares the query and document embeddings and fetches the documents most relevant to the query from the `ChromaDocumentStore` based on the outcome.
|
||||
|
||||
The query needs to be embedded before being passed to this component. For example, you could use a text [embedder](../embedders.mdx) component.
|
||||
|
||||
In addition to the `query_embedding`, the `ChromaEmbeddingRetriever` accepts other optional parameters, including `top_k` (the maximum number of documents to retrieve) and `filters` to narrow down the search space.
|
||||
|
||||
### Usage
|
||||
|
||||
#### On its own
|
||||
|
||||
This Retriever needs the `ChromaDocumentStore` and indexed documents to run.
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.chroma import ChromaDocumentStore
|
||||
from haystack_integrations.components.retrievers.chroma import ChromaEmbeddingRetriever
|
||||
|
||||
document_store = ChromaDocumentStore()
|
||||
|
||||
retriever = ChromaEmbeddingRetriever(document_store=document_store)
|
||||
|
||||
# example run query
|
||||
retriever.run(query_embedding=[0.1] * 384)
|
||||
```
|
||||
|
||||
#### In a pipeline
|
||||
|
||||
Here is how you could use the `ChromaEmbeddingRetriever` in a pipeline. In this example, you would create two pipelines: an indexing one and a querying one.
|
||||
|
||||
In the indexing pipeline, the documents are passed to the Document Embedder and then written into the document Store.
|
||||
|
||||
Then, in the querying pipeline, we use a text embedder to get the vector representation of the input query that will be then passed to the `ChromaEmbeddingRetriever` to get the results.
|
||||
|
||||
```python
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from haystack import Pipeline
|
||||
from haystack.dataclasses import Document
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
# Note: the following requires a "pip install sentence-transformers-haystack"
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
SentenceTransformersTextEmbedder,
|
||||
)
|
||||
|
||||
from haystack_integrations.document_stores.chroma import ChromaDocumentStore
|
||||
from haystack_integrations.components.retrievers.chroma import ChromaEmbeddingRetriever
|
||||
from sentence_transformers import SentenceTransformer
|
||||
|
||||
# Chroma is used in-memory so we use the same instances in the two pipelines below
|
||||
document_store = ChromaDocumentStore()
|
||||
|
||||
documents = [
|
||||
Document(content="This contains variable declarations", meta={"title": "one"}),
|
||||
Document(
|
||||
content="This contains another sort of variable declarations",
|
||||
meta={"title": "two"},
|
||||
),
|
||||
Document(
|
||||
content="This has nothing to do with variable declarations",
|
||||
meta={"title": "three"},
|
||||
),
|
||||
Document(content="A random doc", meta={"title": "four"}),
|
||||
]
|
||||
|
||||
indexing = Pipeline()
|
||||
indexing.add_component("embedder", SentenceTransformersDocumentEmbedder())
|
||||
indexing.add_component("writer", DocumentWriter(document_store))
|
||||
indexing.connect("embedder.documents", "writer.documents")
|
||||
indexing.run({"embedder": {"documents": documents}})
|
||||
|
||||
querying = Pipeline()
|
||||
querying.add_component("query_embedder", SentenceTransformersTextEmbedder())
|
||||
querying.add_component("retriever", ChromaEmbeddingRetriever(document_store))
|
||||
querying.connect("query_embedder.embedding", "retriever.query_embedding")
|
||||
results = querying.run({"query_embedder": {"text": "Variable declarations"}})
|
||||
|
||||
for d in results["retriever"]["documents"]:
|
||||
print(d.meta, d.score)
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Use Chroma for RAG and Indexing](https://haystack.deepset.ai/cookbook/chroma-indexing-and-rag-examples)
|
||||
@@ -0,0 +1,99 @@
|
||||
---
|
||||
title: "ChromaQueryTextRetriever"
|
||||
id: chromaqueryretriever
|
||||
slug: "/chromaqueryretriever"
|
||||
description: "This is a a Retriever compatible with the Chroma Document Store."
|
||||
---
|
||||
|
||||
# ChromaQueryTextRetriever
|
||||
|
||||
This is a a Retriever compatible with the Chroma Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the semantic search pipeline 3. After a Text Embedder and before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a [ChromaDocumentStore](../../document-stores/chromadocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query`: A single query in plain-text format to be processed by the [Retriever](../retrievers.mdx) |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Chroma](/reference/integrations-chroma) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/chroma |
|
||||
| **Package name** | `chroma-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `ChromaQueryTextRetriever` is an embedding-based Retriever compatible with the `ChromaDocumentStore` that uses the Chroma [query API](https://docs.trychroma.com/reference/Collection#query).
|
||||
This component takes a plain-text query string in input and returns the matching documents.
|
||||
Chroma will create the embedding for the query using its [embedding function](https://docs.trychroma.com/embeddings#default-all-minilm-l6-v2); in case you do not want to use the default embedding function, this must be specified at `ChromaDocumentStore` initialization.
|
||||
|
||||
### Usage
|
||||
|
||||
#### On its own
|
||||
|
||||
This Retriever needs the `ChromaDocumentStore` and indexed documents to run.
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.chroma import ChromaDocumentStore
|
||||
from haystack_integrations.components.retrievers.chroma import ChromaQueryTextRetriever
|
||||
|
||||
document_store = ChromaDocumentStore()
|
||||
|
||||
retriever = ChromaQueryTextRetriever(document_store=document_store)
|
||||
|
||||
# example run query
|
||||
retriever.run(query="How does Chroma Retriever work?")
|
||||
```
|
||||
|
||||
#### In a pipeline
|
||||
|
||||
Here is how you could use the `ChromaQueryTextRetriever` in a Pipeline. In this example, you would create two pipelines: an indexing one and a querying one.
|
||||
|
||||
In the indexing pipeline, the documents are written in the Document Store.
|
||||
|
||||
Then, in the querying pipeline, `ChromaQueryTextRetriever` gets the answer from the Document Store based on the provided query.
|
||||
|
||||
```python
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from haystack import Pipeline
|
||||
from haystack.dataclasses import Document
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
from haystack_integrations.document_stores.chroma import ChromaDocumentStore
|
||||
from haystack_integrations.components.retrievers.chroma import ChromaQueryTextRetriever
|
||||
|
||||
# Chroma is used in-memory so we use the same instances in the two pipelines below
|
||||
document_store = ChromaDocumentStore()
|
||||
|
||||
documents = [
|
||||
Document(content="This contains variable declarations", meta={"title": "one"}),
|
||||
Document(
|
||||
content="This contains another sort of variable declarations",
|
||||
meta={"title": "two"},
|
||||
),
|
||||
Document(
|
||||
content="This has nothing to do with variable declarations",
|
||||
meta={"title": "three"},
|
||||
),
|
||||
Document(content="A random doc", meta={"title": "four"}),
|
||||
]
|
||||
|
||||
indexing = Pipeline()
|
||||
indexing.add_component("writer", DocumentWriter(document_store))
|
||||
indexing.run({"writer": {"documents": documents}})
|
||||
|
||||
querying = Pipeline()
|
||||
querying.add_component("retriever", ChromaQueryTextRetriever(document_store))
|
||||
results = querying.run({"retriever": {"query": "Variable declarations", "top_k": 3}})
|
||||
|
||||
for d in results["retriever"]["documents"]:
|
||||
print(d.meta, d.score)
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Use Chroma for RAG and Indexing](https://haystack.deepset.ai/cookbook/chroma-indexing-and-rag-examples)
|
||||
@@ -0,0 +1,139 @@
|
||||
---
|
||||
title: "CogneeRetriever"
|
||||
id: cogneeretriever
|
||||
slug: "/cogneeretriever"
|
||||
description: "Retrieves memories from a CogneeMemoryStore and returns them as system ChatMessage objects."
|
||||
---
|
||||
|
||||
# CogneeRetriever
|
||||
|
||||
Retrieves memories from a `CogneeMemoryStore` and returns them as system `ChatMessage` objects.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before an [`Agent`](../agents-1/agent.mdx) or Chat Generator in memory-augmented pipelines |
|
||||
| **Mandatory init variables** | `memory_store`: A `CogneeMemoryStore` instance |
|
||||
| **Optional init variables** | `top_k`: Maximum number of memories to return (defaults to the store's `top_k`) |
|
||||
| **Mandatory run variables** | `query`: A text query to search memories |
|
||||
| **Optional run variables** | `user_id`: Cognee user ID to scope the retrieval; pass `None` to use Cognee's default user |
|
||||
| **Output variables** | `messages`: A list of system `ChatMessage` objects |
|
||||
| **API reference** | [Cognee](/reference/integrations-cognee#cogneeretriever) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/cognee |
|
||||
| **Package name** | `cognee-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`CogneeRetriever` retrieves memories from a `CogneeMemoryStore` and returns them as system `ChatMessage` objects.
|
||||
|
||||
Use it to inject long-term memory into an Agent or a chat generation pipeline before the model produces a response.
|
||||
|
||||
Search behavior — including the search strategy (`search_type`), dataset, and session tier — is configured on the `CogneeMemoryStore`. The retriever is a thin pipeline adapter over `search_memories`.
|
||||
|
||||
The `user_id` parameter scopes the retrieval to a specific Cognee user. Pass `None` to use Cognee's default user.
|
||||
|
||||
## Installation
|
||||
|
||||
Install the Cognee integration:
|
||||
|
||||
```bash
|
||||
pip install cognee-haystack
|
||||
```
|
||||
|
||||
Set your LLM API key (used by Cognee for graph extraction and queries):
|
||||
|
||||
```bash
|
||||
export LLM_API_KEY="your-llm-api-key"
|
||||
```
|
||||
|
||||
Optionally, set a separate embedding API key (defaults to `LLM_API_KEY` when unset):
|
||||
|
||||
```bash
|
||||
export EMBEDDING_API_KEY="your-embedding-api-key"
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
from haystack_integrations.components.retrievers.cognee import CogneeRetriever
|
||||
from haystack_integrations.memory_stores.cognee import CogneeMemoryStore
|
||||
|
||||
store = CogneeMemoryStore(search_type="GRAPH_COMPLETION", top_k=5)
|
||||
|
||||
# Write some memories first
|
||||
store.add_memories(
|
||||
messages=[ChatMessage.from_user("Alice prefers concise Python examples.")],
|
||||
user_id="a1b2c3d4-e5f6-7890-abcd-ef1234567890",
|
||||
)
|
||||
|
||||
retriever = CogneeRetriever(memory_store=store, top_k=3)
|
||||
|
||||
result = retriever.run(
|
||||
query="What does Alice prefer?",
|
||||
user_id="a1b2c3d4-e5f6-7890-abcd-ef1234567890",
|
||||
)
|
||||
memories = result["messages"]
|
||||
print([message.text for message in memories])
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
This example retrieves memories, prepends them to the current user message, and passes the combined message list to an Agent.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.converters import OutputAdapter
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
from haystack_integrations.components.retrievers.cognee import CogneeRetriever
|
||||
from haystack_integrations.memory_stores.cognee import CogneeMemoryStore
|
||||
|
||||
store = CogneeMemoryStore(dataset_name="my_agent_memory", session_id="alice_session_1")
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("retriever", CogneeRetriever(memory_store=store, top_k=5))
|
||||
pipeline.add_component(
|
||||
"memory_context",
|
||||
OutputAdapter(
|
||||
template="{{ memories + user_messages }}",
|
||||
output_type=list[ChatMessage],
|
||||
unsafe=True,
|
||||
),
|
||||
)
|
||||
pipeline.add_component(
|
||||
"agent",
|
||||
Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-4o-mini"),
|
||||
system_prompt=(
|
||||
"Use any system messages at the start of the conversation as long-term memory. "
|
||||
"Answer concisely."
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
pipeline.connect("retriever.messages", "memory_context.memories")
|
||||
pipeline.connect("memory_context.output", "agent.messages")
|
||||
|
||||
query = "Give me a short implementation tip."
|
||||
|
||||
pipeline.run(
|
||||
{
|
||||
"retriever": {
|
||||
"query": query,
|
||||
"user_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
|
||||
},
|
||||
"memory_context": {
|
||||
"user_messages": [ChatMessage.from_user(query)],
|
||||
},
|
||||
}
|
||||
)
|
||||
```
|
||||
@@ -0,0 +1,176 @@
|
||||
---
|
||||
title: "ElasticsearchBM25Retriever"
|
||||
id: elasticsearchbm25retriever
|
||||
slug: "/elasticsearchbm25retriever"
|
||||
description: "A keyword-based Retriever that fetches Documents matching a query from the Elasticsearch Document Store."
|
||||
---
|
||||
|
||||
# ElasticsearchBM25Retriever
|
||||
|
||||
A keyword-based Retriever that fetches Documents matching a query from the Elasticsearch Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. Before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the semantic search pipeline 3. Before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of [ElasticsearchDocumentStore](../../document-stores/elasticsearch-document-store.mdx) |
|
||||
| **Mandatory run variables** | `query`: A string |
|
||||
| **Output variables** | `documents`: A list of documents (matching the query) |
|
||||
| **API reference** | [Elasticsearch](/reference/integrations-elasticsearch) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/elasticsearch |
|
||||
| **Package name** | `elasticsearch-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`ElasticsearchBM25Retriever` is a keyword-based Retriever that fetches Documents matching a query from an `ElasticsearchDocumentStore`. It determines the similarity between Documents and the query based on the BM25 algorithm, which computes a weighted word overlap between the two strings.
|
||||
|
||||
Since the `ElasticsearchBM25Retriever` matches strings based on word overlap, it’s often used to find exact matches to names of persons or products, IDs, or well-defined error messages. The BM25 algorithm is very lightweight and simple. Nevertheless, it can be hard to beat with more complex embedding-based approaches on out-of-domain data.
|
||||
|
||||
In addition to the `query`, the `ElasticsearchBM25Retriever` accepts other optional parameters, including `top_k` (the maximum number of Documents to retrieve) and `filters` to narrow down the search space.
|
||||
When initializing Retriever, you can also adjust how [inexact fuzzy matching](https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html#fuzziness) is performed, using the `fuzziness` parameter.
|
||||
|
||||
If you want a semantic match between a query and documents, you can use `ElasticsearchEmbeddingRetriever`, which uses vectors created by embedding models to retrieve relevant information.
|
||||
|
||||
## Installation
|
||||
|
||||
[Install](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html) Elasticsearch and then [start](https://www.elastic.co/guide/en/elasticsearch/reference/current/starting-elasticsearch.html) an instance. Haystack supports Elasticsearch 8.
|
||||
|
||||
If you have Docker set up, we recommend pulling the Docker image and running it.
|
||||
|
||||
```shell
|
||||
docker pull docker.elastic.co/elasticsearch/elasticsearch:8.11.1
|
||||
docker run -p 9200:9200 -e "discovery.type=single-node" -e "ES_JAVA_OPTS=-Xms1024m -Xmx1024m" -e "xpack.security.enabled=false" elasticsearch:8.11.1
|
||||
```
|
||||
|
||||
As an alternative, you can go to [Elasticsearch integration GitHub](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/elasticsearch) and start a Docker container running Elasticsearch using the provided `docker-compose.yml`:
|
||||
|
||||
```shell
|
||||
docker compose up
|
||||
```
|
||||
|
||||
Once you have a running Elasticsearch instance, install the `elasticsearch-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install elasticsearch-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.retrievers.elasticsearch import (
|
||||
ElasticsearchBM25Retriever,
|
||||
)
|
||||
from haystack_integrations.document_stores.elasticsearch import (
|
||||
ElasticsearchDocumentStore,
|
||||
)
|
||||
from elasticsearch import Elasticsearch
|
||||
|
||||
document_store = ElasticsearchDocumentStore(hosts="http://localhost:9200/")
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
document_store.write_documents(documents=documents)
|
||||
|
||||
retriever = ElasticsearchBM25Retriever(document_store=document_store)
|
||||
retriever.run(query="How many languages are spoken around the world today?")
|
||||
```
|
||||
|
||||
### In a RAG pipeline
|
||||
|
||||
Set your `OPENAI_API_KEY` as an environment variable and then run the following code:
|
||||
|
||||
```python
|
||||
|
||||
from haystack_integrations.components.retrievers.elasticsearch import (
|
||||
ElasticsearchBM25Retriever,
|
||||
)
|
||||
from haystack_integrations.document_stores.elasticsearch import (
|
||||
ElasticsearchDocumentStore,
|
||||
)
|
||||
|
||||
from elasticsearch import Elasticsearch
|
||||
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders.answer_builder import AnswerBuilder
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
|
||||
# OpenAIChatGenerator reads the OPENAI_API_KEY environment variable by default.
|
||||
|
||||
# Create a RAG query pipeline
|
||||
prompt_template = [
|
||||
ChatMessage.from_user(
|
||||
"""
|
||||
Given these documents, answer the question.\nDocuments:
|
||||
{% for doc in documents %}
|
||||
{{ doc.content }}
|
||||
{% endfor %}
|
||||
|
||||
\nQuestion: {{question}}
|
||||
\nAnswer:
|
||||
""",
|
||||
),
|
||||
]
|
||||
|
||||
document_store = ElasticsearchDocumentStore(hosts="http://localhost:9200/")
|
||||
|
||||
# Add Documents
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
# DuplicatePolicy.SKIP param is optional, but useful to run the script multiple times without throwing errors
|
||||
document_store.write_documents(documents=documents, policy=DuplicatePolicy.SKIP)
|
||||
|
||||
retriever = ElasticsearchBM25Retriever(document_store=document_store)
|
||||
rag_pipeline = Pipeline()
|
||||
rag_pipeline.add_component(name="retriever", instance=retriever)
|
||||
rag_pipeline.add_component(
|
||||
instance=ChatPromptBuilder(template=prompt_template, required_variables="*"),
|
||||
name="prompt_builder",
|
||||
)
|
||||
rag_pipeline.add_component(instance=OpenAIChatGenerator(), name="llm")
|
||||
rag_pipeline.add_component(instance=AnswerBuilder(), name="answer_builder")
|
||||
rag_pipeline.connect("retriever", "prompt_builder.documents")
|
||||
rag_pipeline.connect("prompt_builder.prompt", "llm.messages")
|
||||
rag_pipeline.connect("llm.replies", "answer_builder.replies")
|
||||
rag_pipeline.connect("retriever", "answer_builder.documents")
|
||||
|
||||
question = "How many languages are spoken around the world today?"
|
||||
result = rag_pipeline.run(
|
||||
{
|
||||
"retriever": {"query": question},
|
||||
"prompt_builder": {"question": question},
|
||||
"answer_builder": {"query": question},
|
||||
},
|
||||
)
|
||||
print(result["answer_builder"]["answers"][0].data)
|
||||
```
|
||||
|
||||
Here’s an example output you might get:
|
||||
|
||||
```python
|
||||
"Over 7,000 languages are spoken around the world today"
|
||||
```
|
||||
@@ -0,0 +1,132 @@
|
||||
---
|
||||
title: "ElasticsearchEmbeddingRetriever"
|
||||
id: elasticsearchembeddingretriever
|
||||
slug: "/elasticsearchembeddingretriever"
|
||||
description: "An embedding-based Retriever compatible with the Elasticsearch Document Store."
|
||||
---
|
||||
|
||||
# ElasticsearchEmbeddingRetriever
|
||||
|
||||
An embedding-based Retriever compatible with the Elasticsearch Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the semantic search pipeline 3. After a Text Embedder and before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of [ElasticsearchDocumentStore](../../document-stores/elasticsearch-document-store.mdx) |
|
||||
| **Mandatory run variables** | `query_embedding`: A list of floats |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Elasticsearch](/reference/integrations-elasticsearch) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/elasticsearch |
|
||||
| **Package name** | `elasticsearch-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `ElasticsearchEmbeddingRetriever` is an embedding-based Retriever compatible with the `ElasticsearchDocumentStore`. It compares the query and Document embeddings and fetches the Documents most relevant to the query from the `ElasticsearchDocumentStore` based on the outcome.
|
||||
|
||||
When using the `ElasticsearchEmbeddingRetriever` in your NLP system, ensure it has the query and Document embeddings available. You can do so by adding a Document Embedder to your indexing pipeline and a Text Embedder to your query pipeline.
|
||||
|
||||
In addition to the `query_embedding`, the `ElasticsearchEmbeddingRetriever` accepts other optional parameters, including `top_k` (the maximum number of Documents to retrieve) and `filters` to narrow down the search space.
|
||||
|
||||
When initializing Retriever, you can also set `num_candidates`: the number of approximate nearest neighbor candidates on each shard. It's an advanced setting you can read more about in the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search.html#tune-approximate-knn-for-speed-accuracy).
|
||||
|
||||
The `embedding_similarity_function` to use for embedding retrieval must be defined when the corresponding `ElasticsearchDocumentStore` is initialized.
|
||||
|
||||
## Installation
|
||||
|
||||
[Install](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html) Elasticsearch and then [start](https://www.elastic.co/guide/en/elasticsearch/reference/current/starting-elasticsearch.html) an instance. Haystack supports Elasticsearch 8.
|
||||
|
||||
If you have Docker set up, we recommend pulling the Docker image and running it.
|
||||
|
||||
```shell
|
||||
docker pull docker.elastic.co/elasticsearch/elasticsearch:8.11.1
|
||||
docker run -p 9200:9200 -e "discovery.type=single-node" -e "ES_JAVA_OPTS=-Xms1024m -Xmx1024m" -e "xpack.security.enabled=false" elasticsearch:8.11.1
|
||||
```
|
||||
|
||||
As an alternative, you can go to [Elasticsearch integration GitHub](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/elasticsearch) and start a Docker container running Elasticsearch using the provided `docker-compose.yml`:
|
||||
|
||||
```shell
|
||||
docker compose up
|
||||
```
|
||||
|
||||
Once you have a running Elasticsearch instance, install the `elasticsearch-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install elasticsearch-haystack
|
||||
```
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Use this Retriever in a query Pipeline like this:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.retrievers.elasticsearch import (
|
||||
ElasticsearchEmbeddingRetriever,
|
||||
)
|
||||
from haystack_integrations.document_stores.elasticsearch import (
|
||||
ElasticsearchDocumentStore,
|
||||
)
|
||||
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack import Document, Pipeline
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersTextEmbedder,
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
)
|
||||
|
||||
document_store = ElasticsearchDocumentStore(hosts="http://localhost:9200/")
|
||||
|
||||
model = "BAAI/bge-large-en-v1.5"
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
document_embedder = SentenceTransformersDocumentEmbedder(model=model)
|
||||
documents_with_embeddings = document_embedder.run(documents)
|
||||
|
||||
document_store.write_documents(
|
||||
documents_with_embeddings.get("documents"),
|
||||
policy=DuplicatePolicy.SKIP,
|
||||
)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component(
|
||||
"text_embedder",
|
||||
SentenceTransformersTextEmbedder(model=model),
|
||||
)
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
ElasticsearchEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "How many languages are there?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
```
|
||||
|
||||
The example output would be:
|
||||
|
||||
```python
|
||||
Document(id=cfe93bc1c274908801e6670440bf2bbba54fad792770d57421f85ffa2a4fcc94, content: 'There are over 7,000 languages spoken around the world today.', score: 0.87717235, embedding: vector of size 1024)
|
||||
```
|
||||
@@ -0,0 +1,214 @@
|
||||
---
|
||||
title: "ElasticsearchHybridRetriever"
|
||||
id: elasticsearchhybridretriever
|
||||
slug: "/elasticsearchhybridretriever"
|
||||
description: "This is a SuperComponent that implements a Hybrid Retriever in a single component, relying on Elasticsearch as the backend Document Store."
|
||||
---
|
||||
|
||||
# ElasticsearchHybridRetriever
|
||||
|
||||
This is a [SuperComponent](../../concepts/components/supercomponents.mdx) that implements a Hybrid Retriever in a single component, relying on Elasticsearch as the backend Document Store.
|
||||
|
||||
A Hybrid Retriever uses both traditional keyword-based search (BM25) and embedding-based search to retrieve documents, combining the strengths of both approaches. The Retriever then merges and re-ranks the results from both methods.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. After a TextEmbedder and before a PromptBuilder in a RAG pipeline 2. The last component in a hybrid search pipeline 3. After a TextEmbedder and before an ExtractiveReader in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of [`ElasticsearchDocumentStore`](../../document-stores/elasticsearch-document-store.mdx) <br /> <br />`embedder`: Any [Embedder](../embedders.mdx) implementing the `TextEmbedder` protocol |
|
||||
| **Mandatory run variables** | `query`: A query string |
|
||||
| **Output variables** | `documents`: A list of documents matching the query |
|
||||
| **API reference** | [Elasticsearch](/reference/integrations-elasticsearch) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/elasticsearch |
|
||||
| **Package name** | `elasticsearch-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `ElasticsearchHybridRetriever` combines two retrieval methods:
|
||||
|
||||
1. **BM25 Retrieval**: A keyword-based search that uses the BM25 algorithm to find documents based on term frequency and inverse document frequency. It's based on the [`ElasticsearchBM25Retriever`](elasticsearchbm25retriever.mdx) component and is suitable for finding exact matches to names, IDs, or well-defined terms.
|
||||
2. **Embedding-based Retrieval**: A semantic search that uses vector similarity to find documents that are semantically similar to the query. It's based on the [`ElasticsearchEmbeddingRetriever`](elasticsearchembeddingretriever.mdx) component and is suitable for semantic search.
|
||||
|
||||
The component automatically handles:
|
||||
|
||||
- Converting the query into an embedding using the provided embedder,
|
||||
- Running both retrieval methods in parallel,
|
||||
- Merging and re-ranking the results using the specified join mode (default: Reciprocal Rank Fusion).
|
||||
|
||||
### Installation
|
||||
|
||||
[Install](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html) Elasticsearch and then [start](https://www.elastic.co/guide/en/elasticsearch/reference/current/starting-elasticsearch.html) an instance. Haystack supports Elasticsearch 8.
|
||||
|
||||
If you have Docker set up, we recommend pulling the Docker image and running it.
|
||||
|
||||
```shell
|
||||
docker pull docker.elastic.co/elasticsearch/elasticsearch:8.11.1
|
||||
docker run -p 9200:9200 -e "discovery.type=single-node" -e "ES_JAVA_OPTS=-Xms1024m -Xmx1024m" -e "xpack.security.enabled=false" elasticsearch:8.11.1
|
||||
```
|
||||
|
||||
As an alternative, you can go to the [Elasticsearch integration GitHub](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/elasticsearch) and start a Docker container running Elasticsearch using the provided `docker-compose.yml`:
|
||||
|
||||
```shell
|
||||
docker compose up
|
||||
```
|
||||
|
||||
Once you have a running Elasticsearch instance, install the `elasticsearch-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install elasticsearch-haystack
|
||||
```
|
||||
|
||||
### Optional Parameters
|
||||
|
||||
This Retriever accepts various optional parameters. You can verify the most up-to-date list of parameters in our [API Reference](/reference/integrations-elasticsearch).
|
||||
|
||||
You can pass additional parameters to the underlying BM25 and embedding retriever components using the `top_k_bm25`, `fuzziness`, `filters_bm25`, `scale_score`, `filter_policy_bm25`, `top_k_embedding`, `filters_embedding`, `num_candidates`, and `filter_policy_embedding` parameters.
|
||||
|
||||
The `DocumentJoiner` parameters (`join_mode`, `weights`, `top_k`, and `sort_by_score`) are all exposed directly on the `ElasticsearchHybridRetriever` class.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
This Retriever needs the `ElasticsearchDocumentStore` populated with documents (including embeddings) to run.
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersTextEmbedder,
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
)
|
||||
from haystack_integrations.components.retrievers.elasticsearch import (
|
||||
ElasticsearchHybridRetriever,
|
||||
)
|
||||
from haystack_integrations.document_stores.elasticsearch import (
|
||||
ElasticsearchDocumentStore,
|
||||
)
|
||||
|
||||
document_store = ElasticsearchDocumentStore(hosts="http://localhost:9200/")
|
||||
|
||||
model = "sentence-transformers/all-MiniLM-L6-v2"
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
doc_embedder = SentenceTransformersDocumentEmbedder(model=model)
|
||||
docs_with_embeddings = doc_embedder.run(documents)
|
||||
document_store.write_documents(docs_with_embeddings["documents"])
|
||||
|
||||
embedder = SentenceTransformersTextEmbedder(model=model)
|
||||
|
||||
retriever = ElasticsearchHybridRetriever(
|
||||
document_store=document_store,
|
||||
embedder=embedder,
|
||||
)
|
||||
|
||||
results = retriever.run(query="How many languages are spoken around the world today?")
|
||||
print(results["documents"])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Here's a full example that uses an indexing pipeline to store documents with embeddings, and a query pipeline that uses `ElasticsearchHybridRetriever` for hybrid retrieval.
|
||||
|
||||
Set your `OPENAI_API_KEY` as an environment variable and then run the following code:
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
SentenceTransformersTextEmbedder,
|
||||
)
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack_integrations.components.retrievers.elasticsearch import (
|
||||
ElasticsearchHybridRetriever,
|
||||
)
|
||||
from haystack_integrations.document_stores.elasticsearch import (
|
||||
ElasticsearchDocumentStore,
|
||||
)
|
||||
|
||||
document_store = ElasticsearchDocumentStore(hosts="http://localhost:9200/")
|
||||
|
||||
model = "sentence-transformers/all-MiniLM-L6-v2"
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
# Indexing Pipeline
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component(
|
||||
"doc_embedder",
|
||||
SentenceTransformersDocumentEmbedder(model=model),
|
||||
)
|
||||
indexing_pipeline.add_component(
|
||||
"doc_writer",
|
||||
DocumentWriter(document_store=document_store, policy=DuplicatePolicy.SKIP),
|
||||
)
|
||||
indexing_pipeline.connect("doc_embedder", "doc_writer")
|
||||
indexing_pipeline.run({"doc_embedder": {"documents": documents}})
|
||||
|
||||
# Query Pipeline
|
||||
prompt_template = [
|
||||
ChatMessage.from_user(
|
||||
"""
|
||||
Given these documents, answer the question.\nDocuments:
|
||||
{% for doc in documents %}
|
||||
{{ doc.content }}
|
||||
{% endfor %}
|
||||
|
||||
\nQuestion: {{question}}
|
||||
\nAnswer:
|
||||
""",
|
||||
),
|
||||
]
|
||||
|
||||
embedder = SentenceTransformersTextEmbedder(model=model)
|
||||
retriever = ElasticsearchHybridRetriever(
|
||||
document_store=document_store,
|
||||
embedder=embedder,
|
||||
top_k_bm25=3,
|
||||
top_k_embedding=3,
|
||||
join_mode="reciprocal_rank_fusion",
|
||||
)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("retriever", retriever)
|
||||
query_pipeline.add_component(
|
||||
"prompt_builder",
|
||||
ChatPromptBuilder(template=prompt_template, required_variables="*"),
|
||||
)
|
||||
query_pipeline.add_component("llm", OpenAIChatGenerator())
|
||||
query_pipeline.connect("retriever.documents", "prompt_builder.documents")
|
||||
query_pipeline.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
question = "How many languages are spoken around the world today?"
|
||||
result = query_pipeline.run(
|
||||
{
|
||||
"retriever": {"query": question},
|
||||
"prompt_builder": {"question": question},
|
||||
},
|
||||
)
|
||||
|
||||
print(result["llm"]["replies"][0].text)
|
||||
```
|
||||
@@ -0,0 +1,113 @@
|
||||
---
|
||||
title: ElasticsearchSQLRetriever
|
||||
id: elasticsearchsqlretriever
|
||||
slug: /elasticsearchsqlretriever
|
||||
description: Executes raw Elasticsearch SQL queries against an Elasticsearch Document Store and returns the raw JSON response.
|
||||
---
|
||||
|
||||
# ElasticsearchSQLRetriever
|
||||
|
||||
Executes raw Elasticsearch SQL queries against an Elasticsearch Document Store and returns the raw JSON response.
|
||||
|
||||
| | |
|
||||
| --------------------------------------- | ------------------------------------------------------------------------------------------------ |
|
||||
| **Most common position in a pipeline** | Standalone, or anywhere you need to fetch metadata, aggregations, or other structured data |
|
||||
| **Mandatory init variables** | `document_store`: An instance of `ElasticsearchDocumentStore` |
|
||||
| **Mandatory run variables** | `query`: An Elasticsearch SQL query string |
|
||||
| **Output variables** | `result`: A dictionary with the raw JSON response from the Elasticsearch SQL API |
|
||||
| **API reference** | [Elasticsearch](https://docs.haystack.deepset.ai/reference/integrations-elasticsearch) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/elasticsearch |
|
||||
| **Package name** | `elasticsearch-haystack` |
|
||||
|
||||
## Overview
|
||||
|
||||
`ElasticsearchSQLRetriever` lets you run [Elasticsearch SQL](https://www.elastic.co/guide/en/elasticsearch/reference/current/xpack-sql.html) queries directly against an `ElasticsearchDocumentStore`. Instead of matching a query against documents like the `ElasticsearchBM25Retriever` or `ElasticsearchEmbeddingRetriever`, it executes a SQL statement and returns the **raw JSON response** from the Elasticsearch SQL API.
|
||||
|
||||
This is useful when you need structured access to your index at runtime, for example to fetch specific fields, filter on metadata, or compute aggregations such as counts and averages.
|
||||
|
||||
Unlike the other Elasticsearch retrievers, this component does not return a list of `Document` objects. The output is a single `result` dictionary, where `result["result"]` holds the raw Elasticsearch response. For a typical query, the response contains:
|
||||
|
||||
- `result["result"]["columns"]`: metadata describing each returned column.
|
||||
- `result["result"]["rows"]`: the data rows.
|
||||
|
||||
The component accepts two optional parameters at initialization:
|
||||
|
||||
- `raise_on_failure`: if `True` (the default), an exception is raised when the SQL API call fails. If `False`, the error is logged as a warning and an empty dictionary is returned.
|
||||
- `fetch_size`: the number of results to fetch per page. If not set, the default fetch size configured in Elasticsearch is used.
|
||||
|
||||
## Installation
|
||||
|
||||
Install Elasticsearch and then start an instance. Haystack supports Elasticsearch 8.
|
||||
|
||||
If you have Docker set up, we recommend pulling the Docker image and running it.
|
||||
|
||||
```bash
|
||||
docker pull docker.elastic.co/elasticsearch/elasticsearch:8.11.1
|
||||
docker run -p 9200:9200 -e "discovery.type=single-node" -e "ES_JAVA_OPTS=-Xms1024m -Xmx1024m" -e "xpack.security.enabled=false" elasticsearch:8.11.1
|
||||
```
|
||||
|
||||
As an alternative, you can go to [Elasticsearch integration GitHub](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/elasticsearch) and start a Docker container running Elasticsearch using the provided `docker-compose.yml`:
|
||||
|
||||
```bash
|
||||
docker compose up
|
||||
```
|
||||
|
||||
Once you have a running Elasticsearch instance, install the `elasticsearch-haystack` integration:
|
||||
|
||||
```bash
|
||||
pip install elasticsearch-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Write a few documents to an index, then run a SQL query against it. The example below selects the `content` field from the index and reads the returned columns and rows:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.retrievers.elasticsearch import (
|
||||
ElasticsearchSQLRetriever,
|
||||
)
|
||||
from haystack_integrations.document_stores.elasticsearch import (
|
||||
ElasticsearchDocumentStore,
|
||||
)
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
|
||||
document_store = ElasticsearchDocumentStore(hosts="http://localhost:9200/", index="my_index")
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
# DuplicatePolicy.SKIP is optional, but useful to run the script multiple times without throwing errors
|
||||
document_store.write_documents(documents=documents, policy=DuplicatePolicy.SKIP)
|
||||
|
||||
retriever = ElasticsearchSQLRetriever(document_store=document_store)
|
||||
output = retriever.run(query='SELECT content FROM "my_index" LIMIT 10')
|
||||
|
||||
result = output["result"]
|
||||
print(result["columns"]) # column metadata, e.g. [{"name": "content", "type": "text"}]
|
||||
for row in result["rows"]:
|
||||
print(row)
|
||||
```
|
||||
|
||||
### Running an aggregation query
|
||||
|
||||
Because the component returns the raw SQL response, you can use it for aggregations that the document-based retrievers don't support, such as counting documents:
|
||||
|
||||
```python
|
||||
retriever = ElasticsearchSQLRetriever(document_store=document_store)
|
||||
output = retriever.run(query='SELECT COUNT(*) AS doc_count FROM "my_index"')
|
||||
|
||||
result = output["result"]
|
||||
print(result["rows"]) # e.g. [[3]]
|
||||
```
|
||||
|
||||
To avoid raising an exception on a malformed or failing query, initialize the component with `raise_on_failure=False`. In that case, a failed query logs a warning and returns an empty dictionary instead.
|
||||
@@ -0,0 +1,104 @@
|
||||
---
|
||||
title: "FAISSEmbeddingRetriever"
|
||||
id: faissembeddingretriever
|
||||
slug: "/faissembeddingretriever"
|
||||
description: "An embedding-based Retriever compatible with the FAISSDocumentStore."
|
||||
---
|
||||
|
||||
# FAISSEmbeddingRetriever
|
||||
|
||||
An embedding-based Retriever compatible with the FAISSDocumentStore.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in a semantic search pipeline 3. After a Text Embedder and before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a [`FAISSDocumentStore`](../../document-stores/faissdocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query_embedding`: A vector representing the query (a list of floats) |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [FAISS](/reference/integrations-faiss) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/faiss |
|
||||
| **Package name** | `faiss-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `FAISSEmbeddingRetriever` is an embedding-based Retriever that queries a `FAISSDocumentStore`. It compares the query embedding to document embeddings stored in FAISS and returns the most similar documents.
|
||||
|
||||
This Retriever expects precomputed embeddings in the Document Store and a query embedding at runtime. You can generate them with a Document Embedder in your indexing pipeline and a Text Embedder in your query pipeline.
|
||||
|
||||
In addition to `query_embedding`, you can pass:
|
||||
|
||||
- `top_k`: The maximum number of documents to return.
|
||||
- `filters`: Metadata filters to restrict retrieved documents.
|
||||
|
||||
You can also configure default filters and `filter_policy` at initialization.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.faiss import FAISSDocumentStore
|
||||
from haystack_integrations.components.retrievers.faiss import FAISSEmbeddingRetriever
|
||||
|
||||
document_store = FAISSDocumentStore(embedding_dim=768)
|
||||
retriever = FAISSEmbeddingRetriever(document_store=document_store, top_k=5)
|
||||
|
||||
# Example query embedding
|
||||
result = retriever.run(query_embedding=[0.1] * 768)
|
||||
print(result["documents"])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
SentenceTransformersTextEmbedder,
|
||||
)
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack_integrations.document_stores.faiss import FAISSDocumentStore
|
||||
from haystack_integrations.components.retrievers.faiss import FAISSEmbeddingRetriever
|
||||
|
||||
document_store = FAISSDocumentStore(embedding_dim=768)
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of intelligence.",
|
||||
),
|
||||
Document(
|
||||
content="In certain places, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
document_embedder = SentenceTransformersDocumentEmbedder()
|
||||
documents_with_embeddings = document_embedder.run(documents)["documents"]
|
||||
document_store.write_documents(
|
||||
documents_with_embeddings,
|
||||
policy=DuplicatePolicy.OVERWRITE,
|
||||
)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", SentenceTransformersTextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
FAISSEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "How many languages are there?"
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
```
|
||||
@@ -0,0 +1,153 @@
|
||||
---
|
||||
title: "FalkorDBCypherRetriever"
|
||||
id: falkordbcypherretriever
|
||||
slug: "/falkordbcypherretriever"
|
||||
description: "A Retriever that executes arbitrary OpenCypher queries against a FalkorDB Document Store."
|
||||
---
|
||||
|
||||
# FalkorDBCypherRetriever
|
||||
|
||||
A Retriever that executes arbitrary OpenCypher queries against a FalkorDB Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a query-building component and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a GraphRAG pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a [FalkorDBDocumentStore](../../document-stores/falkordbdocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query`: An OpenCypher query string (or set `custom_cypher_query` at init) |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [FalkorDB](/reference/integrations-falkordb) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/falkordb |
|
||||
| **Package name** | `falkordb-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `FalkorDBCypherRetriever` executes arbitrary OpenCypher queries against a `FalkorDBDocumentStore`, making it suitable for graph traversal and multi-hop queries in GraphRAG pipelines. The query must return nodes or dictionaries that map to Haystack `Document` fields.
|
||||
|
||||
A `custom_cypher_query` can be set at initialization and optionally overridden at runtime by passing `query` to `run()`. Use parameterized queries (`$param_name` in Cypher, passed via `parameters`) rather than string interpolation to avoid injection vulnerabilities.
|
||||
|
||||
:::warning[Security]
|
||||
Raw Cypher queries must only come from trusted sources. Never pass unsanitized user input directly in query strings. Use `parameters` instead.
|
||||
:::
|
||||
|
||||
## Installation
|
||||
|
||||
```shell
|
||||
pip install falkordb-haystack
|
||||
```
|
||||
|
||||
Ensure FalkorDB is running, for example via Docker:
|
||||
|
||||
```shell
|
||||
docker run -d -p 6379:6379 falkordb/falkordb:latest
|
||||
```
|
||||
|
||||
The examples on this page use Transformers components that have moved to the `transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install transformers-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.document_stores.falkordb import FalkorDBDocumentStore
|
||||
from haystack_integrations.components.retrievers.falkordb import FalkorDBCypherRetriever
|
||||
|
||||
document_store = FalkorDBDocumentStore(
|
||||
host="localhost",
|
||||
port=6379,
|
||||
recreate_graph=True,
|
||||
)
|
||||
document_store.write_documents(
|
||||
[
|
||||
Document(
|
||||
content="There are over 7,000 languages spoken around the world today.",
|
||||
meta={"topic": "linguistics"},
|
||||
),
|
||||
Document(
|
||||
content="Elephants have been observed to recognize themselves in mirrors.",
|
||||
meta={"topic": "biology"},
|
||||
),
|
||||
],
|
||||
)
|
||||
|
||||
retriever = FalkorDBCypherRetriever(
|
||||
document_store=document_store,
|
||||
custom_cypher_query="MATCH (d:Document {topic: $topic}) RETURN d",
|
||||
)
|
||||
result = retriever.run(parameters={"topic": "linguistics"})
|
||||
print(result["documents"][0].content)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack_integrations.components.generators.transformers import (
|
||||
TransformersChatGenerator,
|
||||
)
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.document_stores.falkordb import FalkorDBDocumentStore
|
||||
from haystack_integrations.components.retrievers.falkordb import FalkorDBCypherRetriever
|
||||
|
||||
document_store = FalkorDBDocumentStore(
|
||||
host="localhost",
|
||||
port=6379,
|
||||
recreate_graph=True,
|
||||
)
|
||||
document_store.write_documents(
|
||||
[
|
||||
Document(
|
||||
content="There are over 7,000 languages spoken around the world today.",
|
||||
meta={"topic": "linguistics"},
|
||||
),
|
||||
Document(
|
||||
content="Elephants have been observed to recognize themselves in mirrors.",
|
||||
meta={"topic": "biology"},
|
||||
),
|
||||
],
|
||||
)
|
||||
|
||||
prompt_template = [
|
||||
ChatMessage.from_user(
|
||||
"""Given these documents, answer the question.
|
||||
Documents:
|
||||
{% for doc in documents %}
|
||||
{{ doc.content }}
|
||||
{% endfor %}
|
||||
Question: {{ question }}""",
|
||||
),
|
||||
]
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component(
|
||||
"retriever",
|
||||
FalkorDBCypherRetriever(
|
||||
document_store=document_store,
|
||||
custom_cypher_query="MATCH (d:Document {topic: $topic}) RETURN d",
|
||||
),
|
||||
)
|
||||
pipeline.add_component("prompt_builder", ChatPromptBuilder(template=prompt_template))
|
||||
pipeline.add_component(
|
||||
"llm",
|
||||
TransformersChatGenerator(model="HuggingFaceTB/SmolLM2-135M-Instruct"),
|
||||
)
|
||||
pipeline.connect("retriever.documents", "prompt_builder.documents")
|
||||
pipeline.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
result = pipeline.run(
|
||||
{
|
||||
"retriever": {"parameters": {"topic": "linguistics"}},
|
||||
"prompt_builder": {"question": "How many languages are there?"},
|
||||
},
|
||||
)
|
||||
print(result["llm"]["replies"][0].text)
|
||||
```
|
||||
@@ -0,0 +1,143 @@
|
||||
---
|
||||
title: "FalkorDBEmbeddingRetriever"
|
||||
id: falkordbembeddingretriever
|
||||
slug: "/falkordbembeddingretriever"
|
||||
description: "An embedding-based Retriever compatible with the FalkorDB Document Store."
|
||||
---
|
||||
|
||||
# FalkorDBEmbeddingRetriever
|
||||
|
||||
An embedding-based Retriever compatible with the FalkorDB Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline <br /><br /> 2. The last component in a semantic search pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a [FalkorDBDocumentStore](../../document-stores/falkordbdocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query_embedding`: A vector representing the query (a list of floats) |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [FalkorDB](/reference/integrations-falkordb) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/falkordb |
|
||||
| **Package name** | `falkordb-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `FalkorDBEmbeddingRetriever` retrieves documents from a `FalkorDBDocumentStore` using FalkorDB's native vector index. It compares the query embedding with document embeddings and returns the most similar documents.
|
||||
|
||||
In addition to `query_embedding`, the retriever accepts optional `filters` to narrow the search space and `top_k` to limit the number of results.
|
||||
|
||||
The embedding dimension and similarity function are configured on the `FalkorDBDocumentStore` at initialization time.
|
||||
|
||||
## Installation
|
||||
|
||||
```shell
|
||||
pip install falkordb-haystack
|
||||
```
|
||||
|
||||
Ensure FalkorDB is running, for example via Docker:
|
||||
|
||||
```shell
|
||||
docker run -d -p 6379:6379 falkordb/falkordb:latest
|
||||
```
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.document_stores.falkordb import FalkorDBDocumentStore
|
||||
from haystack_integrations.components.retrievers.falkordb import (
|
||||
FalkorDBEmbeddingRetriever,
|
||||
)
|
||||
|
||||
document_store = FalkorDBDocumentStore(
|
||||
host="localhost",
|
||||
port=6379,
|
||||
embedding_dim=3,
|
||||
recreate_graph=True,
|
||||
)
|
||||
document_store.write_documents(
|
||||
[
|
||||
Document(
|
||||
content="There are over 7,000 languages spoken around the world today.",
|
||||
embedding=[0.1, 0.2, 0.3],
|
||||
),
|
||||
Document(
|
||||
content="Elephants have been observed to recognize themselves in mirrors.",
|
||||
embedding=[0.8, 0.1, 0.5],
|
||||
),
|
||||
],
|
||||
)
|
||||
|
||||
retriever = FalkorDBEmbeddingRetriever(document_store=document_store, top_k=1)
|
||||
result = retriever.run(query_embedding=[0.1, 0.2, 0.3])
|
||||
print(result["documents"][0].content)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
SentenceTransformersTextEmbedder,
|
||||
)
|
||||
from haystack_integrations.document_stores.falkordb import FalkorDBDocumentStore
|
||||
from haystack_integrations.components.retrievers.falkordb import (
|
||||
FalkorDBEmbeddingRetriever,
|
||||
)
|
||||
|
||||
document_store = FalkorDBDocumentStore(
|
||||
host="localhost",
|
||||
port=6379,
|
||||
embedding_dim=384,
|
||||
recreate_graph=True,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to recognize themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="Bioluminescent waves can be seen in the Maldives and Puerto Rico.",
|
||||
),
|
||||
]
|
||||
|
||||
document_embedder = SentenceTransformersDocumentEmbedder(
|
||||
model="sentence-transformers/all-MiniLM-L6-v2",
|
||||
)
|
||||
documents_with_embeddings = document_embedder.run(documents)
|
||||
|
||||
document_store.write_documents(
|
||||
documents_with_embeddings["documents"],
|
||||
policy=DuplicatePolicy.OVERWRITE,
|
||||
)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component(
|
||||
"text_embedder",
|
||||
SentenceTransformersTextEmbedder(model="sentence-transformers/all-MiniLM-L6-v2"),
|
||||
)
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
FalkorDBEmbeddingRetriever(document_store=document_store, top_k=3),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
result = query_pipeline.run(
|
||||
{"text_embedder": {"text": "How many languages are there?"}},
|
||||
)
|
||||
print(result["retriever"]["documents"][0].content)
|
||||
```
|
||||
@@ -0,0 +1,137 @@
|
||||
---
|
||||
title: "FilterRetriever"
|
||||
id: filterretriever
|
||||
slug: "/filterretriever"
|
||||
description: "Use this Retriever with any Document Store to get the Documents that match specific filters."
|
||||
---
|
||||
|
||||
# FilterRetriever
|
||||
|
||||
Use this Retriever with any Document Store to get the Documents that match specific filters.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | At the beginning of a Pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a Document Store |
|
||||
| **Mandatory run variables** | `filters`: A dictionary of filters in the same syntax supported by the Document Stores |
|
||||
| **Output variables** | `documents`: All the documents that match these filters |
|
||||
| **API reference** | [Retrievers](/reference/retrievers-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/retrievers/filter_retriever.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`FilterRetriever` retrieves Documents that match the provided filters.
|
||||
|
||||
It’s a special kind of Retriever – it can work with all Document Stores instead of being specialized to work with only one.
|
||||
|
||||
However, as every other Retriever, it needs some Document Store at initialization time, and it will perform filtering on the content of that instance only.
|
||||
|
||||
Therefore, it can be used as any other Retriever in a Pipeline.
|
||||
|
||||
Pay attention when using `FilterRetriever` on a Document Store that contains many Documents, as `FilterRetriever` will return all documents that match the filters. The `run` command with no filters can easily overwhelm other components in the Pipeline (for example, Generators):
|
||||
|
||||
```python
|
||||
filter_retriever.run({})
|
||||
```
|
||||
|
||||
Another thing to note is that `FilterRetriever` does not score your Documents or rank them in any way. If you need to rank the Documents by similarity to a query, consider using Ranker components.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.retrievers import FilterRetriever
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
docs = [
|
||||
Document(content="Python is a popular programming language", meta={"lang": "en"}),
|
||||
Document(
|
||||
content="python ist eine beliebte Programmiersprache",
|
||||
meta={"lang": "de"},
|
||||
),
|
||||
]
|
||||
|
||||
doc_store = InMemoryDocumentStore()
|
||||
doc_store.write_documents(docs)
|
||||
retriever = FilterRetriever(doc_store)
|
||||
result = retriever.run(filters={"field": "lang", "operator": "==", "value": "en"})
|
||||
|
||||
assert "documents" in result
|
||||
assert len(result["documents"]) == 1
|
||||
assert result["documents"][0].content == "Python is a popular programming language"
|
||||
```
|
||||
|
||||
### In a RAG pipeline
|
||||
|
||||
Set your `OPENAI_API_KEY` as an environment variable and then run the following code:
|
||||
|
||||
```python
|
||||
from haystack.components.retrievers.filter_retriever import FilterRetriever
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
|
||||
# OpenAIChatGenerator reads the OPENAI_API_KEY environment variable by default.
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
documents = [
|
||||
Document(content="Mark lives in Berlin.", meta={"year": 2018}),
|
||||
Document(content="Mark lives in Paris.", meta={"year": 2021}),
|
||||
Document(content="Mark is Danish.", meta={"year": 2021}),
|
||||
Document(content="Mark lives in New York.", meta={"year": 2023}),
|
||||
]
|
||||
document_store.write_documents(documents=documents)
|
||||
|
||||
# Create a RAG query pipeline
|
||||
prompt_template = [
|
||||
ChatMessage.from_user(
|
||||
"""
|
||||
Given these documents, answer the question.\nDocuments:
|
||||
{% for doc in documents %}
|
||||
{{ doc.content }}
|
||||
{% endfor %}
|
||||
|
||||
\nQuestion: {{question}}
|
||||
\nAnswer:
|
||||
""",
|
||||
),
|
||||
]
|
||||
|
||||
rag_pipeline = Pipeline()
|
||||
rag_pipeline.add_component(
|
||||
name="retriever",
|
||||
instance=FilterRetriever(document_store=document_store),
|
||||
)
|
||||
rag_pipeline.add_component(
|
||||
instance=ChatPromptBuilder(template=prompt_template, required_variables="*"),
|
||||
name="prompt_builder",
|
||||
)
|
||||
rag_pipeline.add_component(instance=OpenAIChatGenerator(), name="llm")
|
||||
rag_pipeline.connect("retriever", "prompt_builder.documents")
|
||||
rag_pipeline.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
result = rag_pipeline.run(
|
||||
{
|
||||
"retriever": {"filters": {"field": "year", "operator": "==", "value": 2021}},
|
||||
"prompt_builder": {"question": "Where does Mark live?"},
|
||||
},
|
||||
)
|
||||
print(result["llm"]["replies"][0].text)
|
||||
```
|
||||
|
||||
Here’s an example output you might get:
|
||||
|
||||
```
|
||||
According to the provided documents, Mark lives in Paris.
|
||||
```
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
title: "GoogleDriveRetriever"
|
||||
id: googledriveretriever
|
||||
slug: "/googledriveretriever"
|
||||
description: "Retrieves files from Google Drive via the Drive API v3 search endpoint."
|
||||
---
|
||||
|
||||
# GoogleDriveRetriever
|
||||
|
||||
Retrieves files from Google Drive via the Drive API v3 search endpoint.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | At the start of a query pipeline, after an [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx) that provides the `access_token` |
|
||||
| **Mandatory init variables** | None |
|
||||
| **Mandatory run variables** | `query`: The search query string <br /> <br />`access_token`: A delegated Google OAuth bearer token, typically wired from an upstream `OAuthTokenResolver` |
|
||||
| **Output variables** | `documents`: A list of [Documents](../../concepts/data-classes.mdx) holding file metadata (and optionally exported text) |
|
||||
| **API reference** | [Google Drive](/reference/integrations-google-drive) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/google_drive |
|
||||
| **Package name** | `google-drive-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`GoogleDriveRetriever` runs a full-text search over a user's Google Drive (and optionally shared drives) through the [Drive API v3](https://developers.google.com/drive/api/reference/rest/v3/files/list) `files.list` endpoint and maps each matching file to a Haystack `Document`.
|
||||
|
||||
By default, each `Document` carries resource metadata (`file_name`, `file_id`, `web_url`, `mime_type`, `file_extension`, author, and timestamps) and uses the file `description` or `name` as `content`, because the Drive search API does not return a text snippet. Set `include_content=True` to additionally export native Google Docs/Sheets/Slides to text and use that as the `Document` content. Binary files (PDF, DOCX, ...) are never downloaded by the retriever.
|
||||
|
||||
To download the full content of the matching files, compose it with [`GoogleDriveFetcher`](../fetchers/googledrivefetcher.mdx) on the returned `web_url`/`file_id`, followed by a converter.
|
||||
|
||||
### Authentication
|
||||
|
||||
The retriever takes a per-user `access_token` as a run input. The token must carry a delegated Google OAuth scope that allows search, for example `https://www.googleapis.com/auth/drive.readonly`. The metadata-only `drive.metadata.readonly` scope cannot search file content or export documents. Typically you wire the token from an upstream [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx), which emits a plain string. A `Secret` is also accepted and resolved internally.
|
||||
|
||||
### Scoping and filtering the search
|
||||
|
||||
- `query_filter`: an optional Drive query clause AND-ed with the full-text search term, for example `"mimeType != 'application/vnd.google-apps.folder'"` or `"'<folderId>' in parents"`.
|
||||
- `include_shared_drives`: when `True`, the search spans shared drives as well as the user's My Drive.
|
||||
- `order_by`: an optional Drive `orderBy` expression, for example `"modifiedTime desc"`.
|
||||
|
||||
### Installation
|
||||
|
||||
Install the Google Drive integration with:
|
||||
|
||||
```shell
|
||||
pip install google-drive-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
`access_token` below is a per-user delegated Google OAuth bearer token. In production you would obtain it from an [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx) rather than pasting it in.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.retrievers.google_drive import (
|
||||
GoogleDriveRetriever,
|
||||
)
|
||||
|
||||
retriever = GoogleDriveRetriever(top_k=5)
|
||||
|
||||
result = retriever.run(
|
||||
query="quarterly roadmap",
|
||||
access_token="my-delegated-google-token",
|
||||
)
|
||||
|
||||
for doc in result["documents"]:
|
||||
print(doc.meta["file_name"], "-", doc.meta["web_url"])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
The following pipeline obtains a token from an `OAuthTokenResolver` and feeds it into the retriever, so that running the pipeline requires only the query:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.components.connectors.oauth import OAuthTokenResolver
|
||||
from haystack_integrations.utils.oauth import OAuthRefreshTokenSource
|
||||
from haystack_integrations.components.retrievers.google_drive import (
|
||||
GoogleDriveRetriever,
|
||||
)
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component(
|
||||
"resolver",
|
||||
OAuthTokenResolver(
|
||||
token_source=OAuthRefreshTokenSource(
|
||||
token_url="https://oauth2.googleapis.com/token",
|
||||
client_id="aaa-bbb-ccc",
|
||||
refresh_token=Secret.from_env_var("GOOGLE_REFRESH_TOKEN"),
|
||||
scopes=["https://www.googleapis.com/auth/drive.readonly"],
|
||||
),
|
||||
),
|
||||
)
|
||||
pipeline.add_component("retriever", GoogleDriveRetriever(top_k=5))
|
||||
pipeline.connect("resolver.access_token", "retriever.access_token")
|
||||
|
||||
result = pipeline.run({"retriever": {"query": "quarterly roadmap"}})
|
||||
documents = result["retriever"]["documents"]
|
||||
```
|
||||
|
||||
To download and convert the full content of the retrieved files, connect the retriever's `documents` output to a [`GoogleDriveFetcher`](../fetchers/googledrivefetcher.mdx). See that page for an end-to-end retrieve-fetch-convert example.
|
||||
@@ -0,0 +1,170 @@
|
||||
---
|
||||
title: "InMemoryBM25Retriever"
|
||||
id: inmemorybm25retriever
|
||||
slug: "/inmemorybm25retriever"
|
||||
description: "A keyword-based Retriever compatible with InMemoryDocumentStore."
|
||||
---
|
||||
|
||||
# InMemoryBM25Retriever
|
||||
|
||||
A keyword-based Retriever compatible with InMemoryDocumentStore.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In query pipelines: <br />In a RAG pipeline, before a [`PromptBuilder`](../builders/promptbuilder.mdx) <br />In a semantic search pipeline, as the last component <br />In an extractive QA pipeline, before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) |
|
||||
| **Mandatory init variables** | `document_store`: An instance of [InMemoryDocumentStore](../../document-stores/inmemorydocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query`: A query string |
|
||||
| **Output variables** | `documents`: A list of documents (matching the query) |
|
||||
| **API reference** | [Retrievers](/reference/retrievers-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/retrievers/in_memory/bm25_retriever.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`InMemoryBM25Retriever` is a keyword-based Retriever that fetches Documents matching a query from a temporary in-memory database. It determines the similarity between Documents and the query based on the BM25 algorithm, which computes a weighted word overlap between the two strings.
|
||||
|
||||
Since the `InMemoryBM25Retriever` matches strings based on word overlap, it’s often used to find exact matches to names of persons or products, IDs, or well-defined error messages. The BM25 algorithm is very lightweight and simple. Nevertheless, it can be hard to beat with more complex embedding-based approaches on out-of-domain data.
|
||||
|
||||
In addition to the `query`, the `InMemoryBM25Retriever` accepts other optional parameters, including `top_k` (the maximum number of Documents to retrieve) and `filters` to narrow down the search space.
|
||||
Some relevant parameters that impact the BM25 retrieval must be defined when the corresponding `InMemoryDocumentStore` is initialized: these include the specific BM25 algorithm and its parameters.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
document_store.write_documents(documents=documents)
|
||||
|
||||
retriever = InMemoryBM25Retriever(document_store=document_store)
|
||||
retriever.run(query="How many languages are spoken around the world today?")
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
#### In a RAG Pipeline
|
||||
|
||||
Here's an example of the Retriever in a retrieval-augmented generation pipeline:
|
||||
|
||||
```python
|
||||
import os
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders.answer_builder import AnswerBuilder
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
# Create a RAG query pipeline
|
||||
prompt_template = [
|
||||
ChatMessage.from_user(
|
||||
"""
|
||||
Given these documents, answer the question.\nDocuments:
|
||||
{% for doc in documents %}
|
||||
{{ doc.content }}
|
||||
{% endfor %}
|
||||
|
||||
\nQuestion: {{question}}
|
||||
\nAnswer:
|
||||
""",
|
||||
),
|
||||
]
|
||||
|
||||
os.environ["OPENAI_API_KEY"] = "sk-XXXXXX"
|
||||
|
||||
rag_pipeline = Pipeline()
|
||||
rag_pipeline.add_component(
|
||||
instance=InMemoryBM25Retriever(document_store=InMemoryDocumentStore()),
|
||||
name="retriever",
|
||||
)
|
||||
rag_pipeline.add_component(
|
||||
instance=ChatPromptBuilder(template=prompt_template, required_variables="*"),
|
||||
name="prompt_builder",
|
||||
)
|
||||
rag_pipeline.add_component(instance=OpenAIChatGenerator(), name="llm")
|
||||
rag_pipeline.add_component(instance=AnswerBuilder(), name="answer_builder")
|
||||
rag_pipeline.connect("retriever", "prompt_builder.documents")
|
||||
rag_pipeline.connect("prompt_builder.prompt", "llm.messages")
|
||||
rag_pipeline.connect("llm.replies", "answer_builder.replies")
|
||||
rag_pipeline.connect("retriever", "answer_builder.documents")
|
||||
|
||||
# Draw the pipeline
|
||||
rag_pipeline.draw("./rag_pipeline.png")
|
||||
|
||||
# Add Documents
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
rag_pipeline.get_component("retriever").document_store.write_documents(documents)
|
||||
|
||||
# Run the pipeline
|
||||
question = "How many languages are there?"
|
||||
result = rag_pipeline.run(
|
||||
{
|
||||
"retriever": {"query": question},
|
||||
"prompt_builder": {"question": question},
|
||||
"answer_builder": {"query": question},
|
||||
},
|
||||
)
|
||||
print(result["answer_builder"]["answers"][0])
|
||||
```
|
||||
|
||||
#### In a Document Search Pipeline
|
||||
|
||||
Here's how you can use this Retriever in a document search pipeline:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.pipeline import Pipeline
|
||||
|
||||
# Create components and a query pipeline
|
||||
document_store = InMemoryDocumentStore()
|
||||
retriever = InMemoryBM25Retriever(document_store=document_store)
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component(instance=retriever, name="retriever")
|
||||
|
||||
# Add Documents
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
document_store.write_documents(documents)
|
||||
|
||||
# Run the pipeline
|
||||
result = pipeline.run(data={"retriever": {"query": "How many languages are there?"}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
```
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
title: "InMemoryEmbeddingRetriever"
|
||||
id: inmemoryembeddingretriever
|
||||
slug: "/inmemoryembeddingretriever"
|
||||
description: "Use this Retriever with the InMemoryDocumentStore if you're looking for embedding-based retrieval."
|
||||
---
|
||||
|
||||
# InMemoryEmbeddingRetriever
|
||||
|
||||
Use this Retriever with the InMemoryDocumentStore if you're looking for embedding-based retrieval.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In query pipelines: <br />In a RAG pipeline, before a [`PromptBuilder`](../builders/promptbuilder.mdx) <br />In a semantic search pipeline, as the last component <br />In an extractive QA pipeline, after a Tex tEmbedder and before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) |
|
||||
| **Mandatory init variables** | `document_store`: An instance of [InMemoryDocumentStore](../../document-stores/inmemorydocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query_embedding`: A list of floating point numbers |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Retrievers](/reference/retrievers-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/retrievers/in_memory/embedding_retriever.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `InMemoryEmbeddingRetriever` is an embedding-based Retriever compatible with the `InMemoryDocumentStore`. It compares the query and Document embeddings and fetches the Documents most relevant to the query from the `InMemoryDocumentStore` based on the outcome.
|
||||
|
||||
When using the `InMemoryEmbeddingRetriever` in your NLP system, make sure it has the query and Document embeddings available. You can do so by adding a DocumentEmbedder to your indexing pipeline and a Text Embedder to your query pipeline. For details, see [Embedders](../embedders.mdx).
|
||||
|
||||
In addition to the `query_embedding`, the `InMemoryEmbeddingRetriever` accepts other optional parameters, including `top_k` (the maximum number of Documents to retrieve) and `filters` to narrow down the search space.
|
||||
|
||||
The `embedding_similarity_function` to use for embedding retrieval must be defined when the corresponding`InMemoryDocumentStore` is initialized.
|
||||
|
||||
## Usage
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Use this Retriever in a query pipeline like this:
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersTextEmbedder,
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
)
|
||||
from haystack.components.retrievers import InMemoryEmbeddingRetriever
|
||||
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
document_embedder = SentenceTransformersDocumentEmbedder()
|
||||
|
||||
documents_with_embeddings = document_embedder.run(documents)["documents"]
|
||||
document_store.write_documents(documents_with_embeddings)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", SentenceTransformersTextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "How many languages are there?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
```
|
||||
@@ -0,0 +1,146 @@
|
||||
---
|
||||
title: "Mem0MemoryRetriever"
|
||||
id: mem0memoryretriever
|
||||
slug: "/mem0memoryretriever"
|
||||
description: "Retrieves long-term memories from Mem0 as ChatMessage objects."
|
||||
---
|
||||
|
||||
# Mem0MemoryRetriever
|
||||
|
||||
Retrieves long-term memories from Mem0 as `ChatMessage` objects.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before an [`Agent`](../agents-1/agent.mdx) or Chat Generator in memory-augmented pipelines |
|
||||
| **Mandatory init variables** | `memory_store`: A `Mem0MemoryStore` instance |
|
||||
| **Mandatory run variables** | `query`: A text query or `None`; at least one Mem0 scope through `user_id`, `run_id`, `agent_id`, `app_id`, or `filters` |
|
||||
| **Output variables** | `memories`: A list of `ChatMessage` objects |
|
||||
| **Mem0 API docs** | [Search Memories](https://docs.mem0.ai/api-reference/memory/search-memories), [Memory Filters](https://docs.mem0.ai/platform/features/v2-memory-filters) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/mem0 |
|
||||
| **Package name** | `mem0-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`Mem0MemoryRetriever` retrieves memories from a `Mem0MemoryStore` and returns them as system `ChatMessage` objects.
|
||||
Use it to inject long-term memory into an Agent or a chat generation pipeline before the model produces a response.
|
||||
|
||||
The `query` input can be a string or `None`.
|
||||
When `query` is a string, the component searches for relevant memories and applies `top_k`.
|
||||
When `query` is `None`, it returns all memories matching the provided scope.
|
||||
|
||||
Scope the retrieval with at least one Mem0 entity ID: `user_id`, `run_id`, `agent_id`, or `app_id`.
|
||||
You can also pass Haystack-style `filters`; when filters and ID parameters are both provided, they are combined with an `AND` condition.
|
||||
For general filter syntax, see [Metadata Filtering](../../concepts/metadata-filtering.mdx).
|
||||
|
||||
User-provided Mem0 metadata is included in each returned message's `meta`.
|
||||
Mem0 retrieval fields such as `memory_id`, `user_id`, `score`, and timestamps are included under `meta["mem0"]`.
|
||||
|
||||
### Installation
|
||||
|
||||
Install the Mem0 integration:
|
||||
|
||||
```shell
|
||||
pip install mem0-haystack
|
||||
```
|
||||
|
||||
Set your Mem0 API key:
|
||||
|
||||
```shell
|
||||
export MEM0_API_KEY="your-mem0-api-key"
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
from haystack_integrations.components.retrievers.mem0 import Mem0MemoryRetriever
|
||||
from haystack_integrations.memory_stores.mem0 import Mem0MemoryStore
|
||||
|
||||
store = Mem0MemoryStore()
|
||||
store.add_memories(
|
||||
messages=[ChatMessage.from_user("Alice prefers concise Python examples.")],
|
||||
user_id="alice",
|
||||
infer=False,
|
||||
)
|
||||
|
||||
retriever = Mem0MemoryRetriever(memory_store=store, top_k=3)
|
||||
|
||||
result = retriever.run(query="answer style", user_id="alice")
|
||||
memories = result["memories"]
|
||||
|
||||
for memory in memories:
|
||||
print(memory.text)
|
||||
```
|
||||
|
||||
To retrieve all memories in scope, pass `query=None`:
|
||||
|
||||
```python
|
||||
all_memories = retriever.run(query=None, user_id="alice")["memories"]
|
||||
print([memory.text for memory in all_memories])
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
This example retrieves memories, prepends them to the current user message, and passes the combined message list to an Agent.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.converters import OutputAdapter
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
from haystack_integrations.components.retrievers.mem0 import Mem0MemoryRetriever
|
||||
from haystack_integrations.memory_stores.mem0 import Mem0MemoryStore
|
||||
|
||||
store = Mem0MemoryStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("retriever", Mem0MemoryRetriever(memory_store=store, top_k=5))
|
||||
pipeline.add_component(
|
||||
"memory_context",
|
||||
OutputAdapter(
|
||||
template="{{ memories + user_messages }}",
|
||||
output_type=list[ChatMessage],
|
||||
unsafe=True,
|
||||
),
|
||||
)
|
||||
pipeline.add_component(
|
||||
"agent",
|
||||
Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-4o-mini"),
|
||||
system_prompt=(
|
||||
"Use any system messages at the start of the conversation as long-term memory. "
|
||||
"Answer concisely."
|
||||
),
|
||||
streaming_callback=print_streaming_chunk,
|
||||
),
|
||||
)
|
||||
|
||||
pipeline.connect("retriever.memories", "memory_context.memories")
|
||||
pipeline.connect("memory_context.output", "agent.messages")
|
||||
|
||||
query = "Give me a short implementation tip."
|
||||
|
||||
pipeline.run(
|
||||
{
|
||||
"retriever": {
|
||||
"query": query,
|
||||
"user_id": "alice",
|
||||
},
|
||||
"memory_context": {
|
||||
"user_messages": [
|
||||
ChatMessage.from_user(query),
|
||||
],
|
||||
},
|
||||
},
|
||||
)
|
||||
```
|
||||
@@ -0,0 +1,154 @@
|
||||
---
|
||||
title: "MongoDBAtlasEmbeddingRetriever"
|
||||
id: mongodbatlasembeddingretriever
|
||||
slug: "/mongodbatlasembeddingretriever"
|
||||
description: "This is an embedding Retriever compatible with the MongoDB Atlas Document Store."
|
||||
---
|
||||
|
||||
# MongoDBAtlasEmbeddingRetriever
|
||||
|
||||
This is an embedding Retriever compatible with the MongoDB Atlas Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the semantic search pipeline 3. After a Text Embedder and before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a [MongoDBAtlasDocumentStore](../../document-stores/mongodbatlasdocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query_embedding`: A list of floats |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [MongoDB Atlas](/reference/integrations-mongodb-atlas) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/mongodb_atlas |
|
||||
| **Package name** | `mongodb-atlas-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
The `MongoDBAtlasEmbeddingRetriever` is an embedding-based Retriever compatible with the [`MongoDBAtlasDocumentStore`](../../document-stores/mongodbatlasdocumentstore.mdx). It compares the query and Document embeddings and fetches the Documents most relevant to the query from the Document Store based on the outcome.
|
||||
|
||||
### Parameters
|
||||
|
||||
When using the `MongoDBAtlasEmbeddingRetriever` in your NLP system, ensure the query and Document [embeddings](../embedders.mdx) are available. You can do so by adding a Document Embedder to your indexing Pipeline and a Text Embedder to your query Pipeline.
|
||||
|
||||
In addition to the `query_embedding`, the `MongoDBAtlasEmbeddingRetriever` accepts other optional parameters, including `top_k` (the maximum number of Documents to retrieve) and `filters` to narrow down the search space.
|
||||
|
||||
## Usage
|
||||
|
||||
### Installation
|
||||
|
||||
To start using MongoDB Atlas with Haystack, install the package with:
|
||||
|
||||
```shell
|
||||
pip install mongodb-atlas-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
The Retriever needs an instance of `MongoDBAtlasDocumentStore` and indexed Documents to run.
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.mongodb_atlas import (
|
||||
MongoDBAtlasDocumentStore,
|
||||
)
|
||||
from haystack_integrations.components.retrievers.mongodb_atlas import (
|
||||
MongoDBAtlasEmbeddingRetriever,
|
||||
)
|
||||
|
||||
document_store = MongoDBAtlasDocumentStore()
|
||||
|
||||
retriever = MongoDBAtlasEmbeddingRetriever(document_store=document_store)
|
||||
|
||||
# example run query
|
||||
retriever.run(query_embedding=[0.1] * 384)
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
```python
|
||||
from haystack import Pipeline, Document
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
SentenceTransformersTextEmbedder,
|
||||
)
|
||||
from haystack_integrations.document_stores.mongodb_atlas import (
|
||||
MongoDBAtlasDocumentStore,
|
||||
)
|
||||
from haystack_integrations.components.embedders.mongodb_atlas import (
|
||||
MongoDBAtlasEmbeddingRetriever,
|
||||
)
|
||||
|
||||
# Create some example documents
|
||||
documents = [
|
||||
Document(content="My name is Jean and I live in Paris."),
|
||||
Document(content="My name is Mark and I live in Berlin."),
|
||||
Document(content="My name is Giorgio and I live in Rome."),
|
||||
]
|
||||
|
||||
# We support many different databases. Here we load a simple and lightweight in-memory document store.
|
||||
document_store = MongoDBAtlasDocumentStore()
|
||||
|
||||
# Define some more components
|
||||
doc_writer = DocumentWriter(document_store=document_store, policy=DuplicatePolicy.SKIP)
|
||||
doc_embedder = SentenceTransformersDocumentEmbedder(model="intfloat/e5-base-v2")
|
||||
query_embedder = SentenceTransformersTextEmbedder(model="intfloat/e5-base-v2")
|
||||
|
||||
# Pipeline that ingests document for retrieval
|
||||
ingestion_pipe = Pipeline()
|
||||
ingestion_pipe.add_component(instance=doc_embedder, name="doc_embedder")
|
||||
ingestion_pipe.add_component(instance=doc_writer, name="doc_writer")
|
||||
|
||||
ingestion_pipe.connect("doc_embedder.documents", "doc_writer.documents")
|
||||
ingestion_pipe.run({"doc_embedder": {"documents": documents}})
|
||||
|
||||
# Build a RAG pipeline with a Retriever to get relevant documents to
|
||||
# the query and an OpenAIChatGenerator interacting with LLMs using a custom prompt.
|
||||
prompt_template = [
|
||||
ChatMessage.from_user(
|
||||
"""
|
||||
Given these documents, answer the question.\nDocuments:
|
||||
{% for doc in documents %}
|
||||
{{ doc.content }}
|
||||
{% endfor %}
|
||||
|
||||
\nQuestion: {{question}}
|
||||
\nAnswer:
|
||||
""",
|
||||
),
|
||||
]
|
||||
rag_pipeline = Pipeline()
|
||||
rag_pipeline.add_component(instance=query_embedder, name="query_embedder")
|
||||
rag_pipeline.add_component(
|
||||
instance=MongoDBAtlasEmbeddingRetriever(document_store=document_store),
|
||||
name="retriever",
|
||||
)
|
||||
rag_pipeline.add_component(
|
||||
instance=ChatPromptBuilder(template=prompt_template, required_variables="*"),
|
||||
name="prompt_builder",
|
||||
)
|
||||
rag_pipeline.add_component(instance=OpenAIChatGenerator(), name="llm")
|
||||
rag_pipeline.connect("query_embedder", "retriever.query_embedding")
|
||||
rag_pipeline.connect("retriever", "prompt_builder.documents")
|
||||
rag_pipeline.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
# Ask a question on the data you just added.
|
||||
question = "Where does Mark live?"
|
||||
result = rag_pipeline.run(
|
||||
{
|
||||
"query_embedder": {"text": question},
|
||||
"prompt_builder": {"question": question},
|
||||
},
|
||||
)
|
||||
|
||||
# The generated reply is a ChatMessage; its text holds the answer.
|
||||
print(result["llm"]["replies"][0].text)
|
||||
```
|
||||
@@ -0,0 +1,160 @@
|
||||
---
|
||||
title: "MongoDBAtlasFullTextRetriever"
|
||||
id: mongodbatlasfulltextretriever
|
||||
slug: "/mongodbatlasfulltextretriever"
|
||||
description: "This is a full-text search Retriever compatible with the MongoDB Atlas Document Store."
|
||||
---
|
||||
|
||||
# MongoDBAtlasFullTextRetriever
|
||||
|
||||
This is a full-text search Retriever compatible with the MongoDB Atlas Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. Before a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) in a RAG pipeline 2. The last component in the semantic search pipeline 3. Before a [TransformersExtractiveReader](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a [MongoDBAtlasDocumentStore](../../document-stores/mongodbatlasdocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query`: A query string to search for. If the query contains multiple terms, Atlas Search evaluates each term separately for matches. |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [MongoDB Atlas](/reference/integrations-mongodb-atlas) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/mongodb_atlas |
|
||||
| **Package name** | `mongodb-atlas-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
The `MongoDBAtlasFullTextRetriever` is a full-text search Retriever compatible with the [`MongoDBAtlasDocumentStore`](../../document-stores/mongodbatlasdocumentstore.mdx). The full-text search is dependent on the `full_text_search_index` used in the [`MongoDBAtlasDocumentStore`](../../document-stores/mongodbatlasdocumentstore.mdx).
|
||||
|
||||
### Parameters
|
||||
|
||||
In addition to the `query`, the `MongoDBAtlasFullTextRetriever` accepts other optional parameters, including `top_k` (the maximum number of Documents to retrieve) and `filters` to narrow down the search space.
|
||||
|
||||
When running the component, you can specify more optional parameters such as `fuzzy` or `synonyms`, `match_criteria`, `score`. Check out our [MongoDB Atlas](/reference/integrations-mongodb-atlas) API Reference for more details on all parameters.
|
||||
|
||||
## Usage
|
||||
|
||||
### Installation
|
||||
|
||||
To start using MongoDB Atlas with Haystack, install the package with:
|
||||
|
||||
```shell
|
||||
pip install mongodb-atlas-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
The Retriever needs an instance of `MongoDBAtlasDocumentStore` and indexed documents to run.
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.mongodb_atlas import (
|
||||
MongoDBAtlasDocumentStore,
|
||||
)
|
||||
from haystack_integrations.components.retrievers.mongodb_atlas import (
|
||||
MongoDBAtlasFullTextRetriever,
|
||||
)
|
||||
|
||||
store = MongoDBAtlasDocumentStore(
|
||||
database_name="your_existing_db",
|
||||
collection_name="your_existing_collection",
|
||||
vector_search_index="your_existing_index",
|
||||
full_text_search_index="your_existing_index",
|
||||
)
|
||||
retriever = MongoDBAtlasFullTextRetriever(document_store=store)
|
||||
|
||||
results = retriever.run(query="Your search query")
|
||||
print(results["documents"])
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
Here's a Hybrid Retrieval pipeline example that makes use of both available MongoDB Atlas Retrievers:
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
```python
|
||||
from haystack import Pipeline, Document
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
SentenceTransformersTextEmbedder,
|
||||
)
|
||||
from haystack.components.joiners import DocumentJoiner
|
||||
|
||||
from haystack_integrations.document_stores.mongodb_atlas import (
|
||||
MongoDBAtlasDocumentStore,
|
||||
)
|
||||
from haystack_integrations.components.retrievers.mongodb_atlas import (
|
||||
MongoDBAtlasEmbeddingRetriever,
|
||||
MongoDBAtlasFullTextRetriever,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Jean and I live in Paris."),
|
||||
Document(content="My name is Mark and I live in Berlin."),
|
||||
Document(content="My name is Giorgio and I live in Rome."),
|
||||
Document(content="Python is a programming language popular for data science."),
|
||||
Document(
|
||||
content="MongoDB Atlas offers full-text search and vector search capabilities.",
|
||||
),
|
||||
]
|
||||
|
||||
document_store = MongoDBAtlasDocumentStore(
|
||||
database_name="haystack_test",
|
||||
collection_name="test_collection",
|
||||
vector_search_index="test_vector_search_index",
|
||||
full_text_search_index="test_full_text_search_index",
|
||||
)
|
||||
|
||||
# Clean out any old data so this example is repeatable
|
||||
print(f"Clearing collection {document_store.collection_name} …")
|
||||
document_store.collection.delete_many({})
|
||||
|
||||
ingest_pipe = Pipeline()
|
||||
|
||||
doc_embedder = SentenceTransformersDocumentEmbedder(model="intfloat/e5-base-v2")
|
||||
ingest_pipe.add_component(instance=doc_embedder, name="doc_embedder")
|
||||
|
||||
doc_writer = DocumentWriter(document_store=document_store, policy=DuplicatePolicy.SKIP)
|
||||
ingest_pipe.add_component(instance=doc_writer, name="doc_writer")
|
||||
ingest_pipe.connect("doc_embedder.documents", "doc_writer.documents")
|
||||
|
||||
print(f"Running ingestion on {len(documents)} in-memory docs …")
|
||||
ingest_pipe.run({"doc_embedder": {"documents": documents}})
|
||||
|
||||
query_pipe = Pipeline()
|
||||
|
||||
text_embedder = SentenceTransformersTextEmbedder(model="intfloat/e5-base-v2")
|
||||
query_pipe.add_component(instance=text_embedder, name="text_embedder")
|
||||
|
||||
embed_retriever = MongoDBAtlasEmbeddingRetriever(document_store=document_store, top_k=3)
|
||||
query_pipe.add_component(instance=embed_retriever, name="embedding_retriever")
|
||||
query_pipe.connect("text_embedder", "embedding_retriever")
|
||||
|
||||
# (c) full-text retriever
|
||||
ft_retriever = MongoDBAtlasFullTextRetriever(document_store=document_store, top_k=3)
|
||||
query_pipe.add_component(instance=ft_retriever, name="full_text_retriever")
|
||||
|
||||
joiner = DocumentJoiner(join_mode="reciprocal_rank_fusion", top_k=3)
|
||||
query_pipe.add_component(instance=joiner, name="joiner")
|
||||
|
||||
query_pipe.connect("embedding_retriever", "joiner")
|
||||
query_pipe.connect("full_text_retriever", "joiner")
|
||||
|
||||
question = "Where does Mark live?"
|
||||
print(f"Running hybrid retrieval for query: '{question}'")
|
||||
output = query_pipe.run(
|
||||
{
|
||||
"text_embedder": {"text": question},
|
||||
"full_text_retriever": {"query": question},
|
||||
},
|
||||
)
|
||||
|
||||
print("\nFinal fused documents:")
|
||||
for doc in output["joiner"]["documents"]:
|
||||
print(f"- {doc.content}")
|
||||
```
|
||||
@@ -0,0 +1,110 @@
|
||||
---
|
||||
title: "MSSharePointRetriever"
|
||||
id: mssharepointretriever
|
||||
slug: "/mssharepointretriever"
|
||||
description: "Retrieves content from Microsoft SharePoint and OneDrive via the Microsoft Search (Graph) API."
|
||||
---
|
||||
|
||||
# MSSharePointRetriever
|
||||
|
||||
Retrieves content from Microsoft SharePoint and OneDrive via the Microsoft Search (Graph) API.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | At the start of a query pipeline, after an [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx) that provides the `access_token` |
|
||||
| **Mandatory init variables** | None |
|
||||
| **Mandatory run variables** | `query`: The search query string <br /> <br />`access_token`: A delegated Microsoft Graph bearer token, typically wired from an upstream `OAuthTokenResolver` |
|
||||
| **Output variables** | `documents`: A list of [Documents](../../concepts/data-classes.mdx) holding the search snippets and resource metadata |
|
||||
| **API reference** | [Microsoft SharePoint](/reference/integrations-microsoft-sharepoint) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/microsoft_sharepoint |
|
||||
| **Package name** | `microsoft-sharepoint-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`MSSharePointRetriever` searches a user's Microsoft SharePoint and OneDrive content through the [Microsoft Search (Graph) API](https://learn.microsoft.com/en-us/graph/api/resources/search-api-overview). Given a query, it calls `POST /search/query` and maps each hit to a Haystack `Document` whose `content` is the search snippet and whose `meta` carries the resource metadata: `file_name`, `web_url`, `entity_type`, `created_date_time`, `last_modified_date_time`, `created_by`, `last_modified_by`, `mime_type`, and `file_extension`. It also stores the SharePoint identifiers a downstream fetcher needs to read list items and pages by ID (`site_id`, `list_id`, `list_item_id`, `list_item_unique_id`).
|
||||
|
||||
The retriever does **not** download or convert the underlying files – it only returns Search snippets and metadata. To download the full content of the hits, compose it with [`MSSharePointFetcher`](../fetchers/mssharepointfetcher.mdx) followed by a converter.
|
||||
|
||||
### Authentication
|
||||
|
||||
The retriever takes a per-user `access_token` as a run input. The token must carry **delegated** Microsoft Graph permissions (for example `Files.Read.All`, plus `Sites.Read.All` for site and list scoping); the Search API supports delegated permissions only. Typically you wire the token from an upstream [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx), which emits a plain string. A `Secret` is also accepted and resolved internally.
|
||||
|
||||
### Scoping and filtering the search
|
||||
|
||||
You can narrow what is searched in several ways:
|
||||
|
||||
- `entity_types`: which Microsoft Search entity types to query. Defaults to `["driveItem", "listItem"]`, which covers files, folders, SharePoint pages and news, and list items. Other valid values are `"list"` and `"site"`.
|
||||
- KQL operators embedded directly in the query, for example `filetype:docx`, `author:"Jane Doe"`, or `path:"https://contoso.sharepoint.com/sites/Team"`. See the [Keyword Query Language (KQL) syntax reference](https://learn.microsoft.com/en-us/sharepoint/dev/general-development/keyword-query-language-kql-syntax-reference).
|
||||
- `query_template`: a reusable template such as `'{searchTerms} path:"https://contoso.sharepoint.com/sites/Team"'`, where the literal `{searchTerms}` placeholder is replaced by the run-time query.
|
||||
|
||||
### Installation
|
||||
|
||||
Install the Microsoft SharePoint integration with:
|
||||
|
||||
```shell
|
||||
pip install microsoft-sharepoint-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
`access_token` below is a per-user delegated Microsoft Graph bearer token. In production you would obtain it from an [`OAuthTokenResolver`](../connectors/oauthtokenresolver.mdx) rather than pasting it in.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.retrievers.microsoft_sharepoint import (
|
||||
MSSharePointRetriever,
|
||||
)
|
||||
|
||||
retriever = MSSharePointRetriever(top_k=5)
|
||||
|
||||
result = retriever.run(
|
||||
query="quarterly roadmap",
|
||||
access_token="my-delegated-graph-token",
|
||||
)
|
||||
|
||||
for doc in result["documents"]:
|
||||
print(doc.meta["file_name"], "-", doc.meta["web_url"])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
The following pipeline obtains a token from an `OAuthTokenResolver` and feeds it into the retriever, so that running the pipeline requires only the query:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.components.connectors.oauth import OAuthTokenResolver
|
||||
from haystack_integrations.utils.oauth import OAuthRefreshTokenSource
|
||||
from haystack_integrations.components.retrievers.microsoft_sharepoint import (
|
||||
MSSharePointRetriever,
|
||||
)
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component(
|
||||
"resolver",
|
||||
OAuthTokenResolver(
|
||||
token_source=OAuthRefreshTokenSource(
|
||||
token_url="https://login.microsoftonline.com/common/oauth2/v2.0/token",
|
||||
client_id="aaa-bbb-ccc",
|
||||
refresh_token=Secret.from_env_var("MS_REFRESH_TOKEN"),
|
||||
scopes=[
|
||||
"https://graph.microsoft.com/Files.Read.All",
|
||||
"https://graph.microsoft.com/Sites.Read.All",
|
||||
"offline_access",
|
||||
],
|
||||
),
|
||||
),
|
||||
)
|
||||
pipeline.add_component("retriever", MSSharePointRetriever(top_k=5))
|
||||
pipeline.connect("resolver.access_token", "retriever.access_token")
|
||||
|
||||
result = pipeline.run({"retriever": {"query": "quarterly roadmap"}})
|
||||
documents = result["retriever"]["documents"]
|
||||
```
|
||||
|
||||
To download and convert the full content of the retrieved hits, connect the retriever's `documents` output to a [`MSSharePointFetcher`](../fetchers/mssharepointfetcher.mdx). See that page for an end-to-end retrieve-fetch-convert example.
|
||||
@@ -0,0 +1,157 @@
|
||||
---
|
||||
title: "MultiQueryEmbeddingRetriever"
|
||||
id: multiqueryembeddingretriever
|
||||
slug: "/multiqueryembeddingretriever"
|
||||
description: "Retrieves documents using multiple queries in parallel with an embedding-based Retriever."
|
||||
---
|
||||
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
# MultiQueryEmbeddingRetriever
|
||||
|
||||
Retrieves documents using multiple queries in parallel with an embedding-based Retriever.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`QueryExpander`](../query/queryexpander.mdx) component, before a [`ChatPromptBuilder`](../builders/chatpromptbuilder.mdx) in RAG pipelines |
|
||||
| **Mandatory init variables** | `retriever`: An embedding-based Retriever (such as `InMemoryEmbeddingRetriever`)<br />`query_embedder`: A Text Embedder component |
|
||||
| **Mandatory run variables** | `queries`: A list of query strings |
|
||||
| **Output variables** | `documents`: A list of retrieved documents sorted by relevance score |
|
||||
| **API reference** | [Retrievers](/reference/retrievers-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/retrievers/multi_query_embedding_retriever.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`MultiQueryEmbeddingRetriever` improves retrieval recall by searching for documents using multiple queries in parallel. Each query is converted to an embedding using a Text Embedder, and an embedding-based Retriever fetches relevant documents.
|
||||
|
||||
The component:
|
||||
- Processes queries in parallel using a thread pool for better performance
|
||||
- Automatically deduplicates results based on document content
|
||||
- Sorts the final results by relevance score
|
||||
|
||||
This Retriever is particularly effective when combined with [`QueryExpander`](../query/queryexpander.mdx), which generates multiple query variations from a single user query. By searching with these variations, you can find documents that might not match the original query phrasing but are still relevant.
|
||||
|
||||
Use `MultiQueryEmbeddingRetriever` when your documents use different words than your users' queries, or when you want to find more diverse results in RAG pipelines. Running multiple queries takes more time, but you can speed it up by increasing `max_workers` to run queries in parallel.
|
||||
|
||||
:::tip[When to use a `MultiQueryTextRetriever` instead]
|
||||
|
||||
If you need exact keyword matching and don't want to use embeddings, use [`MultiQueryTextRetriever`](multiquerytextretriever.mdx) instead. It works with text-based Retrievers like `InMemoryBM25Retriever` and is better when synonyms can be generated through query expansion.
|
||||
:::
|
||||
|
||||
### Passing Additional Retriever Parameters
|
||||
|
||||
You can pass additional parameters to the underlying Retriever using `retriever_kwargs`:
|
||||
|
||||
```python
|
||||
result = multi_query_retriever.run(
|
||||
queries=["renewable energy", "sustainable power"],
|
||||
retriever_kwargs={"top_k": 5},
|
||||
)
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
This pipeline takes a single query "sustainable power generation" and expands it into multiple variations using an LLM (for example: "renewable energy sources", "green electricity", "clean power"). The Retriever then converts each variation to an embedding and searches for similar documents. This way, documents about "solar energy" or "wind energy" can be found even though they don't contain the words "sustainable power generation".
|
||||
|
||||
Before running the pipeline, documents must be embedded using a Document Embedder and stored in the Document Store.
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="python" label="Python" default>
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersTextEmbedder,
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
)
|
||||
from haystack.components.retrievers import (
|
||||
InMemoryEmbeddingRetriever,
|
||||
MultiQueryEmbeddingRetriever,
|
||||
)
|
||||
from haystack.components.query import QueryExpander
|
||||
|
||||
documents = [
|
||||
Document(
|
||||
content="Renewable energy is energy that is collected from renewable resources.",
|
||||
),
|
||||
Document(
|
||||
content="Solar energy is a type of green energy that is harnessed from the sun.",
|
||||
),
|
||||
Document(
|
||||
content="Wind energy is another type of green energy that is generated by wind turbines.",
|
||||
),
|
||||
Document(
|
||||
content="Geothermal energy is heat that comes from the sub-surface of the earth.",
|
||||
),
|
||||
]
|
||||
|
||||
doc_store = InMemoryDocumentStore()
|
||||
doc_embedder = SentenceTransformersDocumentEmbedder(
|
||||
model="sentence-transformers/all-MiniLM-L6-v2",
|
||||
)
|
||||
documents_with_embeddings = doc_embedder.run(documents)["documents"]
|
||||
doc_store.write_documents(documents_with_embeddings)
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("query_expander", QueryExpander(n_expansions=3))
|
||||
pipeline.add_component(
|
||||
"retriever",
|
||||
MultiQueryEmbeddingRetriever(
|
||||
retriever=InMemoryEmbeddingRetriever(document_store=doc_store, top_k=2),
|
||||
query_embedder=SentenceTransformersTextEmbedder(
|
||||
model="sentence-transformers/all-MiniLM-L6-v2",
|
||||
),
|
||||
),
|
||||
)
|
||||
pipeline.connect("query_expander.queries", "retriever.queries")
|
||||
|
||||
result = pipeline.run({"query_expander": {"query": "sustainable power generation"}})
|
||||
|
||||
for doc in result["retriever"]["documents"]:
|
||||
print(f"Score: {doc.score:.3f} | {doc.content}")
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="yaml" label="YAML">
|
||||
|
||||
```yaml
|
||||
components:
|
||||
query_expander:
|
||||
type: haystack.components.query.query_expander.QueryExpander
|
||||
init_parameters:
|
||||
n_expansions: 3
|
||||
retriever:
|
||||
type: haystack.components.retrievers.multi_query_embedding_retriever.MultiQueryEmbeddingRetriever
|
||||
init_parameters:
|
||||
retriever:
|
||||
type: haystack.components.retrievers.in_memory.embedding_retriever.InMemoryEmbeddingRetriever
|
||||
init_parameters:
|
||||
document_store:
|
||||
type: haystack.document_stores.in_memory.document_store.InMemoryDocumentStore
|
||||
init_parameters: {}
|
||||
top_k: 2
|
||||
query_embedder:
|
||||
type: haystack_integrations.components.embedders.sentence_transformers.sentence_transformers_text_embedder.SentenceTransformersTextEmbedder
|
||||
init_parameters:
|
||||
model: sentence-transformers/all-MiniLM-L6-v2
|
||||
|
||||
connections:
|
||||
- sender: query_expander.queries
|
||||
receiver: retriever.queries
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
@@ -0,0 +1,308 @@
|
||||
---
|
||||
title: "MultiQueryTextRetriever"
|
||||
id: multiquerytextretriever
|
||||
slug: "/multiquerytextretriever"
|
||||
description: "Retrieves documents using multiple queries in parallel with a text-based Retriever."
|
||||
---
|
||||
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
|
||||
# MultiQueryTextRetriever
|
||||
|
||||
Retrieves documents using multiple queries in parallel with a text-based Retriever.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`QueryExpander`](../query/queryexpander.mdx) component, before a [`ChatPromptBuilder`](../builders/chatpromptbuilder.mdx) in RAG pipelines |
|
||||
| **Mandatory init variables** | `retriever`: A text-based Retriever (such as `InMemoryBM25Retriever`) |
|
||||
| **Mandatory run variables** | `queries`: A list of query strings |
|
||||
| **Output variables** | `documents`: A list of retrieved documents sorted by relevance score |
|
||||
| **API reference** | [Retrievers](/reference/retrievers-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/retrievers/multi_query_text_retriever.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`MultiQueryTextRetriever` improves retrieval recall by searching for documents using multiple queries in parallel. It wraps a text-based Retriever (such as `InMemoryBM25Retriever`) and processes multiple query strings simultaneously using a thread pool.
|
||||
|
||||
The component:
|
||||
- Processes queries in parallel for better performance
|
||||
- Automatically deduplicates results based on document content
|
||||
- Sorts the final results by relevance score
|
||||
|
||||
This Retriever is particularly effective when combined with [`QueryExpander`](../query/queryexpander.mdx), which generates multiple query variations from a single user query. By searching with these variations, you can find documents that use different keywords than the original query.
|
||||
|
||||
Use `MultiQueryTextRetriever` when your documents use different words than your users' queries, or when you want to use query expansion with keyword-based search (BM25). Running multiple queries takes more time, but you can speed it up by increasing `max_workers` to run queries in parallel.
|
||||
|
||||
:::tip[When to use `MultiQueryEmbeddingRetriever` instead]
|
||||
|
||||
If you need semantic search where meaning matters more than exact keyword matches, use [`MultiQueryEmbeddingRetriever`](multiqueryembeddingretriever.mdx) instead. It works with embedding-based Retrievers and requires a Text Embedder.
|
||||
:::
|
||||
|
||||
### Passing Additional Retriever Parameters
|
||||
|
||||
You can pass additional parameters to the underlying Retriever using `retriever_kwargs`:
|
||||
|
||||
```python
|
||||
result = multiquery_retriever.run(
|
||||
queries=["renewable energy", "sustainable power"],
|
||||
retriever_kwargs={"top_k": 5},
|
||||
)
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
In this example, we pass three queries manually to the Retriever: "renewable energy", "geothermal", and "hydropower". The Retriever runs a BM25 search for each query (retrieving up to 2 documents per query), then combines all results, removes duplicates, and sorts them by score.
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.retrievers import (
|
||||
InMemoryBM25Retriever,
|
||||
MultiQueryTextRetriever,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(
|
||||
content="Renewable energy is energy that is collected from renewable resources.",
|
||||
),
|
||||
Document(
|
||||
content="Solar energy is a type of green energy that is harnessed from the sun.",
|
||||
),
|
||||
Document(
|
||||
content="Wind energy is another type of green energy that is generated by wind turbines.",
|
||||
),
|
||||
Document(
|
||||
content="Hydropower is a form of renewable energy using the flow of water to generate electricity.",
|
||||
),
|
||||
Document(
|
||||
content="Geothermal energy is heat that comes from the sub-surface of the earth.",
|
||||
),
|
||||
]
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
document_store.write_documents(documents)
|
||||
|
||||
retriever = MultiQueryTextRetriever(
|
||||
retriever=InMemoryBM25Retriever(document_store=document_store, top_k=2),
|
||||
)
|
||||
|
||||
results = retriever.run(queries=["renewable energy", "geothermal", "hydropower"])
|
||||
|
||||
for doc in results["documents"]:
|
||||
print(f"Content: {doc.content}, Score: {doc.score:.4f}")
|
||||
```
|
||||
|
||||
### In a pipeline with QueryExpander
|
||||
|
||||
This pipeline takes a single query "sustainable power" and expands it into multiple variations using an LLM (for example: "renewable energy sources", "green electricity", "clean power"). The Retriever then searches for each variation and combines the results. This way, documents about "solar energy" or "hydropower" can be found even though they don't contain the words "sustainable power".
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="python" label="Python" default>
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.query import QueryExpander
|
||||
from haystack.components.retrievers import (
|
||||
InMemoryBM25Retriever,
|
||||
MultiQueryTextRetriever,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(
|
||||
content="Renewable energy is energy that is collected from renewable resources.",
|
||||
),
|
||||
Document(
|
||||
content="Solar energy is a type of green energy that is harnessed from the sun.",
|
||||
),
|
||||
Document(
|
||||
content="Wind energy is another type of green energy that is generated by wind turbines.",
|
||||
),
|
||||
Document(
|
||||
content="Hydropower is a form of renewable energy using the flow of water to generate electricity.",
|
||||
),
|
||||
Document(
|
||||
content="Geothermal energy is heat that comes from the sub-surface of the earth.",
|
||||
),
|
||||
]
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
document_store.write_documents(documents)
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("query_expander", QueryExpander(n_expansions=3))
|
||||
pipeline.add_component(
|
||||
"retriever",
|
||||
MultiQueryTextRetriever(
|
||||
retriever=InMemoryBM25Retriever(document_store=document_store, top_k=2),
|
||||
),
|
||||
)
|
||||
pipeline.connect("query_expander.queries", "retriever.queries")
|
||||
|
||||
result = pipeline.run({"query_expander": {"query": "sustainable power"}})
|
||||
|
||||
for doc in result["retriever"]["documents"]:
|
||||
print(f"Score: {doc.score:.3f} | {doc.content}")
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="yaml" label="YAML">
|
||||
|
||||
```yaml
|
||||
components:
|
||||
query_expander:
|
||||
type: haystack.components.query.query_expander.QueryExpander
|
||||
init_parameters:
|
||||
n_expansions: 3
|
||||
retriever:
|
||||
type: haystack.components.retrievers.multi_query_text_retriever.MultiQueryTextRetriever
|
||||
init_parameters:
|
||||
retriever:
|
||||
type: haystack.components.retrievers.in_memory.bm25_retriever.InMemoryBM25Retriever
|
||||
init_parameters:
|
||||
document_store:
|
||||
type: haystack.document_stores.in_memory.document_store.InMemoryDocumentStore
|
||||
init_parameters: {}
|
||||
top_k: 2
|
||||
|
||||
connections:
|
||||
- sender: query_expander.queries
|
||||
receiver: retriever.queries
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
### In a RAG pipeline
|
||||
|
||||
This RAG pipeline answers questions using query expansion. When a user asks "What types of energy come from natural sources?", the pipeline:
|
||||
|
||||
1. Expands the question into multiple search queries using an LLM
|
||||
2. Retrieves relevant documents for each query variation
|
||||
3. Builds a prompt containing the retrieved documents and the original question
|
||||
4. Sends the prompt to an LLM to generate an answer
|
||||
|
||||
The question is sent to both the `query_expander` (for generating search queries) and the `prompt_builder` (for the final prompt to the LLM).
|
||||
|
||||
<Tabs>
|
||||
<TabItem value="python" label="Python" default>
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.query import QueryExpander
|
||||
from haystack.components.retrievers import (
|
||||
InMemoryBM25Retriever,
|
||||
MultiQueryTextRetriever,
|
||||
)
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
documents = [
|
||||
Document(
|
||||
content="Renewable energy is energy that is collected from renewable resources.",
|
||||
),
|
||||
Document(
|
||||
content="Solar energy is a type of green energy that is harnessed from the sun.",
|
||||
),
|
||||
Document(
|
||||
content="Wind energy is another type of green energy that is generated by wind turbines.",
|
||||
),
|
||||
]
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
document_store.write_documents(documents)
|
||||
|
||||
prompt_template = [
|
||||
ChatMessage.from_system(
|
||||
"You are a helpful assistant that answers questions based on the provided documents.",
|
||||
),
|
||||
ChatMessage.from_user(
|
||||
"Given these documents, answer the question.\n"
|
||||
"Documents:\n"
|
||||
"{% for doc in documents %}"
|
||||
"{{ doc.content }}\n"
|
||||
"{% endfor %}\n"
|
||||
"Question: {{ question }}",
|
||||
),
|
||||
]
|
||||
|
||||
# Note: This assumes OPENAI_API_KEY environment variable is set
|
||||
rag_pipeline = Pipeline()
|
||||
rag_pipeline.add_component("query_expander", QueryExpander(n_expansions=2))
|
||||
rag_pipeline.add_component(
|
||||
"retriever",
|
||||
MultiQueryTextRetriever(
|
||||
retriever=InMemoryBM25Retriever(document_store=document_store, top_k=2),
|
||||
),
|
||||
)
|
||||
rag_pipeline.add_component(
|
||||
"prompt_builder",
|
||||
ChatPromptBuilder(
|
||||
template=prompt_template,
|
||||
required_variables=["documents", "question"],
|
||||
),
|
||||
)
|
||||
rag_pipeline.add_component("llm", OpenAIChatGenerator())
|
||||
|
||||
rag_pipeline.connect("query_expander.queries", "retriever.queries")
|
||||
rag_pipeline.connect("retriever.documents", "prompt_builder.documents")
|
||||
rag_pipeline.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
question = "What types of energy come from natural sources?"
|
||||
result = rag_pipeline.run(
|
||||
{"query_expander": {"query": question}, "prompt_builder": {"question": question}},
|
||||
)
|
||||
|
||||
print(result["llm"]["replies"][0].text)
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="yaml" label="YAML">
|
||||
|
||||
```yaml
|
||||
components:
|
||||
query_expander:
|
||||
type: haystack.components.query.query_expander.QueryExpander
|
||||
init_parameters:
|
||||
n_expansions: 2
|
||||
retriever:
|
||||
type: haystack.components.retrievers.multi_query_text_retriever.MultiQueryTextRetriever
|
||||
init_parameters:
|
||||
retriever:
|
||||
type: haystack.components.retrievers.in_memory.bm25_retriever.InMemoryBM25Retriever
|
||||
init_parameters:
|
||||
document_store:
|
||||
type: haystack.document_stores.in_memory.document_store.InMemoryDocumentStore
|
||||
init_parameters: {}
|
||||
top_k: 2
|
||||
prompt_builder:
|
||||
type: haystack.components.builders.chat_prompt_builder.ChatPromptBuilder
|
||||
init_parameters:
|
||||
required_variables:
|
||||
- documents
|
||||
- question
|
||||
llm:
|
||||
type: haystack.components.generators.chat.openai.OpenAIChatGenerator
|
||||
init_parameters: {}
|
||||
|
||||
connections:
|
||||
- sender: query_expander.queries
|
||||
receiver: retriever.queries
|
||||
- sender: retriever.documents
|
||||
receiver: prompt_builder.documents
|
||||
- sender: prompt_builder.prompt
|
||||
receiver: llm.messages
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
@@ -0,0 +1,213 @@
|
||||
---
|
||||
title: "MultiRetriever"
|
||||
id: multiretriever
|
||||
slug: "/multiretriever"
|
||||
description: "Runs multiple text retrievers in parallel and combines their results using reciprocal rank fusion or deduplication."
|
||||
---
|
||||
|
||||
# MultiRetriever
|
||||
|
||||
Runs multiple text retrievers in parallel and combines their results using reciprocal rank fusion or deduplication.
|
||||
|
||||
:::warning[Experimental]
|
||||
|
||||
`MultiRetriever` is experimental and may change or be removed in future releases without prior deprecation notice. An `ExperimentalWarning` is printed when initializing this component.
|
||||
|
||||
:::
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After query input, before a [`ChatPromptBuilder`](../builders/chatpromptbuilder.mdx) in RAG pipelines |
|
||||
| **Mandatory init variables** | `retrievers`: A dictionary mapping names to text retrievers (implementing the `TextRetriever` protocol) |
|
||||
| **Optional init variables** | `join_mode`: `"reciprocal_rank_fusion"` (default) or `"concatenate"` |
|
||||
| **Mandatory run variables** | `query`: A query string |
|
||||
| **Output variables** | `documents`: A merged list of retrieved documents |
|
||||
| **API reference** | [Retrievers](/reference/retrievers-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/retrievers/multi_retriever.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`MultiRetriever` composes any number of text retrievers into a single component. All retrievers are queried in parallel using a thread pool, and their results are merged before being returned.
|
||||
|
||||
The component:
|
||||
- Queries all retrievers concurrently for better performance
|
||||
- Merges results across retrievers using the configured `join_mode`
|
||||
- Supports selectively enabling retrievers at runtime via `active_retrievers`
|
||||
|
||||
All retrievers passed to `MultiRetriever` must implement the `TextRetriever` protocol — their `run` method must accept a text `query`, `filters`, and `top_k`. Use [`TextEmbeddingRetriever`](textembeddingretriever.mdx) to wrap an embedding-based retriever so it can be used with this component.
|
||||
|
||||
### Join modes
|
||||
|
||||
The `join_mode` parameter controls how results from multiple retrievers are merged:
|
||||
|
||||
- **`reciprocal_rank_fusion`** (default): Assigns scores based on each document's rank across retrieval lists using the [Reciprocal Rank Fusion](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf) algorithm. Documents appearing highly ranked in multiple lists receive higher scores. Results are deduplicated and returned in descending score order. This is the recommended mode when combining retrievers with incomparable scores, such as BM25 and embedding retrievers.
|
||||
- **`concatenate`**: Combines all results into a single list and deduplicates.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
This example sets up a `MultiRetriever` combining a BM25 retriever and an embedding-based retriever (wrapped with `TextEmbeddingRetriever`). Both are queried in parallel and the results are merged using reciprocal rank fusion.
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
SentenceTransformersTextEmbedder,
|
||||
)
|
||||
from haystack.components.retrievers import (
|
||||
InMemoryBM25Retriever,
|
||||
InMemoryEmbeddingRetriever,
|
||||
)
|
||||
from haystack.components.retrievers import MultiRetriever, TextEmbeddingRetriever
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
documents = [
|
||||
Document(
|
||||
content="Renewable energy is energy that is collected from renewable resources.",
|
||||
),
|
||||
Document(
|
||||
content="Solar energy is a type of green energy that is harnessed from the sun.",
|
||||
),
|
||||
Document(
|
||||
content="Wind energy is another type of green energy that is generated by wind turbines.",
|
||||
),
|
||||
]
|
||||
|
||||
doc_store = InMemoryDocumentStore()
|
||||
doc_embedder = SentenceTransformersDocumentEmbedder(
|
||||
model="sentence-transformers/all-MiniLM-L6-v2",
|
||||
)
|
||||
doc_writer = DocumentWriter(document_store=doc_store, policy=DuplicatePolicy.SKIP)
|
||||
doc_writer.run(documents=doc_embedder.run(documents)["documents"])
|
||||
|
||||
retriever = MultiRetriever(
|
||||
retrievers={
|
||||
"bm25": InMemoryBM25Retriever(document_store=doc_store),
|
||||
"embedding": TextEmbeddingRetriever(
|
||||
retriever=InMemoryEmbeddingRetriever(document_store=doc_store),
|
||||
text_embedder=SentenceTransformersTextEmbedder(
|
||||
model="sentence-transformers/all-MiniLM-L6-v2",
|
||||
),
|
||||
),
|
||||
},
|
||||
top_k=3,
|
||||
)
|
||||
|
||||
result = retriever.run(query="green energy sources")
|
||||
for doc in result["documents"]:
|
||||
print(doc.content)
|
||||
```
|
||||
|
||||
### Selecting retrievers at runtime
|
||||
|
||||
Use the `active_retrievers` parameter to run only a subset of retrievers. Names must match the keys in the `retrievers` dictionary. Building on the example above:
|
||||
|
||||
```python
|
||||
# Run only the BM25 retriever
|
||||
result = retriever.run(query="green energy sources", active_retrievers=["bm25"])
|
||||
for doc in result["documents"]:
|
||||
print(doc.content)
|
||||
```
|
||||
|
||||
### In a RAG pipeline
|
||||
|
||||
This RAG pipeline uses `MultiRetriever` to combine BM25 and embedding retrieval before generating an answer with an LLM.
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
SentenceTransformersTextEmbedder,
|
||||
)
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.retrievers import (
|
||||
InMemoryBM25Retriever,
|
||||
InMemoryEmbeddingRetriever,
|
||||
)
|
||||
from haystack.components.retrievers import MultiRetriever, TextEmbeddingRetriever
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
documents = [
|
||||
Document(
|
||||
content="Renewable energy is energy that is collected from renewable resources.",
|
||||
),
|
||||
Document(
|
||||
content="Solar energy is a type of green energy that is harnessed from the sun.",
|
||||
),
|
||||
Document(
|
||||
content="Wind energy is another type of green energy that is generated by wind turbines.",
|
||||
),
|
||||
]
|
||||
|
||||
doc_store = InMemoryDocumentStore()
|
||||
doc_embedder = SentenceTransformersDocumentEmbedder(
|
||||
model="sentence-transformers/all-MiniLM-L6-v2",
|
||||
)
|
||||
doc_writer = DocumentWriter(document_store=doc_store, policy=DuplicatePolicy.SKIP)
|
||||
doc_writer.run(documents=doc_embedder.run(documents)["documents"])
|
||||
|
||||
prompt_template = [
|
||||
ChatMessage.from_system(
|
||||
"You are a helpful assistant that answers questions based on the provided documents.",
|
||||
),
|
||||
ChatMessage.from_user(
|
||||
"Given these documents, answer the question.\nDocuments:\n"
|
||||
"{% for doc in documents %}{{ doc.content }}\n{% endfor %}\n"
|
||||
"Question: {{ question }}",
|
||||
),
|
||||
]
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component(
|
||||
"retriever",
|
||||
MultiRetriever(
|
||||
retrievers={
|
||||
"bm25": InMemoryBM25Retriever(document_store=doc_store),
|
||||
"embedding": TextEmbeddingRetriever(
|
||||
retriever=InMemoryEmbeddingRetriever(document_store=doc_store),
|
||||
text_embedder=SentenceTransformersTextEmbedder(
|
||||
model="sentence-transformers/all-MiniLM-L6-v2",
|
||||
),
|
||||
),
|
||||
},
|
||||
top_k=3,
|
||||
),
|
||||
)
|
||||
pipeline.add_component(
|
||||
"prompt_builder",
|
||||
ChatPromptBuilder(
|
||||
template=prompt_template,
|
||||
required_variables=["documents", "question"],
|
||||
),
|
||||
)
|
||||
pipeline.add_component("llm", OpenAIChatGenerator())
|
||||
|
||||
pipeline.connect("retriever.documents", "prompt_builder.documents")
|
||||
pipeline.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
result = pipeline.run(
|
||||
{
|
||||
"retriever": {"query": "green energy sources"},
|
||||
"prompt_builder": {"question": "What types of green energy exist?"},
|
||||
},
|
||||
)
|
||||
print(result["llm"]["replies"][0].text)
|
||||
```
|
||||
@@ -0,0 +1,162 @@
|
||||
---
|
||||
title: "OpenSearchBM25Retriever"
|
||||
id: opensearchbm25retriever
|
||||
slug: "/opensearchbm25retriever"
|
||||
description: "This is a keyword-based Retriever that fetches Documents matching a query from an OpenSearch Document Store."
|
||||
---
|
||||
|
||||
# OpenSearchBM25Retriever
|
||||
|
||||
This is a keyword-based Retriever that fetches Documents matching a query from an OpenSearch Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. Before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the semantic search pipeline 3. Before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of an [OpenSearchDocumentStore](../../document-stores/opensearch-document-store.mdx) |
|
||||
| **Mandatory run variables** | `query`: A query string |
|
||||
| **Output variables** | `documents`: A list of documents matching the query |
|
||||
| **API reference** | [OpenSearch](/reference/integrations-opensearch) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opensearch |
|
||||
| **Package name** | `opensearch-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`OpenSearchBM25Retriever` is a keyword-based Retriever that fetches Documents matching a query from an `OpenSearchDocumentStore`. It determines the similarity between Documents and the query based on the BM25 algorithm, which computes a weighted word overlap between the two strings.
|
||||
|
||||
Since the `OpenSearchBM25Retriever` matches strings based on word overlap, it’s often used to find exact matches to names of persons or products, IDs, or well-defined error messages. The BM25 algorithm is very lightweight and simple. Nevertheless, it can be hard to beat with more complex embedding-based approaches on out-of-domain data.
|
||||
|
||||
In addition to the `query`, the `OpenSearchBM25Retriever` accepts other optional parameters, including `top_k` (the maximum number of Documents to retrieve) and `filters` to narrow down the search space.
|
||||
You can adjust how [inexact fuzzy matching](https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html#fuzziness) is performed, using the `fuzziness` parameter.
|
||||
It is also possible to specify if all terms in the query must match using the `all_terms_must_match` parameter, which defaults to `False`.
|
||||
|
||||
If you want more flexible matching of a query to Documents, you can use the `OpenSearchEmbeddingRetriever`, which uses vectors created by LLMs to retrieve relevant information.
|
||||
|
||||
### Setup and installation
|
||||
|
||||
[Install](https://opensearch.org/docs/latest/install-and-configure/install-opensearch/index/) and run an OpenSearch instance.
|
||||
|
||||
If you have Docker set up, we recommend pulling the Docker image and running it.
|
||||
|
||||
```shell
|
||||
docker pull opensearchproject/opensearch:2.11.0
|
||||
docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -e "ES_JAVA_OPTS=-Xms1024m -Xmx1024m" opensearchproject/opensearch:2.11.0
|
||||
```
|
||||
|
||||
As an alternative, you can go to [OpenSearch integration GitHub](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opensearch) and start a Docker container running OpenSearch using the provided `docker-compose.yml`:
|
||||
|
||||
```shell
|
||||
docker compose up
|
||||
```
|
||||
|
||||
Once you have a running OpenSearch instance, install the `opensearch-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install opensearch-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
This Retriever needs the `OpensearchDocumentStore` and indexed Documents to run. You can’t use it on its own.
|
||||
|
||||
### In a RAG pipeline
|
||||
|
||||
Set your `OPENAI_API_KEY` as an environment variable and then run the following code:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.retrievers.opensearch import (
|
||||
OpenSearchBM25Retriever,
|
||||
)
|
||||
from haystack_integrations.document_stores.opensearch import OpenSearchDocumentStore
|
||||
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders.answer_builder import AnswerBuilder
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
|
||||
# OpenAIChatGenerator reads the OPENAI_API_KEY environment variable by default.
|
||||
|
||||
# Create a RAG query pipeline
|
||||
prompt_template = [
|
||||
ChatMessage.from_user(
|
||||
"""
|
||||
Given these documents, answer the question.\nDocuments:
|
||||
{% for doc in documents %}
|
||||
{{ doc.content }}
|
||||
{% endfor %}
|
||||
|
||||
\nQuestion: {{question}}
|
||||
\nAnswer:
|
||||
""",
|
||||
),
|
||||
]
|
||||
|
||||
document_store = OpenSearchDocumentStore(
|
||||
hosts="http://localhost:9200",
|
||||
use_ssl=True,
|
||||
verify_certs=False,
|
||||
http_auth=("admin", "admin"),
|
||||
)
|
||||
|
||||
# Add Documents
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
# DuplicatePolicy.SKIP param is optional, but useful to run the script multiple times without throwing errors
|
||||
document_store.write_documents(documents=documents, policy=DuplicatePolicy.SKIP)
|
||||
|
||||
retriever = OpenSearchBM25Retriever(document_store=document_store)
|
||||
rag_pipeline = Pipeline()
|
||||
rag_pipeline.add_component(name="retriever", instance=retriever)
|
||||
rag_pipeline.add_component(
|
||||
instance=ChatPromptBuilder(template=prompt_template, required_variables="*"),
|
||||
name="prompt_builder",
|
||||
)
|
||||
rag_pipeline.add_component(instance=OpenAIChatGenerator(), name="llm")
|
||||
rag_pipeline.add_component(instance=AnswerBuilder(), name="answer_builder")
|
||||
rag_pipeline.connect("retriever", "prompt_builder.documents")
|
||||
rag_pipeline.connect("prompt_builder.prompt", "llm.messages")
|
||||
rag_pipeline.connect("llm.replies", "answer_builder.replies")
|
||||
rag_pipeline.connect("retriever", "answer_builder.documents")
|
||||
|
||||
question = "How many languages are spoken around the world today?"
|
||||
result = rag_pipeline.run(
|
||||
{
|
||||
"retriever": {"query": question},
|
||||
"prompt_builder": {"question": question},
|
||||
"answer_builder": {"query": question},
|
||||
},
|
||||
)
|
||||
print(result["answer_builder"]["answers"][0])
|
||||
```
|
||||
|
||||
Here’s an example output:
|
||||
|
||||
```python
|
||||
# GeneratedAnswer(
|
||||
# data='Over 7,000 languages are spoken around the world today.',
|
||||
# query='How many languages are spoken around the world today?',
|
||||
# documents=[
|
||||
# Document(id=cfe93bc1c274908801e6670440bf2bbba54fad792770d57421f85ffa2a4fcc94, content: 'There are over 7,000 languages spoken around the world today.', score: 7.179112),
|
||||
# Document(id=7f225626ad1019b273326fbaf11308edfca6d663308a4a3533ec7787367d59a2, content: 'In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the ph...', score: 1.1426818)],
|
||||
# meta={'model': 'gpt-5-mini', 'index': 0, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 86, 'completion_tokens': 13, 'total_tokens': 99}})
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [PDF-Based Question Answering with Amazon Bedrock and Haystack](https://haystack.deepset.ai/cookbook/amazon_bedrock_for_documentation_qa)
|
||||
@@ -0,0 +1,138 @@
|
||||
---
|
||||
title: "OpenSearchEmbeddingRetriever"
|
||||
id: opensearchembeddingretriever
|
||||
slug: "/opensearchembeddingretriever"
|
||||
description: "An embedding-based Retriever compatible with the OpenSearch Document Store."
|
||||
---
|
||||
|
||||
# OpenSearchEmbeddingRetriever
|
||||
|
||||
An embedding-based Retriever compatible with the OpenSearch Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the semantic search pipeline 3. After a Text Embedder and before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of an [OpenSearchDocumentStore](../../document-stores/opensearch-document-store.mdx) |
|
||||
| **Mandatory run variables** | `query_embedding`: A list of floats |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [OpenSearch](/reference/integrations-opensearch) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opensearch |
|
||||
| **Package name** | `opensearch-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `OpenSearchEmbeddingRetriever` is an embedding-based Retriever compatible with the `OpenSearchDocumentStore`. It compares the query and Document embeddings and fetches the Documents most relevant to the query from the `OpenSearchDocumentStore` based on the outcome.
|
||||
|
||||
When using the `OpenSearchEmbeddingRetriever` in your NLP system, make sure it has the query and Document embeddings available. You can do so by adding a Document Embedder to your indexing pipeline and a Text Embedder to your query pipeline.
|
||||
|
||||
In addition to the `query_embedding`, the `OpenSearchEmbeddingRetriever` accepts other optional parameters, including `top_k` (the maximum number of Documents to retrieve) and `filters` to narrow down the search space.
|
||||
|
||||
The `embedding_dim` for storing and retrieving embeddings must be defined when the corresponding `OpenSearchDocumentStore` is initialized.
|
||||
|
||||
### Setup and installation
|
||||
|
||||
[Install](https://opensearch.org/docs/latest/install-and-configure/install-opensearch/index/) and run an OpenSearch instance.
|
||||
|
||||
If you have Docker set up, we recommend pulling the Docker image and running it.
|
||||
|
||||
```shell
|
||||
docker pull opensearchproject/opensearch:2.11.0
|
||||
docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -e "ES_JAVA_OPTS=-Xms1024m -Xmx1024m" opensearchproject/opensearch:2.11.0
|
||||
```
|
||||
|
||||
As an alternative, you can go to [OpenSearch integration GitHub](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opensearch) and start a Docker container running OpenSearch using the provided `docker-compose.yml`:
|
||||
|
||||
```shell
|
||||
docker compose up
|
||||
```
|
||||
|
||||
Once you have a running OpenSearch instance, install the `opensearch-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install opensearch-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Use this Retriever in a query Pipeline like this:
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.retrievers.opensearch import (
|
||||
OpenSearchEmbeddingRetriever,
|
||||
)
|
||||
from haystack_integrations.document_stores.opensearch import OpenSearchDocumentStore
|
||||
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersTextEmbedder,
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
)
|
||||
|
||||
document_store = OpenSearchDocumentStore(
|
||||
hosts="http://localhost:9200",
|
||||
use_ssl=True,
|
||||
verify_certs=False,
|
||||
http_auth=("admin", "admin"),
|
||||
)
|
||||
|
||||
model = "sentence-transformers/all-mpnet-base-v2"
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
document_embedder = SentenceTransformersDocumentEmbedder(model=model)
|
||||
documents_with_embeddings = document_embedder.run(documents)
|
||||
|
||||
document_store.write_documents(
|
||||
documents_with_embeddings.get("documents"),
|
||||
policy=DuplicatePolicy.SKIP,
|
||||
)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component(
|
||||
"text_embedder",
|
||||
SentenceTransformersTextEmbedder(model=model),
|
||||
)
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
OpenSearchEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "How many languages are there?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
```
|
||||
|
||||
The example output would be:
|
||||
|
||||
```python
|
||||
Document(id=cfe93bc1c274908801e6670440bf2bbba54fad792770d57421f85ffa2a4fcc94, content: 'There are over 7,000 languages spoken around the world today.', score: 0.70026743, embedding: vector of size 768)
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [PDF-Based Question Answering with Amazon Bedrock and Haystack](https://haystack.deepset.ai/cookbook/amazon_bedrock_for_documentation_qa)
|
||||
@@ -0,0 +1,139 @@
|
||||
---
|
||||
title: "OpenSearchHybridRetriever"
|
||||
id: opensearchhybridretriever
|
||||
slug: "/opensearchhybridretriever"
|
||||
description: "This is a [SuperComponent](../../concepts/components/supercomponents.mdx) that implements a Hybrid Retriever in a single component, relying on OpenSearch as the backend Document Store."
|
||||
---
|
||||
|
||||
# OpenSearchHybridRetriever
|
||||
|
||||
This is a [SuperComponent](../../concepts/components/supercomponents.mdx) that implements a Hybrid Retriever in a single component, relying on OpenSearch as the backend Document Store.
|
||||
|
||||
A Hybrid Retriever uses both traditional keyword-based search (such as BM25) and embedding-based search to retrieve documents, combining the strengths of both approaches. The Retriever then merges and re-ranks the results from both methods.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| Most common position in a pipeline | 1. After a TextEmbedder and before a PromptBuilder in a RAG pipeline 2. The last component in a hybrid search pipeline 3. After a TextEmbedder and before a TransformersExtractiveReader in an extractive QA pipeline |
|
||||
| Mandatory init variables | `document_store`: An instance of `OpenSearchDocumentStore` to use for retrieval <br /> <br />`embedder`: Any [Embedder](../embedders.mdx) implementing the `TextEmbedder` protocol |
|
||||
| Mandatory run variables | `query`: A query string |
|
||||
| Output variables | `documents`: A list of documents matching the query |
|
||||
| API reference | [OpenSearch](/reference/integrations-opensearch) |
|
||||
| GitHub | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opensearch |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `OpenSearchHybridRetriever` combines two retrieval methods:
|
||||
|
||||
1. **BM25 Retrieval**: A keyword-based search that uses the BM25 algorithm to find documents based on term frequency and inverse document frequency. It's based on the [`OpenSearchBM25Retriever`](opensearchbm25retriever.mdx) component and is suitable for traditional keyword-based search.
|
||||
2. **Embedding-based Retrieval**: A semantic search that uses vector similarity to find documents that are semantically similar to the query. It's based on the [`OpenSearchEmbeddingRetriever`](opensearchembeddingretriever.mdx) component and is suitable for semantic search.
|
||||
|
||||
The component automatically handles:
|
||||
|
||||
- Converting the query into an embedding using the provided embedded,
|
||||
- Running both retrieval methods in parallel,
|
||||
- Merging and re-ranking the results using the specified join mode.
|
||||
|
||||
### Setup and Installation
|
||||
|
||||
```shell
|
||||
pip install opensearch-haystack
|
||||
```
|
||||
|
||||
### Optional Parameters
|
||||
|
||||
This Retriever accepts various optional parameters. You can verify the most up-to-date list of parameters in our [API Reference](/reference/integrations-opensearch#opensearchhybridretriever).
|
||||
|
||||
You can pass additional parameters to the underlying components using the `bm25_retriever` and `embedding_retriever` dictionaries.
|
||||
The `DocumentJoiner` parameters are all exposed on the `OpenSearchHybridRetriever` class, so you can set them directly.
|
||||
|
||||
Here's an example:
|
||||
|
||||
```python
|
||||
retriever = OpenSearchHybridRetriever(
|
||||
document_store=document_store,
|
||||
embedder=embedder,
|
||||
bm25_retriever={"raise_on_failure": True},
|
||||
embedding_retriever={"raise_on_failure": False},
|
||||
)
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
This Retriever needs the `OpensearchDocumentStore` populated with documents to run. You can’t use it on its own.
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Here's a basic example of how to use the `OpenSearchHybridRetriever`:
|
||||
|
||||
You can use the following command to run OpenSearch locally using Docker. Make sure you have Docker installed and running on your machine. Note that this example disables the security plugin for simplicity. In a production environment, you should enable security features.
|
||||
|
||||
```dockerfile
|
||||
docker run -d \\
|
||||
--name opensearch-nosec \\
|
||||
-p 9200:9200 \\
|
||||
-p 9600:9600 \\
|
||||
-e "discovery.type=single-node" \\
|
||||
-e "DISABLE_SECURITY_PLUGIN=true" \\
|
||||
opensearchproject/opensearch:2.12.0
|
||||
```
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.embedders.sentence_transformers import SentenceTransformersTextEmbedder, SentenceTransformersDocumentEmbedder
|
||||
from haystack_integrations.components.retrievers.opensearch import OpenSearchHybridRetriever
|
||||
from haystack_integrations.document_stores.opensearch import OpenSearchDocumentStore
|
||||
|
||||
# Initialize the document store
|
||||
doc_store = OpenSearchDocumentStore(
|
||||
hosts=["http://localhost:9200"],
|
||||
index="document_store",
|
||||
embedding_dim=384,
|
||||
)
|
||||
|
||||
# Create some sample documents
|
||||
docs = [
|
||||
Document(content="Machine learning is a subset of artificial intelligence."),
|
||||
Document(content="Deep learning is a subset of machine learning."),
|
||||
Document(content="Natural language processing is a field of AI."),
|
||||
Document(content="Reinforcement learning is a type of machine learning."),
|
||||
Document(content="Supervised learning is a type of machine learning."),
|
||||
]
|
||||
|
||||
# Embed the documents and add them to the document store
|
||||
doc_embedder = SentenceTransformersDocumentEmbedder(model="sentence-transformers/all-MiniLM-L6-v2")
|
||||
docs = doc_embedder.run(docs)
|
||||
doc_store.write_documents(docs['documents'])
|
||||
|
||||
# Initialize some haystack text embedder, in this case the SentenceTransformersTextEmbedder
|
||||
embedder = SentenceTransformersTextEmbedder(model="sentence-transformers/all-MiniLM-L6-v2")
|
||||
|
||||
# Initialize the hybrid retriever
|
||||
retriever = OpenSearchHybridRetriever(
|
||||
document_store=doc_store,
|
||||
embedder=embedder,
|
||||
top_k_bm25=3,
|
||||
top_k_embedding=3,
|
||||
join_mode="reciprocal_rank_fusion"
|
||||
)
|
||||
|
||||
# Run the retriever
|
||||
results = retriever.run(query="What is reinforcement learning?", filters_bm25=None, filters_embedding=None)
|
||||
|
||||
>> results['documents']
|
||||
{'documents': [Document(id=..., content: 'Reinforcement learning is a type of machine learning.', score: 1.0),
|
||||
Document(id=..., content: 'Supervised learning is a type of machine learning.', score: 0.9760624679979518),
|
||||
Document(id=..., content: 'Deep learning is a subset of machine learning.', score: 0.4919354838709677),
|
||||
Document(id=..., content: 'Machine learning is a subset of artificial intelligence.', score: 0.4841269841269841)]}
|
||||
```
|
||||
@@ -0,0 +1,184 @@
|
||||
---
|
||||
title: OpenSearchMetadataRetriever
|
||||
id: opensearchmetadataretriever
|
||||
slug: /opensearchmetadataretriever
|
||||
description: Searches and ranks the metadata fields of documents stored in an OpenSearch Document Store and returns the matching metadata values.
|
||||
---
|
||||
|
||||
# OpenSearchMetadataRetriever
|
||||
|
||||
Searches and ranks the metadata fields of documents stored in an OpenSearch Document Store and returns the matching metadata values.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Most common position in a pipeline** | The last component in a metadata lookup pipeline, or wherever you need other structured data from an OpenSearchDocumentStore index |
|
||||
| **Mandatory init variables** | `document_store`: An instance of `OpenSearchDocumentStore`; `metadata_fields`: List of metadata field names to search and return |
|
||||
| **Mandatory run variables** | `query`: A search query string (may contain comma-separated parts) |
|
||||
| **Output variables** | `metadata`: A list of dictionaries containing only the requested metadata fields |
|
||||
| **API reference** | [OpenSearch](/reference/integrations-opensearch) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opensearch |
|
||||
| **Package name** | `opensearch-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`OpenSearchMetadataRetriever` searches the metadata of documents stored in an `OpenSearchDocumentStore` and returns the matching metadata values, not the documents themselves. It is useful when the metadata is the answer: for example, listing the categories or tags that match a partial query, building a metadata autocomplete, or surfacing the structured side of an index without pulling back document content.
|
||||
|
||||
Unlike the other OpenSearch retrievers (`OpenSearchBM25Retriever`, `OpenSearchEmbeddingRetriever`, `OpenSearchHybridRetriever`), this component does not return `Document` objects. The output is a list under `metadata`, where each entry is a dictionary containing only the fields you listed in `metadata_fields`. Document content and any other metadata are excluded from the result.
|
||||
|
||||
The retriever supports two search modes:
|
||||
|
||||
- `strict` uses prefix and wildcard matching on the configured metadata fields.
|
||||
- `fuzzy` (the default) uses fuzzy matching with `dis_max` queries, allowing typos and partial matches.
|
||||
|
||||
In both modes, candidate documents are scored server-side with Jaccard similarity on character n-grams (the `jaccard_n` parameter controls the n-gram size), and exact matches receive an additional boost controlled by `exact_match_weight`. Up to 1000 hits are fetched from OpenSearch, and the top `top_k` results are returned.
|
||||
|
||||
Both a synchronous `run` method and an asynchronous `run_async` method are available with the same parameters.
|
||||
|
||||
### Field types
|
||||
|
||||
The matching engine only operates on metadata fields that OpenSearch indexes as text or keyword values. Numeric, boolean, and array-of-non-strings fields are not valid search targets, because prefix, wildcard, and full-text matching do not apply to them. Mixed-type fields, such as a list that combines strings and numbers, are also not supported.
|
||||
|
||||
|
||||
## Installation
|
||||
|
||||
If you have Docker set up, the easiest way to run OpenSearch is to pull and run the Docker image.
|
||||
|
||||
```bash
|
||||
docker pull opensearchproject/opensearch:2
|
||||
docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -e "OPENSEARCH_INITIAL_ADMIN_PASSWORD=<custom-admin-password>" opensearchproject/opensearch:2
|
||||
```
|
||||
|
||||
As an alternative, you can go to the [OpenSearch integration GitHub](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opensearch) and start a Docker container using the provided `docker-compose.yml`:
|
||||
|
||||
```bash
|
||||
docker compose up
|
||||
```
|
||||
|
||||
Once you have a running OpenSearch instance, install the `opensearch-haystack` integration:
|
||||
|
||||
```bash
|
||||
pip install opensearch-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
This Retriever needs an `OpenSearchDocumentStore` with indexed documents. The example below writes three documents with simple categorical metadata and queries the `category` and `status` fields:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.retrievers.opensearch import (
|
||||
OpenSearchMetadataRetriever,
|
||||
)
|
||||
from haystack_integrations.document_stores.opensearch import OpenSearchDocumentStore
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
|
||||
document_store = OpenSearchDocumentStore(
|
||||
hosts="http://localhost:9200",
|
||||
index="my_index",
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(
|
||||
content="Python programming guide",
|
||||
meta={
|
||||
"category": "Python",
|
||||
"status": "active",
|
||||
"priority": 1,
|
||||
"author": "John Doe",
|
||||
},
|
||||
),
|
||||
Document(
|
||||
content="Java tutorial",
|
||||
meta={
|
||||
"category": "Java",
|
||||
"status": "active",
|
||||
"priority": 2,
|
||||
"author": "Jane Smith",
|
||||
},
|
||||
),
|
||||
Document(
|
||||
content="Python advanced topics",
|
||||
meta={
|
||||
"category": "Python",
|
||||
"status": "inactive",
|
||||
"priority": 3,
|
||||
"author": "John Doe",
|
||||
},
|
||||
),
|
||||
]
|
||||
|
||||
document_store.write_documents(documents=documents, policy=DuplicatePolicy.SKIP)
|
||||
|
||||
retriever = OpenSearchMetadataRetriever(
|
||||
document_store=document_store,
|
||||
metadata_fields=["category", "status"],
|
||||
top_k=10,
|
||||
)
|
||||
|
||||
result = retriever.run(query="Python")
|
||||
|
||||
print(result)
|
||||
# {
|
||||
# "metadata": [
|
||||
# {"category": "Python", "status": "active"},
|
||||
# {"category": "Python", "status": "inactive"},
|
||||
# ]
|
||||
# }
|
||||
```
|
||||
|
||||
Only the fields listed in `metadata_fields` appear in each result dictionary. The `author` metadata and the document content are excluded.
|
||||
|
||||
### Multi-part queries
|
||||
|
||||
The `query` string can contain several comma-separated parts. Each part is searched across every field listed in `metadata_fields`, and a document that matches multiple parts is ranked higher (controlled by `exact_match_weight`).
|
||||
|
||||
```python
|
||||
result = retriever.run(query="Python, active")
|
||||
# Returns the metadata of documents whose fields match both "Python" and "active".
|
||||
```
|
||||
|
||||
### Strict mode
|
||||
|
||||
By default the retriever runs in `fuzzy` mode, which tolerates typos and partial matches. For lookups where you only want prefix or wildcard matches and no edit-distance tolerance, switch to `strict`:
|
||||
|
||||
```python
|
||||
retriever = OpenSearchMetadataRetriever(
|
||||
document_store=document_store,
|
||||
metadata_fields=["category"],
|
||||
mode="strict",
|
||||
)
|
||||
|
||||
result = retriever.run(query="Pyth")
|
||||
# Matches "Python" through prefix matching, but not transposed-letter variants.
|
||||
```
|
||||
|
||||
The fuzzy-mode parameters (`fuzziness`, `prefix_length`, `max_expansions`, `tie_breaker`) only take effect when `mode="fuzzy"`.
|
||||
|
||||
### Combining with filters
|
||||
|
||||
You can narrow the candidate set before scoring by passing standard Haystack `filters` at run time. The filters are applied in a `bool` `filter` context, so they exclude non-matching documents without affecting scores:
|
||||
|
||||
```python
|
||||
result = retriever.run(
|
||||
query="Python",
|
||||
filters={"field": "status", "operator": "==", "value": "active"},
|
||||
)
|
||||
```
|
||||
|
||||
### Asynchronous execution
|
||||
|
||||
For pipelines that mix synchronous and asynchronous components, the retriever exposes `run_async` with the same signature:
|
||||
|
||||
```python
|
||||
result = await retriever.run_async(query="Python, active")
|
||||
```
|
||||
|
||||
### Error handling
|
||||
|
||||
By default, a failed OpenSearch request raises an exception. To treat a failure as an empty result instead — for example, when the retriever sits behind a forgiving API — initialize the component with `raise_on_failure=False`. The error is then logged as a warning and `metadata` is returned as an empty list.
|
||||
@@ -0,0 +1,114 @@
|
||||
---
|
||||
title: OpenSearchSQLRetriever
|
||||
id: opensearchsqlretriever
|
||||
slug: /opensearchsqlretriever
|
||||
description: Executes raw OpenSearch SQL queries against an OpenSearch Document Store and returns the raw JSON response.
|
||||
---
|
||||
|
||||
# OpenSearchSQLRetriever
|
||||
|
||||
Executes raw OpenSearch SQL queries against an OpenSearch Document Store and returns the raw JSON response.
|
||||
|
||||
| | |
|
||||
| --------------------------------------- | ------------------------------------------------------------------------------------------------ |
|
||||
| **Most common position in a pipeline** | Standalone, or anywhere you need to fetch metadata, aggregations, or other structured data |
|
||||
| **Mandatory init variables** | `document_store`: An instance of `OpenSearchDocumentStore` |
|
||||
| **Mandatory run variables** | `query`: An OpenSearch SQL query string |
|
||||
| **Output variables** | `result`: A dictionary with the raw JSON response from the OpenSearch SQL API |
|
||||
| **API reference** | [OpenSearch](https://docs.haystack.deepset.ai/reference/integrations-opensearch) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opensearch |
|
||||
| **Package name** | `opensearch-haystack` |
|
||||
|
||||
## Overview
|
||||
|
||||
`OpenSearchSQLRetriever` lets you run [OpenSearch SQL](https://opensearch.org/docs/latest/search-plugins/sql/index/) queries directly against an `OpenSearchDocumentStore`. Instead of matching a query against documents like the `OpenSearchBM25Retriever` or `OpenSearchEmbeddingRetriever`, it executes a SQL statement and returns the **raw JSON response** from the OpenSearch SQL API.
|
||||
|
||||
This is useful when you need structured access to your index at runtime, for example to fetch specific fields, filter on metadata, or compute aggregations such as counts and averages.
|
||||
|
||||
Unlike the other OpenSearch retrievers, this component does not return a list of `Document` objects. The output is a single `result` dictionary, where `result["result"]` holds the raw OpenSearch response. Depending on the query:
|
||||
|
||||
- For regular queries, `result["result"]["hits"]["hits"]` contains the matching documents.
|
||||
- For aggregate queries, `result["result"]["aggregations"]` contains the aggregations.
|
||||
|
||||
The component accepts two optional parameters at initialization:
|
||||
|
||||
- `raise_on_failure`: if `True` (the default), an exception is raised when the SQL API call fails. If `False`, the error is logged as a warning and the result is empty.
|
||||
- `fetch_size`: the number of results to fetch per page. If not set, the default fetch size configured in OpenSearch is used.
|
||||
|
||||
## Installation
|
||||
|
||||
Install OpenSearch and then start an instance.
|
||||
|
||||
If you have Docker set up, we recommend pulling the Docker image and running it.
|
||||
|
||||
```bash
|
||||
docker pull opensearchproject/opensearch:2
|
||||
docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -e "OPENSEARCH_INITIAL_ADMIN_PASSWORD=<custom-admin-password>" opensearchproject/opensearch:2
|
||||
```
|
||||
|
||||
As an alternative, you can go to [OpenSearch integration GitHub](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opensearch) and start a Docker container running OpenSearch using the provided `docker-compose.yml`:
|
||||
|
||||
```bash
|
||||
docker compose up
|
||||
```
|
||||
|
||||
Once you have a running OpenSearch instance, install the `opensearch-haystack` integration:
|
||||
|
||||
```bash
|
||||
pip install opensearch-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Write a few documents to an index, then run a SQL query against it. The example below selects the `content` field from the index and reads the returned hits:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.retrievers.opensearch import (
|
||||
OpenSearchSQLRetriever,
|
||||
)
|
||||
from haystack_integrations.document_stores.opensearch import (
|
||||
OpenSearchDocumentStore,
|
||||
)
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
|
||||
document_store = OpenSearchDocumentStore(hosts="http://localhost:9200", index="my_index")
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
# DuplicatePolicy.SKIP is optional, but useful to run the script multiple times without throwing errors
|
||||
document_store.write_documents(documents=documents, policy=DuplicatePolicy.SKIP)
|
||||
|
||||
retriever = OpenSearchSQLRetriever(document_store=document_store)
|
||||
output = retriever.run(query="SELECT content FROM my_index LIMIT 10")
|
||||
|
||||
result = output["result"]
|
||||
# Regular queries return matching documents under hits
|
||||
for hit in result["hits"]["hits"]:
|
||||
print(hit["_source"])
|
||||
```
|
||||
|
||||
### Running an aggregation query
|
||||
|
||||
Because the component returns the raw SQL response, you can use it for aggregations that the document-based retrievers don't support, such as counting documents:
|
||||
|
||||
```python
|
||||
retriever = OpenSearchSQLRetriever(document_store=document_store)
|
||||
output = retriever.run(query="SELECT COUNT(*) AS doc_count FROM my_index")
|
||||
|
||||
result = output["result"]
|
||||
# Aggregate queries expose their results under aggregations
|
||||
print(result["aggregations"])
|
||||
```
|
||||
|
||||
To avoid raising an exception on a malformed or failing query, initialize the component with `raise_on_failure=False`. In that case, a failed query logs a warning and returns an empty result instead.
|
||||
@@ -0,0 +1,147 @@
|
||||
---
|
||||
title: "OracleEmbeddingRetriever"
|
||||
id: oracleembeddingretriever
|
||||
slug: "/oracleembeddingretriever"
|
||||
description: "An embedding-based Retriever compatible with the Oracle Document Store."
|
||||
---
|
||||
|
||||
# OracleEmbeddingRetriever
|
||||
|
||||
An embedding-based Retriever compatible with the Oracle Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in a semantic search pipeline 3. After a Text Embedder and before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of an [OracleDocumentStore](../../document-stores/oracledocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query_embedding`: A vector representing the query (a list of floats) |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Oracle](/reference/integrations-oracle) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/oracle |
|
||||
| **Package name** | `oracle-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `OracleEmbeddingRetriever` is an embedding-based Retriever compatible with `OracleDocumentStore`. It uses Oracle AI Vector Search to compare query and document embeddings, fetching the most relevant documents based on vector similarity.
|
||||
|
||||
When using `OracleEmbeddingRetriever` in a pipeline, make sure embeddings are available for both documents (at index time) and queries (at query time). Use a Document Embedder in your indexing pipeline and a Text Embedder in your query pipeline.
|
||||
|
||||
The distance metric (COSINE, EUCLIDEAN, or DOT) is configured on the `OracleDocumentStore`. In addition to `query_embedding`, the retriever accepts `top_k` (maximum documents to return) and `filters` to narrow the search space.
|
||||
|
||||
## Installation
|
||||
|
||||
To run Oracle Database 23ai locally with Docker:
|
||||
|
||||
```shell
|
||||
docker run -d --name oracle23ai \
|
||||
-p 1521:1521 \
|
||||
-e ORACLE_PASSWORD=oracle \
|
||||
container-registry.oracle.com/database/free:latest
|
||||
```
|
||||
|
||||
Install the Oracle integration for Haystack:
|
||||
|
||||
```shell
|
||||
pip install oracle-haystack
|
||||
```
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
This Retriever needs an `OracleDocumentStore` and indexed documents with embeddings to run.
|
||||
|
||||
```python
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.document_stores.oracle import (
|
||||
OracleDocumentStore,
|
||||
OracleConnectionConfig,
|
||||
)
|
||||
from haystack_integrations.components.retrievers.oracle import OracleEmbeddingRetriever
|
||||
|
||||
document_store = OracleDocumentStore(
|
||||
connection_config=OracleConnectionConfig(
|
||||
user=Secret.from_env_var("ORACLE_USER"),
|
||||
password=Secret.from_env_var("ORACLE_PASSWORD"),
|
||||
dsn=Secret.from_env_var("ORACLE_DSN"),
|
||||
),
|
||||
embedding_dim=768,
|
||||
)
|
||||
|
||||
retriever = OracleEmbeddingRetriever(document_store=document_store)
|
||||
|
||||
# using a fake vector to keep the example simple
|
||||
retriever.run(query_embedding=[0.1] * 768)
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
SentenceTransformersTextEmbedder,
|
||||
)
|
||||
from haystack.utils import Secret
|
||||
|
||||
from haystack_integrations.document_stores.oracle import (
|
||||
OracleDocumentStore,
|
||||
OracleConnectionConfig,
|
||||
)
|
||||
from haystack_integrations.components.retrievers.oracle import OracleEmbeddingRetriever
|
||||
|
||||
document_store = OracleDocumentStore(
|
||||
connection_config=OracleConnectionConfig(
|
||||
user=Secret.from_env_var("ORACLE_USER"),
|
||||
password=Secret.from_env_var("ORACLE_PASSWORD"),
|
||||
dsn=Secret.from_env_var("ORACLE_DSN"),
|
||||
),
|
||||
embedding_dim=768,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
document_embedder = SentenceTransformersDocumentEmbedder(
|
||||
model="sentence-transformers/all-MiniLM-L6-v2",
|
||||
)
|
||||
documents_with_embeddings = document_embedder.run(documents)
|
||||
|
||||
document_store.write_documents(
|
||||
documents_with_embeddings["documents"],
|
||||
policy=DuplicatePolicy.OVERWRITE,
|
||||
)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component(
|
||||
"text_embedder",
|
||||
SentenceTransformersTextEmbedder(model="sentence-transformers/all-MiniLM-L6-v2"),
|
||||
)
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
OracleEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "How many languages are there?"
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
```
|
||||
@@ -0,0 +1,150 @@
|
||||
---
|
||||
title: "OracleKeywordRetriever"
|
||||
id: oraclekeywordretriever
|
||||
slug: "/oraclekeywordretriever"
|
||||
description: "A keyword-based Retriever that fetches documents matching a query from the Oracle Document Store using Oracle's DBMS_SEARCH full-text index."
|
||||
---
|
||||
|
||||
# OracleKeywordRetriever
|
||||
|
||||
A keyword-based Retriever that fetches documents matching a query from the Oracle Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. Before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in a keyword search pipeline 3. Before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of an [OracleDocumentStore](../../document-stores/oracledocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query`: A string |
|
||||
| **Output variables** | `documents`: A list of documents matching the query |
|
||||
| **API reference** | [Oracle](/reference/integrations-oracle) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/oracle |
|
||||
| **Package name** | `oracle-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `OracleKeywordRetriever` is a keyword-based Retriever compatible with `OracleDocumentStore`. It uses Oracle's DBMS_SEARCH full-text index — automatically created when the document store is initialized — to search documents by keyword relevance.
|
||||
|
||||
This retriever works without embeddings, making it suitable for keyword-only pipelines or as the keyword branch of a hybrid search pipeline.
|
||||
|
||||
In addition to `query`, the retriever accepts `top_k` (maximum documents to return) and `filters` to narrow the search space.
|
||||
|
||||
## Installation
|
||||
|
||||
To run Oracle Database 23ai locally with Docker:
|
||||
|
||||
```shell
|
||||
docker run -d --name oracle23ai \
|
||||
-p 1521:1521 \
|
||||
-e ORACLE_PASSWORD=oracle \
|
||||
container-registry.oracle.com/database/free:latest
|
||||
```
|
||||
|
||||
Install the Oracle integration for Haystack:
|
||||
|
||||
```shell
|
||||
pip install oracle-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
This Retriever needs an `OracleDocumentStore` and indexed documents to run.
|
||||
|
||||
```python
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.document_stores.oracle import (
|
||||
OracleDocumentStore,
|
||||
OracleConnectionConfig,
|
||||
)
|
||||
from haystack_integrations.components.retrievers.oracle import OracleKeywordRetriever
|
||||
|
||||
document_store = OracleDocumentStore(
|
||||
connection_config=OracleConnectionConfig(
|
||||
user=Secret.from_env_var("ORACLE_USER"),
|
||||
password=Secret.from_env_var("ORACLE_PASSWORD"),
|
||||
dsn=Secret.from_env_var("ORACLE_DSN"),
|
||||
),
|
||||
embedding_dim=768,
|
||||
)
|
||||
|
||||
retriever = OracleKeywordRetriever(document_store=document_store)
|
||||
retriever.run(query="my keyword query")
|
||||
```
|
||||
|
||||
### In a RAG pipeline
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack.utils import Secret
|
||||
|
||||
from haystack_integrations.document_stores.oracle import (
|
||||
OracleDocumentStore,
|
||||
OracleConnectionConfig,
|
||||
)
|
||||
from haystack_integrations.components.retrievers.oracle import OracleKeywordRetriever
|
||||
|
||||
prompt_template = [
|
||||
ChatMessage.from_user(
|
||||
"""
|
||||
Given these documents, answer the question.\nDocuments:
|
||||
{% for doc in documents %}
|
||||
{{ doc.content }}
|
||||
{% endfor %}
|
||||
|
||||
\nQuestion: {{question}}
|
||||
\nAnswer:
|
||||
""",
|
||||
),
|
||||
]
|
||||
|
||||
document_store = OracleDocumentStore(
|
||||
connection_config=OracleConnectionConfig(
|
||||
user=Secret.from_env_var("ORACLE_USER"),
|
||||
password=Secret.from_env_var("ORACLE_PASSWORD"),
|
||||
dsn=Secret.from_env_var("ORACLE_DSN"),
|
||||
),
|
||||
embedding_dim=768,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
document_store.write_documents(documents=documents, policy=DuplicatePolicy.SKIP)
|
||||
|
||||
retriever = OracleKeywordRetriever(document_store=document_store)
|
||||
|
||||
rag_pipeline = Pipeline()
|
||||
rag_pipeline.add_component(name="retriever", instance=retriever)
|
||||
rag_pipeline.add_component(
|
||||
instance=ChatPromptBuilder(template=prompt_template, required_variables="*"),
|
||||
name="prompt_builder",
|
||||
)
|
||||
rag_pipeline.add_component(instance=OpenAIChatGenerator(), name="llm")
|
||||
|
||||
rag_pipeline.connect("retriever", "prompt_builder.documents")
|
||||
rag_pipeline.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
question = "How many languages are there?"
|
||||
result = rag_pipeline.run(
|
||||
{
|
||||
"retriever": {"query": question},
|
||||
"prompt_builder": {"question": question},
|
||||
},
|
||||
)
|
||||
print(result["llm"]["replies"][0].text)
|
||||
```
|
||||
@@ -0,0 +1,135 @@
|
||||
---
|
||||
title: "PgvectorEmbeddingRetriever"
|
||||
id: pgvectorembeddingretriever
|
||||
slug: "/pgvectorembeddingretriever"
|
||||
description: "An embedding-based Retriever compatible with the Pgvector Document Store."
|
||||
---
|
||||
|
||||
# PgvectorEmbeddingRetriever
|
||||
|
||||
An embedding-based Retriever compatible with the Pgvector Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the semantic search pipeline 3. After a Text Embedder and before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a [PgvectorDocumentStore](../../document-stores/pgvectordocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query_embedding`: A vector representing the query (a list of floats) |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Pgvector](/reference/integrations-pgvector) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/pgvector |
|
||||
| **Package name** | `pgvector-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `PgvectorEmbeddingRetriever` is an embedding-based Retriever compatible with the `PgvectorDocumentStore`. It compares the query and Document embeddings and fetches the Documents most relevant to the query from the `PgvectorDocumentStore` based on the outcome.
|
||||
|
||||
When using the `PgvectorEmbeddingRetriever` in your Pipeline, make sure it has the query and Document embeddings available. You can do so by adding a Document Embedder to your indexing Pipeline and a Text Embedder to your query Pipeline.
|
||||
|
||||
In addition to the `query_embedding`, the `PgvectorEmbeddingRetriever` accepts other optional parameters, including `top_k` (the maximum number of Documents to retrieve) and `filters` to narrow down the search space.
|
||||
|
||||
Some relevant parameters that impact the embedding retrieval must be defined when the corresponding `PgvectorDocumentStore` is initialized: these include embedding dimension, vector function, and some others related to the search strategy (exact nearest neighbor or HNSW).
|
||||
|
||||
## Installation
|
||||
|
||||
To quickly set up a PostgreSQL database with pgvector, you can use Docker:
|
||||
|
||||
```shell
|
||||
docker run -d -p 5432:5432 -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=postgres ankane/pgvector
|
||||
```
|
||||
|
||||
For more information on installing pgvector, visit the [pgvector GitHub repository](https://github.com/pgvector/pgvector).
|
||||
|
||||
To use pgvector with Haystack, install the `pgvector-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install pgvector-haystack
|
||||
```
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
This Retriever needs the `PgvectorDocumentStore` and indexed Documents to run.
|
||||
|
||||
```python
|
||||
import os
|
||||
from haystack_integrations.document_stores.pgvector import PgvectorDocumentStore
|
||||
from haystack_integrations.components.retrievers.pgvector import (
|
||||
PgvectorEmbeddingRetriever,
|
||||
)
|
||||
|
||||
os.environ["PG_CONN_STR"] = "postgresql://postgres:postgres@localhost:5432/postgres"
|
||||
|
||||
document_store = PgvectorDocumentStore()
|
||||
retriever = PgvectorEmbeddingRetriever(document_store=document_store)
|
||||
|
||||
# using a fake vector to keep the example simple
|
||||
retriever.run(query_embedding=[0.1] * 768)
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
```python
|
||||
import os
|
||||
from haystack.document_stores import DuplicatePolicy
|
||||
from haystack import Document, Pipeline
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersTextEmbedder,
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
)
|
||||
|
||||
from haystack_integrations.document_stores.pgvector import PgvectorDocumentStore
|
||||
from haystack_integrations.components.retrievers.pgvector import (
|
||||
PgvectorEmbeddingRetriever,
|
||||
)
|
||||
|
||||
os.environ["PG_CONN_STR"] = "postgresql://postgres:postgres@localhost:5432/postgres"
|
||||
|
||||
document_store = PgvectorDocumentStore(
|
||||
embedding_dimension=768,
|
||||
vector_function="cosine_similarity",
|
||||
recreate_table=True,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
document_embedder = SentenceTransformersDocumentEmbedder()
|
||||
documents_with_embeddings = document_embedder.run(documents)
|
||||
|
||||
document_store.write_documents(
|
||||
documents_with_embeddings.get("documents"),
|
||||
policy=DuplicatePolicy.OVERWRITE,
|
||||
)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", SentenceTransformersTextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
PgvectorEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "How many languages are there?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
```
|
||||
@@ -0,0 +1,151 @@
|
||||
---
|
||||
title: "PgvectorKeywordRetriever"
|
||||
id: pgvectorkeywordretriever
|
||||
slug: "/pgvectorkeywordretriever"
|
||||
description: "This is a keyword-based Retriever that fetches documents matching a query from the Pgvector Document Store."
|
||||
---
|
||||
|
||||
# PgvectorKeywordRetriever
|
||||
|
||||
This is a keyword-based Retriever that fetches documents matching a query from the Pgvector Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. Before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the semantic search pipeline 3. Before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a [PgvectorDocumentStore](../../document-stores/pgvectordocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query`: A string |
|
||||
| **Output variables** | `document`: A list of documents (matching the query) |
|
||||
| **API reference** | [Pgvector](/reference/integrations-pgvector) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/pgvector |
|
||||
| **Package name** | `pgvector-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `PgvectorKeywordRetriever` is a keyword-based Retriever compatible with the `PgvectorDocumentStore`.
|
||||
|
||||
The component uses the `ts_rank_cd` function of PostgreSQL to rank the documents.
|
||||
It considers how often the query terms appear in the document, how close together the terms are in the document, and how important is the part of the document where they occur.
|
||||
For more details, see [Postgres documentation](https://www.postgresql.org/docs/current/textsearch-controls.html#TEXTSEARCH-RANKING).
|
||||
|
||||
Keep in mind that, unlike similar components such as `ElasticsearchBM25Retriever`, this Retriever does not apply fuzzy search out of the box, so it’s necessary to carefully formulate the query in order to avoid getting zero results.
|
||||
|
||||
In addition to the `query`, the `PgvectorKeywordRetriever` accepts other optional parameters, including `top_k` (the maximum number of documents to retrieve) and `filters` to narrow the search space.
|
||||
|
||||
### Installation
|
||||
|
||||
To quickly set up a PostgreSQL database with pgvector, you can use Docker:
|
||||
|
||||
```shell
|
||||
docker run -d -p 5432:5432 -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=postgres ankane/pgvector
|
||||
```
|
||||
|
||||
For more information on how to install pgvector, visit the [pgvector GitHub repository](https://github.com/pgvector/pgvector).
|
||||
|
||||
Install the `pgvector-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install pgvector-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
This Retriever needs the `PgvectorDocumentStore` and indexed documents to run.
|
||||
|
||||
Set an environment variable `PG_CONN_STR` with the connection string to your PostgreSQL database.
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.pgvector import PgvectorDocumentStore
|
||||
from haystack_integrations.components.retrievers.pgvector import (
|
||||
PgvectorKeywordRetriever,
|
||||
)
|
||||
|
||||
document_store = PgvectorDocumentStore()
|
||||
retriever = PgvectorKeywordRetriever(document_store=document_store)
|
||||
|
||||
retriever.run(query="my nice query")
|
||||
```
|
||||
|
||||
### In a RAG pipeline
|
||||
|
||||
The prerequisites necessary for running this code are:
|
||||
|
||||
- Set an environment variable `OPENAI_API_KEY` with your OpenAI API key.
|
||||
- Set an environment variable `PG_CONN_STR` with the connection string to your PostgreSQL database.
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders.answer_builder import AnswerBuilder
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
|
||||
from haystack_integrations.document_stores.pgvector import PgvectorDocumentStore
|
||||
from haystack_integrations.components.retrievers.pgvector import (
|
||||
PgvectorKeywordRetriever,
|
||||
)
|
||||
|
||||
# Create a RAG query pipeline
|
||||
prompt_template = [
|
||||
ChatMessage.from_user(
|
||||
"""
|
||||
Given these documents, answer the question.\nDocuments:
|
||||
{% for doc in documents %}
|
||||
{{ doc.content }}
|
||||
{% endfor %}
|
||||
|
||||
\nQuestion: {{question}}
|
||||
\nAnswer:
|
||||
""",
|
||||
),
|
||||
]
|
||||
|
||||
document_store = PgvectorDocumentStore(
|
||||
language="english", # this parameter influences text parsing for keyword retrieval
|
||||
recreate_table=True,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
# DuplicatePolicy.SKIP param is optional, but useful to run the script multiple times without throwing errors
|
||||
document_store.write_documents(documents=documents, policy=DuplicatePolicy.SKIP)
|
||||
|
||||
retriever = PgvectorKeywordRetriever(document_store=document_store)
|
||||
rag_pipeline = Pipeline()
|
||||
rag_pipeline.add_component(name="retriever", instance=retriever)
|
||||
rag_pipeline.add_component(
|
||||
instance=ChatPromptBuilder(template=prompt_template, required_variables="*"),
|
||||
name="prompt_builder",
|
||||
)
|
||||
rag_pipeline.add_component(instance=OpenAIChatGenerator(), name="llm")
|
||||
rag_pipeline.add_component(instance=AnswerBuilder(), name="answer_builder")
|
||||
rag_pipeline.connect("retriever", "prompt_builder.documents")
|
||||
rag_pipeline.connect("prompt_builder.prompt", "llm.messages")
|
||||
rag_pipeline.connect("llm.replies", "answer_builder.replies")
|
||||
rag_pipeline.connect("retriever", "answer_builder.documents")
|
||||
|
||||
question = "languages spoken around the world today"
|
||||
result = rag_pipeline.run(
|
||||
{
|
||||
"retriever": {"query": question},
|
||||
"prompt_builder": {"question": question},
|
||||
"answer_builder": {"query": question},
|
||||
},
|
||||
)
|
||||
print(result["answer_builder"])
|
||||
```
|
||||
@@ -0,0 +1,129 @@
|
||||
---
|
||||
title: "PineconeEmbeddingRetriever"
|
||||
id: pineconedenseretriever
|
||||
slug: "/pineconedenseretriever"
|
||||
description: "An embedding-based Retriever compatible with the Pinecone Document Store."
|
||||
---
|
||||
|
||||
# PineconeEmbeddingRetriever
|
||||
|
||||
An embedding-based Retriever compatible with the Pinecone Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the semantic search pipeline 3. After a Text Embedder and before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a [PineconeDocumentStore](../../document-stores/pinecone-document-store.mdx) |
|
||||
| **Mandatory run variables** | `query_embedding`: A vector representing the query (a list of floats) |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Pinecone](/reference/integrations-pinecone) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/pinecone |
|
||||
| **Package name** | `pinecone-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `PineconeEmbeddingRetriever` is an embedding-based Retriever compatible with the `PineconeDocumentStore`. It compares the query and Document embeddings and fetches the Documents most relevant to the query from the `PineconeDocumentStore` based on the outcome.
|
||||
|
||||
When using the `PineconeEmbeddingRetriever` in your NLP system, make sure it has the query and Document embeddings available. You can do so by adding a Document Embedder to your indexing Pipeline and a Text Embedder to your query Pipeline.
|
||||
|
||||
In addition to the `query_embedding`, the `PineconeEmbeddingRetriever` accepts other optional parameters, including `top_k` (the maximum number of Documents to retrieve) and `filters` to narrow down the search space.
|
||||
|
||||
Some relevant parameters that impact the embedding retrieval must be defined when the corresponding `PineconeDocumentStore` is initialized: these include the `dimension` of the embeddings and the distance `metric` to use.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
This Retriever needs the `PineconeDocumentStore` and indexed Documents to run.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.retrievers.pinecone import (
|
||||
PineconeEmbeddingRetriever,
|
||||
)
|
||||
from haystack_integrations.document_stores.pinecone import PineconeDocumentStore
|
||||
|
||||
# Make sure you have the PINECONE_API_KEY environment variable set
|
||||
document_store = PineconeDocumentStore(
|
||||
index="my_index_with_documents",
|
||||
namespace="my_namespace",
|
||||
dimension=768,
|
||||
)
|
||||
|
||||
retriever = PineconeEmbeddingRetriever(document_store=document_store)
|
||||
|
||||
# using an imaginary vector to keep the example simple, example run query:
|
||||
retriever.run(query_embedding=[0.1] * 768)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Install the dependencies you’ll need:
|
||||
|
||||
```shell
|
||||
pip install pinecone-haystack
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
Use this Retriever in a query Pipeline like this:
|
||||
|
||||
```python
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersTextEmbedder,
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
)
|
||||
from haystack_integrations.components.retrievers.pinecone import (
|
||||
PineconeEmbeddingRetriever,
|
||||
)
|
||||
from haystack_integrations.document_stores.pinecone import PineconeDocumentStore
|
||||
|
||||
# Make sure you have the PINECONE_API_KEY environment variable set
|
||||
document_store = PineconeDocumentStore(
|
||||
index="my_index",
|
||||
namespace="my_namespace",
|
||||
dimension=768,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
document_embedder = SentenceTransformersDocumentEmbedder()
|
||||
documents_with_embeddings = document_embedder.run(documents)
|
||||
|
||||
document_store.write_documents(
|
||||
documents_with_embeddings.get("documents"),
|
||||
policy=DuplicatePolicy.OVERWRITE,
|
||||
)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", SentenceTransformersTextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
PineconeEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "How many languages are there?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
```
|
||||
|
||||
The example output would be:
|
||||
|
||||
```python
|
||||
Document(id=cfe93bc1c274908801e6670440bf2bbba54fad792770d57421f85ffa2a4fcc94, content: 'There are over 7,000 languages spoken around the world today.', score: 0.87717235, embedding: vector of size 768)
|
||||
```
|
||||
@@ -0,0 +1,124 @@
|
||||
---
|
||||
title: "QdrantEmbeddingRetriever"
|
||||
id: qdrantembeddingretriever
|
||||
slug: "/qdrantembeddingretriever"
|
||||
description: "An embedding-based Retriever compatible with the Qdrant Document Store."
|
||||
---
|
||||
|
||||
# QdrantEmbeddingRetriever
|
||||
|
||||
An embedding-based Retriever compatible with the Qdrant Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1\. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG Pipeline <br /> <br />2. The last component in the semantic search pipeline <br />3. After a Text Embedder and before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a [QdrantDocumentStore](../../document-stores/qdrant-document-store.mdx) |
|
||||
| **Mandatory run variables** | `query_embedding`: A vector representing the query (a list of floats) |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Qdrant](/reference/integrations-qdrant) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/qdrant |
|
||||
| **Package name** | `qdrant-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `QdrantEmbeddingRetriever` is an embedding-based Retriever compatible with the `QdrantDocumentStore`. It compares the query and Document embeddings and fetches the Documents most relevant to the query from the `QdrantDocumentStore` based on the outcome.
|
||||
|
||||
When using the `QdrantEmbeddingRetriever` in your NLP system, make sure it has the query and Document embeddings available. You can add a Document Embedder to your indexing Pipeline and a Text Embedder to your query Pipeline.
|
||||
|
||||
In addition to the `query_embedding`, the `QdrantEmbeddingRetriever` accepts other optional parameters, including `top_k` (the maximum number of Documents to retrieve) and `filters` to narrow down the search space.
|
||||
|
||||
Some relevant parameters that impact the embedding retrieval must be defined when the corresponding `QdrantDocumentStore` is initialized: these include the embedding dimension (`embedding_dim`), the `similarity` function to use when comparing embeddings and the HNSW configuration (`hnsw_config`).
|
||||
|
||||
### Installation
|
||||
|
||||
To start using Qdrant with Haystack, first install the package with:
|
||||
|
||||
```shell
|
||||
pip install qdrant-haystack
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
#### On its own
|
||||
|
||||
This Retriever needs the `QdrantDocumentStore` and indexed Documents to run.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.retrievers.qdrant import QdrantEmbeddingRetriever
|
||||
from haystack_integrations.document_stores.qdrant import QdrantDocumentStore
|
||||
|
||||
document_store = QdrantDocumentStore(
|
||||
":memory:",
|
||||
recreate_index=True,
|
||||
return_embedding=True,
|
||||
wait_result_from_api=True,
|
||||
)
|
||||
retriever = QdrantEmbeddingRetriever(document_store=document_store)
|
||||
|
||||
# using a fake vector to keep the example simple
|
||||
retriever.run(query_embedding=[0.1] * 768)
|
||||
```
|
||||
|
||||
#### In a Pipeline
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
```python
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersTextEmbedder,
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
)
|
||||
|
||||
from haystack_integrations.components.retrievers.qdrant import QdrantEmbeddingRetriever
|
||||
from haystack_integrations.document_stores.qdrant import QdrantDocumentStore
|
||||
|
||||
document_store = QdrantDocumentStore(
|
||||
":memory:",
|
||||
recreate_index=True,
|
||||
return_embedding=True,
|
||||
wait_result_from_api=True,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
document_embedder = SentenceTransformersDocumentEmbedder()
|
||||
documents_with_embeddings = document_embedder.run(documents)
|
||||
|
||||
document_store.write_documents(
|
||||
documents_with_embeddings.get("documents"),
|
||||
policy=DuplicatePolicy.OVERWRITE,
|
||||
)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", SentenceTransformersTextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
QdrantEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "How many languages are there?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
```
|
||||
@@ -0,0 +1,191 @@
|
||||
---
|
||||
title: "QdrantHybridRetriever"
|
||||
id: qdranthybridretriever
|
||||
slug: "/qdranthybridretriever"
|
||||
description: "A Retriever based both on dense and sparse embeddings, compatible with the Qdrant Document Store."
|
||||
---
|
||||
|
||||
# QdrantHybridRetriever
|
||||
|
||||
A Retriever based both on dense and sparse embeddings, compatible with the Qdrant Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1\. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline <br /> <br />2. The last component in a hybrid search pipeline <br /> 3. After a Text Embedder and before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a [QdrantDocumentStore](../../document-stores/qdrant-document-store.mdx) |
|
||||
| **Mandatory run variables** | `query_embedding`: A dense vector representing the query (a list of floats) <br /> <br />`query_sparse_embedding`: A [`SparseEmbedding`](../../concepts/data-classes.mdx#sparseembedding) object containing a vectorial representation of the query |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Qdrant](/reference/integrations-qdrant) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/qdrant |
|
||||
| **Package name** | `qdrant-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `QdrantHybridRetriever` is a Retriever based both on dense and sparse embeddings, compatible with the [`QdrantDocumentStore`](../../document-stores/qdrant-document-store.mdx).
|
||||
|
||||
It compares the query and document’s dense and sparse embeddings and fetches the documents most relevant to the query from the `QdrantDocumentStore`, fusing the scores with Reciprocal Rank Fusion.
|
||||
|
||||
:::tip[Hybrid Retrieval Pipeline]
|
||||
|
||||
If you want additional customization for merging or fusing results, consider creating a hybrid retrieval pipeline with [`DocumentJoiner`](../joiners/documentjoiner.mdx).
|
||||
|
||||
You can check out our hybrid retrieval pipeline [tutorial](https://haystack.deepset.ai/tutorials/33_hybrid_retrieval) for detailed steps.
|
||||
:::
|
||||
|
||||
When using the `QdrantHybridRetriever`, make sure it has the query and document with dense and sparse embeddings available. You can do so by:
|
||||
|
||||
- Adding a (dense) document Embedder and a sparse document Embedder to your indexing pipeline,
|
||||
- Adding a (dense) text Embedder and a sparse text Embedder to your query pipeline.
|
||||
|
||||
In addition to `query_embedding` and `query_sparse_embedding`, the `QdrantHybridRetriever` accepts other optional parameters, including `top_k` (the maximum number of documents to retrieve) and `filters` to narrow down the search space.
|
||||
|
||||
:::note[Sparse Embedding Support]
|
||||
|
||||
To use Sparse Embedding support, you need to initialize the `QdrantDocumentStore` with `use_sparse_embeddings=True`, which is `False` by default.
|
||||
|
||||
If you want to use Document Store or collection previously created with this feature disabled, you must migrate the existing data. You can do this by taking advantage of the `migrate_to_sparse_embeddings_support` utility function.
|
||||
:::
|
||||
|
||||
### Installation
|
||||
|
||||
To start using Qdrant with Haystack, first install the package with:
|
||||
|
||||
```shell
|
||||
pip install qdrant-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.retrievers.qdrant import QdrantHybridRetriever
|
||||
from haystack_integrations.document_stores.qdrant import QdrantDocumentStore
|
||||
from haystack.dataclasses import Document, SparseEmbedding
|
||||
|
||||
document_store = QdrantDocumentStore(
|
||||
":memory:",
|
||||
use_sparse_embeddings=True,
|
||||
recreate_index=True,
|
||||
return_embedding=True,
|
||||
wait_result_from_api=True,
|
||||
)
|
||||
|
||||
doc = Document(
|
||||
content="test",
|
||||
embedding=[0.5] * 768,
|
||||
sparse_embedding=SparseEmbedding(indices=[0, 3, 5], values=[0.1, 0.5, 0.12]),
|
||||
)
|
||||
|
||||
document_store.write_documents([doc])
|
||||
|
||||
retriever = QdrantHybridRetriever(document_store=document_store)
|
||||
embedding = [0.1] * 768
|
||||
sparse_embedding = SparseEmbedding(indices=[0, 1, 2, 3], values=[0.1, 0.8, 0.05, 0.33])
|
||||
retriever.run(query_embedding=embedding, query_sparse_embedding=sparse_embedding)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Currently, you can compute sparse embeddings using Fastembed Sparse Embedders.
|
||||
First, install the package with:
|
||||
|
||||
```shell
|
||||
pip install fastembed-haystack
|
||||
```
|
||||
|
||||
In the example below, we are using Fastembed Embedders to compute dense embeddings as well.
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack_integrations.components.retrievers.qdrant import QdrantHybridRetriever
|
||||
from haystack_integrations.document_stores.qdrant import QdrantDocumentStore
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack_integrations.components.embedders.fastembed import (
|
||||
FastembedTextEmbedder,
|
||||
FastembedDocumentEmbedder,
|
||||
FastembedSparseTextEmbedder,
|
||||
FastembedSparseDocumentEmbedder,
|
||||
)
|
||||
|
||||
document_store = QdrantDocumentStore(
|
||||
":memory:",
|
||||
recreate_index=True,
|
||||
use_sparse_embeddings=True,
|
||||
embedding_dim=384,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
Document(content="Germany has many big cities"),
|
||||
Document(content="fastembed is supported by and maintained by Qdrant."),
|
||||
]
|
||||
|
||||
indexing = Pipeline()
|
||||
indexing.add_component(
|
||||
"sparse_doc_embedder",
|
||||
FastembedSparseDocumentEmbedder(model="prithvida/Splade_PP_en_v1"),
|
||||
)
|
||||
indexing.add_component(
|
||||
"dense_doc_embedder",
|
||||
FastembedDocumentEmbedder(model="BAAI/bge-small-en-v1.5"),
|
||||
)
|
||||
indexing.add_component(
|
||||
"writer",
|
||||
DocumentWriter(document_store=document_store, policy=DuplicatePolicy.OVERWRITE),
|
||||
)
|
||||
indexing.connect("sparse_doc_embedder", "dense_doc_embedder")
|
||||
indexing.connect("dense_doc_embedder", "writer")
|
||||
|
||||
indexing.run({"sparse_doc_embedder": {"documents": documents}})
|
||||
|
||||
querying = Pipeline()
|
||||
querying.add_component(
|
||||
"sparse_text_embedder",
|
||||
FastembedSparseTextEmbedder(model="prithvida/Splade_PP_en_v1"),
|
||||
)
|
||||
querying.add_component(
|
||||
"dense_text_embedder",
|
||||
FastembedTextEmbedder(
|
||||
model="BAAI/bge-small-en-v1.5",
|
||||
prefix="Represent this sentence for searching relevant passages: ",
|
||||
),
|
||||
)
|
||||
querying.add_component(
|
||||
"retriever",
|
||||
QdrantHybridRetriever(document_store=document_store),
|
||||
)
|
||||
|
||||
querying.connect(
|
||||
"sparse_text_embedder.sparse_embedding",
|
||||
"retriever.query_sparse_embedding",
|
||||
)
|
||||
querying.connect("dense_text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
question = "Who supports fastembed?"
|
||||
|
||||
results = querying.run(
|
||||
{
|
||||
"dense_text_embedder": {"text": question},
|
||||
"sparse_text_embedder": {"text": question},
|
||||
},
|
||||
)
|
||||
|
||||
print(results["retriever"]["documents"][0])
|
||||
|
||||
# Document(id=...,
|
||||
# content: 'fastembed is supported by and maintained by Qdrant.',
|
||||
# score: 1.0)
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
:notebook: Tutorial: [Creating a Hybrid Retrieval Pipeline](https://haystack.deepset.ai/tutorials/33_hybrid_retrieval)
|
||||
|
||||
🧑🍳 Cookbook: [Sparse Embedding Retrieval with Qdrant and FastEmbed](https://haystack.deepset.ai/cookbook/sparse_embedding_retrieval)
|
||||
@@ -0,0 +1,154 @@
|
||||
---
|
||||
title: "QdrantSparseEmbeddingRetriever"
|
||||
id: qdrantsparseembeddingretriever
|
||||
slug: "/qdrantsparseembeddingretriever"
|
||||
description: "A Retriever based on sparse embeddings, compatible with the Qdrant Document Store."
|
||||
---
|
||||
|
||||
# QdrantSparseEmbeddingRetriever
|
||||
|
||||
A Retriever based on sparse embeddings, compatible with the Qdrant Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1\. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline <br /> <br />2. The last component in the semantic search pipeline <br /> 3. After a Text Embedder and before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a [QdrantDocumentStore](../../document-stores/qdrant-document-store.mdx) |
|
||||
| **Mandatory run variables** | `query_sparse_embedding`: A [`SparseEmbedding`](../../concepts/data-classes.mdx#sparseembedding) object containing a vectorial representation of the query |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Qdrant](/reference/integrations-qdrant) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/qdrant |
|
||||
| **Package name** | `qdrant-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `QdrantSparseEmbeddingRetriever` is a Retriever based on sparse embeddings, compatible with the [`QdrantDocumentStore`](../../document-stores/qdrant-document-store.mdx).
|
||||
|
||||
It compares the query and document sparse embeddings and, based on the outcome, fetches the documents most relevant to the query from the `QdrantDocumentStore`.
|
||||
|
||||
When using the `QdrantSparseEmbeddingRetriever`, make sure it has the query and document sparse embeddings available. You can do so by adding a sparse document Embedder to your indexing pipeline and a sparse text Embedder to your query pipeline.
|
||||
|
||||
In addition to the `query_sparse_embedding`, the `QdrantSparseEmbeddingRetriever` accepts other optional parameters, including `top_k` (the maximum number of documents to retrieve) and `filters` to narrow down the search space.
|
||||
|
||||
:::note[Sparse Embedding Support]
|
||||
|
||||
To use Sparse Embedding support, you need to initialize the `QdrantDocumentStore` with `use_sparse_embeddings=True`, which is `False` by default.
|
||||
|
||||
If you want to use Document Store or collection previously created with this feature disabled, you must migrate the existing data. You can do this by taking advantage of the `migrate_to_sparse_embeddings_support` utility function.
|
||||
:::
|
||||
|
||||
### Installation
|
||||
|
||||
To start using Qdrant with Haystack, first install the package with:
|
||||
|
||||
```shell
|
||||
pip install qdrant-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
This Retriever needs the `QdrantDocumentStore` and indexed documents to run.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.retrievers.qdrant import (
|
||||
QdrantSparseEmbeddingRetriever,
|
||||
)
|
||||
from haystack_integrations.document_stores.qdrant import QdrantDocumentStore
|
||||
from haystack.dataclasses import Document, SparseEmbedding
|
||||
|
||||
document_store = QdrantDocumentStore(
|
||||
":memory:",
|
||||
use_sparse_embeddings=True,
|
||||
recreate_index=True,
|
||||
return_embedding=True,
|
||||
)
|
||||
|
||||
doc = Document(
|
||||
content="test",
|
||||
sparse_embedding=SparseEmbedding(indices=[0, 3, 5], values=[0.1, 0.5, 0.12]),
|
||||
)
|
||||
document_store.write_documents([doc])
|
||||
|
||||
retriever = QdrantSparseEmbeddingRetriever(document_store=document_store)
|
||||
sparse_embedding = SparseEmbedding(indices=[0, 1, 2, 3], values=[0.1, 0.8, 0.05, 0.33])
|
||||
retriever.run(query_sparse_embedding=sparse_embedding)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
In Haystack, you can compute sparse embeddings using Fastembed Embedders.
|
||||
|
||||
First, install the package with:
|
||||
|
||||
```shell
|
||||
pip install fastembed-haystack
|
||||
```
|
||||
|
||||
Then, try out this pipeline:
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack_integrations.components.retrievers.qdrant import (
|
||||
QdrantSparseEmbeddingRetriever,
|
||||
)
|
||||
from haystack_integrations.document_stores.qdrant import QdrantDocumentStore
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack_integrations.components.embedders.fastembed import (
|
||||
FastembedSparseDocumentEmbedder,
|
||||
FastembedSparseTextEmbedder,
|
||||
)
|
||||
|
||||
document_store = QdrantDocumentStore(
|
||||
":memory:",
|
||||
recreate_index=True,
|
||||
use_sparse_embeddings=True,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
Document(content="Germany has many big cities"),
|
||||
Document(content="fastembed is supported by and maintained by Qdrant."),
|
||||
]
|
||||
|
||||
sparse_document_embedder = FastembedSparseDocumentEmbedder()
|
||||
writer = DocumentWriter(document_store=document_store, policy=DuplicatePolicy.OVERWRITE)
|
||||
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component("sparse_document_embedder", sparse_document_embedder)
|
||||
indexing_pipeline.add_component("writer", writer)
|
||||
indexing_pipeline.connect("sparse_document_embedder", "writer")
|
||||
|
||||
indexing_pipeline.run({"sparse_document_embedder": {"documents": documents}})
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("sparse_text_embedder", FastembedSparseTextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"sparse_retriever",
|
||||
QdrantSparseEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect(
|
||||
"sparse_text_embedder.sparse_embedding",
|
||||
"sparse_retriever.query_sparse_embedding",
|
||||
)
|
||||
|
||||
query = "Who supports fastembed?"
|
||||
|
||||
result = query_pipeline.run({"sparse_text_embedder": {"text": query}})
|
||||
|
||||
print(result["sparse_retriever"]["documents"][0]) # noqa: T201
|
||||
|
||||
# Document(id=...,
|
||||
# content: 'fastembed is supported by and maintained by Qdrant.',
|
||||
# score: 0.758..)
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Sparse Embedding Retrieval with Qdrant and FastEmbed](https://haystack.deepset.ai/cookbook/sparse_embedding_retrieval)
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
title: "SentenceWindowRetriever"
|
||||
id: sentencewindowretriever
|
||||
slug: "/sentencewindowretriever"
|
||||
description: "Use this component to retrieve neighboring sentences around relevant sentences to get the full context."
|
||||
---
|
||||
|
||||
# SentenceWindowRetriever
|
||||
|
||||
Use this component to retrieve neighboring sentences around relevant sentences to get the full context.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Used after the main Retriever component, like the `InMemoryEmbeddingRetriever` or any other Retriever. |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a Document Store |
|
||||
| **Mandatory run variables** | `retrieved_documents`: A list of already retrieved documents for which you want to get a context window |
|
||||
| **Output variables** | `context_windows`: A list of strings <br /> <br />`context_documents`: A list of documents ordered by `split_idx_start` |
|
||||
| **API reference** | [Retrievers](/reference/retrievers-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/retrievers/sentence_window_retriever.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The "sentence window" is a retrieval technique that allows for the retrieval of the context around relevant sentences.
|
||||
|
||||
During indexing, documents are broken into smaller chunks or sentences and indexed. During retrieval, the sentences most relevant to a given query, based on a certain similarity metric, are retrieved.
|
||||
|
||||
Once we have the relevant sentences, we can retrieve neighboring sentences to provide full context. The number of neighboring sentences to retrieve is defined by a fixed number of sentences before and after the relevant sentence.
|
||||
|
||||
This component is meant to be used with other Retrievers, such as the `InMemoryEmbeddingRetriever`. These Retrievers find relevant sentences by comparing a query against indexed sentences using a similarity metric. Then, the `SentenceWindowRetriever` component retrieves neighboring sentences around the relevant ones by leveraging metadata stored in the `Document` object.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
splitter = DocumentSplitter(split_length=10, split_overlap=5, split_by="word")
|
||||
text = (
|
||||
"This is a text with some words. There is a second sentence. And there is also a third sentence. "
|
||||
"It also contains a fourth sentence. And a fifth sentence. And a sixth sentence. And a seventh sentence"
|
||||
)
|
||||
doc = Document(content=text)
|
||||
|
||||
docs = splitter.run([doc])
|
||||
doc_store = InMemoryDocumentStore()
|
||||
doc_store.write_documents(docs["documents"])
|
||||
|
||||
retriever = SentenceWindowRetriever(document_store=doc_store, window_size=3)
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.components.retrievers import SentenceWindowRetriever
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
splitter = DocumentSplitter(split_length=10, split_overlap=5, split_by="word")
|
||||
text = (
|
||||
"This is a text with some words. There is a second sentence. And there is also a third sentence. "
|
||||
"It also contains a fourth sentence. And a fifth sentence. And a sixth sentence. And a seventh sentence"
|
||||
)
|
||||
doc = Document(content=text)
|
||||
docs = splitter.run([doc])
|
||||
doc_store = InMemoryDocumentStore()
|
||||
doc_store.write_documents(docs["documents"])
|
||||
|
||||
rag = Pipeline()
|
||||
rag.add_component("bm25_retriever", InMemoryBM25Retriever(doc_store, top_k=1))
|
||||
rag.add_component(
|
||||
"sentence_window_retriever",
|
||||
SentenceWindowRetriever(document_store=doc_store, window_size=3),
|
||||
)
|
||||
rag.connect("bm25_retriever", "sentence_window_retriever")
|
||||
|
||||
rag.run({"bm25_retriever": {"query": "third"}})
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
:notebook: Tutorial: [Retrieving a Context Window Around a Sentence](https://haystack.deepset.ai/tutorials/42_sentence_window_retriever)
|
||||
@@ -0,0 +1,92 @@
|
||||
---
|
||||
title: "SnowflakeTableRetriever"
|
||||
id: snowflaketableretriever
|
||||
slug: "/snowflaketableretriever"
|
||||
description: "Connects to a Snowflake database to execute an SQL query."
|
||||
---
|
||||
|
||||
# SnowflakeTableRetriever
|
||||
|
||||
Connects to a Snowflake database to execute an SQL query.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a [`PromptBuilder`](../builders/promptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `user`: User's login <br /> <br />`account`: Snowflake account identifier <br /> <br />`api_key`: Snowflake account password. Can be set with `SNOWFLAKE_API_KEY` env var |
|
||||
| **Mandatory run variables** | `query`: An SQL query to execute |
|
||||
| **Output variables** | `dataframe`: The resulting Pandas dataframe version of the table |
|
||||
| **API reference** | [Snowflake](/reference/integrations-snowflake) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/snowflake |
|
||||
| **Package name** | `snowflake-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `SnowflakeTableRetriever` connects to a Snowflake database and retrieves data using an SQL query. It then returns a Pandas dataframe and a Markdown version of the table:
|
||||
|
||||
To start using the integration, install it with:
|
||||
|
||||
```bash
|
||||
pip install snowflake-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.components.retrievers.snowflake import (
|
||||
SnowflakeTableRetriever,
|
||||
)
|
||||
|
||||
snowflake = SnowflakeTableRetriever(
|
||||
user="<ACCOUNT-USER>",
|
||||
account="<ACCOUNT-IDENTIFIER>",
|
||||
api_key=Secret.from_env_var("SNOWFLAKE_API_KEY"),
|
||||
warehouse="<WAREHOUSE-NAME>",
|
||||
)
|
||||
|
||||
snowflake.run(query="select * from table limit 10;")
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
In the following pipeline example, the `ChatPromptBuilder` is using the table received from the `SnowflakeTableRetriever` to create a prompt and pass it on to an LLM:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.utils import Secret
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.retrievers.snowflake import (
|
||||
SnowflakeTableRetriever,
|
||||
)
|
||||
|
||||
executor = SnowflakeTableRetriever(
|
||||
user="<ACCOUNT-USER>",
|
||||
account="<ACCOUNT-IDENTIFIER>",
|
||||
api_key=Secret.from_env_var("SNOWFLAKE_API_KEY"),
|
||||
warehouse="<WAREHOUSE-NAME>",
|
||||
)
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component(
|
||||
"builder",
|
||||
ChatPromptBuilder(
|
||||
template=[ChatMessage.from_user("Describe this table: {{ table }}")],
|
||||
required_variables="*",
|
||||
),
|
||||
)
|
||||
pipeline.add_component("snowflake", executor)
|
||||
pipeline.add_component("llm", OpenAIChatGenerator(model="gpt-4o"))
|
||||
|
||||
pipeline.connect("snowflake.table", "builder.table")
|
||||
pipeline.connect("builder.prompt", "llm.messages")
|
||||
|
||||
pipeline.run(data={"query": "select employee, salary from table limit 10;"})
|
||||
```
|
||||
@@ -0,0 +1,152 @@
|
||||
---
|
||||
title: "SupabaseGroongaBM25Retriever"
|
||||
id: supabasegroongabm25retriever
|
||||
slug: "/supabasegroongabm25retriever"
|
||||
description: "A full-text Retriever that fetches documents from the SupabaseGroongaDocumentStore using PGroonga search."
|
||||
---
|
||||
|
||||
# SupabaseGroongaBM25Retriever
|
||||
|
||||
A full-text Retriever that fetches documents from the SupabaseGroongaDocumentStore using PGroonga search.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. Before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the full-text search pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a [SupabaseGroongaDocumentStore](../../document-stores/supabasedocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query`: A string |
|
||||
| **Output variables** | `documents`: A list of documents (matching the query) |
|
||||
| **API reference** | [Supabase](/reference/integrations-supabase) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/supabase |
|
||||
| **Package name** | `supabase-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`SupabaseGroongaBM25Retriever` retrieves Documents from the `SupabaseGroongaDocumentStore` using [PGroonga](https://pgroonga.github.io/), a PostgreSQL extension for fast, multilingual full-text search.
|
||||
|
||||
Unlike embedding-based retrievers, this Retriever works with plain text queries and requires no embeddings. It supports a wide range of languages out of the box through PGroonga's multilingual indexing capabilities.
|
||||
|
||||
The Retriever can be combined with `SupabasePgvectorEmbeddingRetriever` and a [`DocumentJoiner`](../joiners/documentjoiner.mdx) for hybrid search pipelines that take advantage of both keyword and semantic retrieval.
|
||||
You can also use of the [Smart Pipeline Connections](https://docs.haystack.deepset.ai/docs/smart-pipeline-connections) and skip the `DocumentJoiner` if you want to combine the results of both retrievers in a RAG pipeline.
|
||||
|
||||
In addition to `query`, the Retriever accepts optional parameters including `top_k` (the maximum number of Documents to retrieve) and `filters` to narrow the search space.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
PGroonga must be enabled in your Supabase project. Run the following SQL in the Supabase SQL editor:
|
||||
|
||||
```sql
|
||||
CREATE EXTENSION IF NOT EXISTS pgroonga;
|
||||
```
|
||||
|
||||
You also need to create a SQL function that PGroonga uses for search. See the [integration README](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/supabase/) for the required function definition.
|
||||
|
||||
## Installation
|
||||
|
||||
```shell
|
||||
pip install supabase-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
This Retriever needs the `SupabaseGroongaDocumentStore` and indexed Documents to run.
|
||||
|
||||
Set the `SUPABASE_URL` and `SUPABASE_SERVICE_KEY` environment variables for your Supabase project.
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.supabase import SupabaseGroongaDocumentStore
|
||||
from haystack_integrations.components.retrievers.supabase import (
|
||||
SupabaseGroongaBM25Retriever,
|
||||
)
|
||||
from haystack.utils import Secret
|
||||
|
||||
document_store = SupabaseGroongaDocumentStore(
|
||||
supabase_url="https://<project-ref>.supabase.co",
|
||||
supabase_key=Secret.from_env_var("SUPABASE_SERVICE_KEY"),
|
||||
table_name="haystack_groonga_documents",
|
||||
)
|
||||
|
||||
retriever = SupabaseGroongaBM25Retriever(document_store=document_store)
|
||||
|
||||
retriever.run(query="my nice query")
|
||||
```
|
||||
|
||||
### In a RAG pipeline
|
||||
|
||||
The prerequisites for running this code are:
|
||||
|
||||
- Set an environment variable `OPENAI_API_KEY` with your OpenAI API key.
|
||||
- Set an environment variable `SUPABASE_SERVICE_KEY` with your Supabase service role key.
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.builders.answer_builder import AnswerBuilder
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack.utils import Secret
|
||||
|
||||
from haystack_integrations.document_stores.supabase import SupabaseGroongaDocumentStore
|
||||
from haystack_integrations.components.retrievers.supabase import (
|
||||
SupabaseGroongaBM25Retriever,
|
||||
)
|
||||
|
||||
document_store = SupabaseGroongaDocumentStore(
|
||||
supabase_url="https://<project-ref>.supabase.co",
|
||||
supabase_key=Secret.from_env_var("SUPABASE_SERVICE_KEY"),
|
||||
table_name="haystack_groonga_documents",
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
document_store.write_documents(documents=documents, policy=DuplicatePolicy.SKIP)
|
||||
|
||||
prompt_template = [
|
||||
ChatMessage.from_user(
|
||||
"Given these documents, answer the question.\nDocuments:\n"
|
||||
"{% for doc in documents %}{{ doc.content }}{% endfor %}\n"
|
||||
"Question: {{question}}\nAnswer:",
|
||||
),
|
||||
]
|
||||
|
||||
retriever = SupabaseGroongaBM25Retriever(document_store=document_store)
|
||||
rag_pipeline = Pipeline()
|
||||
rag_pipeline.add_component(name="retriever", instance=retriever)
|
||||
rag_pipeline.add_component(
|
||||
instance=ChatPromptBuilder(
|
||||
template=prompt_template,
|
||||
required_variables={"question", "documents"},
|
||||
),
|
||||
name="prompt_builder",
|
||||
)
|
||||
rag_pipeline.add_component(instance=OpenAIChatGenerator(), name="llm")
|
||||
rag_pipeline.add_component(instance=AnswerBuilder(), name="answer_builder")
|
||||
rag_pipeline.connect("retriever", "prompt_builder.documents")
|
||||
rag_pipeline.connect("prompt_builder.prompt", "llm.messages")
|
||||
rag_pipeline.connect("llm.replies", "answer_builder.replies")
|
||||
rag_pipeline.connect("retriever", "answer_builder.documents")
|
||||
|
||||
question = "languages spoken around the world today"
|
||||
result = rag_pipeline.run(
|
||||
{
|
||||
"retriever": {"query": question},
|
||||
"prompt_builder": {"question": question},
|
||||
"answer_builder": {"query": question},
|
||||
},
|
||||
)
|
||||
print(result["answer_builder"])
|
||||
```
|
||||
+121
@@ -0,0 +1,121 @@
|
||||
---
|
||||
title: "SupabasePgvectorEmbeddingRetriever"
|
||||
id: supabasepgvectorembeddingretriever
|
||||
slug: "/supabasepgvectorembeddingretriever"
|
||||
description: "An embedding-based Retriever compatible with the SupabasePgvectorDocumentStore."
|
||||
---
|
||||
|
||||
# SupabasePgvectorEmbeddingRetriever
|
||||
|
||||
An embedding-based Retriever compatible with the SupabasePgvectorDocumentStore.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the semantic search pipeline 3. After a Text Embedder and before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a [SupabasePgvectorDocumentStore](../../document-stores/supabasedocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query_embedding`: A vector representing the query (a list of floats) |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Supabase](/reference/integrations-supabase) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/supabase |
|
||||
| **Package name** | `supabase-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`SupabasePgvectorEmbeddingRetriever` is a thin wrapper around [`PgvectorEmbeddingRetriever`](pgvectorembeddingretriever.mdx), adapted for use with `SupabasePgvectorDocumentStore`. It compares the query and Document embeddings and fetches the Documents most relevant to the query based on vector similarity.
|
||||
|
||||
When using this Retriever in your pipeline, make sure embeddings are available. Add a Document Embedder to your indexing pipeline and a Text Embedder to your query pipeline.
|
||||
|
||||
In addition to `query_embedding`, the Retriever accepts optional parameters including `top_k` (the maximum number of Documents to retrieve), `filters` to narrow down the search space, and `vector_function` to override the similarity function set on the Document Store.
|
||||
|
||||
Some relevant parameters that impact embedding retrieval must be defined when the `SupabasePgvectorDocumentStore` is initialized: `embedding_dimension`, `vector_function`, and `search_strategy` (`"exact_nearest_neighbor"` or `"hnsw"`).
|
||||
|
||||
## Installation
|
||||
|
||||
```shell
|
||||
pip install supabase-haystack
|
||||
```
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
This Retriever needs the `SupabasePgvectorDocumentStore` and indexed Documents to run.
|
||||
|
||||
Set the `SUPABASE_DB_URL` environment variable with your Supabase database connection string.
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.supabase import SupabasePgvectorDocumentStore
|
||||
from haystack_integrations.components.retrievers.supabase import (
|
||||
SupabasePgvectorEmbeddingRetriever,
|
||||
)
|
||||
|
||||
document_store = SupabasePgvectorDocumentStore(embedding_dimension=768)
|
||||
retriever = SupabasePgvectorEmbeddingRetriever(document_store=document_store)
|
||||
|
||||
# using a fake vector to keep the example simple
|
||||
retriever.run(query_embedding=[0.1] * 768)
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersTextEmbedder,
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
)
|
||||
|
||||
from haystack_integrations.document_stores.supabase import SupabasePgvectorDocumentStore
|
||||
from haystack_integrations.components.retrievers.supabase import (
|
||||
SupabasePgvectorEmbeddingRetriever,
|
||||
)
|
||||
|
||||
document_store = SupabasePgvectorDocumentStore(
|
||||
embedding_dimension=768,
|
||||
vector_function="cosine_similarity",
|
||||
recreate_table=True,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
document_embedder = SentenceTransformersDocumentEmbedder()
|
||||
documents_with_embeddings = document_embedder.run(documents)
|
||||
|
||||
document_store.write_documents(
|
||||
documents_with_embeddings.get("documents"),
|
||||
policy=DuplicatePolicy.OVERWRITE,
|
||||
)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", SentenceTransformersTextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
SupabasePgvectorEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "How many languages are there?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
```
|
||||
+135
@@ -0,0 +1,135 @@
|
||||
---
|
||||
title: "SupabasePgvectorKeywordRetriever"
|
||||
id: supabasepgvectorkeywordretriever
|
||||
slug: "/supabasepgvectorkeywordretriever"
|
||||
description: "A keyword-based Retriever that fetches documents matching a query from the SupabasePgvectorDocumentStore."
|
||||
---
|
||||
|
||||
# SupabasePgvectorKeywordRetriever
|
||||
|
||||
A keyword-based Retriever that fetches documents matching a query from the SupabasePgvectorDocumentStore.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. Before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the semantic search pipeline 3. Before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a [SupabasePgvectorDocumentStore](../../document-stores/supabasedocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query`: A string |
|
||||
| **Output variables** | `documents`: A list of documents (matching the query) |
|
||||
| **API reference** | [Supabase](/reference/integrations-supabase) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/supabase |
|
||||
| **Package name** | `supabase-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`SupabasePgvectorKeywordRetriever` is a thin wrapper around [`PgvectorKeywordRetriever`](pgvectorkeywordretriever.mdx), adapted for use with `SupabasePgvectorDocumentStore`.
|
||||
|
||||
It uses PostgreSQL full-text search (`to_tsvector` / `plainto_tsquery`) to find Documents and ranks them with the `ts_rank_cd` function. The ranking considers how often the query terms appear in the Document, how close together the terms are, and how important the part of the Document is where they occur. For more details, see the [PostgreSQL documentation](https://www.postgresql.org/docs/current/textsearch-controls.html#TEXTSEARCH-RANKING).
|
||||
|
||||
Keep in mind that, unlike similar components such as `ElasticsearchBM25Retriever`, this Retriever does not apply fuzzy search out of the box, so it's necessary to carefully formulate the query in order to avoid getting zero results.
|
||||
|
||||
The language used to parse query and Document content for keyword retrieval is set via the `language` parameter on the `SupabasePgvectorDocumentStore` (defaults to `"english"`).
|
||||
|
||||
In addition to the `query`, the Retriever accepts optional parameters including `top_k` (the maximum number of Documents to retrieve) and `filters` to narrow the search space.
|
||||
|
||||
## Installation
|
||||
|
||||
```shell
|
||||
pip install supabase-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
This Retriever needs the `SupabasePgvectorDocumentStore` and indexed Documents to run.
|
||||
|
||||
Set the `SUPABASE_DB_URL` environment variable with your Supabase database connection string.
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.supabase import SupabasePgvectorDocumentStore
|
||||
from haystack_integrations.components.retrievers.supabase import (
|
||||
SupabasePgvectorKeywordRetriever,
|
||||
)
|
||||
|
||||
document_store = SupabasePgvectorDocumentStore()
|
||||
retriever = SupabasePgvectorKeywordRetriever(document_store=document_store)
|
||||
|
||||
retriever.run(query="my nice query")
|
||||
```
|
||||
|
||||
### In a RAG pipeline
|
||||
|
||||
The prerequisites for running this code are:
|
||||
|
||||
- Set an environment variable `OPENAI_API_KEY` with your OpenAI API key.
|
||||
- Set an environment variable `SUPABASE_DB_URL` with the connection string to your Supabase database.
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.builders.answer_builder import AnswerBuilder
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
|
||||
from haystack_integrations.document_stores.supabase import SupabasePgvectorDocumentStore
|
||||
from haystack_integrations.components.retrievers.supabase import (
|
||||
SupabasePgvectorKeywordRetriever,
|
||||
)
|
||||
|
||||
prompt_template = [
|
||||
ChatMessage.from_user(
|
||||
"Given these documents, answer the question.\nDocuments:\n"
|
||||
"{% for doc in documents %}{{ doc.content }}{% endfor %}\n"
|
||||
"Question: {{question}}\nAnswer:",
|
||||
),
|
||||
]
|
||||
|
||||
document_store = SupabasePgvectorDocumentStore(
|
||||
language="english",
|
||||
recreate_table=True,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
document_store.write_documents(documents=documents, policy=DuplicatePolicy.SKIP)
|
||||
|
||||
retriever = SupabasePgvectorKeywordRetriever(document_store=document_store)
|
||||
rag_pipeline = Pipeline()
|
||||
rag_pipeline.add_component(name="retriever", instance=retriever)
|
||||
rag_pipeline.add_component(
|
||||
instance=ChatPromptBuilder(
|
||||
template=prompt_template,
|
||||
required_variables={"question", "documents"},
|
||||
),
|
||||
name="prompt_builder",
|
||||
)
|
||||
rag_pipeline.add_component(instance=OpenAIChatGenerator(), name="llm")
|
||||
rag_pipeline.add_component(instance=AnswerBuilder(), name="answer_builder")
|
||||
rag_pipeline.connect("retriever", "prompt_builder.documents")
|
||||
rag_pipeline.connect("prompt_builder.prompt", "llm.messages")
|
||||
rag_pipeline.connect("llm.replies", "answer_builder.replies")
|
||||
rag_pipeline.connect("retriever", "answer_builder.documents")
|
||||
|
||||
question = "languages spoken around the world today"
|
||||
result = rag_pipeline.run(
|
||||
{
|
||||
"retriever": {"query": question},
|
||||
"prompt_builder": {"question": question},
|
||||
"answer_builder": {"query": question},
|
||||
},
|
||||
)
|
||||
print(result["answer_builder"])
|
||||
```
|
||||
@@ -0,0 +1,115 @@
|
||||
---
|
||||
title: "TextEmbeddingRetriever"
|
||||
id: textembeddingretriever
|
||||
slug: "/textembeddingretriever"
|
||||
description: "Wraps an embedding-based retriever with a text embedder into a single component that accepts a text query."
|
||||
---
|
||||
|
||||
# TextEmbeddingRetriever
|
||||
|
||||
Wraps an embedding-based retriever with a text embedder into a single component that accepts a text query.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In query pipelines: <br />In a RAG pipeline, before a [`ChatPromptBuilder`](../builders/chatpromptbuilder.mdx) <br />In a semantic search pipeline, as the last component <br />As a retriever inside [`MultiRetriever`](multiretriever.mdx) |
|
||||
| **Mandatory init variables** | `retriever`: An embedding-based Retriever<br />`text_embedder`: A Text Embedder component |
|
||||
| **Mandatory run variables** | `query`: A query string |
|
||||
| **Output variables** | `documents`: A list of retrieved documents sorted by relevance score |
|
||||
| **API reference** | [Retrievers](/reference/retrievers-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/retrievers/text_embedding_retriever.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`TextEmbeddingRetriever` bundles a text embedder and an embedding-based retriever into a single component. It accepts a plain text query, converts it to an embedding internally, and returns documents sorted by relevance score.
|
||||
|
||||
You can use it anywhere an embedding-based retriever fits: in RAG pipelines before a prompt builder, as the final component in a semantic search pipeline, or as a drop-in retriever inside [`MultiRetriever`](multiretriever.mdx).
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
SentenceTransformersTextEmbedder,
|
||||
)
|
||||
from haystack.components.retrievers import (
|
||||
InMemoryEmbeddingRetriever,
|
||||
TextEmbeddingRetriever,
|
||||
)
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
documents = [
|
||||
Document(
|
||||
content="Renewable energy is energy that is collected from renewable resources.",
|
||||
),
|
||||
Document(
|
||||
content="Solar energy is a type of green energy that is harnessed from the sun.",
|
||||
),
|
||||
Document(
|
||||
content="Wind energy is another type of green energy that is generated by wind turbines.",
|
||||
),
|
||||
Document(
|
||||
content="Geothermal energy is heat that comes from the sub-surface of the earth.",
|
||||
),
|
||||
]
|
||||
|
||||
doc_store = InMemoryDocumentStore()
|
||||
doc_embedder = SentenceTransformersDocumentEmbedder(
|
||||
model="sentence-transformers/all-MiniLM-L6-v2",
|
||||
)
|
||||
doc_writer = DocumentWriter(document_store=doc_store, policy=DuplicatePolicy.SKIP)
|
||||
doc_writer.run(documents=doc_embedder.run(documents)["documents"])
|
||||
|
||||
retriever = TextEmbeddingRetriever(
|
||||
retriever=InMemoryEmbeddingRetriever(document_store=doc_store, top_k=2),
|
||||
text_embedder=SentenceTransformersTextEmbedder(
|
||||
model="sentence-transformers/all-MiniLM-L6-v2",
|
||||
),
|
||||
)
|
||||
|
||||
result = retriever.run(query="Geothermal energy")
|
||||
for doc in result["documents"]:
|
||||
print(f"Content: {doc.content}, Score: {doc.score}")
|
||||
```
|
||||
|
||||
### As part of MultiRetriever
|
||||
|
||||
`TextEmbeddingRetriever` is most commonly used as one of the retrievers inside a [`MultiRetriever`](multiretriever.mdx):
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersTextEmbedder,
|
||||
)
|
||||
from haystack.components.retrievers import (
|
||||
InMemoryBM25Retriever,
|
||||
InMemoryEmbeddingRetriever,
|
||||
)
|
||||
from haystack.components.retrievers import MultiRetriever, TextEmbeddingRetriever
|
||||
|
||||
retriever = MultiRetriever(
|
||||
retrievers={
|
||||
"bm25": InMemoryBM25Retriever(document_store=doc_store),
|
||||
"embedding": TextEmbeddingRetriever(
|
||||
retriever=InMemoryEmbeddingRetriever(document_store=doc_store),
|
||||
text_embedder=SentenceTransformersTextEmbedder(
|
||||
model="sentence-transformers/all-MiniLM-L6-v2",
|
||||
),
|
||||
),
|
||||
},
|
||||
)
|
||||
```
|
||||
@@ -0,0 +1,122 @@
|
||||
---
|
||||
title: "ValkeyEmbeddingRetriever"
|
||||
id: valkeyembeddingretriever
|
||||
slug: "/valkeyembeddingretriever"
|
||||
description: "This is an embedding Retriever compatible with the Valkey Document Store."
|
||||
---
|
||||
|
||||
# ValkeyEmbeddingRetriever
|
||||
|
||||
This is an embedding Retriever compatible with the Valkey Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. After a Text Embedder and before a [`ChatPromptBuilder`](../builders/chatpromptbuilder.mdx) or [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in a semantic search pipeline 3. After a Text Embedder and before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a [ValkeyDocumentStore](../../document-stores/valkeydocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query_embedding`: A list of floats |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Valkey](/reference/integrations-valkey) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/valkey |
|
||||
| **Package name** | `valkey-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `ValkeyEmbeddingRetriever` is an embedding-based Retriever compatible with the [`ValkeyDocumentStore`](../../document-stores/valkeydocumentstore.mdx). It compares the query and Document embeddings and fetches the Documents most relevant to the query from the `ValkeyDocumentStore` based on vector similarity.
|
||||
|
||||
### Parameters
|
||||
|
||||
When using the `ValkeyEmbeddingRetriever` in your system, ensure the query and Document [embeddings](../embedders.mdx) are available. You can do so by adding a Document embedder to your indexing pipeline and a text embedder to your query pipeline.
|
||||
|
||||
In addition to the `query_embedding`, the `ValkeyEmbeddingRetriever` accepts other optional parameters, including `top_k` (the maximum number of Documents to retrieve) and `filters` to narrow down the search space.
|
||||
|
||||
## Usage
|
||||
|
||||
### Installation
|
||||
|
||||
To start using Valkey with Haystack, install the package with:
|
||||
|
||||
```shell
|
||||
pip install valkey-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
This Retriever needs an instance of `ValkeyDocumentStore` and indexed Documents to run.
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.valkey import ValkeyDocumentStore
|
||||
from haystack_integrations.components.retrievers.valkey import ValkeyEmbeddingRetriever
|
||||
|
||||
document_store = ValkeyDocumentStore(
|
||||
nodes_list=[("localhost", 6379)],
|
||||
index_name="my_documents",
|
||||
embedding_dim=768,
|
||||
distance_metric="cosine",
|
||||
)
|
||||
|
||||
retriever = ValkeyEmbeddingRetriever(document_store=document_store)
|
||||
|
||||
# Using a fake vector to keep the example simple
|
||||
retriever.run(query_embedding=[0.1] * 768)
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
SentenceTransformersTextEmbedder,
|
||||
)
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack_integrations.document_stores.valkey import ValkeyDocumentStore
|
||||
from haystack_integrations.components.retrievers.valkey import ValkeyEmbeddingRetriever
|
||||
|
||||
document_store = ValkeyDocumentStore(
|
||||
nodes_list=[("localhost", 6379)],
|
||||
index_name="my_documents",
|
||||
embedding_dim=768,
|
||||
distance_metric="cosine",
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
indexing = Pipeline()
|
||||
indexing.add_component("embedder", SentenceTransformersDocumentEmbedder())
|
||||
indexing.add_component("writer", DocumentWriter(document_store))
|
||||
indexing.connect("embedder.documents", "writer.documents")
|
||||
indexing.run({"embedder": {"documents": documents}})
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", SentenceTransformersTextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
ValkeyEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "How many languages are there?"
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
```
|
||||
|
||||
For a full RAG example with `ValkeyEmbeddingRetriever`, see the [ValkeyDocumentStore](../../document-stores/valkeydocumentstore.mdx#using-valkey-in-a-rag-pipeline) documentation.
|
||||
@@ -0,0 +1,134 @@
|
||||
---
|
||||
title: "VespaEmbeddingRetriever"
|
||||
id: vespaembeddingretriever
|
||||
slug: "/vespaembeddingretriever"
|
||||
description: "An embedding-based Retriever compatible with the Vespa Document Store."
|
||||
---
|
||||
|
||||
# VespaEmbeddingRetriever
|
||||
|
||||
An embedding-based Retriever compatible with the Vespa Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the semantic search pipeline 3. After a Text Embedder and before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a [VespaDocumentStore](../../document-stores/vespadocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query_embedding`: A vector representing the query (a list of floats) |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Vespa](/reference/integrations-vespa) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/vespa |
|
||||
| **Package name** | `vespa-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `VespaEmbeddingRetriever` is a dense embedding-based Retriever compatible with the `VespaDocumentStore`. It uses Vespa's [nearest-neighbor search](https://docs.vespa.ai/en/nearest-neighbor-search.html) to find Documents whose embedding is closest to the query embedding and applies a configurable rank profile to score them.
|
||||
|
||||
When using the `VespaEmbeddingRetriever` in your Pipeline, make sure it has the query and Document embeddings available. You can do so by adding a Document Embedder to your indexing Pipeline and a Text Embedder to your query Pipeline.
|
||||
|
||||
In addition to the `query_embedding`, the `VespaEmbeddingRetriever` accepts other optional parameters, including `top_k` (the maximum number of Documents to retrieve) and `filters` to narrow down the search space.
|
||||
|
||||
The retriever expects the underlying Vespa application to expose:
|
||||
|
||||
- A tensor field for embeddings (named `embedding` by default, configurable on the Document Store via `embedding_field`).
|
||||
- A rank profile that scores nearest-neighbor candidates (named `semantic` by default, configurable via the `ranking` parameter). The profile typically uses `closeness(field, embedding)` and takes a query input tensor (named `query_embedding` by default, configurable via `query_tensor_name`).
|
||||
|
||||
You can additionally tune retrieval with `target_hits`, which sets how many neighbors each Vespa content node considers per query before first-phase ranking.
|
||||
|
||||
## Installation
|
||||
|
||||
Install the `vespa-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install vespa-haystack
|
||||
```
|
||||
|
||||
To run Vespa locally, see the [Vespa quick start](https://docs.vespa.ai/en/vespa-quick-start.html).
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
This Retriever needs the `VespaDocumentStore` and indexed Documents to run. Set the `VESPA_URL` environment variable (or pass `url=...` to the Document Store) to connect to your Vespa application.
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.vespa import VespaDocumentStore
|
||||
from haystack_integrations.components.retrievers.vespa import (
|
||||
VespaEmbeddingRetriever,
|
||||
)
|
||||
|
||||
document_store = VespaDocumentStore(schema="doc", namespace="doc")
|
||||
retriever = VespaEmbeddingRetriever(document_store=document_store)
|
||||
|
||||
## using a fake vector to keep the example simple
|
||||
retriever.run(query_embedding=[0.1] * 768)
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
SentenceTransformersTextEmbedder,
|
||||
)
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
from haystack_integrations.document_stores.vespa import VespaDocumentStore
|
||||
from haystack_integrations.components.retrievers.vespa import (
|
||||
VespaEmbeddingRetriever,
|
||||
)
|
||||
|
||||
document_store = VespaDocumentStore(
|
||||
schema="doc",
|
||||
namespace="doc",
|
||||
content_field="content",
|
||||
embedding_field="embedding",
|
||||
metadata_fields=["category"],
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(
|
||||
content="Haystack integrates with Vespa for search.",
|
||||
meta={"category": "docs"},
|
||||
),
|
||||
Document(
|
||||
content="Vespa supports lexical and vector retrieval.",
|
||||
meta={"category": "docs"},
|
||||
),
|
||||
Document(content="Cats sleep most of the day.", meta={"category": "animals"}),
|
||||
]
|
||||
|
||||
indexing = Pipeline()
|
||||
indexing.add_component("embedder", SentenceTransformersDocumentEmbedder())
|
||||
indexing.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
indexing.connect("embedder", "writer")
|
||||
indexing.run({"embedder": {"documents": documents}})
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", SentenceTransformersTextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
VespaEmbeddingRetriever(
|
||||
document_store=document_store,
|
||||
top_k=2,
|
||||
query_tensor_name="query_embedding",
|
||||
),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "semantic vector search"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
```
|
||||
@@ -0,0 +1,149 @@
|
||||
---
|
||||
title: "VespaKeywordRetriever"
|
||||
id: vespakeywordretriever
|
||||
slug: "/vespakeywordretriever"
|
||||
description: "A keyword-based Retriever that fetches documents matching a query from the Vespa Document Store."
|
||||
---
|
||||
|
||||
# VespaKeywordRetriever
|
||||
|
||||
A keyword-based Retriever that fetches documents matching a query from the Vespa Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. Before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the keyword search pipeline 3. Before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a [VespaDocumentStore](../../document-stores/vespadocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query`: A string |
|
||||
| **Output variables** | `documents`: A list of documents (matching the query) |
|
||||
| **API reference** | [Vespa](/reference/integrations-vespa) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/vespa |
|
||||
| **Package name** | `vespa-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `VespaKeywordRetriever` is a keyword-based Retriever compatible with the `VespaDocumentStore`. It runs a [YQL](https://docs.vespa.ai/en/query-language.html) `userQuery()` against your Vespa application and ranks results with a configurable rank profile (defaults to `bm25`, which typically uses Vespa's [BM25 ranking feature](https://docs.vespa.ai/en/reference/bm25.html)).
|
||||
|
||||
The retriever expects the underlying Vespa application to expose:
|
||||
|
||||
- A text field for the Document body (named `content` by default, configurable on the Document Store via `content_field`). The field needs to be indexed for text matching in your Vespa schema.
|
||||
- A rank profile that scores lexical matches (named `bm25` by default, configurable via the `ranking` parameter). Pass `ranking=None` to use the schema default profile.
|
||||
|
||||
In addition to the `query`, the `VespaKeywordRetriever` accepts other optional parameters, including `top_k` (the maximum number of Documents to retrieve) and `filters` to narrow the search space.
|
||||
|
||||
## Installation
|
||||
|
||||
Install the `vespa-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install vespa-haystack
|
||||
```
|
||||
|
||||
To run Vespa locally, see the [Vespa quick start](https://docs.vespa.ai/en/vespa-quick-start.html).
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
This Retriever needs the `VespaDocumentStore` and indexed Documents to run. Set the `VESPA_URL` environment variable (or pass `url=...` to the Document Store) to connect to your Vespa application.
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.vespa import VespaDocumentStore
|
||||
from haystack_integrations.components.retrievers.vespa import (
|
||||
VespaKeywordRetriever,
|
||||
)
|
||||
|
||||
document_store = VespaDocumentStore(schema="doc", namespace="doc")
|
||||
retriever = VespaKeywordRetriever(document_store=document_store)
|
||||
|
||||
retriever.run(query="my nice query")
|
||||
```
|
||||
|
||||
### In a RAG pipeline
|
||||
|
||||
The prerequisites necessary for running this code are:
|
||||
|
||||
- Set an environment variable `OPENAI_API_KEY` with your OpenAI API key.
|
||||
- Set the `VESPA_URL` environment variable (or pass `url=...` to the Document Store) to connect to your Vespa application.
|
||||
- A deployed Vespa schema with a `content` text field, a `category` metadata field, and a `bm25` rank profile.
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.builders.answer_builder import AnswerBuilder
|
||||
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
|
||||
from haystack_integrations.document_stores.vespa import VespaDocumentStore
|
||||
from haystack_integrations.components.retrievers.vespa import (
|
||||
VespaKeywordRetriever,
|
||||
)
|
||||
|
||||
## Create a RAG query pipeline
|
||||
prompt_template = [
|
||||
ChatMessage.from_system("You are a helpful assistant."),
|
||||
ChatMessage.from_user(
|
||||
"Given these documents, answer the question.\nDocuments:\n"
|
||||
"{% for doc in documents %}{{ doc.content }}{% endfor %}\n"
|
||||
"Question: {{question}}\nAnswer:",
|
||||
),
|
||||
]
|
||||
|
||||
document_store = VespaDocumentStore(
|
||||
schema="doc",
|
||||
namespace="doc",
|
||||
content_field="content",
|
||||
metadata_fields=["category"],
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(
|
||||
content="Haystack integrates with Vespa for search.",
|
||||
meta={"category": "docs"},
|
||||
),
|
||||
Document(
|
||||
content="Vespa supports lexical and vector retrieval.",
|
||||
meta={"category": "docs"},
|
||||
),
|
||||
Document(
|
||||
content="This note is about something else entirely.",
|
||||
meta={"category": "misc"},
|
||||
),
|
||||
]
|
||||
|
||||
document_store.write_documents(documents=documents, policy=DuplicatePolicy.OVERWRITE)
|
||||
|
||||
retriever = VespaKeywordRetriever(
|
||||
document_store=document_store,
|
||||
filters={"field": "meta.category", "operator": "==", "value": "docs"},
|
||||
)
|
||||
rag_pipeline = Pipeline()
|
||||
rag_pipeline.add_component(name="retriever", instance=retriever)
|
||||
rag_pipeline.add_component(
|
||||
instance=ChatPromptBuilder(
|
||||
template=prompt_template,
|
||||
required_variables={"question", "documents"},
|
||||
),
|
||||
name="prompt_builder",
|
||||
)
|
||||
rag_pipeline.add_component(instance=OpenAIChatGenerator(), name="llm")
|
||||
rag_pipeline.add_component(instance=AnswerBuilder(), name="answer_builder")
|
||||
rag_pipeline.connect("retriever", "prompt_builder.documents")
|
||||
rag_pipeline.connect("prompt_builder.prompt", "llm.messages")
|
||||
rag_pipeline.connect("llm.replies", "answer_builder.replies")
|
||||
rag_pipeline.connect("retriever", "answer_builder.documents")
|
||||
|
||||
question = "How does Haystack work with Vespa?"
|
||||
result = rag_pipeline.run(
|
||||
{
|
||||
"retriever": {"query": question},
|
||||
"prompt_builder": {"question": question},
|
||||
"answer_builder": {"query": question},
|
||||
},
|
||||
)
|
||||
print(result["answer_builder"])
|
||||
```
|
||||
@@ -0,0 +1,140 @@
|
||||
---
|
||||
title: "WeaviateBM25Retriever"
|
||||
id: weaviatebm25retriever
|
||||
slug: "/weaviatebm25retriever"
|
||||
description: "This is a keyword-based Retriever that fetches Documents matching a query from the Weaviate Document Store."
|
||||
---
|
||||
|
||||
# WeaviateBM25Retriever
|
||||
|
||||
This is a keyword-based Retriever that fetches Documents matching a query from the Weaviate Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. Before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the semantic search pipeline 3. Before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a [WeaviateDocumentStore](../../document-stores/weaviatedocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query`: A string |
|
||||
| **Output variables** | `documents`: A list of documents (matching the query) |
|
||||
| **API reference** | [Weaviate](/reference/integrations-weaviate) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/weaviate |
|
||||
| **Package name** | `weaviate-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`WeaviateBM25Retriever` is a keyword-based Retriever that fetches Documents matching a query from [`WeaviateDocumentStore`](../../document-stores/weaviatedocumentstore.mdx). It determines the similarity between Documents and the query based on the BM25 algorithm, which computes a weighted word overlap between the
|
||||
two strings.
|
||||
|
||||
Since the `WeaviateBM25Retriever` matches strings based on word overlap, it’s often used to find exact matches to names of persons or products, IDs, or well-defined error messages. The BM25 algorithm is very lightweight and simple. Beating it with more complex embedding-based approaches on out-of-domain data can be hard.
|
||||
|
||||
If you want a semantic match between a query and documents, use the [`WeaviateEmbeddingRetriever`](weaviateembeddingretriever.mdx), which uses vectors created by embedding models to retrieve relevant information.
|
||||
|
||||
### Parameters
|
||||
|
||||
In addition to the `query`, the `WeaviateBM25Retriever` accepts other optional parameters, including `top_k` (the maximum number of Documents to retrieve) and `filters` to narrow down the search space.
|
||||
|
||||
### Usage
|
||||
|
||||
### Installation
|
||||
|
||||
To start using Weaviate with Haystack, install the package with:
|
||||
|
||||
```shell
|
||||
pip install weaviate-haystack
|
||||
```
|
||||
|
||||
#### On its own
|
||||
|
||||
This Retriever needs an instance of `WeaviateDocumentStore` and indexed Documents to run.
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.weaviate.document_store import (
|
||||
WeaviateDocumentStore,
|
||||
)
|
||||
from haystack_integrations.components.retrievers.weaviate import WeaviateBM25Retriever
|
||||
|
||||
document_store = WeaviateDocumentStore(url="http://localhost:8080")
|
||||
|
||||
retriever = WeaviateBM25Retriever(document_store=document_store)
|
||||
|
||||
retriever.run(query="How to make a pizza", top_k=3)
|
||||
```
|
||||
|
||||
#### In a Pipeline
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.weaviate.document_store import (
|
||||
WeaviateDocumentStore,
|
||||
)
|
||||
from haystack_integrations.components.retrievers.weaviate import (
|
||||
WeaviateBM25Retriever,
|
||||
)
|
||||
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders.answer_builder import AnswerBuilder
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
|
||||
# Create a RAG query pipeline
|
||||
prompt_template = [
|
||||
ChatMessage.from_user(
|
||||
"""
|
||||
Given these documents, answer the question.\nDocuments:
|
||||
{% for doc in documents %}
|
||||
{{ doc.content }}
|
||||
{% endfor %}
|
||||
|
||||
\nQuestion: {{question}}
|
||||
\nAnswer:
|
||||
""",
|
||||
),
|
||||
]
|
||||
|
||||
document_store = WeaviateDocumentStore(url="http://localhost:8080")
|
||||
|
||||
# Add Documents
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
# DuplicatePolicy.SKIP param is optional, but useful to run the script multiple times without throwing errors
|
||||
document_store.write_documents(documents=documents, policy=DuplicatePolicy.SKIP)
|
||||
|
||||
rag_pipeline = Pipeline()
|
||||
rag_pipeline.add_component(
|
||||
name="retriever",
|
||||
instance=WeaviateBM25Retriever(document_store=document_store),
|
||||
)
|
||||
rag_pipeline.add_component(
|
||||
instance=ChatPromptBuilder(template=prompt_template, required_variables="*"),
|
||||
name="prompt_builder",
|
||||
)
|
||||
rag_pipeline.add_component(instance=OpenAIChatGenerator(), name="llm")
|
||||
rag_pipeline.add_component(instance=AnswerBuilder(), name="answer_builder")
|
||||
rag_pipeline.connect("retriever", "prompt_builder.documents")
|
||||
rag_pipeline.connect("prompt_builder.prompt", "llm.messages")
|
||||
rag_pipeline.connect("llm.replies", "answer_builder.replies")
|
||||
rag_pipeline.connect("retriever", "answer_builder.documents")
|
||||
|
||||
question = "How many languages are spoken around the world today?"
|
||||
result = rag_pipeline.run(
|
||||
{
|
||||
"retriever": {"query": question},
|
||||
"prompt_builder": {"question": question},
|
||||
"answer_builder": {"query": question},
|
||||
},
|
||||
)
|
||||
print(result["answer_builder"]["answers"][0])
|
||||
```
|
||||
@@ -0,0 +1,127 @@
|
||||
---
|
||||
title: "WeaviateEmbeddingRetriever"
|
||||
id: weaviateembeddingretriever
|
||||
slug: "/weaviateembeddingretriever"
|
||||
description: "This is an embedding Retriever compatible with the Weaviate Document Store."
|
||||
---
|
||||
|
||||
# WeaviateEmbeddingRetriever
|
||||
|
||||
This is an embedding Retriever compatible with the Weaviate Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the semantic search pipeline 3. After a Text Embedder and before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a [WeaviateDocumentStore](../../document-stores/weaviatedocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query_embedding`: A list of floats |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Weaviate](/reference/integrations-weaviate) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/weaviate |
|
||||
| **Package name** | `weaviate-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `WeaviateEmbeddingRetriever` is an embedding-based Retriever compatible with the [`WeaviateDocumentStore`](../../document-stores/weaviatedocumentstore.mdx). It compares the query and Document embeddings and fetches the Documents most relevant to the query from the `WeaviateDocumentStore` based on the outcome.
|
||||
|
||||
### Parameters
|
||||
|
||||
When using the `WeaviateEmbeddingRetriever` in your NLP system, ensure the query and Document [embeddings](../embedders.mdx) are available. You can do so by adding a Document Embedder to your indexing Pipeline and a Text Embedder to your query Pipeline.
|
||||
|
||||
In addition to the `query_embedding`, the `WeaviateEmbeddingRetriever` accepts other optional parameters, including `top_k` (the maximum number of Documents to retrieve) and `filters` to narrow down the search space.
|
||||
|
||||
You can also specify `distance`, the maximum allowed distance between embeddings, and `certainty`, the normalized distance between the result items and the search embedding. The behavior of `distance` depends on the Collection’s distance metric used. See the [official Weaviate documentation](https://weaviate.io/developers/weaviate/api/graphql/search-operators#variables) for more information.
|
||||
|
||||
The embedding similarity function depends on the vectorizer used in the `WeaviateDocumentStore` collection. Check out the [official Weaviate documentation](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules) to see all the supported vectorizers.
|
||||
|
||||
## Usage
|
||||
|
||||
### Installation
|
||||
|
||||
To start using Weaviate with Haystack, install the package with:
|
||||
|
||||
```shell
|
||||
pip install weaviate-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
This Retriever needs an instance of `WeaviateDocumentStore` and indexed Documents to run.
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.weaviate.document_store import (
|
||||
WeaviateDocumentStore,
|
||||
)
|
||||
from haystack_integrations.components.retrievers.weaviate import (
|
||||
WeaviateEmbeddingRetriever,
|
||||
)
|
||||
|
||||
document_store = WeaviateDocumentStore(url="http://localhost:8080")
|
||||
|
||||
retriever = WeaviateEmbeddingRetriever(document_store=document_store)
|
||||
|
||||
# using a fake vector to keep the example simple
|
||||
retriever.run(query_embedding=[0.1] * 768)
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
```python
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersTextEmbedder,
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
)
|
||||
|
||||
from haystack_integrations.document_stores.weaviate.document_store import (
|
||||
WeaviateDocumentStore,
|
||||
)
|
||||
from haystack_integrations.components.retrievers.weaviate import (
|
||||
WeaviateEmbeddingRetriever,
|
||||
)
|
||||
|
||||
document_store = WeaviateDocumentStore(url="http://localhost:8080")
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
document_embedder = SentenceTransformersDocumentEmbedder()
|
||||
documents_with_embeddings = document_embedder.run(documents)
|
||||
|
||||
document_store.write_documents(
|
||||
documents_with_embeddings.get("documents"),
|
||||
policy=DuplicatePolicy.OVERWRITE,
|
||||
)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", SentenceTransformersTextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
WeaviateEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "How many languages are there?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
```
|
||||
@@ -0,0 +1,165 @@
|
||||
---
|
||||
title: "WeaviateHybridRetriever"
|
||||
id: weaviatehybridretriever
|
||||
slug: "/weaviatehybridretriever"
|
||||
description: "A Retriever that combines BM25 keyword search and vector similarity to fetch documents from the Weaviate Document Store."
|
||||
---
|
||||
|
||||
# WeaviateHybridRetriever
|
||||
|
||||
A Retriever that combines BM25 keyword search and vector similarity to fetch documents from the Weaviate Document Store.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | 1. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in a hybrid search pipeline 3. After a Text Embedder and before a [`TransformersExtractiveReader`](../readers/transformersextractivereader.mdx) in an extractive QA pipeline |
|
||||
| **Mandatory init variables** | `document_store`: An instance of a [WeaviateDocumentStore](../../document-stores/weaviatedocumentstore.mdx) |
|
||||
| **Mandatory run variables** | `query`: A string <br /> <br />`query_embedding`: A list of floats |
|
||||
| **Output variables** | `documents`: A list of documents (matching the query) |
|
||||
| **API reference** | [Weaviate](/reference/integrations-weaviate) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/weaviate |
|
||||
| **Package name** | `weaviate-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `WeaviateHybridRetriever` combines keyword-based (BM25) and vector similarity search to fetch documents from the [`WeaviateDocumentStore`](../../document-stores/weaviatedocumentstore.mdx). Weaviate executes both searches in parallel and fuses the results into a single ranked list. The Retriever requires both a text query and its corresponding embedding.
|
||||
|
||||
The `alpha` parameter controls how much each search method contributes to the final results:
|
||||
|
||||
- `alpha = 0.0`: only keyword (BM25) scoring is used,
|
||||
- `alpha = 1.0`: only vector similarity scoring is used,
|
||||
- Values in between blend the two; higher values favor the vector score, lower values favor BM25.
|
||||
|
||||
If you don't specify `alpha`, the Weaviate server default is used.
|
||||
|
||||
You can also use the `max_vector_distance` parameter to set a threshold for the vector component. Candidates with a distance larger than this threshold are excluded from the vector portion before blending.
|
||||
|
||||
See the [official Weaviate documentation](https://weaviate.io/developers/weaviate/search/hybrid#parameters) for more details on hybrid search parameters.
|
||||
|
||||
### Parameters
|
||||
|
||||
When using the `WeaviateHybridRetriever`, you need to provide both the query text and its embedding. You can do this by adding a Text Embedder to your query pipeline.
|
||||
|
||||
In addition to `query` and `query_embedding`, the retriever accepts optional parameters including `top_k` (the maximum number of documents to return), `filters` to narrow down the search space, and `filter_policy` to determine how filters are applied.
|
||||
|
||||
## Usage
|
||||
|
||||
### Installation
|
||||
|
||||
To start using Weaviate with Haystack, install the package with:
|
||||
|
||||
```shell
|
||||
pip install weaviate-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
This Retriever needs an instance of `WeaviateDocumentStore` and indexed documents to run.
|
||||
|
||||
```python
|
||||
from haystack_integrations.document_stores.weaviate.document_store import (
|
||||
WeaviateDocumentStore,
|
||||
)
|
||||
from haystack_integrations.components.retrievers.weaviate import WeaviateHybridRetriever
|
||||
|
||||
document_store = WeaviateDocumentStore(url="http://localhost:8080")
|
||||
|
||||
retriever = WeaviateHybridRetriever(document_store=document_store)
|
||||
|
||||
# using a fake vector to keep the example simple
|
||||
retriever.run(query="How many languages are there?", query_embedding=[0.1] * 768)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
```python
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersTextEmbedder,
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
)
|
||||
|
||||
from haystack_integrations.document_stores.weaviate.document_store import (
|
||||
WeaviateDocumentStore,
|
||||
)
|
||||
from haystack_integrations.components.retrievers.weaviate import (
|
||||
WeaviateHybridRetriever,
|
||||
)
|
||||
|
||||
document_store = WeaviateDocumentStore(url="http://localhost:8080")
|
||||
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
|
||||
document_embedder = SentenceTransformersDocumentEmbedder()
|
||||
documents_with_embeddings = document_embedder.run(documents)
|
||||
|
||||
document_store.write_documents(
|
||||
documents_with_embeddings.get("documents"),
|
||||
policy=DuplicatePolicy.OVERWRITE,
|
||||
)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", SentenceTransformersTextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
WeaviateHybridRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "How many languages are there?"
|
||||
|
||||
result = query_pipeline.run(
|
||||
{"text_embedder": {"text": query}, "retriever": {"query": query}},
|
||||
)
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
```
|
||||
|
||||
### Adjusting the Alpha Parameter
|
||||
|
||||
You can set the `alpha` parameter at initialization or override it at query time:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.retrievers.weaviate import WeaviateHybridRetriever
|
||||
|
||||
# Favor keyword search (good for exact matches)
|
||||
retriever_keyword_heavy = WeaviateHybridRetriever(
|
||||
document_store=document_store,
|
||||
alpha=0.25,
|
||||
)
|
||||
|
||||
# Balanced hybrid search
|
||||
retriever_balanced = WeaviateHybridRetriever(document_store=document_store, alpha=0.5)
|
||||
|
||||
# Favor vector search (good for semantic similarity)
|
||||
retriever_vector_heavy = WeaviateHybridRetriever(
|
||||
document_store=document_store,
|
||||
alpha=0.75,
|
||||
)
|
||||
|
||||
# Override alpha at query time
|
||||
result = retriever_balanced.run(
|
||||
query="artificial intelligence",
|
||||
query_embedding=embedding,
|
||||
alpha=0.8,
|
||||
)
|
||||
```
|
||||
Reference in New Issue
Block a user