c56bef871b
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
116 lines
4.3 KiB
Plaintext
116 lines
4.3 KiB
Plaintext
---
|
|
title: "TextEmbeddingRetriever"
|
|
id: textembeddingretriever
|
|
slug: "/textembeddingretriever"
|
|
description: "Wraps an embedding-based retriever with a text embedder into a single component that accepts a text query."
|
|
---
|
|
|
|
# TextEmbeddingRetriever
|
|
|
|
Wraps an embedding-based retriever with a text embedder into a single component that accepts a text query.
|
|
|
|
<div className="key-value-table">
|
|
|
|
| | |
|
|
| --- | --- |
|
|
| **Most common position in a pipeline** | In query pipelines: <br />In a RAG pipeline, before a [`ChatPromptBuilder`](../builders/chatpromptbuilder.mdx) <br />In a semantic search pipeline, as the last component <br />As a retriever inside [`MultiRetriever`](multiretriever.mdx) |
|
|
| **Mandatory init variables** | `retriever`: An embedding-based Retriever<br />`text_embedder`: A Text Embedder component |
|
|
| **Mandatory run variables** | `query`: A query string |
|
|
| **Output variables** | `documents`: A list of retrieved documents sorted by relevance score |
|
|
| **API reference** | [Retrievers](/reference/retrievers-api) |
|
|
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/retrievers/text_embedding_retriever.py |
|
|
| **Package name** | `haystack-ai` |
|
|
|
|
</div>
|
|
|
|
## Overview
|
|
|
|
`TextEmbeddingRetriever` bundles a text embedder and an embedding-based retriever into a single component. It accepts a plain text query, converts it to an embedding internally, and returns documents sorted by relevance score.
|
|
|
|
You can use it anywhere an embedding-based retriever fits: in RAG pipelines before a prompt builder, as the final component in a semantic search pipeline, or as a drop-in retriever inside [`MultiRetriever`](multiretriever.mdx).
|
|
|
|
## Usage
|
|
|
|
### On its own
|
|
|
|
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
|
|
|
```shell
|
|
pip install sentence-transformers-haystack
|
|
```
|
|
|
|
```python
|
|
from haystack import Document
|
|
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
|
from haystack.document_stores.types import DuplicatePolicy
|
|
from haystack_integrations.components.embedders.sentence_transformers import (
|
|
SentenceTransformersDocumentEmbedder,
|
|
SentenceTransformersTextEmbedder,
|
|
)
|
|
from haystack.components.retrievers import (
|
|
InMemoryEmbeddingRetriever,
|
|
TextEmbeddingRetriever,
|
|
)
|
|
from haystack.components.writers import DocumentWriter
|
|
|
|
documents = [
|
|
Document(
|
|
content="Renewable energy is energy that is collected from renewable resources.",
|
|
),
|
|
Document(
|
|
content="Solar energy is a type of green energy that is harnessed from the sun.",
|
|
),
|
|
Document(
|
|
content="Wind energy is another type of green energy that is generated by wind turbines.",
|
|
),
|
|
Document(
|
|
content="Geothermal energy is heat that comes from the sub-surface of the earth.",
|
|
),
|
|
]
|
|
|
|
doc_store = InMemoryDocumentStore()
|
|
doc_embedder = SentenceTransformersDocumentEmbedder(
|
|
model="sentence-transformers/all-MiniLM-L6-v2",
|
|
)
|
|
doc_writer = DocumentWriter(document_store=doc_store, policy=DuplicatePolicy.SKIP)
|
|
doc_writer.run(documents=doc_embedder.run(documents)["documents"])
|
|
|
|
retriever = TextEmbeddingRetriever(
|
|
retriever=InMemoryEmbeddingRetriever(document_store=doc_store, top_k=2),
|
|
text_embedder=SentenceTransformersTextEmbedder(
|
|
model="sentence-transformers/all-MiniLM-L6-v2",
|
|
),
|
|
)
|
|
|
|
result = retriever.run(query="Geothermal energy")
|
|
for doc in result["documents"]:
|
|
print(f"Content: {doc.content}, Score: {doc.score}")
|
|
```
|
|
|
|
### As part of MultiRetriever
|
|
|
|
`TextEmbeddingRetriever` is most commonly used as one of the retrievers inside a [`MultiRetriever`](multiretriever.mdx):
|
|
|
|
```python
|
|
from haystack_integrations.components.embedders.sentence_transformers import (
|
|
SentenceTransformersTextEmbedder,
|
|
)
|
|
from haystack.components.retrievers import (
|
|
InMemoryBM25Retriever,
|
|
InMemoryEmbeddingRetriever,
|
|
)
|
|
from haystack.components.retrievers import MultiRetriever, TextEmbeddingRetriever
|
|
|
|
retriever = MultiRetriever(
|
|
retrievers={
|
|
"bm25": InMemoryBM25Retriever(document_store=doc_store),
|
|
"embedding": TextEmbeddingRetriever(
|
|
retriever=InMemoryEmbeddingRetriever(document_store=doc_store),
|
|
text_embedder=SentenceTransformersTextEmbedder(
|
|
model="sentence-transformers/all-MiniLM-L6-v2",
|
|
),
|
|
),
|
|
},
|
|
)
|
|
```
|