Files
wehub-resource-sync c56bef871b
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
chore: import upstream snapshot with attribution
2026-07-13 13:22:28 +08:00

154 lines
5.8 KiB
Plaintext
Raw Permalink Blame History

This file contains invisible Unicode characters
This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: "QdrantSparseEmbeddingRetriever"
id: qdrantsparseembeddingretriever
slug: "/qdrantsparseembeddingretriever"
description: "A Retriever based on sparse embeddings, compatible with the Qdrant Document Store."
---
# QdrantSparseEmbeddingRetriever
A Retriever based on sparse embeddings, compatible with the Qdrant Document Store.
<div className="key-value-table">
| | |
| --- | --- |
| **Most common position in a pipeline** | 1\. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline <br /> <br />2. The last component in the semantic search pipeline <br /> 3. After a Text Embedder and before an [`ExtractiveReader`](../readers/extractivereader.mdx) in an extractive QA pipeline |
| **Mandatory init variables** | `document_store`: An instance of a [QdrantDocumentStore](../../document-stores/qdrant-document-store.mdx) |
| **Mandatory run variables** | `query_sparse_embedding`: A [`SparseEmbedding`](../../concepts/data-classes.mdx#sparseembedding) object containing a vectorial representation of the query |
| **Output variables** | `documents`: A list of documents |
| **API reference** | [Qdrant](/reference/integrations-qdrant) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/qdrant |
</div>
## Overview
The `QdrantSparseEmbeddingRetriever` is a Retriever based on sparse embeddings, compatible with the [`QdrantDocumentStore`](../../document-stores/qdrant-document-store.mdx).
It compares the query and document sparse embeddings and, based on the outcome, fetches the documents most relevant to the query from the `QdrantDocumentStore`.
When using the `QdrantSparseEmbeddingRetriever`, make sure it has the query and document sparse embeddings available. You can do so by adding a sparse document Embedder to your indexing pipeline and a sparse text Embedder to your query pipeline.
In addition to the `query_sparse_embedding`, the `QdrantSparseEmbeddingRetriever` accepts other optional parameters, including `top_k` (the maximum number of documents to retrieve) and `filters` to narrow down the search space.
:::note[Sparse Embedding Support]
To use Sparse Embedding support, you need to initialize the `QdrantDocumentStore` with `use_sparse_embeddings=True`, which is `False` by default.
If you want to use Document Store or collection previously created with this feature disabled, you must migrate the existing data. You can do this by taking advantage of the `migrate_to_sparse_embeddings_support` utility function.
:::
### Installation
To start using Qdrant with Haystack, first install the package with:
```shell
pip install qdrant-haystack
```
## Usage
### On its own
This Retriever needs the `QdrantDocumentStore` and indexed documents to run.
```python
from haystack_integrations.components.retrievers.qdrant import (
QdrantSparseEmbeddingRetriever,
)
from haystack_integrations.document_stores.qdrant import QdrantDocumentStore
from haystack.dataclasses import Document, SparseEmbedding
document_store = QdrantDocumentStore(
":memory:",
use_sparse_embeddings=True,
recreate_index=True,
return_embedding=True,
)
doc = Document(
content="test",
sparse_embedding=SparseEmbedding(indices=[0, 3, 5], values=[0.1, 0.5, 0.12]),
)
document_store.write_documents([doc])
retriever = QdrantSparseEmbeddingRetriever(document_store=document_store)
sparse_embedding = SparseEmbedding(indices=[0, 1, 2, 3], values=[0.1, 0.8, 0.05, 0.33])
retriever.run(query_sparse_embedding=sparse_embedding)
```
### In a pipeline
In Haystack, you can compute sparse embeddings using Fastembed Embedders.
First, install the package with:
```shell
pip install fastembed-haystack
```
Then, try out this pipeline:
```python
from haystack import Document, Pipeline
from haystack.components.writers import DocumentWriter
from haystack_integrations.components.retrievers.qdrant import (
QdrantSparseEmbeddingRetriever,
)
from haystack_integrations.document_stores.qdrant import QdrantDocumentStore
from haystack.document_stores.types import DuplicatePolicy
from haystack_integrations.components.embedders.fastembed import (
FastembedDocumentEmbedder,
FastembedTextEmbedder,
)
document_store = QdrantDocumentStore(
":memory:",
recreate_index=True,
use_sparse_embeddings=True,
)
documents = [
Document(content="My name is Wolfgang and I live in Berlin"),
Document(content="I saw a black horse running"),
Document(content="Germany has many big cities"),
Document(content="fastembed is supported by and maintained by Qdrant."),
]
sparse_document_embedder = FastembedSparseDocumentEmbedder()
writer = DocumentWriter(document_store=document_store, policy=DuplicatePolicy.OVERWRITE)
indexing_pipeline = Pipeline()
indexing_pipeline.add_component("sparse_document_embedder", sparse_document_embedder)
indexing_pipeline.add_component("writer", writer)
indexing_pipeline.connect("sparse_document_embedder", "writer")
indexing_pipeline.run({"sparse_document_embedder": {"documents": documents}})
query_pipeline = Pipeline()
query_pipeline.add_component("sparse_text_embedder", FastembedSparseTextEmbedder())
query_pipeline.add_component(
"sparse_retriever",
QdrantSparseEmbeddingRetriever(document_store=document_store),
)
query_pipeline.connect(
"sparse_text_embedder.sparse_embedding",
"sparse_retriever.query_sparse_embedding",
)
query = "Who supports fastembed?"
result = query_pipeline.run({"sparse_text_embedder": {"text": query}})
print(result["sparse_retriever"]["documents"][0]) # noqa: T201
## Document(id=...,
## content: 'fastembed is supported by and maintained by Qdrant.',
## score: 0.758..)
```
## Additional References
🧑‍🍳 Cookbook: [Sparse Embedding Retrieval with Qdrant and FastEmbed](https://haystack.deepset.ai/cookbook/sparse_embedding_retrieval)