wehub-resource-sync
c56bef871b
Sync docs with Docusaurus / sync (push) Waiting to run
Tests / Check if changed (push) Waiting to run
Tests / format (push) Blocked by required conditions
Tests / check-imports (push) Blocked by required conditions
Tests / Unit / macos-latest (push) Blocked by required conditions
Tests / Unit / ubuntu-latest (push) Blocked by required conditions
Tests / Unit / windows-latest (push) Blocked by required conditions
Tests / mypy (push) Blocked by required conditions
Tests / Integration / ubuntu-latest (push) Blocked by required conditions
Tests / Integration / macos-latest (push) Blocked by required conditions
Tests / Integration / windows-latest (push) Blocked by required conditions
Tests / notify-slack-on-failure (push) Blocked by required conditions
Tests / Mark tests as completed (push) Blocked by required conditions
Docker image release / Build base image (push) Waiting to run
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
599 lines
18 KiB
Markdown
599 lines
18 KiB
Markdown
---
|
||
title: "Document Stores"
|
||
id: document-stores-api
|
||
description: "Stores your texts and meta data and provides them to the Retriever at query time."
|
||
slug: "/document-stores-api"
|
||
---
|
||
|
||
|
||
## document_store
|
||
|
||
### BM25DocumentStats
|
||
|
||
A dataclass for managing document statistics for BM25 retrieval.
|
||
|
||
**Parameters:**
|
||
|
||
- **freq_token** (<code>dict\[str, int\]</code>) – A Counter of token frequencies in the document.
|
||
- **doc_len** (<code>int</code>) – Number of tokens in the document.
|
||
|
||
### InMemoryDocumentStore
|
||
|
||
Stores data in-memory. It's ephemeral and cannot be saved to disk.
|
||
|
||
#### __init__
|
||
|
||
```python
|
||
__init__(
|
||
bm25_tokenization_regex: str = "(?u)\\b\\w+\\b",
|
||
bm25_algorithm: Literal["BM25Okapi", "BM25L", "BM25Plus"] = "BM25L",
|
||
bm25_parameters: dict | None = None,
|
||
embedding_similarity_function: Literal[
|
||
"dot_product", "cosine"
|
||
] = "dot_product",
|
||
index: str | None = None,
|
||
shared: bool = True,
|
||
async_executor: ThreadPoolExecutor | None = None,
|
||
return_embedding: bool = True,
|
||
) -> None
|
||
```
|
||
|
||
Initializes the DocumentStore.
|
||
|
||
**Parameters:**
|
||
|
||
- **bm25_tokenization_regex** (<code>str</code>) – The regular expression used to tokenize the text for BM25 retrieval.
|
||
- **bm25_algorithm** (<code>Literal['BM25Okapi', 'BM25L', 'BM25Plus']</code>) – The BM25 algorithm to use. One of "BM25Okapi", "BM25L", or "BM25Plus".
|
||
- **bm25_parameters** (<code>dict | None</code>) – Parameters for BM25 implementation in a dictionary format.
|
||
For example: `{'k1':1.5, 'b':0.75, 'epsilon':0.25}`
|
||
You can learn more about these parameters by visiting https://github.com/dorianbrown/rank_bm25.
|
||
- **embedding_similarity_function** (<code>Literal['dot_product', 'cosine']</code>) – The similarity function used to compare Documents embeddings.
|
||
One of "dot_product" (default) or "cosine". To choose the most appropriate function, look for information
|
||
about your embedding model.
|
||
- **index** (<code>str | None</code>) – A specific index to store the documents. If not specified, a random UUID is used.
|
||
When `shared` is True, instances using the same index share the same documents.
|
||
- **shared** (<code>bool</code>) – Whether the documents live in process-global storage shared across instances using the same
|
||
index (True, the default), or are kept instance-local and freed when this instance is garbage collected
|
||
(False). Shared storage persists for the lifetime of the process, so prefer `shared=False` for stores
|
||
that are created frequently (for example per request) to avoid unbounded memory growth.
|
||
- **async_executor** (<code>ThreadPoolExecutor | None</code>) – Optional ThreadPoolExecutor to use for async calls. If not provided, a single-threaded
|
||
executor will be initialized and used.
|
||
- **return_embedding** (<code>bool</code>) – Whether to return the embedding of the retrieved Documents. Default is True.
|
||
|
||
#### shutdown
|
||
|
||
```python
|
||
shutdown() -> None
|
||
```
|
||
|
||
Explicitly shutdown the executor if we own it.
|
||
|
||
#### storage
|
||
|
||
```python
|
||
storage: dict[str, Document]
|
||
```
|
||
|
||
Utility property that returns the storage used by this instance of InMemoryDocumentStore.
|
||
|
||
#### to_dict
|
||
|
||
```python
|
||
to_dict() -> dict[str, Any]
|
||
```
|
||
|
||
Serializes the component to a dictionary.
|
||
|
||
**Returns:**
|
||
|
||
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
|
||
|
||
#### from_dict
|
||
|
||
```python
|
||
from_dict(data: dict[str, Any]) -> InMemoryDocumentStore
|
||
```
|
||
|
||
Deserializes the component from a dictionary.
|
||
|
||
**Parameters:**
|
||
|
||
- **data** (<code>dict\[str, Any\]</code>) – The dictionary to deserialize from.
|
||
|
||
**Returns:**
|
||
|
||
- <code>InMemoryDocumentStore</code> – The deserialized component.
|
||
|
||
#### save_to_disk
|
||
|
||
```python
|
||
save_to_disk(path: str) -> None
|
||
```
|
||
|
||
Write the database and its data to disk as a JSON file.
|
||
|
||
**Parameters:**
|
||
|
||
- **path** (<code>str</code>) – The path to the JSON file.
|
||
|
||
#### load_from_disk
|
||
|
||
```python
|
||
load_from_disk(path: str) -> InMemoryDocumentStore
|
||
```
|
||
|
||
Load the database and its data from disk as a JSON file.
|
||
|
||
**Parameters:**
|
||
|
||
- **path** (<code>str</code>) – The path to the JSON file.
|
||
|
||
**Returns:**
|
||
|
||
- <code>InMemoryDocumentStore</code> – The loaded InMemoryDocumentStore.
|
||
|
||
#### count_documents
|
||
|
||
```python
|
||
count_documents() -> int
|
||
```
|
||
|
||
Returns the number of documents present in the DocumentStore.
|
||
|
||
#### filter_documents
|
||
|
||
```python
|
||
filter_documents(filters: dict[str, Any] | None = None) -> list[Document]
|
||
```
|
||
|
||
Returns the documents that match the filters provided.
|
||
|
||
**Parameters:**
|
||
|
||
- **filters** (<code>dict\[str, Any\] | None</code>) – The filters to apply. For a detailed specification of the filters, refer to the
|
||
[documentation](https://docs.haystack.deepset.ai/docs/metadata-filtering).
|
||
|
||
**Returns:**
|
||
|
||
- <code>list\[Document\]</code> – A list of Documents that match the given filters.
|
||
|
||
#### write_documents
|
||
|
||
```python
|
||
write_documents(
|
||
documents: list[Document], policy: DuplicatePolicy = DuplicatePolicy.NONE
|
||
) -> int
|
||
```
|
||
|
||
Refer to the DocumentStore.write_documents() protocol documentation.
|
||
|
||
If `policy` is set to `DuplicatePolicy.NONE` defaults to `DuplicatePolicy.FAIL`.
|
||
|
||
#### delete_documents
|
||
|
||
```python
|
||
delete_documents(document_ids: list[str]) -> None
|
||
```
|
||
|
||
Deletes all documents with matching document_ids from the DocumentStore.
|
||
|
||
**Parameters:**
|
||
|
||
- **document_ids** (<code>list\[str\]</code>) – The document_ids to delete.
|
||
|
||
#### delete_all_documents
|
||
|
||
```python
|
||
delete_all_documents() -> None
|
||
```
|
||
|
||
Deletes all documents in the document store.
|
||
|
||
#### update_by_filter
|
||
|
||
```python
|
||
update_by_filter(filters: dict[str, Any], meta: dict[str, Any]) -> int
|
||
```
|
||
|
||
Updates the metadata of all documents that match the provided filters.
|
||
|
||
**Parameters:**
|
||
|
||
- **filters** (<code>dict\[str, Any\]</code>) – The filters to apply to select documents for updating.
|
||
For filter syntax, see filter_documents.
|
||
- **meta** (<code>dict\[str, Any\]</code>) – The metadata fields to update. These will be merged with existing metadata.
|
||
|
||
**Returns:**
|
||
|
||
- <code>int</code> – The number of documents updated.
|
||
|
||
**Raises:**
|
||
|
||
- <code>ValueError</code> – if filters have invalid syntax.
|
||
|
||
#### delete_by_filter
|
||
|
||
```python
|
||
delete_by_filter(filters: dict[str, Any]) -> int
|
||
```
|
||
|
||
Deletes all documents that match the provided filters.
|
||
|
||
**Parameters:**
|
||
|
||
- **filters** (<code>dict\[str, Any\]</code>) – The filters to apply to select documents for deletion.
|
||
For filter syntax, see filter_documents.
|
||
|
||
**Returns:**
|
||
|
||
- <code>int</code> – The number of documents deleted.
|
||
|
||
**Raises:**
|
||
|
||
- <code>ValueError</code> – if filters have invalid syntax.
|
||
|
||
#### count_documents_by_filter
|
||
|
||
```python
|
||
count_documents_by_filter(filters: dict[str, Any]) -> int
|
||
```
|
||
|
||
Returns the number of documents that match the provided filters.
|
||
|
||
**Parameters:**
|
||
|
||
- **filters** (<code>dict\[str, Any\]</code>) – The filters to apply.
|
||
For a detailed specification of the filters, refer to the
|
||
[documentation](https://docs.haystack.deepset.ai/docs/metadata-filtering).
|
||
|
||
**Returns:**
|
||
|
||
- <code>int</code> – The number of documents that match the filters.
|
||
|
||
#### count_unique_metadata_by_filter
|
||
|
||
```python
|
||
count_unique_metadata_by_filter(
|
||
filters: dict[str, Any], metadata_fields: list[str]
|
||
) -> dict[str, int]
|
||
```
|
||
|
||
Returns the number of unique values for each specified metadata field from documents matching the filters.
|
||
|
||
**Parameters:**
|
||
|
||
- **filters** (<code>dict\[str, Any\]</code>) – The filters to apply.
|
||
For a detailed specification of the filters, refer to the
|
||
[documentation](https://docs.haystack.deepset.ai/docs/metadata-filtering).
|
||
- **metadata_fields** (<code>list\[str\]</code>) – List of field names to count unique values for.
|
||
Field names can include or omit the "meta." prefix.
|
||
|
||
**Returns:**
|
||
|
||
- <code>dict\[str, int\]</code> – A dictionary mapping each metadata field name (without "meta." prefix)
|
||
to the count of its unique values among the filtered documents.
|
||
|
||
#### get_metadata_fields_info
|
||
|
||
```python
|
||
get_metadata_fields_info() -> dict[str, dict[str, str]]
|
||
```
|
||
|
||
Returns information about the metadata fields present in the stored documents.
|
||
|
||
Types are inferred from the stored values (keyword, int, float, boolean).
|
||
|
||
**Returns:**
|
||
|
||
- <code>dict\[str, dict\[str, str\]\]</code> – A dictionary mapping each metadata field name to a dict with a "type" key.
|
||
|
||
#### get_metadata_field_min_max
|
||
|
||
```python
|
||
get_metadata_field_min_max(metadata_field: str) -> dict[str, Any]
|
||
```
|
||
|
||
Returns the minimum and maximum values for the given metadata field across all documents.
|
||
|
||
**Parameters:**
|
||
|
||
- **metadata_field** (<code>str</code>) – The metadata field name. Can include or omit the "meta." prefix.
|
||
|
||
**Returns:**
|
||
|
||
- <code>dict\[str, Any\]</code> – A dictionary with "min" and "max" keys. Returns `{"min": None, "max": None}`
|
||
if the field is missing or has no values.
|
||
|
||
#### get_metadata_field_unique_values
|
||
|
||
```python
|
||
get_metadata_field_unique_values(
|
||
metadata_field: str, search_term: str | None = None
|
||
) -> tuple[list[str], int]
|
||
```
|
||
|
||
Returns unique values for a metadata field, optionally filtered by a search term in content.
|
||
|
||
**Parameters:**
|
||
|
||
- **metadata_field** (<code>str</code>) – The metadata field name. Can include or omit the "meta." prefix.
|
||
- **search_term** (<code>str | None</code>) – If set, only documents whose content contains this term (case-insensitive)
|
||
are considered.
|
||
|
||
**Returns:**
|
||
|
||
- <code>tuple\[list\[str\], int\]</code> – A tuple of (list of unique values, total count of unique values).
|
||
|
||
#### bm25_retrieval
|
||
|
||
```python
|
||
bm25_retrieval(
|
||
query: str,
|
||
filters: dict[str, Any] | None = None,
|
||
top_k: int = 10,
|
||
scale_score: bool = False,
|
||
) -> list[Document]
|
||
```
|
||
|
||
Retrieves documents that are most relevant to the query using BM25 algorithm.
|
||
|
||
**Parameters:**
|
||
|
||
- **query** (<code>str</code>) – The query string.
|
||
- **filters** (<code>dict\[str, Any\] | None</code>) – A dictionary with filters to narrow down the search space.
|
||
- **top_k** (<code>int</code>) – The number of top documents to retrieve. Default is 10.
|
||
- **scale_score** (<code>bool</code>) – Whether to scale the scores of the retrieved documents. Default is False.
|
||
|
||
**Returns:**
|
||
|
||
- <code>list\[Document\]</code> – A list of the top_k documents most relevant to the query.
|
||
|
||
#### embedding_retrieval
|
||
|
||
```python
|
||
embedding_retrieval(
|
||
query_embedding: list[float],
|
||
filters: dict[str, Any] | None = None,
|
||
top_k: int = 10,
|
||
scale_score: bool = False,
|
||
return_embedding: bool | None = False,
|
||
) -> list[Document]
|
||
```
|
||
|
||
Retrieves documents that are most similar to the query embedding using a vector similarity metric.
|
||
|
||
**Parameters:**
|
||
|
||
- **query_embedding** (<code>list\[float\]</code>) – Embedding of the query.
|
||
- **filters** (<code>dict\[str, Any\] | None</code>) – A dictionary with filters to narrow down the search space.
|
||
- **top_k** (<code>int</code>) – The number of top documents to retrieve. Default is 10.
|
||
- **scale_score** (<code>bool</code>) – Whether to scale the scores of the retrieved Documents. Default is False.
|
||
- **return_embedding** (<code>bool | None</code>) – Whether to return the embedding of the retrieved Documents.
|
||
If not provided, the value of the `return_embedding` parameter set at component
|
||
initialization will be used. Default is False.
|
||
|
||
**Returns:**
|
||
|
||
- <code>list\[Document\]</code> – A list of the top_k documents most relevant to the query.
|
||
|
||
**Raises:**
|
||
|
||
- <code>ValueError</code> – if filters have invalid syntax.
|
||
|
||
#### count_documents_async
|
||
|
||
```python
|
||
count_documents_async() -> int
|
||
```
|
||
|
||
Returns the number of documents present in the DocumentStore.
|
||
|
||
#### filter_documents_async
|
||
|
||
```python
|
||
filter_documents_async(filters: dict[str, Any] | None = None) -> list[Document]
|
||
```
|
||
|
||
Returns the documents that match the filters provided.
|
||
|
||
**Parameters:**
|
||
|
||
- **filters** (<code>dict\[str, Any\] | None</code>) – The filters to apply. For a detailed specification of the filters, refer to the
|
||
[documentation](https://docs.haystack.deepset.ai/docs/metadata-filtering).
|
||
|
||
**Returns:**
|
||
|
||
- <code>list\[Document\]</code> – A list of Documents that match the given filters.
|
||
|
||
#### write_documents_async
|
||
|
||
```python
|
||
write_documents_async(
|
||
documents: list[Document], policy: DuplicatePolicy = DuplicatePolicy.NONE
|
||
) -> int
|
||
```
|
||
|
||
Refer to the DocumentStore.write_documents() protocol documentation.
|
||
|
||
If `policy` is set to `DuplicatePolicy.NONE` defaults to `DuplicatePolicy.FAIL`.
|
||
|
||
#### delete_documents_async
|
||
|
||
```python
|
||
delete_documents_async(document_ids: list[str]) -> None
|
||
```
|
||
|
||
Deletes all documents with matching document_ids from the DocumentStore.
|
||
|
||
**Parameters:**
|
||
|
||
- **document_ids** (<code>list\[str\]</code>) – The document_ids to delete.
|
||
|
||
#### update_by_filter_async
|
||
|
||
```python
|
||
update_by_filter_async(filters: dict[str, Any], meta: dict[str, Any]) -> int
|
||
```
|
||
|
||
Updates the metadata of all documents that match the provided filters.
|
||
|
||
**Parameters:**
|
||
|
||
- **filters** (<code>dict\[str, Any\]</code>) – The filters to apply to select documents for updating.
|
||
For filter syntax, see filter_documents.
|
||
- **meta** (<code>dict\[str, Any\]</code>) – The metadata fields to update. These will be merged with existing metadata.
|
||
|
||
**Returns:**
|
||
|
||
- <code>int</code> – The number of documents updated.
|
||
|
||
#### count_documents_by_filter_async
|
||
|
||
```python
|
||
count_documents_by_filter_async(filters: dict[str, Any]) -> int
|
||
```
|
||
|
||
Returns the number of documents that match the provided filters.
|
||
|
||
**Parameters:**
|
||
|
||
- **filters** (<code>dict\[str, Any\]</code>) – The filters to apply.
|
||
For a detailed specification of the filters, refer to the
|
||
[documentation](https://docs.haystack.deepset.ai/docs/metadata-filtering).
|
||
|
||
**Returns:**
|
||
|
||
- <code>int</code> – The number of documents that match the filters.
|
||
|
||
#### count_unique_metadata_by_filter_async
|
||
|
||
```python
|
||
count_unique_metadata_by_filter_async(
|
||
filters: dict[str, Any], metadata_fields: list[str]
|
||
) -> dict[str, int]
|
||
```
|
||
|
||
Returns the number of unique values for each specified metadata field from documents matching the filters.
|
||
|
||
**Parameters:**
|
||
|
||
- **filters** (<code>dict\[str, Any\]</code>) – The filters to apply.
|
||
For a detailed specification of the filters, refer to the
|
||
[documentation](https://docs.haystack.deepset.ai/docs/metadata-filtering).
|
||
- **metadata_fields** (<code>list\[str\]</code>) – List of field names to count unique values for.
|
||
Field names can include or omit the "meta." prefix.
|
||
|
||
**Returns:**
|
||
|
||
- <code>dict\[str, int\]</code> – A dictionary mapping each metadata field name (without "meta." prefix)
|
||
to the count of its unique values among the filtered documents.
|
||
|
||
#### get_metadata_fields_info_async
|
||
|
||
```python
|
||
get_metadata_fields_info_async() -> dict[str, dict[str, str]]
|
||
```
|
||
|
||
Returns information about the metadata fields present in the stored documents.
|
||
|
||
Types are inferred from the stored values (keyword, int, float, boolean).
|
||
|
||
**Returns:**
|
||
|
||
- <code>dict\[str, dict\[str, str\]\]</code> – A dictionary mapping each metadata field name to a dict with a "type" key.
|
||
|
||
#### get_metadata_field_min_max_async
|
||
|
||
```python
|
||
get_metadata_field_min_max_async(metadata_field: str) -> dict[str, Any]
|
||
```
|
||
|
||
Returns the minimum and maximum values for the given metadata field across all documents.
|
||
|
||
**Parameters:**
|
||
|
||
- **metadata_field** (<code>str</code>) – The metadata field name. Can include or omit the "meta." prefix.
|
||
|
||
**Returns:**
|
||
|
||
- <code>dict\[str, Any\]</code> – A dictionary with "min" and "max" keys. Returns `{"min": None, "max": None}`
|
||
if the field is missing or has no values.
|
||
|
||
#### get_metadata_field_unique_values_async
|
||
|
||
```python
|
||
get_metadata_field_unique_values_async(
|
||
metadata_field: str, search_term: str | None = None
|
||
) -> tuple[list[str], int]
|
||
```
|
||
|
||
Returns unique values for a metadata field, optionally filtered by a search term in content.
|
||
|
||
**Parameters:**
|
||
|
||
- **metadata_field** (<code>str</code>) – The metadata field name. Can include or omit the "meta." prefix.
|
||
- **search_term** (<code>str | None</code>) – If set, only documents whose content contains this term (case-insensitive)
|
||
are considered.
|
||
|
||
**Returns:**
|
||
|
||
- <code>tuple\[list\[str\], int\]</code> – A tuple of (list of unique values, total count of unique values).
|
||
|
||
#### delete_all_documents_async
|
||
|
||
```python
|
||
delete_all_documents_async() -> None
|
||
```
|
||
|
||
Deletes all documents in the document store.
|
||
|
||
#### bm25_retrieval_async
|
||
|
||
```python
|
||
bm25_retrieval_async(
|
||
query: str,
|
||
filters: dict[str, Any] | None = None,
|
||
top_k: int = 10,
|
||
scale_score: bool = False,
|
||
) -> list[Document]
|
||
```
|
||
|
||
Retrieves documents that are most relevant to the query using BM25 algorithm.
|
||
|
||
**Parameters:**
|
||
|
||
- **query** (<code>str</code>) – The query string.
|
||
- **filters** (<code>dict\[str, Any\] | None</code>) – A dictionary with filters to narrow down the search space.
|
||
- **top_k** (<code>int</code>) – The number of top documents to retrieve. Default is 10.
|
||
- **scale_score** (<code>bool</code>) – Whether to scale the scores of the retrieved documents. Default is False.
|
||
|
||
**Returns:**
|
||
|
||
- <code>list\[Document\]</code> – A list of the top_k documents most relevant to the query.
|
||
|
||
#### embedding_retrieval_async
|
||
|
||
```python
|
||
embedding_retrieval_async(
|
||
query_embedding: list[float],
|
||
filters: dict[str, Any] | None = None,
|
||
top_k: int = 10,
|
||
scale_score: bool = False,
|
||
return_embedding: bool = False,
|
||
) -> list[Document]
|
||
```
|
||
|
||
Retrieves documents that are most similar to the query embedding using a vector similarity metric.
|
||
|
||
**Parameters:**
|
||
|
||
- **query_embedding** (<code>list\[float\]</code>) – Embedding of the query.
|
||
- **filters** (<code>dict\[str, Any\] | None</code>) – A dictionary with filters to narrow down the search space.
|
||
- **top_k** (<code>int</code>) – The number of top documents to retrieve. Default is 10.
|
||
- **scale_score** (<code>bool</code>) – Whether to scale the scores of the retrieved Documents. Default is False.
|
||
- **return_embedding** (<code>bool</code>) – Whether to return the embedding of the retrieved Documents. Default is False.
|
||
|
||
**Returns:**
|
||
|
||
- <code>list\[Document\]</code> – A list of the top_k documents most relevant to the query.
|