chore: import upstream snapshot with attribution

docs-website/reference/haystack-api/document_stores_api.md

@@ -0,0 +1,598 @@
+---
+title: "Document Stores"
+id: document-stores-api
+description: "Stores your texts and meta data and provides them to the Retriever at query time."
+slug: "/document-stores-api"
+---
+
+
+## document_store
+
+### BM25DocumentStats
+
+A dataclass for managing document statistics for BM25 retrieval.
+
+**Parameters:**
+
+- **freq_token** (<code>dict\[str, int\]</code>) – A Counter of token frequencies in the document.
+- **doc_len** (<code>int</code>) – Number of tokens in the document.
+
+### InMemoryDocumentStore
+
+Stores data in-memory. It's ephemeral and cannot be saved to disk.
+
+#### __init__
+
+```python
+__init__(
+    bm25_tokenization_regex: str = "(?u)\\b\\w+\\b",
+    bm25_algorithm: Literal["BM25Okapi", "BM25L", "BM25Plus"] = "BM25L",
+    bm25_parameters: dict | None = None,
+    embedding_similarity_function: Literal[
+        "dot_product", "cosine"
+    ] = "dot_product",
+    index: str | None = None,
+    shared: bool = True,
+    async_executor: ThreadPoolExecutor | None = None,
+    return_embedding: bool = True,
+) -> None
+```
+
+Initializes the DocumentStore.
+
+**Parameters:**
+
+- **bm25_tokenization_regex** (<code>str</code>) – The regular expression used to tokenize the text for BM25 retrieval.
+- **bm25_algorithm** (<code>Literal['BM25Okapi', 'BM25L', 'BM25Plus']</code>) – The BM25 algorithm to use. One of "BM25Okapi", "BM25L", or "BM25Plus".
+- **bm25_parameters** (<code>dict | None</code>) – Parameters for BM25 implementation in a dictionary format.
+  For example: `{'k1':1.5, 'b':0.75, 'epsilon':0.25}`
+  You can learn more about these parameters by visiting https://github.com/dorianbrown/rank_bm25.
+- **embedding_similarity_function** (<code>Literal['dot_product', 'cosine']</code>) – The similarity function used to compare Documents embeddings.
+  One of "dot_product" (default) or "cosine". To choose the most appropriate function, look for information
+  about your embedding model.
+- **index** (<code>str | None</code>) – A specific index to store the documents. If not specified, a random UUID is used.
+  When `shared` is True, instances using the same index share the same documents.
+- **shared** (<code>bool</code>) – Whether the documents live in process-global storage shared across instances using the same
+  index (True, the default), or are kept instance-local and freed when this instance is garbage collected
+  (False). Shared storage persists for the lifetime of the process, so prefer `shared=False` for stores
+  that are created frequently (for example per request) to avoid unbounded memory growth.
+- **async_executor** (<code>ThreadPoolExecutor | None</code>) – Optional ThreadPoolExecutor to use for async calls. If not provided, a single-threaded
+  executor will be initialized and used.
+- **return_embedding** (<code>bool</code>) – Whether to return the embedding of the retrieved Documents. Default is True.
+
+#### shutdown
+
+```python
+shutdown() -> None
+```
+
+Explicitly shutdown the executor if we own it.
+
+#### storage
+
+```python
+storage: dict[str, Document]
+```
+
+Utility property that returns the storage used by this instance of InMemoryDocumentStore.
+
+#### to_dict
+
+```python
+to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+**Returns:**
+
+- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
+
+#### from_dict
+
+```python
+from_dict(data: dict[str, Any]) -> InMemoryDocumentStore
+```
+
+Deserializes the component from a dictionary.
+
+**Parameters:**
+
+- **data** (<code>dict\[str, Any\]</code>) – The dictionary to deserialize from.
+
+**Returns:**
+
+- <code>InMemoryDocumentStore</code> – The deserialized component.
+
+#### save_to_disk
+
+```python
+save_to_disk(path: str) -> None
+```
+
+Write the database and its data to disk as a JSON file.
+
+**Parameters:**
+
+- **path** (<code>str</code>) – The path to the JSON file.
+
+#### load_from_disk
+
+```python
+load_from_disk(path: str) -> InMemoryDocumentStore
+```
+
+Load the database and its data from disk as a JSON file.
+
+**Parameters:**
+
+- **path** (<code>str</code>) – The path to the JSON file.
+
+**Returns:**
+
+- <code>InMemoryDocumentStore</code> – The loaded InMemoryDocumentStore.
+
+#### count_documents
+
+```python
+count_documents() -> int
+```
+
+Returns the number of documents present in the DocumentStore.
+
+#### filter_documents
+
+```python
+filter_documents(filters: dict[str, Any] | None = None) -> list[Document]
+```
+
+Returns the documents that match the filters provided.
+
+**Parameters:**
+
+- **filters** (<code>dict\[str, Any\] | None</code>) – The filters to apply. For a detailed specification of the filters, refer to the
+  [documentation](https://docs.haystack.deepset.ai/docs/metadata-filtering).
+
+**Returns:**
+
+- <code>list\[Document\]</code> – A list of Documents that match the given filters.
+
+#### write_documents
+
+```python
+write_documents(
+    documents: list[Document], policy: DuplicatePolicy = DuplicatePolicy.NONE
+) -> int
+```
+
+Refer to the DocumentStore.write_documents() protocol documentation.
+
+If `policy` is set to `DuplicatePolicy.NONE` defaults to `DuplicatePolicy.FAIL`.
+
+#### delete_documents
+
+```python
+delete_documents(document_ids: list[str]) -> None
+```
+
+Deletes all documents with matching document_ids from the DocumentStore.
+
+**Parameters:**
+
+- **document_ids** (<code>list\[str\]</code>) – The document_ids to delete.
+
+#### delete_all_documents
+
+```python
+delete_all_documents() -> None
+```
+
+Deletes all documents in the document store.
+
+#### update_by_filter
+
+```python
+update_by_filter(filters: dict[str, Any], meta: dict[str, Any]) -> int
+```
+
+Updates the metadata of all documents that match the provided filters.
+
+**Parameters:**
+
+- **filters** (<code>dict\[str, Any\]</code>) – The filters to apply to select documents for updating.
+  For filter syntax, see filter_documents.
+- **meta** (<code>dict\[str, Any\]</code>) – The metadata fields to update. These will be merged with existing metadata.
+
+**Returns:**
+
+- <code>int</code> – The number of documents updated.
+
+**Raises:**
+
+- <code>ValueError</code> – if filters have invalid syntax.
+
+#### delete_by_filter
+
+```python
+delete_by_filter(filters: dict[str, Any]) -> int
+```
+
+Deletes all documents that match the provided filters.
+
+**Parameters:**
+
+- **filters** (<code>dict\[str, Any\]</code>) – The filters to apply to select documents for deletion.
+  For filter syntax, see filter_documents.
+
+**Returns:**
+
+- <code>int</code> – The number of documents deleted.
+
+**Raises:**
+
+- <code>ValueError</code> – if filters have invalid syntax.
+
+#### count_documents_by_filter
+
+```python
+count_documents_by_filter(filters: dict[str, Any]) -> int
+```
+
+Returns the number of documents that match the provided filters.
+
+**Parameters:**
+
+- **filters** (<code>dict\[str, Any\]</code>) – The filters to apply.
+  For a detailed specification of the filters, refer to the
+  [documentation](https://docs.haystack.deepset.ai/docs/metadata-filtering).
+
+**Returns:**
+
+- <code>int</code> – The number of documents that match the filters.
+
+#### count_unique_metadata_by_filter
+
+```python
+count_unique_metadata_by_filter(
+    filters: dict[str, Any], metadata_fields: list[str]
+) -> dict[str, int]
+```
+
+Returns the number of unique values for each specified metadata field from documents matching the filters.
+
+**Parameters:**
+
+- **filters** (<code>dict\[str, Any\]</code>) – The filters to apply.
+  For a detailed specification of the filters, refer to the
+  [documentation](https://docs.haystack.deepset.ai/docs/metadata-filtering).
+- **metadata_fields** (<code>list\[str\]</code>) – List of field names to count unique values for.
+  Field names can include or omit the "meta." prefix.
+
+**Returns:**
+
+- <code>dict\[str, int\]</code> – A dictionary mapping each metadata field name (without "meta." prefix)
+  to the count of its unique values among the filtered documents.
+
+#### get_metadata_fields_info
+
+```python
+get_metadata_fields_info() -> dict[str, dict[str, str]]
+```
+
+Returns information about the metadata fields present in the stored documents.
+
+Types are inferred from the stored values (keyword, int, float, boolean).
+
+**Returns:**
+
+- <code>dict\[str, dict\[str, str\]\]</code> – A dictionary mapping each metadata field name to a dict with a "type" key.
+
+#### get_metadata_field_min_max
+
+```python
+get_metadata_field_min_max(metadata_field: str) -> dict[str, Any]
+```
+
+Returns the minimum and maximum values for the given metadata field across all documents.
+
+**Parameters:**
+
+- **metadata_field** (<code>str</code>) – The metadata field name. Can include or omit the "meta." prefix.
+
+**Returns:**
+
+- <code>dict\[str, Any\]</code> – A dictionary with "min" and "max" keys. Returns `{"min": None, "max": None}`
+  if the field is missing or has no values.
+
+#### get_metadata_field_unique_values
+
+```python
+get_metadata_field_unique_values(
+    metadata_field: str, search_term: str | None = None
+) -> tuple[list[str], int]
+```
+
+Returns unique values for a metadata field, optionally filtered by a search term in content.
+
+**Parameters:**
+
+- **metadata_field** (<code>str</code>) – The metadata field name. Can include or omit the "meta." prefix.
+- **search_term** (<code>str | None</code>) – If set, only documents whose content contains this term (case-insensitive)
+  are considered.
+
+**Returns:**
+
+- <code>tuple\[list\[str\], int\]</code> – A tuple of (list of unique values, total count of unique values).
+
+#### bm25_retrieval
+
+```python
+bm25_retrieval(
+    query: str,
+    filters: dict[str, Any] | None = None,
+    top_k: int = 10,
+    scale_score: bool = False,
+) -> list[Document]
+```
+
+Retrieves documents that are most relevant to the query using BM25 algorithm.
+
+**Parameters:**
+
+- **query** (<code>str</code>) – The query string.
+- **filters** (<code>dict\[str, Any\] | None</code>) – A dictionary with filters to narrow down the search space.
+- **top_k** (<code>int</code>) – The number of top documents to retrieve. Default is 10.
+- **scale_score** (<code>bool</code>) – Whether to scale the scores of the retrieved documents. Default is False.
+
+**Returns:**
+
+- <code>list\[Document\]</code> – A list of the top_k documents most relevant to the query.
+
+#### embedding_retrieval
+
+```python
+embedding_retrieval(
+    query_embedding: list[float],
+    filters: dict[str, Any] | None = None,
+    top_k: int = 10,
+    scale_score: bool = False,
+    return_embedding: bool | None = False,
+) -> list[Document]
+```
+
+Retrieves documents that are most similar to the query embedding using a vector similarity metric.
+
+**Parameters:**
+
+- **query_embedding** (<code>list\[float\]</code>) – Embedding of the query.
+- **filters** (<code>dict\[str, Any\] | None</code>) – A dictionary with filters to narrow down the search space.
+- **top_k** (<code>int</code>) – The number of top documents to retrieve. Default is 10.
+- **scale_score** (<code>bool</code>) – Whether to scale the scores of the retrieved Documents. Default is False.
+- **return_embedding** (<code>bool | None</code>) – Whether to return the embedding of the retrieved Documents.
+  If not provided, the value of the `return_embedding` parameter set at component
+  initialization will be used. Default is False.
+
+**Returns:**
+
+- <code>list\[Document\]</code> – A list of the top_k documents most relevant to the query.
+
+**Raises:**
+
+- <code>ValueError</code> – if filters have invalid syntax.
+
+#### count_documents_async
+
+```python
+count_documents_async() -> int
+```
+
+Returns the number of documents present in the DocumentStore.
+
+#### filter_documents_async
+
+```python
+filter_documents_async(filters: dict[str, Any] | None = None) -> list[Document]
+```
+
+Returns the documents that match the filters provided.
+
+**Parameters:**
+
+- **filters** (<code>dict\[str, Any\] | None</code>) – The filters to apply. For a detailed specification of the filters, refer to the
+  [documentation](https://docs.haystack.deepset.ai/docs/metadata-filtering).
+
+**Returns:**
+
+- <code>list\[Document\]</code> – A list of Documents that match the given filters.
+
+#### write_documents_async
+
+```python
+write_documents_async(
+    documents: list[Document], policy: DuplicatePolicy = DuplicatePolicy.NONE
+) -> int
+```
+
+Refer to the DocumentStore.write_documents() protocol documentation.
+
+If `policy` is set to `DuplicatePolicy.NONE` defaults to `DuplicatePolicy.FAIL`.
+
+#### delete_documents_async
+
+```python
+delete_documents_async(document_ids: list[str]) -> None
+```
+
+Deletes all documents with matching document_ids from the DocumentStore.
+
+**Parameters:**
+
+- **document_ids** (<code>list\[str\]</code>) – The document_ids to delete.
+
+#### update_by_filter_async
+
+```python
+update_by_filter_async(filters: dict[str, Any], meta: dict[str, Any]) -> int
+```
+
+Updates the metadata of all documents that match the provided filters.
+
+**Parameters:**
+
+- **filters** (<code>dict\[str, Any\]</code>) – The filters to apply to select documents for updating.
+  For filter syntax, see filter_documents.
+- **meta** (<code>dict\[str, Any\]</code>) – The metadata fields to update. These will be merged with existing metadata.
+
+**Returns:**
+
+- <code>int</code> – The number of documents updated.
+
+#### count_documents_by_filter_async
+
+```python
+count_documents_by_filter_async(filters: dict[str, Any]) -> int
+```
+
+Returns the number of documents that match the provided filters.
+
+**Parameters:**
+
+- **filters** (<code>dict\[str, Any\]</code>) – The filters to apply.
+  For a detailed specification of the filters, refer to the
+  [documentation](https://docs.haystack.deepset.ai/docs/metadata-filtering).
+
+**Returns:**
+
+- <code>int</code> – The number of documents that match the filters.
+
+#### count_unique_metadata_by_filter_async
+
+```python
+count_unique_metadata_by_filter_async(
+    filters: dict[str, Any], metadata_fields: list[str]
+) -> dict[str, int]
+```
+
+Returns the number of unique values for each specified metadata field from documents matching the filters.
+
+**Parameters:**
+
+- **filters** (<code>dict\[str, Any\]</code>) – The filters to apply.
+  For a detailed specification of the filters, refer to the
+  [documentation](https://docs.haystack.deepset.ai/docs/metadata-filtering).
+- **metadata_fields** (<code>list\[str\]</code>) – List of field names to count unique values for.
+  Field names can include or omit the "meta." prefix.
+
+**Returns:**
+
+- <code>dict\[str, int\]</code> – A dictionary mapping each metadata field name (without "meta." prefix)
+  to the count of its unique values among the filtered documents.
+
+#### get_metadata_fields_info_async
+
+```python
+get_metadata_fields_info_async() -> dict[str, dict[str, str]]
+```
+
+Returns information about the metadata fields present in the stored documents.
+
+Types are inferred from the stored values (keyword, int, float, boolean).
+
+**Returns:**
+
+- <code>dict\[str, dict\[str, str\]\]</code> – A dictionary mapping each metadata field name to a dict with a "type" key.
+
+#### get_metadata_field_min_max_async
+
+```python
+get_metadata_field_min_max_async(metadata_field: str) -> dict[str, Any]
+```
+
+Returns the minimum and maximum values for the given metadata field across all documents.
+
+**Parameters:**
+
+- **metadata_field** (<code>str</code>) – The metadata field name. Can include or omit the "meta." prefix.
+
+**Returns:**
+
+- <code>dict\[str, Any\]</code> – A dictionary with "min" and "max" keys. Returns `{"min": None, "max": None}`
+  if the field is missing or has no values.
+
+#### get_metadata_field_unique_values_async
+
+```python
+get_metadata_field_unique_values_async(
+    metadata_field: str, search_term: str | None = None
+) -> tuple[list[str], int]
+```
+
+Returns unique values for a metadata field, optionally filtered by a search term in content.
+
+**Parameters:**
+
+- **metadata_field** (<code>str</code>) – The metadata field name. Can include or omit the "meta." prefix.
+- **search_term** (<code>str | None</code>) – If set, only documents whose content contains this term (case-insensitive)
+  are considered.
+
+**Returns:**
+
+- <code>tuple\[list\[str\], int\]</code> – A tuple of (list of unique values, total count of unique values).
+
+#### delete_all_documents_async
+
+```python
+delete_all_documents_async() -> None
+```
+
+Deletes all documents in the document store.
+
+#### bm25_retrieval_async
+
+```python
+bm25_retrieval_async(
+    query: str,
+    filters: dict[str, Any] | None = None,
+    top_k: int = 10,
+    scale_score: bool = False,
+) -> list[Document]
+```
+
+Retrieves documents that are most relevant to the query using BM25 algorithm.
+
+**Parameters:**
+
+- **query** (<code>str</code>) – The query string.
+- **filters** (<code>dict\[str, Any\] | None</code>) – A dictionary with filters to narrow down the search space.
+- **top_k** (<code>int</code>) – The number of top documents to retrieve. Default is 10.
+- **scale_score** (<code>bool</code>) – Whether to scale the scores of the retrieved documents. Default is False.
+
+**Returns:**
+
+- <code>list\[Document\]</code> – A list of the top_k documents most relevant to the query.
+
+#### embedding_retrieval_async
+
+```python
+embedding_retrieval_async(
+    query_embedding: list[float],
+    filters: dict[str, Any] | None = None,
+    top_k: int = 10,
+    scale_score: bool = False,
+    return_embedding: bool = False,
+) -> list[Document]
+```
+
+Retrieves documents that are most similar to the query embedding using a vector similarity metric.
+
+**Parameters:**
+
+- **query_embedding** (<code>list\[float\]</code>) – Embedding of the query.
+- **filters** (<code>dict\[str, Any\] | None</code>) – A dictionary with filters to narrow down the search space.
+- **top_k** (<code>int</code>) – The number of top documents to retrieve. Default is 10.
+- **scale_score** (<code>bool</code>) – Whether to scale the scores of the retrieved Documents. Default is False.
+- **return_embedding** (<code>bool</code>) – Whether to return the embedding of the retrieved Documents. Default is False.
+
+**Returns:**
+
+- <code>list\[Document\]</code> – A list of the top_k documents most relevant to the query.