wehub-resource-sync
c56bef871b
Sync docs with Docusaurus / sync (push) Waiting to run
Tests / Check if changed (push) Waiting to run
Tests / format (push) Blocked by required conditions
Tests / check-imports (push) Blocked by required conditions
Tests / Unit / macos-latest (push) Blocked by required conditions
Tests / Unit / ubuntu-latest (push) Blocked by required conditions
Tests / Unit / windows-latest (push) Blocked by required conditions
Tests / mypy (push) Blocked by required conditions
Tests / Integration / ubuntu-latest (push) Blocked by required conditions
Tests / Integration / macos-latest (push) Blocked by required conditions
Tests / Integration / windows-latest (push) Blocked by required conditions
Tests / notify-slack-on-failure (push) Blocked by required conditions
Tests / Mark tests as completed (push) Blocked by required conditions
Docker image release / Build base image (push) Waiting to run
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
521 lines
14 KiB
Markdown
521 lines
14 KiB
Markdown
---
|
||
title: "Astra"
|
||
id: integrations-astra
|
||
description: "Astra integration for Haystack"
|
||
slug: "/integrations-astra"
|
||
---
|
||
|
||
|
||
## haystack_integrations.components.retrievers.astra.retriever
|
||
|
||
### AstraEmbeddingRetriever
|
||
|
||
A component for retrieving documents from an AstraDocumentStore.
|
||
|
||
Usage example:
|
||
|
||
```python
|
||
from haystack_integrations.document_stores.astra import AstraDocumentStore
|
||
from haystack_integrations.components.retrievers.astra import AstraEmbeddingRetriever
|
||
|
||
document_store = AstraDocumentStore(
|
||
api_endpoint=api_endpoint,
|
||
token=token,
|
||
collection_name=collection_name,
|
||
duplicates_policy=DuplicatePolicy.SKIP,
|
||
embedding_dim=384,
|
||
)
|
||
|
||
retriever = AstraEmbeddingRetriever(document_store=document_store)
|
||
```
|
||
|
||
#### __init__
|
||
|
||
```python
|
||
__init__(
|
||
document_store: AstraDocumentStore,
|
||
filters: dict[str, Any] | None = None,
|
||
top_k: int = 10,
|
||
filter_policy: str | FilterPolicy = FilterPolicy.REPLACE,
|
||
) -> None
|
||
```
|
||
|
||
Initialize the AstraEmbeddingRetriever.
|
||
|
||
**Parameters:**
|
||
|
||
- **document_store** (<code>AstraDocumentStore</code>) – An instance of AstraDocumentStore.
|
||
- **filters** (<code>dict\[str, Any\] | None</code>) – a dictionary with filters to narrow down the search space.
|
||
- **top_k** (<code>int</code>) – the maximum number of documents to retrieve.
|
||
- **filter_policy** (<code>str | FilterPolicy</code>) – Policy to determine how filters are applied.
|
||
|
||
#### run
|
||
|
||
```python
|
||
run(
|
||
query_embedding: list[float],
|
||
filters: dict[str, Any] | None = None,
|
||
top_k: int | None = None,
|
||
) -> dict[str, list[Document]]
|
||
```
|
||
|
||
Retrieve documents from the AstraDocumentStore.
|
||
|
||
**Parameters:**
|
||
|
||
- **query_embedding** (<code>list\[float\]</code>) – floats representing the query embedding
|
||
- **filters** (<code>dict\[str, Any\] | None</code>) – Filters applied to the retrieved Documents. The way runtime filters are applied depends on
|
||
the `filter_policy` chosen at retriever initialization. See init method docstring for more
|
||
details.
|
||
- **top_k** (<code>int | None</code>) – the maximum number of documents to retrieve.
|
||
|
||
**Returns:**
|
||
|
||
- <code>dict\[str, list\[Document\]\]</code> – a dictionary with the following keys:
|
||
- `documents`: A list of documents retrieved from the AstraDocumentStore.
|
||
|
||
#### run_async
|
||
|
||
```python
|
||
run_async(
|
||
query_embedding: list[float],
|
||
filters: dict[str, Any] | None = None,
|
||
top_k: int | None = None,
|
||
) -> dict[str, list[Document]]
|
||
```
|
||
|
||
Retrieve documents from the AstraDocumentStore asynchronously.
|
||
|
||
Runs the sync search in a thread pool to avoid blocking the event loop.
|
||
|
||
**Parameters:**
|
||
|
||
- **query_embedding** (<code>list\[float\]</code>) – floats representing the query embedding
|
||
- **filters** (<code>dict\[str, Any\] | None</code>) – Filters applied to the retrieved Documents. The way runtime filters are applied depends on
|
||
the `filter_policy` chosen at retriever initialization. See init method docstring for more
|
||
details.
|
||
- **top_k** (<code>int | None</code>) – the maximum number of documents to retrieve.
|
||
|
||
**Returns:**
|
||
|
||
- <code>dict\[str, list\[Document\]\]</code> – a dictionary with the following keys:
|
||
- `documents`: A list of documents retrieved from the AstraDocumentStore.
|
||
|
||
#### to_dict
|
||
|
||
```python
|
||
to_dict() -> dict[str, Any]
|
||
```
|
||
|
||
Serializes the component to a dictionary.
|
||
|
||
**Returns:**
|
||
|
||
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
|
||
|
||
#### from_dict
|
||
|
||
```python
|
||
from_dict(data: dict[str, Any]) -> AstraEmbeddingRetriever
|
||
```
|
||
|
||
Deserializes the component from a dictionary.
|
||
|
||
**Parameters:**
|
||
|
||
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
|
||
|
||
**Returns:**
|
||
|
||
- <code>AstraEmbeddingRetriever</code> – Deserialized component.
|
||
|
||
## haystack_integrations.document_stores.astra.document_store
|
||
|
||
### AstraDocumentStore
|
||
|
||
An AstraDocumentStore document store for Haystack.
|
||
|
||
Example Usage:
|
||
|
||
```python
|
||
from haystack_integrations.document_stores.astra import AstraDocumentStore
|
||
|
||
document_store = AstraDocumentStore(
|
||
api_endpoint=api_endpoint,
|
||
token=token,
|
||
collection_name=collection_name,
|
||
duplicates_policy=DuplicatePolicy.SKIP,
|
||
embedding_dim=384,
|
||
)
|
||
```
|
||
|
||
#### __init__
|
||
|
||
```python
|
||
__init__(
|
||
api_endpoint: Secret = Secret.from_env_var("ASTRA_DB_API_ENDPOINT"),
|
||
token: Secret = Secret.from_env_var("ASTRA_DB_APPLICATION_TOKEN"),
|
||
collection_name: str = "documents",
|
||
embedding_dimension: int = 768,
|
||
duplicates_policy: DuplicatePolicy = DuplicatePolicy.NONE,
|
||
similarity: str = "cosine",
|
||
namespace: str | None = None,
|
||
) -> None
|
||
```
|
||
|
||
The connection to Astra DB is established and managed through the JSON API.
|
||
|
||
The required credentials (api endpoint and application token) can be generated
|
||
through the UI by clicking and the connect tab, and then selecting JSON API and
|
||
Generate Configuration.
|
||
|
||
**Parameters:**
|
||
|
||
- **api_endpoint** (<code>Secret</code>) – the Astra DB API endpoint.
|
||
- **token** (<code>Secret</code>) – the Astra DB application token.
|
||
- **collection_name** (<code>str</code>) – the current collection in the keyspace in the current Astra DB.
|
||
- **embedding_dimension** (<code>int</code>) – dimension of embedding vector.
|
||
- **duplicates_policy** (<code>DuplicatePolicy</code>) – handle duplicate documents based on DuplicatePolicy parameter options.
|
||
Parameter options : (`SKIP`, `OVERWRITE`, `FAIL`, `NONE`)
|
||
- `DuplicatePolicy.NONE`: Default policy, If a Document with the same ID already exists,
|
||
it is skipped and not written.
|
||
- `DuplicatePolicy.SKIP`: if a Document with the same ID already exists, it is skipped and not written.
|
||
- `DuplicatePolicy.OVERWRITE`: if a Document with the same ID already exists, it is overwritten.
|
||
- `DuplicatePolicy.FAIL`: if a Document with the same ID already exists, an error is raised.
|
||
- **similarity** (<code>str</code>) – the similarity function used to compare document vectors.
|
||
|
||
**Raises:**
|
||
|
||
- <code>ValueError</code> – if the API endpoint or token is not set.
|
||
|
||
#### index
|
||
|
||
```python
|
||
index: AstraClient
|
||
```
|
||
|
||
Return the AstraClient index, initializing it if necessary.
|
||
|
||
#### from_dict
|
||
|
||
```python
|
||
from_dict(data: dict[str, Any]) -> AstraDocumentStore
|
||
```
|
||
|
||
Deserializes the component from a dictionary.
|
||
|
||
**Parameters:**
|
||
|
||
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
|
||
|
||
**Returns:**
|
||
|
||
- <code>AstraDocumentStore</code> – Deserialized component.
|
||
|
||
#### to_dict
|
||
|
||
```python
|
||
to_dict() -> dict[str, Any]
|
||
```
|
||
|
||
Serializes the component to a dictionary.
|
||
|
||
**Returns:**
|
||
|
||
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
|
||
|
||
#### write_documents
|
||
|
||
```python
|
||
write_documents(
|
||
documents: list[Document], policy: DuplicatePolicy = DuplicatePolicy.NONE
|
||
) -> int
|
||
```
|
||
|
||
Indexes documents for later queries.
|
||
|
||
**Parameters:**
|
||
|
||
- **documents** (<code>list\[Document\]</code>) – a list of Haystack Document objects.
|
||
- **policy** (<code>DuplicatePolicy</code>) – handle duplicate documents based on DuplicatePolicy parameter options.
|
||
Parameter options : (`SKIP`, `OVERWRITE`, `FAIL`, `NONE`)
|
||
- `DuplicatePolicy.NONE`: Default policy, If a Document with the same ID already exists,
|
||
it is skipped and not written.
|
||
- `DuplicatePolicy.SKIP`: If a Document with the same ID already exists,
|
||
it is skipped and not written.
|
||
- `DuplicatePolicy.OVERWRITE`: If a Document with the same ID already exists, it is overwritten.
|
||
- `DuplicatePolicy.FAIL`: If a Document with the same ID already exists, an error is raised.
|
||
|
||
**Returns:**
|
||
|
||
- <code>int</code> – number of documents written.
|
||
|
||
**Raises:**
|
||
|
||
- <code>ValueError</code> – if the documents are not of type Document or dict.
|
||
- <code>DuplicateDocumentError</code> – if a document with the same ID already exists and policy is set to FAIL.
|
||
- <code>Exception</code> – if the document ID is not a string or if `id` and `_id` are both present in the document.
|
||
|
||
#### count_documents
|
||
|
||
```python
|
||
count_documents() -> int
|
||
```
|
||
|
||
Counts the number of documents in the document store.
|
||
|
||
**Returns:**
|
||
|
||
- <code>int</code> – the number of documents in the document store.
|
||
|
||
#### filter_documents
|
||
|
||
```python
|
||
filter_documents(filters: dict[str, Any] | None = None) -> list[Document]
|
||
```
|
||
|
||
Returns at most 1000 documents that match the filter.
|
||
|
||
**Parameters:**
|
||
|
||
- **filters** (<code>dict\[str, Any\] | None</code>) – filters to apply.
|
||
|
||
**Returns:**
|
||
|
||
- <code>list\[Document\]</code> – matching documents.
|
||
|
||
**Raises:**
|
||
|
||
- <code>AstraDocumentStoreFilterError</code> – if the filter is invalid or not supported by this class.
|
||
|
||
#### get_documents_by_id
|
||
|
||
```python
|
||
get_documents_by_id(ids: list[str]) -> list[Document]
|
||
```
|
||
|
||
Gets documents by their IDs.
|
||
|
||
**Parameters:**
|
||
|
||
- **ids** (<code>list\[str\]</code>) – the IDs of the documents to retrieve.
|
||
|
||
**Returns:**
|
||
|
||
- <code>list\[Document\]</code> – the matching documents.
|
||
|
||
#### get_document_by_id
|
||
|
||
```python
|
||
get_document_by_id(document_id: str) -> Document
|
||
```
|
||
|
||
Gets a document by its ID.
|
||
|
||
**Parameters:**
|
||
|
||
- **document_id** (<code>str</code>) – the ID to filter by
|
||
|
||
**Returns:**
|
||
|
||
- <code>Document</code> – the found document
|
||
|
||
**Raises:**
|
||
|
||
- <code>MissingDocumentError</code> – if the document is not found
|
||
|
||
#### search
|
||
|
||
```python
|
||
search(
|
||
query_embedding: list[float],
|
||
top_k: int,
|
||
filters: dict[str, Any] | None = None,
|
||
) -> list[Document]
|
||
```
|
||
|
||
Perform a search for a list of queries.
|
||
|
||
**Parameters:**
|
||
|
||
- **query_embedding** (<code>list\[float\]</code>) – a list of query embeddings.
|
||
- **top_k** (<code>int</code>) – the number of results to return.
|
||
- **filters** (<code>dict\[str, Any\] | None</code>) – filters to apply during search.
|
||
|
||
**Returns:**
|
||
|
||
- <code>list\[Document\]</code> – matching documents.
|
||
|
||
#### delete_documents
|
||
|
||
```python
|
||
delete_documents(document_ids: list[str]) -> None
|
||
```
|
||
|
||
Deletes documents from the document store.
|
||
|
||
**Parameters:**
|
||
|
||
- **document_ids** (<code>list\[str\]</code>) – IDs of the documents to delete.
|
||
|
||
**Raises:**
|
||
|
||
- <code>MissingDocumentError</code> – if no document was deleted but document IDs were provided.
|
||
|
||
#### delete_all_documents
|
||
|
||
```python
|
||
delete_all_documents() -> None
|
||
```
|
||
|
||
Deletes all documents from the document store.
|
||
|
||
#### delete_by_filter
|
||
|
||
```python
|
||
delete_by_filter(filters: dict[str, Any]) -> int
|
||
```
|
||
|
||
Deletes documents that match the provided filters.
|
||
|
||
**Parameters:**
|
||
|
||
- **filters** (<code>dict\[str, Any\]</code>) – The filters to apply to find documents to delete.
|
||
|
||
**Returns:**
|
||
|
||
- <code>int</code> – The number of documents deleted.
|
||
|
||
**Raises:**
|
||
|
||
- <code>AstraDocumentStoreFilterError</code> – if the filter is invalid or not supported.
|
||
|
||
#### update_by_filter
|
||
|
||
```python
|
||
update_by_filter(filters: dict[str, Any], meta: dict[str, Any]) -> int
|
||
```
|
||
|
||
Updates documents that match the provided filters with the given metadata.
|
||
|
||
**Parameters:**
|
||
|
||
- **filters** (<code>dict\[str, Any\]</code>) – The filters to apply to find documents to update.
|
||
- **meta** (<code>dict\[str, Any\]</code>) – The metadata fields to update. This will be merged with existing metadata.
|
||
|
||
**Returns:**
|
||
|
||
- <code>int</code> – The number of documents updated.
|
||
|
||
**Raises:**
|
||
|
||
- <code>AstraDocumentStoreFilterError</code> – if the filter is invalid or not supported.
|
||
|
||
#### count_documents_by_filter
|
||
|
||
```python
|
||
count_documents_by_filter(filters: dict[str, Any]) -> int
|
||
```
|
||
|
||
Applies a filter and counts the documents that matched it.
|
||
|
||
**Parameters:**
|
||
|
||
- **filters** (<code>dict\[str, Any\]</code>) – The filters to apply to the document list.
|
||
|
||
**Returns:**
|
||
|
||
- <code>int</code> – The number of documents that match the filter.
|
||
|
||
#### count_unique_metadata_by_filter
|
||
|
||
```python
|
||
count_unique_metadata_by_filter(
|
||
filters: dict[str, Any], metadata_fields: list[str]
|
||
) -> dict[str, int]
|
||
```
|
||
|
||
Applies a filter selecting documents and counts the unique values for each meta field of the matched documents.
|
||
|
||
**Parameters:**
|
||
|
||
- **filters** (<code>dict\[str, Any\]</code>) – The filters to apply to the document list.
|
||
- **metadata_fields** (<code>list\[str\]</code>) – The metadata fields to count unique values for.
|
||
|
||
**Returns:**
|
||
|
||
- <code>dict\[str, int\]</code> – A dictionary where the keys are the metadata field names and the values are the count of unique
|
||
values.
|
||
|
||
#### get_metadata_fields_info
|
||
|
||
```python
|
||
get_metadata_fields_info() -> dict[str, dict[str, str]]
|
||
```
|
||
|
||
Returns the metadata fields and the corresponding types.
|
||
|
||
**Returns:**
|
||
|
||
- <code>dict\[str, dict\[str, str\]\]</code> – A dictionary mapping field names to dictionaries with a `type` key.
|
||
|
||
#### get_metadata_field_min_max
|
||
|
||
```python
|
||
get_metadata_field_min_max(metadata_field: str) -> dict[str, Any]
|
||
```
|
||
|
||
For a given metadata field, find its max and min value.
|
||
|
||
**Parameters:**
|
||
|
||
- **metadata_field** (<code>str</code>) – The metadata field to inspect.
|
||
|
||
**Returns:**
|
||
|
||
- <code>dict\[str, Any\]</code> – A dictionary with `min` and `max`.
|
||
|
||
#### get_metadata_field_unique_values
|
||
|
||
```python
|
||
get_metadata_field_unique_values(
|
||
metadata_field: str,
|
||
search_term: str | None = None,
|
||
from_: int = 0,
|
||
size: int = 10,
|
||
) -> tuple[list[str], int]
|
||
```
|
||
|
||
Retrieves unique values for a field matching a search term or all possible values if no search term is given.
|
||
|
||
**Parameters:**
|
||
|
||
- **metadata_field** (<code>str</code>) – The metadata field to inspect.
|
||
- **search_term** (<code>str | None</code>) – Optional case-insensitive substring search term.
|
||
- **from\_** (<code>int</code>) – The starting index for pagination.
|
||
- **size** (<code>int</code>) – The number of values to return.
|
||
|
||
**Returns:**
|
||
|
||
- <code>tuple\[list\[str\], int\]</code> – A tuple containing the paginated values and the total count.
|
||
|
||
## haystack_integrations.document_stores.astra.errors
|
||
|
||
### AstraDocumentStoreError
|
||
|
||
Bases: <code>DocumentStoreError</code>
|
||
|
||
Parent class for all AstraDocumentStore errors.
|
||
|
||
### AstraDocumentStoreFilterError
|
||
|
||
Bases: <code>FilterError</code>
|
||
|
||
Raised when an invalid filter is passed to AstraDocumentStore.
|
||
|
||
### AstraDocumentStoreConfigError
|
||
|
||
Bases: <code>AstraDocumentStoreError</code>
|
||
|
||
Raised when an invalid configuration is passed to AstraDocumentStore.
|