chore: import upstream snapshot with attribution

docs-website/versioned_docs/version-2.26/concepts/document-store/choosing-a-document-store.mdx

@@ -0,0 +1,109 @@
+---
+title: "Choosing a Document Store"
+id: choosing-a-document-store
+slug: "/choosing-a-document-store"
+description: "This article goes through different types of Document Stores and explains their advantages and disadvantages."
+---
+
+import ClickableImage from "@site/src/components/ClickableImage";
+
+# Choosing a Document Store
+
+This article goes through different types of Document Stores and explains their advantages and disadvantages.
+
+### Introduction
+
+Whether you are developing a chatbot, a RAG system, or an image captioner, at some point, it’ll be likely for your AI application to compare the input it gets with the information it already knows. Most of the time, this comparison is performed through vector similarity search.
+
+If you’re unfamiliar with vectors, think about them as a way to represent text, images, or audio/video in a numerical form called vector embeddings. Vector databases are specifically designed to store such vectors efficiently, providing all the functionalities an AI application needs to implement data retrieval and similarity search.
+
+Document Stores are special objects in Haystack that abstract all the different vector databases into a common interface that can be easily integrated into a pipeline, most commonly through a Retriever component. Normally, you will find specialized Document Store and Retriever objects for each vector database Haystack supports.
+
+### Types of vector databases
+
+But why are vector databases so different, and which one should you use in your Haystack pipeline?
+
+We can group vector databases into five categories, from more specialized to general purpose:
+
+- Vector libraries
+- Pure vector databases
+- Vector-capable SQL databases
+- Vector-capable NoSQL databases
+- Full-text search databases
+
+We are working on supporting all these types in Haystack.
+
+In the meantime, here’s the most recent overview of available integrations:
+<ClickableImage src="/img/2c188e9-2.0_Document_Stores_6.png" alt="Document store categories diagram showing four types: pure vector databases (Chroma, Milvus, Pinecone, Weaviate, Qdrant), full-text search databases (Elasticsearch, OpenSearch), vector-capable SQL databases (Pgvector for PostgreSQL), and vector-capable NoSQL databases (DataStax Astra, MongoDB, neo4j)" className="img-light-bg" />
+
+#### Summary
+
+Here is a quick summary of different Document Stores available in Haystack.
+
+Continue further down the article for a more complex explanation of the strengths and disadvantages of each type.
+
+<div className="key-value-table">
+
+|  |  |
+| --- | --- |
+| Type                     | Best for                                                                                            |
+| Vector libraries         | Managing hardware resources effectively.                                                            |
+| Pure vector DBs          | Managing lots of high-dimensional data.                                                             |
+| Vector-capable SQL DBs   | Lower maintenance costs with focus on structured data and less on vectors.                          |
+| Vector-capable NoSQL DBs | Combining vectors with structured data without the limitations of the traditional relational model. |
+| Full-text search DBs     | Superior full-text search, reliable for production.                                                 |
+| In-memory                | Fast, minimal prototypes on small datasets.                                                         |
+
+</div>
+
+#### Vector libraries
+
+Vector libraries are often included in the “vector database” category improperly, as they are limited to handling only vectors, are designed to work in-memory, and normally don’t have a clean way to store data on disk. Still, they are the way to go every time performance and speed are the top requirements for your AI application, as these libraries can use hardware resources very effectively.
+
+:::warning[In progress]
+
+We are currently developing the support for vector libraries in Haystack.
+:::
+
+#### Pure vector databases
+
+Pure vector databases, also known as just “vector databases”, offer efficient similarity search capabilities through advanced indexing techniques. Most of them support metadata, and despite a recent trend to add more text-search features on top of it, you should consider pure vector databases closer to vector libraries than a regular database. Pick a pure vector database when your application needs to manage huge amounts of high-dimensional data effectively: they are designed to be highly scalable and highly available. Most are open source, but companies usually provide them “as a service” through paid subscriptions.
+
+- [Chroma](../../document-stores/chromadocumentstore.mdx)
+- [Pinecone](../../document-stores/pinecone-document-store.mdx)
+- [Qdrant](../../document-stores/qdrant-document-store.mdx)
+- [Weaviate](../../document-stores/weaviatedocumentstore.mdx)
+- [Milvus](https://haystack.deepset.ai/integrations/milvus-document-store) (external integration)
+
+#### Vector-capable SQL databases
+
+This category is relatively small but growing fast and includes well-known relational databases where vector capabilities were added through plugins or extensions. They are not as performant as the previous categories, but the main advantage of these databases is the opportunity to easily combine vectors with structured data, having a one-stop data shop for your application. You should pick a vector-capable SQL database when the performance trade-off is paid off by the lower cost of maintaining a single database instance for your application or when the structured data plays a more fundamental role in your business logic, with vectors being more of a nice-to-have.
+
+- [Pgvector](../../document-stores/pgvectordocumentstore.mdx)
+
+#### Vector-capable NoSQL databases
+
+Historically, the killer features of NoSQL databases were the ability to scale horizontally and the adoption of a flexible data model to overcome certain limitations of the traditional relational model. This stays true for databases in this category, where the vector capabilities are added on top of the existing features. Similarly to the previous category, vector support might not be as good as pure vector databases, but once again, there is a tradeoff that might be convenient to bear depending on the use case. For example, if a certain NoSQL database is already part of the stack of your application and a lower performance is not a show-stopper, you might give it a shot.
+
+- [Astra](../../document-stores/astradocumentstore.mdx)
+- [MongoDB](../../document-stores/mongodbatlasdocumentstore.mdx)
+- [Neo4j](https://haystack.deepset.ai/integrations/neo4j-document-store) (external)
+
+#### Full-text search databases
+
+The main advantage of full-text search databases is they are already designed to work with text, so you can expect a high level of support for text data along with good performance and the opportunity to scale both horizontally and vertically. Initially, vector capabilities were subpar and provided through plugins or extensions, but this is rapidly changing. You can see how the market leaders in this category have recently added first-class support for vectors. Pick a full-text search database if text data plays a central role in your business logic so that you can easily and effectively implement techniques like hybrid search with a good level of support for similarity search and state-of-the-art support for full-text search.
+
+- [Elasticsearch](../../document-stores/elasticsearch-document-store.mdx)
+- [OpenSearch](../../document-stores/opensearch-document-store.mdx)
+
+#### The in-memory Document Store
+
+Haystack ships with an ephemeral document store that relies on pure Python data structures stored in memory, so it doesn’t fall into any of the vector database categories above. This special Document Store is ideal for creating quick prototypes with small datasets. It doesn’t require any special setup, and it can be used right away without installing additional dependencies.
+
+- [InMemory](../../document-stores/inmemorydocumentstore.mdx)
+
+### Final considerations
+
+It can be very challenging to pick one vector database over another by only looking at pure performance, as even the slightest difference in the benchmark can produce a different leaderboard (for example, some benchmarks test the cloud services while others work on a reference machine). Thinking about including features like filtering or not can bring in a whole new set of complexities that make the comparison even harder.
+
+What’s important for you to know is that the Document Store interface doesn’t add much to the costs, and the relative performance of one vector database over another should stay the same when used within Haystack pipelines.

docs-website/versioned_docs/version-2.26/concepts/document-store/creating-custom-document-stores.mdx

@@ -0,0 +1,174 @@
+---
+title: "Creating Custom Document Stores"
+id: creating-custom-document-stores
+slug: "/creating-custom-document-stores"
+description: "Create your own Document Stores to manage your documents."
+---
+
+# Creating Custom Document Stores
+
+Create your own Document Stores to manage your documents.
+
+Custom Document Stores are resources that you can build and leverage in situations where a ready-made solution is not available in Haystack. For example:
+
+- You’re working with a vector store that’s not yet supported in Haystack.
+- You need a very specific retrieval strategy to search for your documents.
+- You want to customize the way Haystack reads and writes documents.
+
+Similar to [custom components](../components/custom-components.mdx), you can use a custom Document Store in a Haystack pipeline as long as you can import its code into your Python program. The best practice is distributing a custom Document Store as a standalone integration package.
+
+## Recommendations
+
+Before you start, there are a few recommendations we provide to ensure a custom Document Store behaves consistently with the rest of the Haystack ecosystem. At the end of the day, a Document Store is just Python code written in a way that Haystack can understand, but the way you name it, organize it, and distribute it can make a difference. None of these recommendations are mandatory, but we encourage you to follow as many as you can.
+
+### Naming Convention
+
+We recommend naming your Document Store following the format `<TECHNOLOGY>-haystack`, for example, `chroma-haystack`. This makes it consistent with the others, lowering the cognitive load for your users and easing discoverability.
+
+This naming convention applies to the name of the git repository (`https://github.com/your-org/example-haystack`) and the name of the Python package (`example-haystack`).
+
+### Structure
+
+More often than not, a Document Store can be fairly complex, and setting up a dedicated Git repository can be handy and future-proof. To ease this step, we prepared a [GitHub template](https://github.com/deepset-ai/custom-component) that provides the structure you need to host a custom Document Store in a dedicated repository. It includes the boilerplate for packaging, testing, and distributing your custom Document Store as a standalone Python package.
+
+See the instructions in the [template repository](https://github.com/deepset-ai/custom-component) to get started.
+
+### Packaging
+
+As with any other [Haystack integration](../integrations.mdx), a Document Store can be added to your Haystack applications by installing an additional Python package, for example, with `pip`. Once you have a Git repository hosting your Document Store and a `pyproject.toml` file to create an `example-haystack` package (using our [GitHub template](https://github.com/deepset-ai/custom-component)), it will be possible to `pip install` it directly from sources, for example:
+
+```shell
+pip install git+https://github.com/your-org/example-haystack.git
+```
+
+Though very practical to quickly deliver prototypes, if you want others to use your custom Document Store, we recommend you publish a package on PyPI so that it will be versioned and installable with simply:
+
+```shell
+pip install example-haystack
+```
+
+:::tip
+👍
+
+Our [GitHub template](https://github.com/deepset-ai/custom-component) ships a GitHub workflow that will automatically publish the Document Store package on PyPI.
+:::
+
+### Documentation
+
+We recommend thoroughly documenting your custom Document Store with a detailed README file and possibly generating API documentation using a static generator.
+
+For inspiration, see the [neo4j-haystack](https://github.com/prosto/neo4j-haystack) repository and its [documentation](https://prosto.github.io/neo4j-haystack/) pages.
+
+## Implementation
+
+### DocumentStore Protocol
+
+You can use any Python class as a Document Store, provided that it implements all the methods of the `DocumentStore` Python protocol defined in Haystack:
+
+```python
+class DocumentStore(Protocol):
+    def to_dict(self) -> Dict[str, Any]:
+        """
+        Serializes this store to a dictionary.
+        """
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "DocumentStore":
+        """
+        Deserializes the store from a dictionary.
+        """
+
+    def count_documents(self) -> int:
+        """
+        Returns the number of documents stored.
+        """
+
+    def filter_documents(
+        self,
+        filters: Optional[Dict[str, Any]] = None,
+    ) -> List[Document]:
+        """
+        Returns the documents that match the filters provided.
+        """
+
+    def write_documents(
+        self,
+        documents: List[Document],
+        policy: DuplicatePolicy = DuplicatePolicy.FAIL,
+    ) -> int:
+        """
+        Writes (or overwrites) documents into the DocumentStore, return the number of documents that was written.
+        """
+
+    def delete_documents(self, document_ids: List[str]) -> None:
+        """
+        Deletes all documents with a matching document_ids from the DocumentStore.
+        """
+```
+
+The `DocumentStore` interface supports the basic CRUD operations you would normally perform on a database or a storage system, and mostly generic components like [`DocumentWriter`](../../pipeline-components/writers/documentwriter.mdx) use it.
+
+### Additional Methods
+
+Usually, a Document Store comes with additional methods that can provide advanced search functionalities. These methods are not part of the `DocumentStore` protocol and don’t follow any particular convention. We designed it like this to provide maximum flexibility to the Document Store when using any specific features of the underlying database.
+
+Some additional methods that are not part of the `DocumentStore` protocol, but are implemented by most Document Stores in Haystack, include:
+
+```python
+def delete_all_documents(recreate_index: bool = False)
+def update_by_filter(filters: dict[str, Any], meta: dict[str, Any], refresh: bool = False) -> int:
+def delete_by_filter(filters: dict[str, Any]) -> int:
+```
+These methods are not part of the Protocol but highly recommended to implement in your custom Document Store, as users often expect them to be available.
+
+For example, Haystack wouldn’t get in the way when your Document Store defines a specific `search` method that takes a long list of parameters that only make sense in the context of a particular vector database. Normally, a [Retriever](../../pipeline-components/retrievers.mdx) component would then use this additional search method.
+
+### Retrievers
+
+To get the most out of your custom Document Store, in most cases, you would need to create one or more accompanying Retrievers that use the additional search methods mentioned above. Before proceeding and implementing your custom Retriever, it might be helpful to learn more about  [Retrievers](../../pipeline-components/retrievers.mdx) in general through the Haystack documentation.
+
+From the implementation perspective, Retrievers in Haystack are like any other custom component. For more details, refer to the [creating custom components](../components/custom-components.mdx) documentation page.
+
+Although not mandatory, we encourage you to follow more specific [naming conventions](../../pipeline-components/retrievers.mdx#naming-conventions) for your custom Retriever.
+
+### Serialization
+
+Haystack requires every component to be representable by a Python dictionary for correct serialization implementation. Some components, such as Retrievers and Writers, maintain a reference to a Document Store instance. Therefore, `DocumentStore` classes should implement the `from_dict` and `to_dict` methods. This allows to rebuild an instance after reading a pipeline from a file.
+
+For a practical example of what to serialize in a custom Document Store, consider a database client you created using an IP address and a database name. When constructing the dictionary to return in `to_dict`, you would store the IP address and the database name, not the database client instance.
+
+### Secrets Management
+
+There's a likelihood that users will need to provide sensitive data, such as passwords, API keys, or private URLs, to create a Document Store instance. This sensitive data could potentially be leaked if it's passed around in plain text.
+
+Haystack has a specific way to wrap sensitive data into special objects called Secrets. This prevents the data from being leaked during serialization roundtrips. We strongly recommend using this feature extensively for data security (better safe than sorry!).
+
+You can read more about Secret Management in Haystack [documentation](../secret-management.mdx).
+
+### Testing
+
+Haystack comes with some testing functionalities you can use in a custom Document Store. In particular, an empty class inheriting from `DocumentStoreBaseTests` would already run the standard tests that any Document Store is expected to pass in order to work properly.
+
+### Implementation Tips
+
+- The best way to learn how to write a custom Document Store is to look at the existing ones: the `InMemoryDocumentStore`, which is part of Haystack, or the [`ElasticsearchDocumentStore`](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/elasticsearch), which is a Core Integration, are good places to start.
+- When starting from scratch, it might be easier to create the four CRUD methods of the `DocumentStore` protocol one at a time and test them one at a time as well. For example:
+  1. Implement the logic for `count_documents`.
+  2. In your `test_document_store.py` module, define the test class `TestDocumentStore(CountDocumentsTest)`. Note how we only inherit from the specific testing mix-in `CountDocumentsTest`.
+  3. Make the tests pass.
+  4. Implement the logic for `write_documents`.
+  5. Change `test_document_store.py` so that your class now also derives from the `WriteDocumentsTest` mix-in: `TestDocumentStore(CountDocumentsTest, WriteDocumentsTest)`.
+  6. Keep iterating with the remaining methods.
+- Having a notebook where users can try out your Document Store in a full pipeline can really help adoption, and it’s a great source of documentation. Our [haystack-cookbook](https://github.com/deepset-ai/haystack-cookbook) repository has good visibility, and we encourage contributors to create a PR and add their own.
+
+Verifying that the implementation meets all `DocumentStoreBaseTests` [tests](https://github.com/deepset-ai/haystack/blob/main/haystack/testing/document_store.py) is the minimum requirement for a custom Document Store to be consistent with the rest of the Haystack ecosystem.
+
+But, ideally making it compatible with the ``DocumentStoreBaseExtendedTests`` tests is a good way to ensure that your Document Store meets all the common used functionalities that users expect from a Document Store, such as `delete_all_documents` or `update_by_filter`.
+
+If the technology you are using for your Document Store supports asynchronous operations, we recommend implementing `async` versions of the methods in the `DocumentStore` protocol as well. This allows users to take advantage of async features in their applications and pipelines, improving performance and scalability.
+
+## Get Featured on the Integrations Page
+
+The [Integrations web page](https://haystack.deepset.ai/integrations) makes Haystack integrations visible to the community, and it’s a great opportunity to showcase your work. Once your Document Store is usable and properly packaged, you can open a pull request in the [haystack-integrations](https://github.com/deepset-ai/haystack-integrations) GitHub repository to add an integration tile.
+
+See the [integrations documentation page](../integrations.mdx#how-do-i-showcase-my-integration) for more details.