chore: import upstream snapshot with attribution

docs-website/versioned_docs/version-2.24/concepts/components/custom-components.mdx

@@ -0,0 +1,176 @@
+---
+title: "Creating Custom Components"
+id: custom-components
+slug: "/custom-components"
+description: "Create your own components and use them standalone or in pipelines."
+---
+
+# Creating Custom Components
+
+Create your own components and use them standalone or in pipelines.
+
+With Haystack, you can easily create any custom components for various tasks, from filtering results to integrating with external software. You can then insert, reuse, and share these components within Haystack or even with an external audience by packaging them and submitting them to [Haystack Integrations](../integrations.mdx)!
+
+## Requirements
+
+Here are the requirements for all custom components:
+
+- `@component`: This decorator marks a class as a component, allowing it to be used in a pipeline.
+- `run()`: This is a required method in every component. It accepts input arguments and returns a `dict`. The inputs can either come from the pipeline when it’s executed, or from the output of another component when connected using `connect()`. The `run()` method should be compatible with the input/output definitions declared for the component. See an [Extended Example](#extended-example) below to check how it works.
+
+
+:::note[Avoid in-place input mutation]
+
+When building custom components, do not change the component's inputs directly. Instead, work on a copy or a new version of the input, modify that, and return it. The reason for this is that the original input values might be reused by other components or by later pipeline steps. Mutating the input directly can lead to unintended side effects and bugs in the pipeline, as other components might rely on the original input values.
+
+When only one or a few fields of the input need to be changed (for example, `meta` on a `Document`), use `dataclasses.replace()` to create a new instance with the updated fields. This is simpler and more efficient than deep-copying the whole object:
+
+```python
+from dataclasses import replace
+
+
+def run(self, documents):
+    updated = [replace(doc, meta={**doc.meta, "processed": True}) for doc in documents]
+    return {"documents": updated}
+```
+
+When you need to modify nested mutable structures, for example `list` or `dict` attributes, or update many fields of the dataclass instance, use a full deep copy instead:
+
+```python
+import copy
+
+
+def run(self, documents):
+    documents_copy = copy.deepcopy(documents)
+    # mutate documents_copy safely here
+    return {"documents": documents_copy}
+```
+
+:::
+
+
+### Inputs and Outputs
+
+Next, define the inputs and outputs for your component.
+
+#### Inputs
+
+You can choose between three input options:
+
+- `set_input_type`: This method defines or updates a single input socket for a component instance. It’s ideal for adding or modifying a specific input at runtime without affecting others. Use this when you need to dynamically set or modify a single input based on specific conditions.
+- `set_input_types`: This method allows you to define multiple input sockets at once, replacing any existing inputs. It’s useful when you know all the inputs the component will need and want to configure them in bulk. Use this when you want to define multiple inputs during initialization.
+- Declaring arguments directly in the `run()` method. Use this method when the component’s inputs are static and known at the time of class definition.
+
+#### Outputs
+
+You can choose between two output options:
+
+- `@component.output_types`: This decorator defines the output types and names at the time of class definition. The output names and types must match the `dict` returned by the `run()` method. Use this when the output types are static and known in advance. This decorator is cleaner and more readable for static components.
+- `set_output_types`: This method defines or updates multiple output sockets for a component instance at runtime. It’s useful when you need flexibility in configuring outputs dynamically. Use this when the output types need to be set at runtime for greater flexibility.
+
+## Short Example
+
+Here is an example of a simple minimal component setup:
+
+```python
+from haystack import component
+
+
+@component
+class WelcomeTextGenerator:
+    """
+    A component generating personal welcome message and making it upper case
+    """
+
+    @component.output_types(welcome_text=str, note=str)
+    def run(self, name: str):
+        return {
+            "welcome_text": f"Hello {name}, welcome to Haystack!".upper(),
+            "note": "welcome message is ready",
+        }
+```
+
+Here, the custom component `WelcomeTextGenerator` accepts one input: `name` string and returns two outputs: `welcome_text` and `note`.
+
+## Extended Example
+
+Check out an example below on how to create two custom components and connect them in a Haystack pipeline.
+
+```python
+# import necessary dependencies
+from haystack import component, Pipeline
+
+
+# Create two custom components. Note the mandatory @component decorator and @component.output_types, as well as the mandatory run method.
+@component
+class WelcomeTextGenerator:
+    """
+    A component generating personal welcome message and making it upper case
+    """
+
+    @component.output_types(welcome_text=str, note=str)
+    def run(self, name: str):
+        return {
+            "welcome_text": (
+                "Hello {name}, welcome to Haystack!".format(name=name)
+            ).upper(),
+            "note": "welcome message is ready",
+        }
+
+
+@component
+class WhitespaceSplitter:
+    """
+    A component for splitting the text by whitespace
+    """
+
+    @component.output_types(split_text=list[str])
+    def run(self, text: str):
+        return {"split_text": text.split()}
+
+
+# create a pipeline and add the custom components to it
+text_pipeline = Pipeline()
+text_pipeline.add_component(
+    name="welcome_text_generator",
+    instance=WelcomeTextGenerator(),
+)
+text_pipeline.add_component(name="splitter", instance=WhitespaceSplitter())
+
+# connect the components
+text_pipeline.connect(
+    sender="welcome_text_generator.welcome_text",
+    receiver="splitter.text",
+)
+
+# define the result and run the pipeline
+result = text_pipeline.run({"welcome_text_generator": {"name": "Bilge"}})
+
+print(result["splitter"]["split_text"])
+```
+
+## Extending the Existing Components
+
+To extend already existing components in Haystack, subclass an existing component and use the `@component` decorator to mark it. Override or extend the `run()` method to process inputs and outputs. Call `super()` with the derived class name from the init of the derived class to avoid initialization issues:
+
+```python
+class DerivedComponent(BaseComponent):
+    def __init__(self):
+        super(DerivedComponent, self).__init__()
+
+
+## ...
+
+dc = DerivedComponent()  # ok
+```
+
+An example of an extended component is Haystack's [FaithfulnessEvaluator](https://github.com/deepset-ai/haystack/blob/e5a80722c22c59eb99416bf0cd712f6de7cd581a/haystack/components/evaluators/faithfulness.py) derived from LLMEvaluator.
+
+## Additional References
+
+🧑‍🍳 Cookbooks:
+
+- [Build quizzes and adventures with Character Codex and llamafile](https://haystack.deepset.ai/cookbook/charactercodex_llamafile/)
+- [Run tasks concurrently within a custom component](https://haystack.deepset.ai/cookbook/concurrent_tasks/)
+- [Chat With Your SQL Database](https://haystack.deepset.ai/cookbook/chat_with_sql_3_ways/)
+- [Hacker News Summaries with Custom Components](https://haystack.deepset.ai/cookbook/hackernews-custom-component-rag/)

docs-website/versioned_docs/version-2.24/concepts/components/supercomponents.mdx

@@ -0,0 +1,187 @@
+---
+title: "SuperComponents"
+id: supercomponents
+slug: "/supercomponents"
+description: "`SuperComponent` lets you wrap a complete pipeline and use it like a single component. This is helpful when you want to simplify the interface of a complex pipeline, reuse it in different contexts, or expose only the necessary inputs and outputs."
+---
+
+# SuperComponents
+
+`SuperComponent` lets you wrap a complete pipeline and use it like a single component. This is helpful when you want to simplify the interface of a complex pipeline, reuse it in different contexts, or expose only the necessary inputs and outputs.
+
+## `@super_component` decorator (recommended)
+
+Haystack now provides a simple `@super_component` decorator for wrapping a pipeline as a component. All you need is to create a class with the decorator, and to include an `pipeline` attribute.
+
+With this decorator, the `to_dict` and `from_dict` serialization is optional, as is the input and output mapping.
+
+### Example
+
+The custom HybridRetriever example SuperComponent below turns your query into embeddings, then runs both a BM25 search and an embedding-based search at the same time. It finally merges those two result sets and returns the combined documents.
+
+```python
+## pip install haystack-ai datasets "sentence-transformers>=3.0.0"
+
+from haystack import Document, Pipeline, super_component
+from haystack.components.joiners import DocumentJoiner
+from haystack.components.embedders import SentenceTransformersTextEmbedder
+from haystack.components.retrievers import (
+    InMemoryBM25Retriever,
+    InMemoryEmbeddingRetriever,
+)
+from haystack.document_stores.in_memory import InMemoryDocumentStore
+
+from datasets import load_dataset
+
+
+@super_component
+class HybridRetriever:
+    def __init__(
+        self,
+        document_store: InMemoryDocumentStore,
+        embedder_model: str = "BAAI/bge-small-en-v1.5",
+    ):
+        embedding_retriever = InMemoryEmbeddingRetriever(document_store)
+        bm25_retriever = InMemoryBM25Retriever(document_store)
+        text_embedder = SentenceTransformersTextEmbedder(embedder_model)
+        document_joiner = DocumentJoiner()
+
+        self.pipeline = Pipeline()
+        self.pipeline.add_component("text_embedder", text_embedder)
+        self.pipeline.add_component("embedding_retriever", embedding_retriever)
+        self.pipeline.add_component("bm25_retriever", bm25_retriever)
+        self.pipeline.add_component("document_joiner", document_joiner)
+
+        self.pipeline.connect("text_embedder", "embedding_retriever")
+        self.pipeline.connect("bm25_retriever", "document_joiner")
+        self.pipeline.connect("embedding_retriever", "document_joiner")
+
+
+dataset = load_dataset("HaystackBot/medrag-pubmed-chunk-with-embeddings", split="train")
+docs = [
+    Document(content=doc["contents"], embedding=doc["embedding"]) for doc in dataset
+]
+document_store = InMemoryDocumentStore()
+document_store.write_documents(docs)
+
+query = "What treatments are available for chronic bronchitis?"
+
+result = HybridRetriever(document_store).run(text=query, query=query)
+print(result)
+```
+
+### Input Mapping
+
+You can optionally map the input names of your SuperComponent to the actual sockets inside the pipeline.
+
+```python
+input_mapping = {"query": ["retriever.query", "prompt.query"]}
+```
+
+### Output Mapping
+
+You can also map the pipeline's output sockets that you want to expose to the SuperComponent's output names.
+
+```python
+output_mapping = {"llm.replies": "replies"}
+```
+
+If you don’t provide mappings, SuperComponent will try to auto-detect them. So, if multiple components have outputs with the same name, we recommend using `output_mapping` to avoid conflicts.
+
+## SuperComponent class
+
+Haystack also gives you an option to inherit from SuperComponent class. This option requires `to_dict` and `from_dict` serialization, as well as the input and output mapping described above.
+
+### Example
+
+Here is a simple example of initializing a `SuperComponent` with a pipeline:
+
+```python
+from haystack import Pipeline, SuperComponent
+
+with open("pipeline.yaml", "r") as file:
+    pipeline = Pipeline.load(file)
+
+super_component = SuperComponent(pipeline)
+```
+
+The example pipeline below retrieves relevant documents based on a user query, builds a custom prompt using those documents, then sends the prompt to an `OpenAIChatGenerator` to create an answer. The `SuperComponent` wraps the pipeline so it can be run with a simple input (`query`) and returns a clean output (`replies`).
+
+```python
+from haystack import Pipeline, SuperComponent
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.components.builders import ChatPromptBuilder
+from haystack.components.retrievers import InMemoryBM25Retriever
+from haystack.dataclasses.chat_message import ChatMessage
+from haystack.document_stores.in_memory import InMemoryDocumentStore
+from haystack.dataclasses import Document
+
+document_store = InMemoryDocumentStore()
+documents = [
+    Document(content="Paris is the capital of France."),
+    Document(content="London is the capital of England."),
+]
+document_store.write_documents(documents)
+
+prompt_template = [
+    ChatMessage.from_user(
+    '''
+    According to the following documents:
+    {% for document in documents %}
+    {{document.content}}
+    {% endfor %}
+    Answer the given question: {{query}}
+    Answer:
+    '''
+    )
+]
+
+prompt_builder = ChatPromptBuilder(template=prompt_template, required_variables="*")
+
+pipeline = Pipeline()
+pipeline.add_component("retriever", InMemoryBM25Retriever(document_store=document_store))
+pipeline.add_component("prompt_builder", prompt_builder)
+pipeline.add_component("llm", OpenAIChatGenerator())
+pipeline.connect("retriever.documents", "prompt_builder.documents")
+pipeline.connect("prompt_builder.prompt", "llm.messages")
+
+## Create a super component with simplified input/output mapping
+wrapper = SuperComponent(
+    pipeline=pipeline,
+    input_mapping={
+        "query": ["retriever.query", "prompt_builder.query"],
+    },
+    output_mapping={
+        "llm.replies": "replies",
+        "retriever.documents": "documents"
+    }
+)
+
+## Run the pipeline with simplified interface
+result = wrapper.run(query="What is the capital of France?")
+print(result)
+{'replies': [ChatMessage(_role=<ChatRole.ASSISTANT: 'assistant'>,
+ _content=[TextContent(text='The capital of France is Paris.')],...)
+```
+
+## Type Checking and Static Code Analysis
+
+Creating SuperComponents using the @super_component decorator can induce type or linting errors. One way to avoid these issues is to add the exposed public methods to your SuperComponent. Here's an example:
+
+```python
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+
+    def run(self, *, documents: list[Document]) -> dict[str, list[Document]]: ...
+    def warm_up(self) -> None:  # noqa: D102
+        ...
+```
+
+## Ready-Made SuperComponents
+
+You can see two implementations of SuperComponents already integrated in Haystack:
+
+- [DocumentPreprocessor](../../pipeline-components/preprocessors/documentpreprocessor.mdx)
+- [MultiFileConverter](../../pipeline-components/converters/multifileconverter.mdx)
+- [OpenSearchHybridRetriever](../../pipeline-components/retrievers/opensearchhybridretriever.mdx)