chore: import upstream snapshot with attribution

docs-website/versioned_docs/version-2.29/concepts/agents.mdx

@@ -0,0 +1,121 @@
+---
+title: "Agents"
+id: agents
+slug: "/agents"
+description: "This page explains how to create an AI agent in Haystack capable of retrieving information, generating responses, and taking actions using various Haystack components."
+---
+
+# Agents
+
+This page explains how to create an AI agent in Haystack capable of retrieving information, generating responses, and taking actions using various Haystack components.
+
+## What’s an AI Agent?
+
+An AI agent is a system that can:
+
+- Understand user input (text, image, audio, and other queries),
+- Retrieve relevant information (documents or structured data),
+- Generate intelligent responses (using LLMs like OpenAI or Hugging Face models),
+- Perform actions (calling APIs, fetching live data, executing functions).
+
+AI agents are autonomous systems that use large language models (LLMs) to make decisions and solve complex tasks.
+They interact with their environment using tools, memory, and reasoning.
+An AI agent is more than a chatbot — it actively plans, chooses the right tools, and executes tasks to achieve a goal.
+Unlike traditional software, it adapts to new information and refines its process as needed.
+
+1. **LLM as the Brain**: The agent’s core is an LLM, which understands context, processes natural language and serves as the central intelligence system.
+2. **Tools for Interaction**: Agents connect to external tools, APIs, and databases to gather information and take action.
+3. **Memory for Context**: Short-term memory helps track conversations, while long-term memory stores knowledge for future interactions.
+4. **Reasoning and Planning**: Agents break down complex problems, come up with step-by-step action plans, and adapt based on new data and feedback.
+
+An AI agent starts with a prompt that defines its role and objectives.
+It decides when to use tools, gathers data, and refines its approach through loops of reasoning and action.
+For example, a customer service agent answers queries using a database — if it lacks an answer, it fetches real-time data, summarizes it, and responds.
+A coding assistant understands project requirements, suggests solutions, and writes code.
+
+## Key Components
+
+### Agent Component
+
+Haystack has a built-in [Agent](../pipeline-components/agents-1/agent.mdx) component that manages the full tool-calling loop — it calls the LLM, invokes tools, updates state, and continues until a stopping condition is met.
+Key capabilities include:
+
+- **State management**: Share typed data between tools, accumulate results across iterations, and surface them in the result dict using `state_schema`. See [State](../pipeline-components/agents-1/state.mdx).
+- **Streaming**: Stream token-by-token output with a `streaming_callback`.
+- **Human-in-the-loop**: Intercept tool calls for human review before execution. See [Human in the Loop](../pipeline-components/agents-1/human-in-the-loop.mdx).
+- **Multi-agent systems**: Wrap an `Agent` as a `ComponentTool` to build coordinator/specialist architectures. See [Multi-Agent Systems](./agents/multi-agent-systems.mdx).
+- **MCP server exposure**: Expose your agent as an MCP server using [Hayhooks](../development/hayhooks.mdx), making it callable from any MCP-compatible client such as Claude Desktop or Cursor.
+- **Multimodal inputs**: Pass images alongside text using `ImageContent` in `ChatMessage` content parts, or return `ImageContent` from tools for dynamic image analysis. Requires a vision-capable model such as `gpt-5` or `gemini-2.5-flash`. See [Multimodal Inputs](../pipeline-components/agents-1/agent.mdx#multimodal-inputs).
+
+Check out the [Agent](../pipeline-components/agents-1/agent.mdx) documentation, or the [example](#tool-calling-agent) below to get started.
+
+### State
+
+[`State`](../pipeline-components/agents-1/state.mdx) is Haystack's built-in mechanism for sharing data between tools and accumulating results across multiple tool calls.
+You define a `state_schema` on the `Agent`, and any keys declared there are returned alongside `messages` and `last_message` in the agent's result dict.
+
+### Tools
+
+Haystack provides several ways to create and manage tools:
+
+- [`Tool`](../tools/tool.mdx) class / [`@tool`](../tools/tool.mdx#tool-decorator) decorator – Define a tool from a Python function. The `@tool` decorator automatically uses the function's name and docstring; the `Tool` class gives full control over the name, description, and schema.
+- [`ComponentTool`](../tools/componenttool.mdx) – Wraps any Haystack component as a callable tool.
+- [`PipelineTool`](../tools/pipelinetool.mdx) – Wraps a full Haystack pipeline as a callable tool.
+- [`MCPTool`](../tools/mcptool.mdx) / [`MCPToolset`](../tools/mcptoolset.mdx) – Connects to Model Context Protocol (MCP) servers to load external tools.
+- [`Toolset`](../tools/toolset.mdx) – Groups multiple tools into a single unit to pass to an Agent or Generator.
+- [`SearchableToolset`](../tools/searchabletoolset.mdx) – Enables keyword-based tool discovery for large catalogs, so the LLM only sees relevant tools at each step.
+
+## Example
+
+### Tool-Calling Agent
+
+Create a tool-calling agent with the `Agent` component. This example requires `OPENAI_API_KEY` and `SERPERDEV_API_KEY` to be set as environment variables:
+
+```shell
+export OPENAI_API_KEY=<your-openai-key>
+export SERPERDEV_API_KEY=<your-serperdev-key>
+```
+
+```python
+from haystack.components.agents import Agent
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.components.generators.utils import print_streaming_chunk
+from haystack.components.websearch import SerperDevWebSearch
+from haystack.dataclasses import ChatMessage
+from haystack.tools import ComponentTool
+
+# Wrap the web search component as a tool
+web_tool = ComponentTool(
+    component=SerperDevWebSearch(top_k=3),
+    name="web_search",
+    description="Search the web for current information like weather, news, or facts.",
+)
+
+tool_calling_agent = Agent(
+    chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
+    system_prompt=(
+        "You're a helpful agent. When asked about current information like weather, news, or facts, "
+        "use the web_search tool to find the information and then summarize the findings."
+    ),
+    tools=[web_tool],
+    streaming_callback=print_streaming_chunk,
+)
+
+result = tool_calling_agent.run(
+    messages=[ChatMessage.from_user("How is the weather in Berlin?")],
+)
+print(result["last_message"].text)
+```
+
+Resulting in:
+
+```python
+>>> The current weather in Berlin is approximately 60°F. The forecast for today includes clouds in the morning with some sunshine later. The high temperature is expected to be around 65°F, and the low tonight will drop to 40°F.
+
+- **Morning**: 49°F
+- **Afternoon**: 57°F
+- **Evening**: 47°F
+- **Overnight**: 39°F
+
+For more details, you can check the full forecasts on [AccuWeather](https://www.accuweather.com/en/de/berlin/10178/current-weather/178087) or [Weather.com](https://weather.com/weather/today/l/5ca23443513a0fdc1d37ae2ffaf5586162c6fe592a66acc9320a0d0536be1bb9).
+```

docs-website/versioned_docs/version-2.29/concepts/agents/multi-agent-systems.mdx

@@ -0,0 +1,338 @@
+---
+title: "Multi-Agent Systems"
+id: multi-agent-systems
+slug: "/multi-agent-systems"
+description: "Learn how to build multi-agent systems in Haystack by spawning agents as tools. Use the @tool decorator or ComponentTool to connect specialist agents to a coordinator."
+---
+
+# Multi-Agent Systems
+
+Multi-agent systems let you compose multiple `Agent` instances into larger architectures where a **coordinator** agent delegates to **specialist** agents.
+Each specialist focuses on a specific task with its own tools and system prompt - the coordinator plans and routes work without needing to know how each task gets done.
+
+Spawning agents as tools is useful when:
+
+- A task is too broad for a single agent to handle reliably,
+- You want to isolate different capabilities into focused, reusable agents,
+- You need to keep the coordinator's context lean for better decisions and lower token usage.
+
+In Haystack, you spawn a specialist agent as a tool using either the `@tool` decorator (recommended) or `ComponentTool`.
+
+## Converting an Agent to a Tool
+
+### `@tool` Decorator (Recommended)
+
+Wrapping an agent inside a `@tool` function gives you full control over what the coordinator LLM sees:
+
+- **Simplified parameters**: define explicit `Annotated` arguments instead of exposing `agent.run()`'s full interface
+- **Formatted output**: extract and return only what the coordinator needs, rather than the full result dict
+- **Error handling**: catch exceptions and return a clean message so the coordinator can recover
+
+This approach works better with smaller LLMs because the tool has a clean, minimal signature.
+The coordinator only needs to provide a query string - all the `ChatMessage` construction and result unpacking is hidden inside the function.
+
+```python
+from typing import Annotated
+from haystack.components.agents import Agent
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.components.generators.utils import print_streaming_chunk
+from haystack.dataclasses import ChatMessage
+from haystack.tools import ComponentTool, tool
+from haystack.components.websearch import SerperDevWebSearch
+from haystack.utils import Secret
+
+
+research_agent = Agent(
+    chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
+    tools=[
+        ComponentTool(
+            component=SerperDevWebSearch(
+                api_key=Secret.from_env_var("SERPERDEV_API_KEY"),
+                top_k=3,
+            ),
+            name="web_search",
+            description="Search the web for current information on any topic",
+        ),
+    ],
+    system_prompt="You are a research specialist. Search the web to find information.",
+)
+
+
+@tool
+def research(query: Annotated[str, "The research question to investigate"]) -> str:
+    """Research a topic and return a summary of findings."""
+    try:
+        result = research_agent.run(messages=[ChatMessage.from_user(query)])
+        return result["last_message"].text
+    except Exception as e:
+        return f"Research failed: {e}"
+
+
+coordinator = Agent(
+    chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
+    tools=[research],
+    system_prompt="You are a coordinator. Delegate research tasks to the research tool.",
+    streaming_callback=print_streaming_chunk,
+)
+
+result = coordinator.run(
+    messages=[
+        ChatMessage.from_user("What are the latest developments in Haystack AI?"),
+    ],
+)
+```
+
+### `ComponentTool`
+
+`ComponentTool` wraps an agent directly without a wrapper function.
+Choose it when you want **declarative configuration**: the full specialist setup (model, tools, system prompt) lives in one serializable object alongside the coordinator.
+
+Use `outputs_to_string={"source": "last_message"}` to surface only the specialist's final reply to the coordinator rather than the full result dict.
+
+```python
+from haystack.tools import ComponentTool
+
+research_tool = ComponentTool(
+    component=research_agent,
+    name="research_specialist",
+    description="A specialist that researches topics on the web",
+    outputs_to_string={"source": "last_message"},
+)
+
+coordinator = Agent(
+    chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
+    tools=[research_tool],
+    system_prompt="You are a coordinator. Delegate research tasks to the research specialist.",
+    streaming_callback=print_streaming_chunk,
+)
+
+result = coordinator.run(
+    messages=[
+        ChatMessage.from_user("What are the latest developments in Haystack AI?"),
+    ],
+)
+```
+
+The full specialist configuration is captured inline when serialized.
+Wrap the coordinator in a `Pipeline` and call `pipeline.dumps()` to get the YAML, which can be loaded back with `Pipeline.loads()`.
+
+<details>
+<summary>View YAML</summary>
+
+```yaml
+components:
+  coordinator:
+    init_parameters:
+      chat_generator:
+        init_parameters:
+          api_base_url: null
+          api_key:
+            env_vars:
+            - OPENAI_API_KEY
+            strict: true
+            type: env_var
+          generation_kwargs: {}
+          http_client_kwargs: null
+          max_retries: null
+          model: gpt-5.4-nano
+          organization: null
+          streaming_callback: null
+          timeout: null
+          tools: null
+          tools_strict: false
+        type: haystack.components.generators.chat.openai.OpenAIChatGenerator
+      confirmation_strategies: null
+      exit_conditions:
+      - text
+      max_agent_steps: 100
+      raise_on_tool_invocation_failure: false
+      required_variables: null
+      state_schema: {}
+      streaming_callback: null
+      system_prompt: You are a coordinator. Delegate research tasks to the research
+        specialist. Keep your final answer concise.
+      tool_invoker_kwargs: null
+      tools:
+      - data:
+          component:
+            init_parameters:
+              chat_generator:
+                init_parameters:
+                  api_base_url: null
+                  api_key:
+                    env_vars:
+                    - OPENAI_API_KEY
+                    strict: true
+                    type: env_var
+                  generation_kwargs: {}
+                  http_client_kwargs: null
+                  max_retries: null
+                  model: gpt-5.4-nano
+                  organization: null
+                  streaming_callback: null
+                  timeout: null
+                  tools: null
+                  tools_strict: false
+                type: haystack.components.generators.chat.openai.OpenAIChatGenerator
+              confirmation_strategies: null
+              exit_conditions:
+              - text
+              max_agent_steps: 100
+              raise_on_tool_invocation_failure: false
+              required_variables: null
+              state_schema: {}
+              streaming_callback: null
+              system_prompt: You are a research specialist. Search the web to find
+                information. Return a concise summary of your findings in 3-5 sentences.
+              tool_invoker_kwargs: null
+              tools:
+              - data:
+                  component:
+                    init_parameters:
+                      allowed_domains: null
+                      api_key:
+                        env_vars:
+                        - SERPERDEV_API_KEY
+                        strict: true
+                        type: env_var
+                      exclude_subdomains: false
+                      search_params: {}
+                      top_k: 3
+                    type: haystack.components.websearch.serper_dev.SerperDevWebSearch
+                  description: Search the web for current information on any topic
+                  inputs_from_state: null
+                  name: web_search
+                  outputs_to_state: null
+                  outputs_to_string: null
+                  parameters: null
+                type: haystack.tools.component_tool.ComponentTool
+              user_prompt: null
+            type: haystack.components.agents.agent.Agent
+          description: A specialist that researches topics on the web
+          inputs_from_state: null
+          name: research_specialist
+          outputs_to_state: null
+          outputs_to_string:
+            source: last_message
+          parameters: null
+        type: haystack.tools.component_tool.ComponentTool
+      user_prompt: null
+    type: haystack.components.agents.agent.Agent
+connection_type_validation: true
+connections: []
+max_runs_per_component: 100
+metadata: {}
+```
+
+</details>
+
+## Coordinator / Specialist Pattern
+
+The coordinator/specialist pattern cleanly splits responsibilities: the coordinator handles planning and delegation, while each specialist owns a focused toolset and a targeted system prompt.
+
+This is also a form of **context engineering**: deliberately controlling what each agent sees.
+A specialist accumulates its own tool call trace, but the coordinator only needs the final answer.
+Returning just `result["last_message"].text` (with `@tool`) or using `outputs_to_string` (with `ComponentTool`) surfaces only the specialist's final reply, keeping the coordinator's context lean.
+
+When covering multiple topics, the coordinator can call the same specialist tool several times in a single response.
+All tool calls from one LLM response are executed concurrently using a thread pool.
+Control the level of parallelism with `max_workers` in `tool_invoker_kwargs` (default: `4`).
+
+The example below asks the coordinator about two topics: it calls `research` twice and both specialists run in parallel.
+
+`HTMLToDocument` uses [Trafilatura](https://trafilatura.readthedocs.io) to extract clean text from HTML pages.
+Install it before running:
+
+```shell
+pip install trafilatura
+```
+
+```python
+from typing import Annotated
+from haystack.components.agents import Agent
+from haystack.components.converters import HTMLToDocument
+from haystack.components.fetchers.link_content import LinkContentFetcher
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.components.generators.utils import print_streaming_chunk
+from haystack.components.websearch import SerperDevWebSearch
+from haystack.dataclasses import ChatMessage
+from haystack.tools import ComponentTool, tool
+from haystack.utils import Secret
+
+
+search_tool = ComponentTool(
+    component=SerperDevWebSearch(
+        api_key=Secret.from_env_var("SERPERDEV_API_KEY"),
+        top_k=3,
+    ),
+    name="web_search",
+    description="Search the web for current information on any topic",
+)
+
+
+@tool
+def fetch_page(url: Annotated[str, "The URL of the web page to fetch"]) -> str:
+    """Fetch the content of a web page given its URL."""
+    try:
+        streams = LinkContentFetcher().run(urls=[url])["streams"]
+        if not streams:
+            return "No content found."
+        documents = HTMLToDocument().run(sources=streams)["documents"]
+        return documents[0].content if documents else "No content extracted."
+    except Exception as e:
+        return f"Failed to fetch page: {e}"
+
+
+research_agent = Agent(
+    chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
+    tools=[search_tool, fetch_page],
+    system_prompt=(
+        "You are a research specialist. Search the web to find relevant pages, "
+        "then fetch their full content for detailed information. "
+        "Return a concise summary of your findings in 3-5 sentences."
+    ),
+)
+
+
+@tool
+def research(query: Annotated[str, "The research question to investigate"]) -> str:
+    """Research a topic and return a summary of findings."""
+    try:
+        result = research_agent.run(messages=[ChatMessage.from_user(query)])
+        return result["last_message"].text
+    except Exception as e:
+        return f"Research failed: {e}"
+
+
+coordinator = Agent(
+    chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
+    tools=[research],
+    system_prompt=(
+        "You are a coordinator. Delegate research tasks to the research tool. "
+        "For questions covering multiple topics, research each one independently. "
+        "Keep your final answer concise."
+    ),
+    streaming_callback=print_streaming_chunk,
+    tool_invoker_kwargs={"max_workers": 4},  # run up to 4 specialist calls in parallel
+)
+
+result = coordinator.run(
+    messages=[
+        ChatMessage.from_user(
+            "What are the latest developments in large language models and retrieval-augmented generation?",
+        ),
+    ],
+)
+```
+
+## Additional References
+
+📖 Related docs:
+
+- [Agent](../../pipeline-components/agents-1/agent.mdx)
+- [State](../../pipeline-components/agents-1/state.mdx)
+- [ComponentTool](../../tools/componenttool.mdx)
+
+📚 Tutorials:
+
+- [Creating a Multi-Agent System](https://haystack.deepset.ai/tutorials/45_creating_a_multi_agent_system)

docs-website/versioned_docs/version-2.29/concepts/components.mdx

@@ -0,0 +1,59 @@
+---
+title: "Components"
+id: components
+slug: "/components"
+description: "Components are the building blocks of a pipeline. They perform tasks such as preprocessing, retrieving, or summarizing text while routing queries through different branches of a pipeline. This page is a summary of all component types available in Haystack."
+---
+
+import ClickableImage from "@site/src/components/ClickableImage";
+
+# Components
+
+Components are the building blocks of a pipeline. They perform tasks such as preprocessing, retrieving, or summarizing text while routing queries through different branches of a pipeline. This page is a summary of all component types available in Haystack.
+
+Components are connected to each other using a [pipeline](pipelines.mdx), and they function like building blocks that can be easily switched out for each other. A component can take the selected outputs of other components as input. You can also provide input to a component when you call `pipeline.run()`.
+
+## Stand-Alone or In a Pipeline
+
+You can integrate components in a pipeline to perform a specific task. But you can also use some of them stand-alone, outside of a pipeline. For example, you can run `DocumentWriter` on its own, to write documents into a Document Store. To check how to use a component and if it's usable outside of a pipeline, check the _Usage_ section on the component's documentation page.
+
+Each component has a `run()` method. When you connect components in a pipeline, and you run the pipeline by calling `Pipeline.run()`, it invokes the `run()` method for each component sequentially.
+
+## Input and Output
+
+To connect components in a pipeline, you need to know the names of the inputs and outputs they accept. The output of one component must be compatible with the input the subsequent component accepts. For example, to connect Retriever and Ranker in a pipeline, you must know that the Retriever outputs `documents` and the Ranker accepts `documents` as input.
+
+The mandatory inputs and outputs are listed in a table at the top of each component's documentation page so that you can quickly check them:
+<ClickableImage src="/img/3a53f3e-inputs_and_outputs.png" alt="DocumentWriter component specification table showing Name, Folder Path, Position in Pipeline, Inputs (documents list), and Outputs (documents_written integer)" />
+
+You can also look them up in the code in the component`run()` method. Here's an example of the inputs and outputs of `TransformerSimilarityRanker`:
+
+```python
+@component.output_types(documents=List[Document]) # "documents" is the output name you need when connecting components in a pipeline
+def run(self, query: str, documents: List[Document], top_k: Optional[int] = None):# "query" and "documents" are the mandatory inputs, additionally you can also specify the optional top_k parameter
+"""
+Returns a list of Documents ranked by their similarity to the given query.
+
+:param query: Query string.
+:param documents: List of Documents.
+:param top_k: The maximum number of Documents you want the Ranker to return.
+:return: List of Documents sorted by their similarity to the query with the most similar Documents appearing first.
+"""
+```
+
+## Warming Up Components
+
+Components that use heavy resources, like LLMs or embedding models, have a `warm_up()` method that loads the necessary resources (such as models) into memory. This method is automatically called the first time the component runs, so you can use components directly without explicitly calling `warm_up()`:
+
+```python
+from haystack import Document
+from haystack.components.embedders import SentenceTransformersDocumentEmbedder
+
+doc = Document(content="I love pizza!")
+doc_embedder = SentenceTransformersDocumentEmbedder()
+
+result = doc_embedder.run([doc])  # warm_up() is called automatically on first run
+print(result["documents"][0].embedding)
+```
+
+You can still call `warm_up()` explicitly if you want to control when resources are loaded.

docs-website/versioned_docs/version-2.29/concepts/components/custom-components.mdx

@@ -0,0 +1,182 @@
+---
+title: "Creating Custom Components"
+id: custom-components
+slug: "/custom-components"
+description: "Create your own components and use them standalone or in pipelines."
+---
+
+# Creating Custom Components
+
+Create your own components and use them standalone or in pipelines.
+
+With Haystack, you can easily create any custom components for various tasks, from filtering results to integrating with external software. You can then insert, reuse, and share these components within Haystack or even with an external audience by packaging them and submitting them to [Haystack Integrations](../integrations.mdx)!
+
+## Requirements
+
+Here are the requirements for all custom components:
+
+- `@component`: This decorator marks a class as a component, allowing it to be used in a pipeline.
+- `run()`: This is a required method in every component. It accepts input arguments and returns a `dict`. The inputs can either come from the pipeline when it’s executed, or from the output of another component when connected using `connect()`. The `run()` method should be compatible with the input/output definitions declared for the component. See an [Extended Example](#extended-example) below to check how it works.
+
+
+:::note[Avoid in-place input mutation]
+
+When building custom components, do not change the component's inputs directly. Instead, work on a copy or a new version of the input, modify that, and return it. The reason for this is that the original input values might be reused by other components or by later pipeline steps. Mutating the input directly can lead to unintended side effects and bugs in the pipeline, as other components might rely on the original input values.
+
+When only one or a few fields of the input need to be changed (for example, `meta` on a `Document`), use `dataclasses.replace()` to create a new instance with the updated fields. This is simpler and more efficient than deep-copying the whole object:
+
+```python
+from dataclasses import replace
+
+
+def run(self, documents):
+    updated = [replace(doc, meta={**doc.meta, "processed": True}) for doc in documents]
+    return {"documents": updated}
+```
+
+When you need to modify nested mutable structures, for example `list` or `dict` attributes, or update many fields of the dataclass instance, use a full deep copy instead:
+
+```python
+import copy
+
+
+def run(self, documents):
+    documents_copy = copy.deepcopy(documents)
+    # mutate documents_copy safely here
+    return {"documents": documents_copy}
+```
+
+:::
+
+
+### Inputs and Outputs
+
+Next, define the inputs and outputs for your component.
+
+#### Inputs
+
+You can choose between three input options:
+
+- `set_input_type`: This method defines or updates a single input socket for a component instance. It’s ideal for adding or modifying a specific input at runtime without affecting others. Use this when you need to dynamically set or modify a single input based on specific conditions.
+- `set_input_types`: This method allows you to define multiple input sockets at once, replacing any existing inputs. It’s useful when you know all the inputs the component will need and want to configure them in bulk. Use this when you want to define multiple inputs during initialization.
+- Declaring arguments directly in the `run()` method. Use this method when the component’s inputs are static and known at the time of class definition.
+
+#### Outputs
+
+You can choose between two output options:
+
+- `@component.output_types`: This decorator defines the output types and names at the time of class definition. The output names and types must match the `dict` returned by the `run()` method. Use this when the output types are static and known in advance. This decorator is cleaner and more readable for static components.
+- `set_output_types`: This method defines or updates multiple output sockets for a component instance at runtime. It’s useful when you need flexibility in configuring outputs dynamically. Use this when the output types need to be set at runtime for greater flexibility.
+
+## Short Example
+
+Here is an example of a simple minimal component setup:
+
+```python
+from haystack import component
+
+
+@component
+class WelcomeTextGenerator:
+    """
+    A component generating personal welcome message and making it upper case
+    """
+
+    @component.output_types(welcome_text=str, note=str)
+    def run(self, name: str):
+        return {
+            "welcome_text": f"Hello {name}, welcome to Haystack!".upper(),
+            "note": "welcome message is ready",
+        }
+```
+
+Here, the custom component `WelcomeTextGenerator` accepts one input: `name` string and returns two outputs: `welcome_text` and `note`.
+
+## Extended Example
+
+Check out an example below on how to create two custom components and connect them in a Haystack pipeline.
+
+```python
+# import necessary dependencies
+from haystack import component, Pipeline
+
+
+# Create two custom components. Note the mandatory @component decorator and @component.output_types, as well as the mandatory run method.
+@component
+class WelcomeTextGenerator:
+    """
+    A component generating personal welcome message and making it upper case
+    """
+
+    @component.output_types(welcome_text=str, note=str)
+    def run(self, name: str):
+        return {
+            "welcome_text": (
+                "Hello {name}, welcome to Haystack!".format(name=name)
+            ).upper(),
+            "note": "welcome message is ready",
+        }
+
+
+@component
+class WhitespaceSplitter:
+    """
+    A component for splitting the text by whitespace
+    """
+
+    @component.output_types(split_text=list[str])
+    def run(self, text: str):
+        return {"split_text": text.split()}
+
+
+# create a pipeline and add the custom components to it
+text_pipeline = Pipeline()
+text_pipeline.add_component(
+    name="welcome_text_generator",
+    instance=WelcomeTextGenerator(),
+)
+text_pipeline.add_component(name="splitter", instance=WhitespaceSplitter())
+
+# connect the components
+text_pipeline.connect(
+    sender="welcome_text_generator.welcome_text",
+    receiver="splitter.text",
+)
+
+# define the result and run the pipeline
+result = text_pipeline.run({"welcome_text_generator": {"name": "Bilge"}})
+
+print(result["splitter"]["split_text"])
+```
+
+## Extending the Existing Components
+
+To extend already existing components in Haystack, subclass an existing component and use the `@component` decorator to mark it. Override or extend the `run()` method to process inputs and outputs. Call `super()` with the derived class name from the init of the derived class to avoid initialization issues:
+
+```python
+class DerivedComponent(BaseComponent):
+    def __init__(self):
+        super(DerivedComponent, self).__init__()
+
+
+# ...
+
+dc = DerivedComponent()  # ok
+```
+
+An example of an extended component is Haystack's [FaithfulnessEvaluator](https://github.com/deepset-ai/haystack/blob/e5a80722c22c59eb99416bf0cd712f6de7cd581a/haystack/components/evaluators/faithfulness.py) derived from LLMEvaluator.
+
+## Project Template
+
+If you're building a custom component that you want to package and share, we provide a [GitHub template repository](https://github.com/deepset-ai/custom-component) that gives you a ready-made project structure. It includes the boilerplate for packaging, testing, and distributing your custom component as a standalone Python package. Use it to quickly scaffold a new integration or reusable component without setting up the project from scratch.
+
+Check out the [video walkthrough](https://www.youtube.com/watch?v=SWC0QecAMcI) for a step-by-step guide on how to use the template.
+
+## Additional References
+
+🧑‍🍳 Cookbooks:
+
+- [Build quizzes and adventures with Character Codex and llamafile](https://haystack.deepset.ai/cookbook/charactercodex_llamafile/)
+- [Run tasks concurrently within a custom component](https://haystack.deepset.ai/cookbook/concurrent_tasks/)
+- [Chat With Your SQL Database](https://haystack.deepset.ai/cookbook/chat_with_sql_3_ways/)
+- [Hacker News Summaries with Custom Components](https://haystack.deepset.ai/cookbook/hackernews-custom-component-rag/)

docs-website/versioned_docs/version-2.29/concepts/components/supercomponents.mdx

@@ -0,0 +1,187 @@
+---
+title: "SuperComponents"
+id: supercomponents
+slug: "/supercomponents"
+description: "`SuperComponent` lets you wrap a complete pipeline and use it like a single component. This is helpful when you want to simplify the interface of a complex pipeline, reuse it in different contexts, or expose only the necessary inputs and outputs."
+---
+
+# SuperComponents
+
+`SuperComponent` lets you wrap a complete pipeline and use it like a single component. This is helpful when you want to simplify the interface of a complex pipeline, reuse it in different contexts, or expose only the necessary inputs and outputs.
+
+## `@super_component` decorator (recommended)
+
+Haystack now provides a simple `@super_component` decorator for wrapping a pipeline as a component. All you need is to create a class with the decorator, and to include an `pipeline` attribute.
+
+With this decorator, the `to_dict` and `from_dict` serialization is optional, as is the input and output mapping.
+
+### Example
+
+The custom HybridRetriever example SuperComponent below turns your query into embeddings, then runs both a BM25 search and an embedding-based search at the same time. It finally merges those two result sets and returns the combined documents.
+
+```python
+# pip install haystack-ai datasets "sentence-transformers>=3.0.0"
+
+from haystack import Document, Pipeline, super_component
+from haystack.components.joiners import DocumentJoiner
+from haystack.components.embedders import SentenceTransformersTextEmbedder
+from haystack.components.retrievers import (
+    InMemoryBM25Retriever,
+    InMemoryEmbeddingRetriever,
+)
+from haystack.document_stores.in_memory import InMemoryDocumentStore
+
+from datasets import load_dataset
+
+
+@super_component
+class HybridRetriever:
+    def __init__(
+        self,
+        document_store: InMemoryDocumentStore,
+        embedder_model: str = "BAAI/bge-small-en-v1.5",
+    ):
+        embedding_retriever = InMemoryEmbeddingRetriever(document_store)
+        bm25_retriever = InMemoryBM25Retriever(document_store)
+        text_embedder = SentenceTransformersTextEmbedder(embedder_model)
+        document_joiner = DocumentJoiner()
+
+        self.pipeline = Pipeline()
+        self.pipeline.add_component("text_embedder", text_embedder)
+        self.pipeline.add_component("embedding_retriever", embedding_retriever)
+        self.pipeline.add_component("bm25_retriever", bm25_retriever)
+        self.pipeline.add_component("document_joiner", document_joiner)
+
+        self.pipeline.connect("text_embedder", "embedding_retriever")
+        self.pipeline.connect("bm25_retriever", "document_joiner")
+        self.pipeline.connect("embedding_retriever", "document_joiner")
+
+
+dataset = load_dataset("HaystackBot/medrag-pubmed-chunk-with-embeddings", split="train")
+docs = [
+    Document(content=doc["contents"], embedding=doc["embedding"]) for doc in dataset
+]
+document_store = InMemoryDocumentStore()
+document_store.write_documents(docs)
+
+query = "What treatments are available for chronic bronchitis?"
+
+result = HybridRetriever(document_store).run(text=query, query=query)
+print(result)
+```
+
+### Input Mapping
+
+You can optionally map the input names of your SuperComponent to the actual sockets inside the pipeline.
+
+```python
+input_mapping = {"query": ["retriever.query", "prompt.query"]}
+```
+
+### Output Mapping
+
+You can also map the pipeline's output sockets that you want to expose to the SuperComponent's output names.
+
+```python
+output_mapping = {"llm.replies": "replies"}
+```
+
+If you don’t provide mappings, SuperComponent will try to auto-detect them. So, if multiple components have outputs with the same name, we recommend using `output_mapping` to avoid conflicts.
+
+## SuperComponent class
+
+Haystack also gives you an option to inherit from SuperComponent class. This option requires `to_dict` and `from_dict` serialization, as well as the input and output mapping described above.
+
+### Example
+
+Here is a simple example of initializing a `SuperComponent` with a pipeline:
+
+```python
+from haystack import Pipeline, SuperComponent
+
+with open("pipeline.yaml", "r") as file:
+    pipeline = Pipeline.load(file)
+
+super_component = SuperComponent(pipeline)
+```
+
+The example pipeline below retrieves relevant documents based on a user query, builds a custom prompt using those documents, then sends the prompt to an `OpenAIChatGenerator` to create an answer. The `SuperComponent` wraps the pipeline so it can be run with a simple input (`query`) and returns a clean output (`replies`).
+
+```python
+from haystack import Pipeline, SuperComponent
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.components.builders import ChatPromptBuilder
+from haystack.components.retrievers import InMemoryBM25Retriever
+from haystack.dataclasses.chat_message import ChatMessage
+from haystack.document_stores.in_memory import InMemoryDocumentStore
+from haystack.dataclasses import Document
+
+document_store = InMemoryDocumentStore()
+documents = [
+    Document(content="Paris is the capital of France."),
+    Document(content="London is the capital of England."),
+]
+document_store.write_documents(documents)
+
+prompt_template = [
+    ChatMessage.from_user(
+    '''
+    According to the following documents:
+    {% for document in documents %}
+    {{document.content}}
+    {% endfor %}
+    Answer the given question: {{query}}
+    Answer:
+    '''
+    )
+]
+
+prompt_builder = ChatPromptBuilder(template=prompt_template, required_variables="*")
+
+pipeline = Pipeline()
+pipeline.add_component("retriever", InMemoryBM25Retriever(document_store=document_store))
+pipeline.add_component("prompt_builder", prompt_builder)
+pipeline.add_component("llm", OpenAIChatGenerator())
+pipeline.connect("retriever.documents", "prompt_builder.documents")
+pipeline.connect("prompt_builder.prompt", "llm.messages")
+
+# Create a super component with simplified input/output mapping
+wrapper = SuperComponent(
+    pipeline=pipeline,
+    input_mapping={
+        "query": ["retriever.query", "prompt_builder.query"],
+    },
+    output_mapping={
+        "llm.replies": "replies",
+        "retriever.documents": "documents"
+    }
+)
+
+# Run the pipeline with simplified interface
+result = wrapper.run(query="What is the capital of France?")
+print(result)
+{'replies': [ChatMessage(_role=<ChatRole.ASSISTANT: 'assistant'>,
+ _content=[TextContent(text='The capital of France is Paris.')],...)
+```
+
+## Type Checking and Static Code Analysis
+
+Creating SuperComponents using the @super_component decorator can induce type or linting errors. One way to avoid these issues is to add the exposed public methods to your SuperComponent. Here's an example:
+
+```python
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+
+    def run(self, *, documents: list[Document]) -> dict[str, list[Document]]: ...
+    def warm_up(self) -> None:  # noqa: D102
+        ...
+```
+
+## Ready-Made SuperComponents
+
+You can see two implementations of SuperComponents already integrated in Haystack:
+
+- [DocumentPreprocessor](../../pipeline-components/preprocessors/documentpreprocessor.mdx)
+- [MultiFileConverter](../../pipeline-components/converters/multifileconverter.mdx)
+- [OpenSearchHybridRetriever](../../pipeline-components/retrievers/opensearchhybridretriever.mdx)

docs-website/versioned_docs/version-2.29/concepts/concepts-overview.mdx

@@ -0,0 +1,56 @@
+---
+title: "Haystack Concepts Overview"
+id: concepts-overview
+slug: "/concepts-overview"
+description: "Haystack provides all the tools you need to build custom agents and RAG pipelines with LLMs that work for you. This includes everything from prototyping to deployment. This page discusses the most important concepts Haystack operates on."
+---
+
+import ClickableImage from "@site/src/components/ClickableImage";
+
+# Haystack Concepts Overview
+
+Haystack provides all the tools you need to build custom agents and RAG pipelines with LLMs that work for you. This includes everything from prototyping to deployment. This page discusses the most important concepts Haystack operates on.
+
+### Components
+
+Haystack offers various components, each performing different kinds of tasks. You can see the whole variety in the **PIPELINE COMPONENTS** section in the left-side navigation. These are often powered by the latest Large Language Models (LLMs) and transformer models. Code-wise, they are Python classes with methods you can directly call. Most commonly, all you need to do is initialize the component with the required parameters and then run it with a `run()` method.
+
+Working on this level with Haystack components is a hands-on approach. Components define the name and the type of all of their inputs and outputs. The Component API reduces complexity and makes it easier to [create custom components](components/custom-components.mdx), for example, for third-party APIs and databases. Haystack validates the connections between components before running the pipeline and, if needed, generates error messages with instructions on fixing the errors.
+
+#### Generators
+
+[Generators](../pipeline-components/generators.mdx) are responsible for generating text responses after you give them a prompt. They are specific for each LLM technology (OpenAI, Cohere, local models, and others). There are two types of Generators: chat and non-chat:
+
+- The chat ones enable chat completion and are designed for conversational contexts. It expects a list of messages to interact with the user.
+- The non-chat Generators use LLMs for simpler text generation (for example, translating or summarizing text).
+
+Read more about various Generators in our [guides](../pipeline-components/generators/guides-to-generators/choosing-the-right-generator.mdx).
+
+#### Retrievers
+
+[Retrievers](../pipeline-components/retrievers.mdx) go through all the documents in a Document Store, select the ones that match the user query, and pass it on to the next component. There are various Retrievers that are customized for specific Document Stores. This means that they can handle specific requirements for each database using customized parameters.
+
+For example, for Elasticsearch Document Store, you will find both the Document Store and Retriever packages in its GitHub [repo](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/elasticsearch).
+
+### Document Stores
+
+[Document Store](document-store.mdx) is an object that stores your documents in Haystack, like an interface to a storage database. It uses specific functions like `write_documents()` or `delete_documents()` to work with data. Various components have access to the Document Store and can interact with it by, for example, reading or writing Documents.
+
+If you are working with more complex pipelines in Haystack, you can use a [`DocumentWriter`](../pipeline-components/writers/documentwriter.mdx) component to write data into Document Stores for you
+
+### Data Classes
+
+You can use different [data classes](data-classes.mdx) in Haystack to carry the data through the system. The data classes are mostly likely to appear as inputs or outputs of your pipelines.
+
+`Document` class contains information to be carried through the pipeline. It can be text, metadata, tables, or binary data. Documents can be written into Document Stores but also written and read by other components.
+
+`Answer` class holds not only the answer generated in a pipeline but also the originating query and metadata.
+
+### Pipelines
+
+Finally, you can combine various components, Document Stores, and integrations into [pipelines](pipelines.mdx) to create powerful and customizable systems. It is a highly flexible system that allows you to have simultaneous flows, standalone components, loops, and other types of connections. You can have the preprocessing, indexing, and querying steps all in one pipeline, or you can split them up according to your needs.
+
+If you want to reuse pipelines, you can save them into a convenient format (YAML, TOML, and more) on a disk or share them around using the [serialization](pipelines/serialization.mdx) process.
+
+Here is a short Haystack pipeline, illustrated:
+<ClickableImage src="/img/00f5fe8-Pipeline_Illustrations_2.png" alt="RAG architecture overview showing query flow through retrieval and generation stages, with document stores providing context for the language model" />

docs-website/versioned_docs/version-2.29/concepts/data-classes.mdx

@@ -0,0 +1,309 @@
+---
+title: "Data Classes"
+id: data-classes
+slug: "/data-classes"
+description: "In Haystack, there are a handful of core classes that are regularly used in many different places. These are classes that carry data through the system and you are likely to interact with these as either the input or output of your pipeline."
+---
+
+# Data Classes
+
+In Haystack, there are a handful of core classes that are regularly used in many different places. These are classes that carry data through the system and you are likely to interact with these as either the input or output of your pipeline.
+
+Haystack uses data classes to help components communicate with each other in a simple and modular way. By doing this, data flows seamlessly through the Haystack pipelines. This page goes over the available data classes in Haystack: ByteStream, Answer (along with its variants ExtractedAnswer and GeneratedAnswer), ChatMessage, FileContent, Document, and StreamingChunk, explaining how they contribute to the Haystack ecosystem.
+
+You can check out the detailed parameters in our [Data Classes](/reference/data-classes-api) API reference.
+
+### Answer
+
+#### Overview
+
+The `Answer` class serves as the base for responses generated within Haystack, containing the answer's data, the originating query, and additional metadata.
+
+#### Key Features
+
+- Adaptable data handling, accommodating any data type (`data`).
+- Query tracking for contextual relevance (`query`).
+- Extensive metadata support for detailed answer description.
+
+#### Attributes
+
+```python
+@dataclass
+class Answer:
+    data: Any
+    query: str
+    meta: Dict[str, Any]
+```
+
+### ExtractedAnswer
+
+#### Overview
+
+`ExtractedAnswer` is a subclass of `Answer` that deals explicitly with answers derived from Documents, offering more detailed attributes.
+
+#### Key Features
+
+- Includes reference to the originating `Document`.
+- Score attribute to quantify the answer's confidence level.
+- Optional start and end indices for pinpointing answer location within the source.
+
+#### Attributes
+
+```python
+@dataclass
+class ExtractedAnswer:
+    query: str
+    score: float
+    data: Optional[str] = None
+    document: Optional[Document] = None
+    context: Optional[str] = None
+    document_offset: Optional["Span"] = None
+    context_offset: Optional["Span"] = None
+    meta: Dict[str, Any] = field(default_factory=dict)
+```
+
+### GeneratedAnswer
+
+#### Overview
+
+`GeneratedAnswer` extends the `Answer` class to accommodate answers generated from multiple Documents.
+
+#### Key Features
+
+- Handles string-type data.
+- Links to a list of `Document` objects, enhancing answer traceability.
+
+#### Attributes
+
+```python
+@dataclass
+class GeneratedAnswer:
+    data: str
+    query: str
+    documents: List[Document]
+    meta: Dict[str, Any] = field(default_factory=dict)
+```
+
+### ByteStream
+
+#### Overview
+
+`ByteStream` represents binary object abstraction in the Haystack framework and is crucial for handling various binary data formats.
+
+#### Key Features
+
+- Holds binary data and associated metadata.
+- Optional MIME type specification for flexibility.
+- File interaction methods (`to_file`, `from_file_path`, `from_string`) for easy data manipulation.
+
+#### Attributes
+
+```python
+@dataclass(repr=False)
+class ByteStream:
+    data: bytes
+    meta: Dict[str, Any] = field(default_factory=dict, hash=False)
+    mime_type: Optional[str] = field(default=None)
+```
+
+#### Example
+
+```python
+from haystack.dataclasses.byte_stream import ByteStream
+
+image = ByteStream.from_file_path("dog.jpg")
+```
+
+### ChatMessage
+
+`ChatMessage` is the central abstraction to represent a message for a LLM. It contains role, metadata and several types of content, including text, tool calls and tool calls results.
+
+Read the detailed documentation for the `ChatMessage` data class on a dedicated [ChatMessage](data-classes/chatmessage.mdx) page.
+
+### FileContent
+
+`FileContent` represents a file payload that can be attached to a `ChatMessage`, including base64 data, MIME type, filename, and provider-specific metadata.
+
+Read the detailed documentation for the `FileContent` data class on a dedicated [FileContent](data-classes/filecontent.mdx) page.
+
+### Document
+
+#### Overview
+
+`Document` represents a central data abstraction in Haystack, capable of holding text, tables, and binary data.
+
+#### Key Features
+
+- Unique ID for each document.
+- Multiple content types are supported: text, binary (`blob`).
+- Custom metadata and scoring for advanced document management.
+- Optional embedding for AI-based applications.
+
+#### Attributes
+
+```python
+@dataclass
+class Document(metaclass=_BackwardCompatible):
+    id: str = field(default="")
+    content: Optional[str] = field(default=None)
+    blob: Optional[ByteStream] = field(default=None)
+    meta: Dict[str, Any] = field(default_factory=dict)
+    score: Optional[float] = field(default=None)
+    embedding: Optional[List[float]] = field(default=None)
+    sparse_embedding: Optional[SparseEmbedding] = field(default=None)
+```
+
+#### Example
+
+```python
+from haystack import Document
+
+documents = Document(
+    content="Here are the contents of your document",
+    embedding=[0.1] * 768,
+)
+```
+
+### StreamingChunk
+
+#### Overview
+
+`StreamingChunk` represents a partially streamed LLM response, enabling real-time LLM response processing. It encapsulates a segment of streamed content along with associated metadata and provides comprehensive information about the streaming state.
+
+#### Key Features
+
+- String-based content representation for text chunks
+- Support for tool calls and tool call results
+- Component tracking and metadata management
+- Streaming state indicators (start, finish reason)
+- Content block indexing for multi-part responses
+
+#### Attributes
+
+```python
+@dataclass
+class StreamingChunk:
+    content: str
+    meta: dict[str, Any] = field(default_factory=dict, hash=False)
+    component_info: Optional[ComponentInfo] = field(default=None)
+    index: Optional[int] = field(default=None)
+    tool_calls: Optional[list[ToolCallDelta]] = field(default=None)
+    tool_call_result: Optional[ToolCallResult] = field(default=None)
+    start: bool = field(default=False)
+    finish_reason: Optional[FinishReason] = field(default=None)
+    reasoning: Optional[ReasoningContent] = field(default=None)
+```
+
+#### Example
+
+```python
+from haystack.dataclasses import StreamingChunk, ToolCallDelta, ReasoningContent
+
+# Basic text chunk
+chunk = StreamingChunk(
+    content="Hello world",
+    start=True,
+    meta={"model": "gpt-5-mini"},
+)
+
+# Tool call chunk
+tool_chunk = StreamingChunk(
+    content="",
+    tool_calls=[
+        ToolCallDelta(
+            index=0,
+            tool_name="calculator",
+            arguments='{"operation": "add", "a": 2, "b": 3}',
+        ),
+    ],
+    index=0,
+    start=False,
+    finish_reason="tool_calls",
+)
+
+# Reasoning chunk
+reasoning_chunk = StreamingChunk(
+    content="",
+    reasoning=ReasoningContent(
+        reasoning_text="Thinking step by step about the answer.",
+    ),
+    index=0,
+    start=True,
+    meta={"model": "gpt-4.1-mini"},
+)
+```
+
+### ToolCallDelta
+
+#### Overview
+
+`ToolCallDelta` represents a tool call prepared by the model, usually contained in an assistant message during streaming.
+
+#### Attributes
+
+```python
+@dataclass
+class ToolCallDelta:
+    index: int
+    tool_name: Optional[str] = field(default=None)
+    arguments: Optional[str] = field(default=None)
+    id: Optional[str] = field(default=None)
+    extra: Optional[Dict[str, Any]] = field(default=None)
+```
+
+### ComponentInfo
+
+#### Overview
+
+The `ComponentInfo` class represents information about a component within a Haystack pipeline. It is used to track the type and name of components that generate or process data, aiding in debugging, tracing, and metadata management throughout the pipeline.
+
+#### Key Features
+
+- Stores the type of the component (including module and class name).
+- Optionally stores the name assigned to the component in the pipeline.
+- Provides a convenient class method to create a `ComponentInfo` instance from a `Component` object.
+
+#### Attributes
+
+```python
+@dataclass
+class ComponentInfo:
+    type: str
+    name: Optional[str] = field(default=None)
+
+    @classmethod
+    def from_component(cls, component: Component) -> "ComponentInfo": ...
+```
+
+#### Example
+
+```python
+from haystack.dataclasses.streaming_chunk import ComponentInfo
+from haystack.core.component import Component
+
+
+class MyComponent(Component): ...
+
+
+component = MyComponent()
+info = ComponentInfo.from_component(component)
+print(info.type)  # e.g., 'my_module.MyComponent'
+print(info.name)  # Name assigned in the pipeline, if any
+```
+
+### SparseEmbedding
+
+#### Overview
+
+The `SparseEmbedding` class represents a sparse embedding: a vector where most values are zeros.
+
+#### Attributes
+
+- `indices`: List of indices of non-zero elements in the embedding.
+- `values`: List of values of non-zero elements in the embedding.
+
+### Tool
+
+`Tool` is a data class representing a tool that Language Models can prepare a call for.
+
+Read the detailed documentation for the `Tool` data class on a dedicated [Tool](../tools/tool.mdx) page.

docs-website/versioned_docs/version-2.29/concepts/data-classes/chatmessage.mdx

@@ -0,0 +1,413 @@
+---
+title: "ChatMessage"
+id: chatmessage
+slug: "/chatmessage"
+description: "`ChatMessage` is the central abstraction to represent a message for a LLM. It contains role, metadata and several types of content, including text, images, tool calls, tool call results, and reasoning content."
+---
+
+# ChatMessage
+
+`ChatMessage` is the central abstraction to represent a message for a LLM. It contains role, metadata and several types of content, including text, images, tool calls, tool call results, and reasoning content.
+
+To create a `ChatMessage` instance, use `from_user`, `from_system`, `from_assistant`, and `from_tool` class methods.
+
+The [content](#types-of-content) of the `ChatMessage` can then be inspected using the `text`, `texts`, `image`, `images`, `file`, `files`, `tool_call`, `tool_calls`, `tool_call_result`, `tool_call_results`, `reasoning`, and `reasonings` properties.
+
+If you are looking for the details of this data class methods and parameters, head over to our [API documentation](/reference/data-classes-api#chatmessage).
+
+## Types of Content
+
+`ChatMessage` currently supports `TextContent`, `ImageContent`, `FileContent`, `ToolCall`, `ToolCallResult`, and `ReasoningContent` types of content:
+
+```python
+@dataclass
+class TextContent:
+    """
+    The textual content of a chat message.
+
+    :param text: The text content of the message.
+    """
+
+    text: str
+
+
+@dataclass
+class ToolCall:
+    """
+    Represents a Tool call prepared by the model, usually contained in an assistant message.
+
+    :param tool_name: The name of the Tool to call.
+    :param arguments: The arguments to call the Tool with.
+    :param id: The ID of the Tool call.
+    :param extra: Dictionary of extra information about the Tool call. Use to store provider-specific
+        information. To avoid serialization issues, values should be JSON serializable.
+    """
+
+    tool_name: str
+    arguments: Dict[str, Any]
+    id: Optional[str] = None  # noqa: A003
+    extra: Optional[Dict[str, Any]] = None
+
+
+@dataclass
+class ToolCallResult:
+    """
+    Represents the result of a Tool invocation.
+
+    :param result: The result of the Tool invocation.
+    :param origin: The Tool call that produced this result.
+    :param error: Whether the Tool invocation resulted in an error.
+    """
+
+    result: str | Sequence[TextContent | ImageContent]
+    origin: ToolCall
+    error: bool
+
+
+@dataclass
+class ImageContent:
+    """
+    The image content of a chat message.
+
+    :param base64_image: A base64 string representing the image.
+    :param mime_type: The MIME type of the image (e.g. "image/png", "image/jpeg").
+        Providing this value is recommended, as most LLM providers require it.
+        If not provided, the MIME type is guessed from the base64 string, which can be slow and not always reliable.
+    :param detail: Optional detail level of the image (only supported by OpenAI). One of "auto", "high", or "low".
+    :param meta: Optional metadata for the image.
+    :param validation: If True (default), a validation process is performed:
+        - Check whether the base64 string is valid;
+        - Guess the MIME type if not provided;
+        - Check if the MIME type is a valid image MIME type.
+        Set to False to skip validation and speed up initialization.
+    """
+
+    base64_image: str
+    mime_type: Optional[str] = None
+    detail: Optional[Literal["auto", "high", "low"]] = None
+    meta: Dict[str, Any] = field(default_factory=dict)
+    validation: bool = True
+
+
+@dataclass
+class FileContent:
+    """
+    The file content of a chat message.
+
+    :param base64_data: A base64 string representing the file.
+    :param mime_type: The MIME type of the file (e.g. "application/pdf").
+        Providing this value is recommended, as most LLM providers require it.
+        If not provided, the MIME type is guessed from the base64 string, which can be slow and not always reliable.
+    :param filename: Optional filename of the file. Some LLM providers use this information.
+    :param extra: Dictionary of extra information about the file. Can be used to store provider-specific information.
+        To avoid serialization issues, values should be JSON serializable.
+    :param validation: If True (default), a validation process is performed:
+        - Check whether the base64 string is valid;
+        - Guess the MIME type if not provided.
+        Set to False to skip validation and speed up initialization.
+    """
+
+    base64_data: str
+    mime_type: str | None = None
+    filename: str | None = None
+    extra: dict[str, Any] = field(default_factory=dict)
+    validation: bool = True
+
+
+@dataclass
+class ReasoningContent:
+    """
+    Represents the optional reasoning content prepared by the model, usually contained in an assistant message.
+
+    :param reasoning_text: The reasoning text produced by the model.
+    :param extra: Dictionary of extra information about the reasoning content. Use to store provider-specific
+        information. To avoid serialization issues, values should be JSON serializable.
+    """
+
+    reasoning_text: str
+    extra: Dict[str, Any] = field(default_factory=dict)
+```
+
+The `ImageContent` and `FileContent` dataclasses also provide two convenience class methods: `from_file_path` and `from_url`.
+For more details, refer to our [API documentation](/reference/data-classes-api).
+
+## Working with a ChatMessage
+
+The following examples demonstrate how to create a `ChatMessage` and inspect its properties.
+
+### from_user with TextContent
+
+```python
+from haystack.dataclasses import ChatMessage
+
+user_message = ChatMessage.from_user("What is the capital of Australia?")
+
+print(user_message)
+>>> ChatMessage(
+>>>    _role=<ChatRole.USER: 'user'>,
+>>>    _content=[TextContent(text='What is the capital of Australia?')],
+>>>    _name=None,
+>>>    _meta={}
+>>>)
+
+print(user_message.text)
+>>> What is the capital of Australia?
+
+print(user_message.texts)
+>>> ['What is the capital of Australia?']
+```
+
+### from_user with TextContent and ImageContent
+
+```python
+from haystack.dataclasses import ChatMessage, ImageContent
+
+lion_image_url = (
+    "https://images.unsplash.com/photo-1546182990-dffeafbe841d?"
+	"ixlib=rb-4.0&q=80&w=1080&fit=max"
+)
+
+image_content = ImageContent.from_url(lion_image_url, detail="low")
+
+user_message = ChatMessage.from_user(
+	content_parts=[
+		"What does the image show?",
+		image_content
+		])
+
+print(user_message)
+>>> ChatMessage(
+>>>     _role=<ChatRole.USER: 'user'>,
+>>>     _content=[
+>>>         TextContent(text='What does the image show?'),
+>>>         ImageContent(
+>>>             base64_image='/9j/4...',
+>>>             mime_type='image/jpeg',
+>>>             detail='low',
+>>>             meta={
+>>>                 'content_type': 'image/jpeg',
+>>>                 'url': '...'
+>>>             }
+>>>         )
+>>>     ],
+>>>     _name=None,
+>>>     _meta={}
+>>> )
+
+print(user_message.text)
+>>> What does the image show?
+
+print(user_message.texts)
+>>> ['What does the image show?']
+
+print(user_message.image)
+>>> ImageContent(
+>>>     base64_image='/9j/4...',
+>>>     mime_type='image/jpeg',
+>>>     detail='low',
+>>>     meta={
+>>>         'content_type': 'image/jpeg',
+>>>         'url': '...'
+>>>     }
+>>> )
+```
+
+### from_user with TextContent and FileContent
+
+```python
+from haystack.dataclasses import ChatMessage, FileContent
+
+paper_url = "https://arxiv.org/pdf/2309.08632"
+
+file_content = FileContent.from_url(paper_url)
+
+user_message = ChatMessage.from_user(
+	content_parts=[
+        file_content,
+        "Summarize this paper in 100 words."
+		])
+
+print(user_message)
+>>> ChatMessage(
+>>>     _role=<ChatRole.USER: 'user'>,
+>>>     _content=[
+>>>         FileContent(
+>>>             base64_data='JVBERi0...',
+>>>             mime_type='application/pdf',
+>>>             filename='2309.08632',
+>>>             extra={}
+>>>         ),
+>>>         TextContent(text='Summarize this paper in 100 words.')
+>>>     ],
+>>>     _name=None,
+>>>     _meta={}
+>>> )
+
+print(user_message.text)
+>>> Summarize this paper in 100 words.
+
+print(user_message.texts)
+>>> ['Summarize this paper in 100 words.']
+
+print(user_message.file)
+>>> FileContent(
+>>>     base64_data='JVBERi0...',
+>>>     mime_type='application/pdf',
+>>>     filename='2309.08632',
+>>>     extra={}
+>>> )
+```
+
+### from_assistant with TextContent
+
+```python
+from haystack.dataclasses import ChatMessage
+
+assistant_message = ChatMessage.from_assistant("How can I assist you today?")
+
+print(assistant_message)
+>>> ChatMessage(
+>>>    _role=<ChatRole.ASSISTANT: 'assistant'>,
+>>>    _content=[TextContent(text='How can I assist you today?')],
+>>>    _name=None,
+>>>    _meta={}
+>>>)
+
+print(assistant_message.text)
+>>> How can I assist you today?
+
+print(assistant_message.texts)
+>>> ['How can I assist you today?']
+```
+
+### from_assistant with ToolCall
+
+```python
+from haystack.dataclasses import ChatMessage, ToolCall
+
+tool_call = ToolCall(tool_name="weather_tool", arguments={"location": "Rome"})
+
+assistant_message_w_tool_call = ChatMessage.from_assistant(tool_calls=[tool_call])
+
+print(assistant_message_w_tool_call)
+>>> ChatMessage(
+>>>    _role=<ChatRole.ASSISTANT: 'assistant'>,
+>>>    _content=[ToolCall(tool_name='weather_tool', arguments={'location': 'Rome'}, id=None)],
+>>>    _name=None,
+>>>    _meta={}
+>>>)
+
+print(assistant_message_w_tool_call.text)
+>>> None
+
+print(assistant_message_w_tool_call.texts)
+>>> []
+
+print(assistant_message_w_tool_call.tool_call)
+>>> ToolCall(tool_name='weather_tool', arguments={'location': 'Rome'}, id=None)
+
+print(assistant_message_w_tool_call.tool_calls)
+>>> [ToolCall(tool_name='weather_tool', arguments={'location': 'Rome'}, id=None)]
+
+print(assistant_message_w_tool_call.tool_call_result)
+>>> None
+
+print(assistant_message_w_tool_call.tool_call_results)
+>>> []
+```
+
+### from_tool
+
+```python
+from haystack.dataclasses import ChatMessage
+
+tool_message = ChatMessage.from_tool(tool_result="temperature: 25°C", origin=tool_call, error=False)
+
+print(tool_message)
+>>> ChatMessage(
+>>>    _role=<ChatRole.TOOL: 'tool'>,
+>>>    _content=[ToolCallResult(
+>>>							   result='temperature: 25°C',
+>>>                origin=ToolCall(tool_name='weather_tool', arguments={'location': 'Rome'}, id=None),
+>>>                error=False
+>>>                )],
+>>>    _name=None,
+>>>    _meta={}
+>>>)
+
+print(tool_message.text)
+>>> None
+
+print(tool_message.texts)
+>>> []
+
+print(tool_message.tool_call)
+>>> None
+
+print(tool_message.tool_calls)
+>>> []
+
+print(tool_message.tool_call_result)
+>>> ToolCallResult(
+>>>     result='temperature: 25°C',
+>>>     origin=ToolCall(tool_name='weather_tool', arguments={'location': 'Rome'}, id=None),
+>>>     error=False
+>>> )
+
+print(tool_message.tool_call_results)
+>>> [
+>>>     ToolCallResult(
+>>>         result='temperature: 25°C',
+>>>         origin=ToolCall(tool_name='weather_tool', arguments={'location': 'Rome'}, id=None),
+>>>         error=False
+>>>     )
+>>> ]
+```
+
+## Migrating from Legacy ChatMessage (before v2.9)
+
+In Haystack 2.9, we updated the `ChatMessage` data class for greater flexibility and support for multiple content types: text, tool calls, and tool call results.
+
+There are some breaking changes involved, so we recommend reviewing this guide to migrate smoothly.
+
+### Creating a ChatMessage
+
+You can no longer directly initialize `ChatMessage` using `role`, `content`, and `meta`.
+
+- Use the following class methods instead: `from_assistant`, `from_user`, `from_system`, and `from_tool`.
+- Replace the `content` parameter with `text`.
+
+```python
+from haystack.dataclasses import ChatMessage
+
+# LEGACY - DOES NOT WORK IN 2.9.0
+message = ChatMessage(role=ChatRole.USER, content="Hello!")
+
+# Use the class method instead
+message = ChatMessage.from_user("Hello!")
+```
+
+### Accessing ChatMessage Attributes
+
+- The legacy `content` attribute is now internal (`_content`).
+- Inspect `ChatMessage` attributes using the following properties:
+  - `role`
+  - `meta`
+  - `name`
+  - `text` and `texts`
+  - `image` and `images`
+  - `tool_call` and `tool_calls`
+  - `tool_call_result` and `tool_call_results`
+  - `reasoning` and `reasonings`
+
+```python
+from haystack.dataclasses import ChatMessage
+
+message = ChatMessage.from_user("Hello!")
+
+# LEGACY - DOES NOT WORK IN 2.9.0
+print(message.content)
+
+# Use the appropriate property instead
+print(message.text)
+```

docs-website/versioned_docs/version-2.29/concepts/data-classes/filecontent.mdx

@@ -0,0 +1,119 @@
+---
+title: "FileContent"
+id: filecontent
+slug: "/filecontent"
+description: "`FileContent` represents file payloads in chat messages, including base64 data, MIME type, filename, and provider-specific metadata."
+---
+
+# FileContent
+
+`FileContent` represents a file payload that can be attached to a [`ChatMessage`](chatmessage.mdx). Use it when a chat model accepts file inputs, such as PDFs or other documents, together with the user's text prompt.
+
+If you need the full list of parameters and methods, see the [`FileContent` API reference](/reference/data-classes-api#filecontent).
+
+## Attributes
+
+```python
+@dataclass
+class FileContent:
+    base64_data: str
+    mime_type: str | None = None
+    filename: str | None = None
+    extra: dict[str, Any] = field(default_factory=dict)
+    validation: bool = True
+```
+
+- `base64_data` stores the file content as a base64-encoded string.
+- `mime_type` identifies the file type, for example `application/pdf`. Providing it explicitly is recommended because many model providers require it.
+- `filename` is optional, but some providers use it when processing uploaded files.
+- `extra` can store provider-specific metadata. Values should be JSON serializable.
+- `validation` checks that `base64_data` is valid and tries to infer the MIME type when one is not provided.
+
+## Create from a file path
+
+Use `from_file_path` to read a local file, base64-encode it, infer the MIME type from the path, and populate the filename.
+
+```python
+from haystack.dataclasses import ChatMessage, FileContent
+
+file_content = FileContent.from_file_path("data/attention-is-all-you-need.pdf")
+
+message = ChatMessage.from_user(
+    content_parts=[
+        file_content,
+        "Summarize the key ideas in this paper.",
+    ]
+)
+```
+
+Pass `filename` or `extra` when a provider expects a specific filename or provider-specific options:
+
+```python
+file_content = FileContent.from_file_path(
+    "data/report.pdf",
+    filename="quarterly-report.pdf",
+    extra={"source": "finance"},
+)
+```
+
+## Create from a URL
+
+Use `from_url` to download a file and convert it into a `FileContent` instance.
+
+```python
+from haystack.dataclasses import FileContent
+
+file_content = FileContent.from_url(
+    "https://example.com/reports/quarterly-report.pdf",
+    timeout=30,
+)
+```
+
+If no filename is provided, Haystack uses the final path segment of the URL.
+
+## Create from base64 data
+
+If you already have file bytes, encode them and pass the MIME type explicitly.
+
+```python
+import base64
+from pathlib import Path
+
+from haystack.dataclasses import FileContent
+
+data = Path("data/manual.pdf").read_bytes()
+file_content = FileContent(
+    base64_data=base64.b64encode(data).decode("utf-8"),
+    mime_type="application/pdf",
+    filename="manual.pdf",
+)
+```
+
+Set `validation=False` only when the base64 data and MIME type are already trusted and you want to skip validation.
+
+## Inspect files in a ChatMessage
+
+After adding `FileContent` to a `ChatMessage`, use the `file` and `files` properties to access file payloads.
+
+```python
+from haystack.dataclasses import ChatMessage, FileContent
+
+file_content = FileContent.from_file_path("data/invoice.pdf")
+message = ChatMessage.from_user(content_parts=[file_content, "Extract the invoice total."])
+
+print(message.file)
+print(message.files)
+```
+
+`message.file` returns the first file payload, or `None` if there are no files. `message.files` returns all file payloads.
+
+## Serialization
+
+Use `to_dict` and `from_dict` to serialize and restore file content.
+
+```python
+payload = file_content.to_dict()
+restored = FileContent.from_dict(payload)
+```
+
+For tracing, Haystack replaces the full base64 payload with a placeholder so large files are not sent to the tracing backend.

docs-website/versioned_docs/version-2.29/concepts/device-management.mdx

@@ -0,0 +1,127 @@
+---
+title: "Device Management"
+id: device-management
+slug: "/device-management"
+description: "This page discusses the concept of device management in the context of Haystack."
+---
+
+# Device Management
+
+This page discusses the concept of device management in the context of Haystack.
+
+Many Haystack components, such as `HuggingFaceLocalGenerator` , `AzureOpenAIGenerator`, and others, allow users the ability to pick and choose which language model is to be queried and executed. For components that interface with cloud-based services, the service provider automatically takes care of the details of provisioning the requisite hardware (like GPUs). However, if you wish to use models on your local machine, you’ll need to figure out how to deploy them on your hardware. Further complicating things, different ML libraries have different APIs to launch models on specific devices.
+
+To make the process of running inference on local models as straightforward as possible, Haystack uses a framework-agnostic device management implementation. Exposing devices through this interface means you no longer need to worry about library-specific invocations and device representations.
+
+## Concepts
+
+Haystack’s device management is built on the following abstractions:
+
+- `DeviceType`  - An enumeration that lists all the different types of supported devices.
+- `Device`  - A generic representation of a device composed of a `DeviceType` and a unique identifier. Together, it represents a single device in the group of all available devices.
+- `DeviceMap` - A mapping of strings to `Device` instances. The strings represent model-specific identifiers, usually model parameters. This allows us to map specific parts of a model to specific devices.
+- `ComponentDevice` - A tagged union of a single `Device` or a `DeviceMap` instance. Components that support local inference will expose an optional `device` parameter of this type in their constructor.
+
+With the above abstractions, Haystack can fully address any supported device that’s part of your local machine and can support the usage of multiple devices at the same time. Every component that supports local inference will internally handle the conversion of these generic representations to their backend-specific representations.
+
+:::info[Source Code]
+
+Find the full code for the abstractions above in the Haystack GitHub [repo](https://github.com/deepset-ai/haystack/blob/6a776e672fb69cc4ee42df9039066200f1baf24e/haystack/utils/device.py).
+:::
+
+## Usage
+
+To use a single device for inference, use either the `ComponentDevice.from_single` or `ComponentDevice.from_str` class method:
+
+```python
+from haystack.utils import ComponentDevice, Device
+
+device = ComponentDevice.from_single(Device.gpu(id=1))
+# Alternatively, use a PyTorch device string
+device = ComponentDevice.from_str("cuda:1")
+generator = HuggingFaceLocalGenerator(model="llama2", device=device)
+```
+
+To use multiple devices, use the `ComponentDevice.from_multiple` class method:
+
+```python
+from haystack.utils import ComponentDevice, Device, DeviceMap
+
+device_map = DeviceMap(
+    {
+        "encoder.layer1": Device.gpu(id=0),
+        "decoder.layer2": Device.gpu(id=1),
+        "self_attention": Device.disk(),
+        "lm_head": Device.cpu(),
+    },
+)
+device = ComponentDevice.from_multiple(device_map)
+generator = HuggingFaceLocalGenerator(model="llama2", device=device)
+```
+
+### Integrating Devices in Custom Components
+
+Components should expose an optional `device` parameter of type `ComponentDevice`.  Once exposed, they can determine what to do with it:
+
+- If `device=None`, the component can pass that to the backend. In this case, the backend decides which device the model will be placed on.
+- Alternatively, the component can attempt to automatically pick an available device before passing it to the backend using the `ComponentDevice.resolve_device` class method.
+
+Once the device has been resolved, the component can use the `ComponentDevice.to_*` methods to get the backend-specific representation of the underlying device, which is then passed to the backend.
+
+The `ComponentDevice` instance should be serialized in the component’s `to_dict` and `from_dict` methods.
+
+```python
+from haystack.utils import ComponentDevice, Device, DeviceMap
+
+class MyComponent(Component):
+    def __init__(self, device: Optional[ComponentDevice] = None):
+        # If device is None, automatically select a device.
+        self.device = ComponentDevice.resolve_device(device)
+
+    def warm_up(self):
+        # Call the framework-specific conversion method.
+        self.model = AutoModel.from_pretrained(
+            "deepset/bert-base-cased-squad2", device=self.device.to_hf()
+            )
+
+    def to_dict(self):
+        # Serialize the policy like any other (custom) data.
+        return default_to_dict(
+            self, device=self.device.to_dict() if self.device else None, ...
+        )
+
+    @classmethod
+    def from_dict(cls, data):
+        # Deserialize the device data inplace before passing
+        # it to the generic from_dict function.
+	    init_params = data["init_parameters"]
+        init_params["device"] = ComponentDevice.from_dict(init_params["device"])
+        return default_from_dict(cls, data)
+
+# Automatically selects a device.
+c = MyComponent(device=None)
+
+# Uses the first GPU available.
+c = MyComponent(device=ComponentDevice.from_str("cuda:0"))
+
+# Uses the CPU.
+c = MyComponent(device=ComponentDevice.from_single(Device.cpu()))
+
+# Allow the component to use multiple devices using a device map.
+c = MyComponent(device=ComponentDevice.from_multiple(DeviceMap({
+    "layer1": Device.cpu(),
+    "layer2": Device.gpu(1),
+    "layer3": Device.disk()
+})))
+```
+
+If the component’s backend provides a more specialized API to manage devices, it could add an additional init parameter that acts as a conduit. For instance, `HuggingFaceLocalGenerator` exposes a `huggingface_pipeline_kwargs` parameter through which Hugging Face-specific `device_map`  arguments can be passed:
+
+```python
+generator = HuggingFaceLocalGenerator(
+    model="llama2",
+    huggingface_pipeline_kwargs={"device_map": "balanced"},
+)
+```
+
+In such cases, ensure that the parameter precedence and selection behavior is clearly documented. In the case of `HuggingFaceLocalGenerator`, the device map passed through the `huggingface_pipeline_kwargs` parameter overrides the explicit `device` parameter and is documented as such.

docs-website/versioned_docs/version-2.29/concepts/document-store.mdx

@@ -0,0 +1,103 @@
+---
+title: "Document Store"
+id: document-store
+slug: "/document-store"
+description: "You can think of the Document Store as a database that stores your data and provides them to the Retriever at query time. Learn how to use Document Store in a pipeline or how to create your own."
+---
+
+# Document Store
+
+You can think of the Document Store as a database that stores your data and provides them to the Retriever at query time. Learn how to use Document Store in a pipeline or how to create your own.
+
+Document Store is an object that stores your documents. In Haystack, a Document Store is different from a component, as it doesn't have the `run()` method. You can think of it as an interface to your database – you put the information there, or you can look through it. This means that a Document Store is not a piece of a pipeline but rather a tool that the components of a pipeline have access to and can interact with.
+
+:::tip[Work with Retrievers]
+
+The most common way to use a Document Store in Haystack is to fetch documents using a Retriever. A Document Store will often have a corresponding Retriever to get the most out of specific technologies. See more information in our [Retriever](../pipeline-components/retrievers.mdx) documentation.
+:::
+
+:::note[How to choose a Document Store?]
+
+To learn about different types of Document Stores and their strengths and disadvantages, head to the [Choosing a Document Store](document-store/choosing-a-document-store.mdx) page.
+:::
+
+### DocumentStore Protocol
+
+Document Stores in Haystack are designed to use the following methods as part of their protocol:
+
+- `count_documents` returns the number of documents stored in the given store as an integer.
+- `filter_documents` returns a list of documents that match the provided filters.
+- `write_documents` writes or overwrites documents into the given store and returns the number of documents that were written as an integer.
+- `delete_documents` deletes all documents with given `document_ids` from the Document Store.
+
+### Initialization
+
+To use a Document Store in a pipeline, you must initialize it first.
+
+See the installation and initialization details for each Document Store in the "Document Stores" section in the navigation panel on your left.
+
+### Work with Documents
+
+Convert your data into `Document` objects before writing them into a Document Store along with its metadata and document ID.
+
+The ID field is mandatory, so if you don’t choose a specific ID yourself, Haystack will do its best to come up with a unique ID based on the document’s information and assign it automatically. However, since Haystack uses the document’s contents to create an ID, two identical documents might have identical IDs. Keep it in mind as you update your documents, as the ID will not be updated automatically.
+
+```python
+document_store = ChromaDocumentStore()
+documents = [
+    Document(
+        meta={"name": DOCUMENT_NAME, ...}, id="document_unique_id", content="this is content"
+    ),
+  	...
+]
+document_store.write_documents(documents)
+```
+
+To write documents into the `InMemoryDocumentStore`, simply call the `.write_documents()` function:
+
+```python
+document_store.write_documents(
+    [
+        Document(content="My name is Jean and I live in Paris."),
+        Document(content="My name is Mark and I live in Berlin."),
+        Document(content="My name is Giorgio and I live in Rome."),
+    ],
+)
+```
+
+:::note[`DocumentWriter`]
+
+See `DocumentWriter` component [docs](../pipeline-components/writers/documentwriter.mdx) to write your documents into a Document Store in a pipeline.
+:::
+
+### DuplicatePolicy
+
+The `DuplicatePolicy` is a class that defines the different options for handling documents with the same ID in a `DocumentStore`. It has three possible values:
+
+- **OVERWRITE**: Indicates that if a document with the same ID already exists in the `DocumentStore`, it should be overwritten with the new document.
+- **SKIP**: If a document with the same ID already exists, the new document will be skipped and not added to the `DocumentStore`.
+- **FAIL**: Raises an error if a document with the same ID already exists in the `DocumentStore`. It prevents duplicate documents from being added.
+
+Here is an example of how you could apply the policy to skip the existing document:
+
+```python
+from haystack.document_stores.in_memory import InMemoryDocumentStore
+from haystack.components.writers import DocumentWriter
+from haystack.document_stores.types import DuplicatePolicy
+
+document_store = InMemoryDocumentStore()
+document_writer = DocumentWriter(
+    document_store=document_store,
+    policy=DuplicatePolicy.SKIP,
+)
+```
+
+### Custom Document Store
+
+All custom document stores must implement the [protocol](https://github.com/deepset-ai/haystack/blob/13804293b1bb79743e5a30e980b76a0561dcfaf8/haystack/document_stores/types/protocol.py) with four mandatory methods: `count_documents`,`filter_documents`, `write_documents`, and `delete_documents`.
+
+The `init` function should indicate all the specifics for the chosen database or vector store.
+
+We also recommend having a custom corresponding Retriever to get the most out of a specific Document Store.
+
+See [Creating Custom Document Stores](document-store/creating-custom-document-stores.mdx) page for more details.

docs-website/versioned_docs/version-2.29/concepts/document-store/choosing-a-document-store.mdx

@@ -0,0 +1,109 @@
+---
+title: "Choosing a Document Store"
+id: choosing-a-document-store
+slug: "/choosing-a-document-store"
+description: "This article goes through different types of Document Stores and explains their advantages and disadvantages."
+---
+
+import ClickableImage from "@site/src/components/ClickableImage";
+
+# Choosing a Document Store
+
+This article goes through different types of Document Stores and explains their advantages and disadvantages.
+
+### Introduction
+
+Whether you are developing a chatbot, a RAG system, or an image captioner, at some point, it’ll be likely for your AI application to compare the input it gets with the information it already knows. Most of the time, this comparison is performed through vector similarity search.
+
+If you’re unfamiliar with vectors, think about them as a way to represent text, images, or audio/video in a numerical form called vector embeddings. Vector databases are specifically designed to store such vectors efficiently, providing all the functionalities an AI application needs to implement data retrieval and similarity search.
+
+Document Stores are special objects in Haystack that abstract all the different vector databases into a common interface that can be easily integrated into a pipeline, most commonly through a Retriever component. Normally, you will find specialized Document Store and Retriever objects for each vector database Haystack supports.
+
+### Types of vector databases
+
+But why are vector databases so different, and which one should you use in your Haystack pipeline?
+
+We can group vector databases into five categories, from more specialized to general purpose:
+
+- Vector libraries
+- Pure vector databases
+- Vector-capable SQL databases
+- Vector-capable NoSQL databases
+- Full-text search databases
+
+We are working on supporting all these types in Haystack.
+
+In the meantime, here’s the most recent overview of available integrations:
+<ClickableImage src="/img/2c188e9-2.0_Document_Stores_6.png" alt="Document store categories diagram showing four types: pure vector databases (Chroma, Milvus, Pinecone, Weaviate, Qdrant), full-text search databases (Elasticsearch, OpenSearch), vector-capable SQL databases (Pgvector for PostgreSQL), and vector-capable NoSQL databases (DataStax Astra, MongoDB, neo4j)" className="img-light-bg" />
+
+#### Summary
+
+Here is a quick summary of different Document Stores available in Haystack.
+
+Continue further down the article for a more complex explanation of the strengths and disadvantages of each type.
+
+<div className="key-value-table">
+
+|  |  |
+| --- | --- |
+| Type                     | Best for                                                                                            |
+| Vector libraries         | Managing hardware resources effectively.                                                            |
+| Pure vector DBs          | Managing lots of high-dimensional data.                                                             |
+| Vector-capable SQL DBs   | Lower maintenance costs with focus on structured data and less on vectors.                          |
+| Vector-capable NoSQL DBs | Combining vectors with structured data without the limitations of the traditional relational model. |
+| Full-text search DBs     | Superior full-text search, reliable for production.                                                 |
+| In-memory                | Fast, minimal prototypes on small datasets.                                                         |
+
+</div>
+
+#### Vector libraries
+
+Vector libraries are often included in the “vector database” category improperly, as they are limited to handling only vectors, are designed to work in-memory, and normally don’t have a clean way to store data on disk. Still, they are the way to go every time performance and speed are the top requirements for your AI application, as these libraries can use hardware resources very effectively.
+
+:::warning[In progress]
+
+We are currently developing the support for vector libraries in Haystack.
+:::
+
+#### Pure vector databases
+
+Pure vector databases, also known as just “vector databases”, offer efficient similarity search capabilities through advanced indexing techniques. Most of them support metadata, and despite a recent trend to add more text-search features on top of it, you should consider pure vector databases closer to vector libraries than a regular database. Pick a pure vector database when your application needs to manage huge amounts of high-dimensional data effectively: they are designed to be highly scalable and highly available. Most are open source, but companies usually provide them “as a service” through paid subscriptions.
+
+- [Chroma](../../document-stores/chromadocumentstore.mdx)
+- [Pinecone](../../document-stores/pinecone-document-store.mdx)
+- [Qdrant](../../document-stores/qdrant-document-store.mdx)
+- [Weaviate](../../document-stores/weaviatedocumentstore.mdx)
+- [Milvus](https://haystack.deepset.ai/integrations/milvus-document-store) (external integration)
+
+#### Vector-capable SQL databases
+
+This category is relatively small but growing fast and includes well-known relational databases where vector capabilities were added through plugins or extensions. They are not as performant as the previous categories, but the main advantage of these databases is the opportunity to easily combine vectors with structured data, having a one-stop data shop for your application. You should pick a vector-capable SQL database when the performance trade-off is paid off by the lower cost of maintaining a single database instance for your application or when the structured data plays a more fundamental role in your business logic, with vectors being more of a nice-to-have.
+
+- [Pgvector](../../document-stores/pgvectordocumentstore.mdx)
+
+#### Vector-capable NoSQL databases
+
+Historically, the killer features of NoSQL databases were the ability to scale horizontally and the adoption of a flexible data model to overcome certain limitations of the traditional relational model. This stays true for databases in this category, where the vector capabilities are added on top of the existing features. Similarly to the previous category, vector support might not be as good as pure vector databases, but once again, there is a tradeoff that might be convenient to bear depending on the use case. For example, if a certain NoSQL database is already part of the stack of your application and a lower performance is not a show-stopper, you might give it a shot.
+
+- [Astra](../../document-stores/astradocumentstore.mdx)
+- [MongoDB](../../document-stores/mongodbatlasdocumentstore.mdx)
+- [Neo4j](https://haystack.deepset.ai/integrations/neo4j-document-store) (external)
+
+#### Full-text search databases
+
+The main advantage of full-text search databases is they are already designed to work with text, so you can expect a high level of support for text data along with good performance and the opportunity to scale both horizontally and vertically. Initially, vector capabilities were subpar and provided through plugins or extensions, but this is rapidly changing. You can see how the market leaders in this category have recently added first-class support for vectors. Pick a full-text search database if text data plays a central role in your business logic so that you can easily and effectively implement techniques like hybrid search with a good level of support for similarity search and state-of-the-art support for full-text search.
+
+- [Elasticsearch](../../document-stores/elasticsearch-document-store.mdx)
+- [OpenSearch](../../document-stores/opensearch-document-store.mdx)
+
+#### The in-memory Document Store
+
+Haystack ships with an ephemeral document store that relies on pure Python data structures stored in memory, so it doesn’t fall into any of the vector database categories above. This special Document Store is ideal for creating quick prototypes with small datasets. It doesn’t require any special setup, and it can be used right away without installing additional dependencies.
+
+- [InMemory](../../document-stores/inmemorydocumentstore.mdx)
+
+### Final considerations
+
+It can be very challenging to pick one vector database over another by only looking at pure performance, as even the slightest difference in the benchmark can produce a different leaderboard (for example, some benchmarks test the cloud services while others work on a reference machine). Thinking about including features like filtering or not can bring in a whole new set of complexities that make the comparison even harder.
+
+What’s important for you to know is that the Document Store interface doesn’t add much to the costs, and the relative performance of one vector database over another should stay the same when used within Haystack pipelines.

docs-website/versioned_docs/version-2.29/concepts/document-store/creating-custom-document-stores.mdx

@@ -0,0 +1,174 @@
+---
+title: "Creating Custom Document Stores"
+id: creating-custom-document-stores
+slug: "/creating-custom-document-stores"
+description: "Create your own Document Stores to manage your documents."
+---
+
+# Creating Custom Document Stores
+
+Create your own Document Stores to manage your documents.
+
+Custom Document Stores are resources that you can build and leverage in situations where a ready-made solution is not available in Haystack. For example:
+
+- You’re working with a vector store that’s not yet supported in Haystack.
+- You need a very specific retrieval strategy to search for your documents.
+- You want to customize the way Haystack reads and writes documents.
+
+Similar to [custom components](../components/custom-components.mdx), you can use a custom Document Store in a Haystack pipeline as long as you can import its code into your Python program. The best practice is distributing a custom Document Store as a standalone integration package.
+
+## Recommendations
+
+Before you start, there are a few recommendations we provide to ensure a custom Document Store behaves consistently with the rest of the Haystack ecosystem. At the end of the day, a Document Store is just Python code written in a way that Haystack can understand, but the way you name it, organize it, and distribute it can make a difference. None of these recommendations are mandatory, but we encourage you to follow as many as you can.
+
+### Naming Convention
+
+We recommend naming your Document Store following the format `<TECHNOLOGY>-haystack`, for example, `chroma-haystack`. This makes it consistent with the others, lowering the cognitive load for your users and easing discoverability.
+
+This naming convention applies to the name of the git repository (`https://github.com/your-org/example-haystack`) and the name of the Python package (`example-haystack`).
+
+### Structure
+
+More often than not, a Document Store can be fairly complex, and setting up a dedicated Git repository can be handy and future-proof. To ease this step, we prepared a [GitHub template](https://github.com/deepset-ai/custom-component) that provides the structure you need to host a custom Document Store in a dedicated repository. It includes the boilerplate for packaging, testing, and distributing your custom Document Store as a standalone Python package.
+
+See the instructions in the [template repository](https://github.com/deepset-ai/custom-component) to get started. You can also watch the [video walkthrough](https://www.youtube.com/watch?v=SWC0QecAMcI) for a step-by-step guide.
+
+### Packaging
+
+As with any other [Haystack integration](../integrations.mdx), a Document Store can be added to your Haystack applications by installing an additional Python package, for example, with `pip`. Once you have a Git repository hosting your Document Store and a `pyproject.toml` file to create an `example-haystack` package (using our [GitHub template](https://github.com/deepset-ai/custom-component)), it will be possible to `pip install` it directly from sources, for example:
+
+```shell
+pip install git+https://github.com/your-org/example-haystack.git
+```
+
+Though very practical to quickly deliver prototypes, if you want others to use your custom Document Store, we recommend you publish a package on PyPI so that it will be versioned and installable with simply:
+
+```shell
+pip install example-haystack
+```
+
+:::tip
+👍
+
+Our [GitHub template](https://github.com/deepset-ai/custom-component) ships a GitHub workflow that will automatically publish the Document Store package on PyPI.
+:::
+
+### Documentation
+
+We recommend thoroughly documenting your custom Document Store with a detailed README file and possibly generating API documentation using a static generator.
+
+For inspiration, see the [neo4j-haystack](https://github.com/prosto/neo4j-haystack) repository and its [documentation](https://prosto.github.io/neo4j-haystack/) pages.
+
+## Implementation
+
+### DocumentStore Protocol
+
+You can use any Python class as a Document Store, provided that it implements all the methods of the `DocumentStore` Python protocol defined in Haystack:
+
+```python
+class DocumentStore(Protocol):
+    def to_dict(self) -> Dict[str, Any]:
+        """
+        Serializes this store to a dictionary.
+        """
+
+    @classmethod
+    def from_dict(cls, data: Dict[str, Any]) -> "DocumentStore":
+        """
+        Deserializes the store from a dictionary.
+        """
+
+    def count_documents(self) -> int:
+        """
+        Returns the number of documents stored.
+        """
+
+    def filter_documents(
+        self,
+        filters: Optional[Dict[str, Any]] = None,
+    ) -> List[Document]:
+        """
+        Returns the documents that match the filters provided.
+        """
+
+    def write_documents(
+        self,
+        documents: List[Document],
+        policy: DuplicatePolicy = DuplicatePolicy.FAIL,
+    ) -> int:
+        """
+        Writes (or overwrites) documents into the DocumentStore, return the number of documents that was written.
+        """
+
+    def delete_documents(self, document_ids: List[str]) -> None:
+        """
+        Deletes all documents with a matching document_ids from the DocumentStore.
+        """
+```
+
+The `DocumentStore` interface supports the basic CRUD operations you would normally perform on a database or a storage system, and mostly generic components like [`DocumentWriter`](../../pipeline-components/writers/documentwriter.mdx) use it.
+
+### Additional Methods
+
+Usually, a Document Store comes with additional methods that can provide advanced search functionalities. These methods are not part of the `DocumentStore` protocol and don’t follow any particular convention. We designed it like this to provide maximum flexibility to the Document Store when using any specific features of the underlying database.
+
+Some additional methods that are not part of the `DocumentStore` protocol, but are implemented by most Document Stores in Haystack, include:
+
+```python
+def delete_all_documents(recreate_index: bool = False)
+def update_by_filter(filters: dict[str, Any], meta: dict[str, Any], refresh: bool = False) -> int:
+def delete_by_filter(filters: dict[str, Any]) -> int:
+```
+These methods are not part of the Protocol but highly recommended to implement in your custom Document Store, as users often expect them to be available.
+
+For example, Haystack wouldn’t get in the way when your Document Store defines a specific `search` method that takes a long list of parameters that only make sense in the context of a particular vector database. Normally, a [Retriever](../../pipeline-components/retrievers.mdx) component would then use this additional search method.
+
+### Retrievers
+
+To get the most out of your custom Document Store, in most cases, you would need to create one or more accompanying Retrievers that use the additional search methods mentioned above. Before proceeding and implementing your custom Retriever, it might be helpful to learn more about  [Retrievers](../../pipeline-components/retrievers.mdx) in general through the Haystack documentation.
+
+From the implementation perspective, Retrievers in Haystack are like any other custom component. For more details, refer to the [creating custom components](../components/custom-components.mdx) documentation page.
+
+Although not mandatory, we encourage you to follow more specific [naming conventions](../../pipeline-components/retrievers.mdx#naming-conventions) for your custom Retriever.
+
+### Serialization
+
+Haystack requires every component to be representable by a Python dictionary for correct serialization implementation. Some components, such as Retrievers and Writers, maintain a reference to a Document Store instance. Therefore, `DocumentStore` classes should implement the `from_dict` and `to_dict` methods. This allows to rebuild an instance after reading a pipeline from a file.
+
+For a practical example of what to serialize in a custom Document Store, consider a database client you created using an IP address and a database name. When constructing the dictionary to return in `to_dict`, you would store the IP address and the database name, not the database client instance.
+
+### Secrets Management
+
+There's a likelihood that users will need to provide sensitive data, such as passwords, API keys, or private URLs, to create a Document Store instance. This sensitive data could potentially be leaked if it's passed around in plain text.
+
+Haystack has a specific way to wrap sensitive data into special objects called Secrets. This prevents the data from being leaked during serialization roundtrips. We strongly recommend using this feature extensively for data security (better safe than sorry!).
+
+You can read more about Secret Management in Haystack [documentation](../secret-management.mdx).
+
+### Testing
+
+Haystack comes with some testing functionalities you can use in a custom Document Store. In particular, an empty class inheriting from `DocumentStoreBaseTests` would already run the standard tests that any Document Store is expected to pass in order to work properly.
+
+### Implementation Tips
+
+- The best way to learn how to write a custom Document Store is to look at the existing ones: the `InMemoryDocumentStore`, which is part of Haystack, or the [`ElasticsearchDocumentStore`](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/elasticsearch), which is a Core Integration, are good places to start.
+- When starting from scratch, it might be easier to create the four CRUD methods of the `DocumentStore` protocol one at a time and test them one at a time as well. For example:
+  1. Implement the logic for `count_documents`.
+  2. In your `test_document_store.py` module, define the test class `TestDocumentStore(CountDocumentsTest)`. Note how we only inherit from the specific testing mix-in `CountDocumentsTest`.
+  3. Make the tests pass.
+  4. Implement the logic for `write_documents`.
+  5. Change `test_document_store.py` so that your class now also derives from the `WriteDocumentsTest` mix-in: `TestDocumentStore(CountDocumentsTest, WriteDocumentsTest)`.
+  6. Keep iterating with the remaining methods.
+- Having a notebook where users can try out your Document Store in a full pipeline can really help adoption, and it’s a great source of documentation. Our [haystack-cookbook](https://github.com/deepset-ai/haystack-cookbook) repository has good visibility, and we encourage contributors to create a PR and add their own.
+
+Verifying that the implementation meets all `DocumentStoreBaseTests` [tests](https://github.com/deepset-ai/haystack/blob/main/haystack/testing/document_store.py) is the minimum requirement for a custom Document Store to be consistent with the rest of the Haystack ecosystem.
+
+But, ideally making it compatible with the ``DocumentStoreBaseExtendedTests`` tests is a good way to ensure that your Document Store meets all the common used functionalities that users expect from a Document Store, such as `delete_all_documents` or `update_by_filter`.
+
+If the technology you are using for your Document Store supports asynchronous operations, we recommend implementing `async` versions of the methods in the `DocumentStore` protocol as well. This allows users to take advantage of async features in their applications and pipelines, improving performance and scalability.
+
+## Get Featured on the Integrations Page
+
+The [Integrations web page](https://haystack.deepset.ai/integrations) makes Haystack integrations visible to the community, and it’s a great opportunity to showcase your work. Once your Document Store is usable and properly packaged, you can open a pull request in the [haystack-integrations](https://github.com/deepset-ai/haystack-integrations) GitHub repository to add an integration tile.
+
+See the [integrations documentation page](../integrations.mdx#how-do-i-showcase-my-integration) for more details.

docs-website/versioned_docs/version-2.29/concepts/experimental-package.mdx

@@ -0,0 +1,67 @@
+---
+title: "Experimental Package"
+id: experimental-package
+slug: "/experimental-package"
+description: "Try out new experimental features with Haystack."
+---
+
+# Experimental Package
+
+Try out new experimental features with Haystack.
+
+The `haystack-experimental` package allows you to test new experimental features without committing to their official release. Its main goal is to gather user feedback and iterate on new features quickly.
+
+Check out the `haystack-experimental` [GitHub repository](https://github.com/deepset-ai/haystack-experimental) for the latest catalog of available features, or take a look at our [Experiments API Reference](/reference).
+
+### Installation
+
+For simplicity, every release of `haystack-experimental` includes all the available experiments at that time. To install the latest features, run:
+
+```shell
+pip install -U haystack-experimental
+```
+
+:::info
+The latest version of the experimental package is only tested against the latest version of Haystack. Compatibility with older versions of Haystack is not guaranteed.
+:::
+
+### Lifecycle
+
+Each experimental feature has a default lifespan of 3 months starting from the date of the first non-pre-release build that includes it. Once it reaches the end of its lifespan, we will remove it from `haystack-experimental` and either:
+
+- Merge the feature into Haystack and publish it with the next minor release,
+- Release the feature as an integration, or
+- Drop the feature.
+
+### Usage
+
+You can import the experimental new features like any other Haystack integration package:
+
+```python
+from haystack.dataclasses import ChatMessage
+from haystack_experimental.components.generators import FoobarGenerator
+
+c = FoobarGenerator()
+c.run([ChatMessage.from_user("What's an experiment? Be brief.")])
+```
+
+Experiments can also override existing Haystack features. For example, you can opt into an experimental type of `Pipeline` by changing the usual import:
+
+```python
+# from haystack import Pipeline
+from haystack_experimental import Pipeline
+
+pipe = Pipeline()
+# ...
+pipe.run(...)
+```
+
+## Additional References
+
+🧑‍🍳 Cookbooks:
+
+- [Improving Retrieval with Auto-Merging and Hierarchical Document Retrieval](https://haystack.deepset.ai/cookbook/auto_merging_retriever)
+- [Invoking APIs with OpenAPITool](https://haystack.deepset.ai/cookbook/openapitool)
+- [Conversational RAG using Memory](https://haystack.deepset.ai/cookbook/conversational_rag_using_memory)
+- [Define & Run Tools](https://haystack.deepset.ai/cookbook/tools_support)
+- [Newsletter Sending Agent with Experimental Haystack Tools](https://haystack.deepset.ai/cookbook/newsletter-agent)

docs-website/versioned_docs/version-2.29/concepts/integrations.mdx

@@ -0,0 +1,60 @@
+---
+title: "Introduction to Integrations"
+id: integrations
+slug: "/integrations"
+description: "The Haystack ecosystem integrates with many other technologies, such as vector databases, model providers and even custom components made by the community.  Here you can explore our integrations, which may be maintined by deepset, or submitted by others."
+---
+
+# Introduction to Integrations
+
+The Haystack ecosystem integrates with many other technologies, such as vector databases, model providers and even custom components made by the community.  Here you can explore our integrations, which may be maintined by deepset, or submitted by others.
+
+Haystack integrates with a number of other technologies and tools. For example, you can use a number of different model providers or databases with Haystack. 
+
+There are two main types of integrations:
+
+- **Maintained by deepset:** All of the integrations we maintain are hosted in the [haystack-core-integrations](https://github.com/deepset-ai/haystack-core-integrations) repository.
+- **Maintained by our partners or community:** These are integrations that you, our partners, or anyone else can build and maintain themselves. Given they comply with some of our requirements, we will also showcase these on our website.
+
+## What are integrations?
+
+An integration is any type of external technology that can be used to extend the capabilities of the Haystack framework. Some integration examples are those providing access to model providers like OpenAI or Cohere, to databases like Weaviate and Qdrant, or even to monitoring tools such as Traceloop. They can be components, Document Stores, or any other feature that can be used with Haystack.
+
+We maintain a list of available integrations on the [Haystack Integrations](https://haystack.deepset.ai/integrations) page, where you can see which integrations we maintain or which have been contributed by the community.
+
+An integrations page focuses on explaining how Haystack integrates with that technology. For example, the OpenAI integration page will provide a summary of the various ways Haystack and OpenAI can work together.
+
+Here are the integration types you can currently choose from:
+
+- **Model Provider**: You can see how we integrate with different model providers and the available components through these integrations
+- **Document Store**: These are the databases and vector stores you can use with your Haystack pipelines.
+- **Evaluation Framework**: Evaluation frameworks that are supported by Haystack that you can use to evaluate Haystack pipelines.
+- **Monitoring Tool**: These are tools like Chainlit and Traceloop that integrate with Haystack and provide monitoring and observability capabilities.
+- **Data Ingestion**: These are the integrations that allow you to ingest and use data from different resources, such as Notion, Mastodon, and others.
+- **Custom Component**: Some integrations that cover very unique use cases are often contributed and maintained by our community members. We list these integrations under the _Custom Component_ tag.
+
+## How do I use an integration?
+
+Each page dedicated to an integration contains installation instructions and basic usage instructions. For example, the OpenAI integration page gives you an overview of the different ways in which you can interact with OpenAI.
+
+## How can I create an integration?
+
+The most common types of integrations are custom components and Document Stores. Integrations such as model providers might even include multiple custom components. Have a look at these documentation pages that will guide you through the requirements for each integration type:
+
+- [Creating Custom Components](components/custom-components.mdx)
+- [Creating Custom Document Stores](document-store/creating-custom-document-stores.mdx)
+
+Check out the [video walkthrough](https://www.youtube.com/watch?v=SWC0QecAMcI) for a step-by-step guide on how to use the [custom-component template](https://github.com/deepset-ai/custom-component) to create a Haystack integration.
+
+## How do I showcase my integration?
+
+To make your integration visible to the Haystack community, contribute it to our [haystack-integrations](https://github.com/deepset-ai/haystack-integrations) GitHub repository. There are several requirements you have to follow:
+
+- Make sure your contribution is [packaged](https://packaging.python.org/en/latest/), installable, and runnable. We suggest using [hatch](https://hatch.pypa.io/latest/) for this purpose.
+- Provide the GitHub repo and issue link.
+- Create a Pull Request in the [haystack-integrations](https://github.com/deepset-ai/haystack-integrations) repo by following the [draft-integration.md](https://github.com/deepset-ai/haystack-integrations/blob/main/draft-integration.md) and include a clear explanation of what your integration is. This page should include:
+  - Installation instructions
+  - A list of the components the integration includes
+  - Examples of how to use it with clear/runnable code
+  - Licensing information
+  - (Optionally) Documentation and/or API docs that you’ve generated for your repository

docs-website/versioned_docs/version-2.29/concepts/jinja-templates.mdx

@@ -0,0 +1,58 @@
+---
+title: "Jinja Templates"
+id: jinja-templates
+slug: "/jinja-templates"
+description: "Learn how Jinja templates work with Haystack components."
+---
+
+# Jinja Templates
+
+Learn how Jinja templates work with Haystack components.
+
+Jinja templates are text structures that contain placeholders for generating dynamic content. These placeholders are filled in when the template is rendered. You can check out the full list of Jinja2 features in the [original documentation](https://jinja.palletsprojects.com/en/3.0.x/templates/).
+
+You can use these templates in Haystack [Builders](../pipeline-components/builders.mdx), [OutputAdapter](../pipeline-components/converters/outputadapter.mdx), and [ConditionalRouter](../pipeline-components/routers/conditionalrouter.mdx) components.
+
+Here is an example of `OutputAdapter` using a short Jinja template to output only the content field of the first document in the arrays of documents:
+
+```python
+from haystack import Document
+from haystack.components.converters import OutputAdapter
+
+adapter = OutputAdapter(template="{{ documents[0].content }}", output_type=str)
+input_data = {"documents": [Document(content="Test content")]}
+expected_output = {"output": "Test content"}
+assert adapter.run(**input_data) == expected_output
+```
+
+### Using Python f‑strings with Jinja
+
+When you embed Jinja placeholders inside a Python f‑string, you must escape Jinja’s `{` and `}` by doubling them (so `{{ var }}` becomes `{{{{ var }}}}`). Otherwise, Python will consume the braces and the Jinja variable won’t be found.
+
+Preferred template:
+
+```python
+template = """
+Language: {{ language }}
+Question: {{ question }}
+"""
+# pass both variables when rendering
+```
+
+It you need to use an f‑string (escape braces):
+
+```python
+language = "en"
+template = f"""
+Language: {language}
+Question: {{{{ question }}}}
+"""
+```
+
+## Safety Features
+
+Due to how we use Jinja in some Components, there are some security considerations to take into account. Jinja works by executing embedded in templates, so it’s _imperative_ that they stem from a trusted source. If the template is allowed to be customized by the end user, it can potentially lead to remote code execution.
+
+To mitigate this risk, Jinja templates are executed and rendered in a [sandbox environment](https://jinja.palletsprojects.com/en/3.1.x/sandbox/). While this approach is safer, it's also less flexible and limits the expressiveness of the template. If you need the more advanced functionality of Jinja templates, components that use them provide an `unsafe` init parameter - setting it to `False` will disable the sandbox environment and enable unsafe template rendering.
+
+With unsafe template rendering, the [OutputAdapter](../pipeline-components/converters/outputadapter.mdx) and [ConditionalRouter](../pipeline-components/routers/conditionalrouter.mdx) components allow their `output_type` to be set to one of the [Haystack data classes](data-classes.mdx) such as `ChatMessage`, `Document`, or `Answer`.

docs-website/versioned_docs/version-2.29/concepts/metadata-filtering.mdx

@@ -0,0 +1,155 @@
+---
+title: "Metadata Filtering"
+id: metadata-filtering
+slug: "/metadata-filtering"
+description: "This page provides a detailed explanation of how to apply metadata filters at query time."
+---
+
+# Metadata Filtering
+
+This page provides a detailed explanation of how to apply metadata filters at query time.
+
+When you index documents into your Document Store, you can attach metadata to them. One example is the `DocumentLanguageClassifier`, which adds the language of the document's content to its metadata. Components like `MetadataRouter` can then route documents based on their metadata.
+
+You can then use the metadata to filter your search queries, allowing you to narrow down the results by focusing on specific criteria. This ensures your Retriever fetches answers from the most relevant subset of your data.
+
+To illustrate how metadata filters work, imagine you have a set of annual reports from various companies. You may want to perform a search on just a specific year and just on a small selection of companies. This can reduce the workload of the Retriever and also ensure that you get more relevant results.
+
+## Filtering Types
+
+Filters are defined as a dictionary or nested dictionaries that can be of two types: Comparison or Logic.
+
+### Comparison
+
+Comparison operators help search your metadata fields according the specified conditions.
+
+Comparison dictionaries must contain the following keys:
+
+\-`field`: the name of one of the meta fields of a document, such as `meta.years`.
+
+\-`operator`: must be one of the following:
+
+```
+    - `==`
+    - `!=`
+    - `>`
+    - `>=`
+    - `<`
+    - `<=`
+    - `in`
+    - `not in`
+```
+
+:::info
+The available comparison operators may vary depending on the specific Document Store integration. For example, the `ChromaDocumentStore` supports two additional operators: `contains` and `not contains`. Find the details about the supported filters in the specific integration’s API reference.
+:::
+
+\-`value`: takes a single value or (in the case of "in" and “not in”) a list of values.
+
+#### Example
+
+Here is an example of a simple filter in the form of a dictionary. The filter selects documents classified as “article” in the `type` meta field of the document:
+
+```python
+filters = {"field": "meta.type", "operator": "==", "value": "article"}
+```
+
+### Logic
+
+Logical operators can be used to create a nested dictionary, allowing you to apply multiple `fields` as filter conditions. Logic dictionaries must contain the following keys:
+
+\-`operator`: usually one of the following:
+
+```
+    - `NOT`
+    - `OR`
+    - `AND`
+```
+
+:::info
+The available logic operators may vary depending on the specific Document Store integration. For example, the `ChromaDocumentStore` doesn’t support the `NOT` operator. Find the details about the supported filters in the specific integration’s API reference.
+:::
+
+\-`conditions`: must be a list of dictionaries, either of type Comparison or Logic.
+
+#### Nested Filter Example
+
+Here is a more complex filter that uses both Comparison and Logic to find documents where:
+
+- Meta field `type` is "article",
+- Meta field `date` is between 1420066800 and 1609455600 (a specific date range),
+- Meta field `rating` is greater than or equal to 3,
+- Documents are either classified as `genre`  ["economy", "politics"] `OR` the meta field `publisher` is "nytimes".
+
+```python
+filters = {
+    "operator": "AND",
+    "conditions": [
+        {"field": "meta.type", "operator": "==", "value": "article"},
+        {"field": "meta.date", "operator": ">=", "value": 1420066800},
+        {"field": "meta.date", "operator": "<", "value": 1609455600},
+        {"field": "meta.rating", "operator": ">=", "value": 3},
+        {
+            "operator": "OR",
+            "conditions": [
+                {
+                    "field": "meta.genre",
+                    "operator": "in",
+                    "value": ["economy", "politics"],
+                },
+                {"field": "meta.publisher", "operator": "==", "value": "nytimes"},
+            ],
+        },
+    ],
+}
+```
+
+## Filters Usage
+
+Filters can be applied either through the `Retriever` class or directly within Document Stores.
+
+In the `Retriever` class, filters are passed through the `filters` argument. When working with a pipeline, filters can be provided to `Pipeline.run()`, which will automatically route them to the `Retriever` class (refer to the [pipelines documentation](pipelines.mdx) for more information on working with pipelines).
+
+The example below shows how filters can be passed to Retrievers within a pipeline:
+
+```python
+pipeline.run(
+    data={
+        "retriever": {
+            "query": "Why did the revenue increase?",
+            "filters": {
+                "operator": "AND",
+                "conditions": [
+                    {"field": "meta.years", "operator": "==", "value": "2019"},
+                    {
+                        "field": "meta.companies",
+                        "operator": "in",
+                        "value": ["BMW", "Mercedes"],
+                    },
+                ],
+            },
+        },
+    },
+)
+```
+
+In Document Stores, the `filter_documents` method is used to apply filters to stored documents, if the specific integration supports filtering.
+
+The example below shows how filters can be passed to the `QdrantDocumentStore`:
+
+```python
+filters = {
+    "operator": "AND",
+    "conditions": [
+        {"field": "meta.type", "operator": "==", "value": "article"},
+        {"field": "meta.genre", "operator": "in", "value": ["economy", "politics"]},
+    ],
+}
+results = QdrantDocumentStore.filter_documents(filters=filters)
+```
+
+## Additional References
+
+:notebook: Tutorial: [Filtering Documents with Metadata](https://haystack.deepset.ai/tutorials/31_metadata_filtering)
+
+🧑‍🍳 Cookbook: [Extracting Metadata Filters from a Query](https://haystack.deepset.ai/cookbook/extracting_metadata_filters_from_a_user_query)

docs-website/versioned_docs/version-2.29/concepts/pipelines.mdx

@@ -0,0 +1,116 @@
+---
+title: "Pipelines"
+id: pipelines
+slug: "/pipelines"
+description: "To build modern search pipelines with LLMs, you need two things: powerful components and an easy way to put them together. The Haystack pipeline is built for this purpose and enables you to design and scale your interactions with LLMs."
+---
+
+import ClickableImage from "@site/src/components/ClickableImage";
+import YoutubeEmbed from "@site/src/components/YoutubeEmbed";
+
+# Pipelines
+
+To build modern search pipelines with LLMs, you need two things: powerful components and an easy way to put them together. The Haystack pipeline is built for this purpose and enables you to design and scale your interactions with LLMs.
+
+The pipelines in Haystack are directed multigraphs of different Haystack components and integrations. They give you the freedom to connect these components in various ways. This means that the pipeline doesn't need to be a continuous stream of information. With the flexibility of Haystack pipelines, you can have simultaneous flows, standalone components, loops, and other types of connections.
+
+## Flexibility
+
+Haystack pipelines are much more than just query and indexing pipelines. What a pipeline does, whether that be indexing, querying, fetching from an API, preprocessing or more, completely depends on how you design your pipeline and what components you use. While you can still create single-function pipelines, like indexing pipelines using ready-made components to clean up, split, and write the documents into a Document Store, or query pipelines that just take a query and return an answer, Haystack allows you to combine multiple use cases into one pipeline with decision components (like the `ConditionalRouter`) as well.
+
+### Agentic Pipelines
+
+Haystack loops and branches enable the creation of complex applications like agents. Here are a few examples on how to create them:
+
+- [Tutorial: Building a Chat Agent with Function Calling](https://haystack.deepset.ai/tutorials/40_building_chat_application_with_function_calling)
+- [Tutorial: Building an Agentic RAG with Fallback to Websearch](https://haystack.deepset.ai/tutorials/36_building_fallbacks_with_conditional_routing)
+- [Tutorial: Generating Structured Output with Loop-Based Auto-Correction](https://haystack.deepset.ai/tutorials/28_structured_output_with_loop)
+- [Cookbook: Define & Run Tools](https://haystack.deepset.ai/cookbook/tools_support)
+- [Cookbook: Conversational RAG using Memory](https://haystack.deepset.ai/cookbook/conversational_rag_using_memory)
+- [Cookbook: Newsletter Sending Agent with Experimental Haystack Tools](https://haystack.deepset.ai/cookbook/newsletter-agent)
+
+### Branching
+
+A pipeline can have multiple branches that process data concurrently. For example, to process different file types, you can have a pipeline with a bunch of converters, each handling a specific file type. You then feed all your files to the pipeline and it smartly divides and routes them to appropriate converters all at once, saving you the effort of sending your files one by one for processing.
+<ClickableImage src="/img/83f686b-Pipeline_Illustrations_1_1.png" alt="Pipeline architecture diagram showing components arranged in parallel branches that converge into a single pipeline flow" size="large" />
+
+### Loops
+
+Components in a pipeline can work in iterative loops, which you can cap at a desired number. This can be handy for scenarios like self-correcting loops, where you have a generator producing some output and then a validator component to check if the output is correct. If the generator's output has errors, the validator component can loop back to the generator for a corrected output. The loop goes on until the output passes the validation and can be sent further down the pipeline.
+
+See [Pipeline Loops](pipelines/pipeline-loops.mdx) for a deeper explanation of how loops are executed, how they terminate, and how to use them safely.
+
+<ClickableImage src="/img/2390eea-Pipeline_Illustrations_1_2.png" alt="Pipeline architecture diagram illustrating a feedback loop where output from later components loops back to earlier components" size="large" />
+
+### Async Pipelines
+
+The AsyncPipeline enables parallel execution of Haystack components when their dependencies allow it. This improves performance in complex pipelines with independent operations. For example, it can run multiple Retrievers or LLM calls simultaneously, execute independent pipeline branches in parallel, and efficiently handle I/O-bound operations that would otherwise cause delays. Through concurrent execution, the AsyncPipeline significantly reduces total processing time compared to sequential execution.
+
+Find out more in our [AsyncPipeline](pipelines/asyncpipeline.mdx) documentation.
+
+## SuperComponents
+
+To simplify your code, we have introduced [SuperComponents](components/supercomponents.mdx) that allow you to wrap complete pipelines and reuse them as a single component. Check out their documentation page for the details and examples.
+
+## Data Flow
+
+While the data (the initial query) flows through the entire pipeline, individual values are only passed from one component to another when they are connected. Therefore, not all components have access to all the data. This approach offers the benefits of speed and ease of debugging.
+
+To connect components and integrations in a pipeline, you must know the names of their inputs and outputs. The output of one component must be accepted as input by the following component. When you connect components in a pipeline with `Pipeline.connect()`, it validates if the input and output types match.
+
+### Smart Pipeline Connections
+
+Pipelines support smarter connection semantics that simplify how components are wired together.
+
+Compatible outputs can be implicitly combined when connected to a single input.
+Pipelines also perform implicit type adaptation at connection time for some selected types.
+
+These behaviors reduce the need for glue components like `Joiners` and `OutputAdapters`, keeping pipelines concise and easier to read.
+
+See [Smart Pipeline Connections](pipelines/smart-pipeline-connections.mdx) for details and examples.
+
+<YoutubeEmbed videoId="SxAwyeCkguc" title="Introduction to Haystack Pipelines" />
+
+## Steps to Create a Pipeline Explained
+
+Once all your components are created and ready to be combined in a pipeline, there are four steps to make it work:
+
+1. Create the pipeline with `Pipeline()`.
+   This creates the Pipeline object.
+2. Add components to the pipeline, one by one, with `.add_component(name, component)`.
+   This just adds components to the pipeline without connecting them yet. It's especially useful for loops as it allows the smooth connection of the components in the next step because they all already exist in the pipeline.
+3. Connect components with `.connect("producer_component.output_name", "consumer_component.input_name")`.
+   At this step, you explicitly connect one of the outputs of a component to one of the inputs of the next component. This is also when the pipeline validates the connection without running the components. It makes the validation fast.
+4. Run the pipeline with `.run({"component_1": {"mandatory_inputs": value}})`.
+   Finally, you run the Pipeline by specifying the first component in the pipeline and passing its mandatory inputs. Optionally, you can pass inputs to other components, for example: `.run({"component_1": {"mandatory_inputs": value}, "component_2": {"inputs": value}})`.
+
+The full pipeline [example](pipelines/creating-pipelines.mdx#example) in [Creating Pipelines](pipelines/creating-pipelines.mdx) shows how all the elements come together to create a working RAG pipeline.
+
+Once you create your pipeline, you can [visualize it in a graph](pipelines/visualizing-pipelines.mdx) to understand how the components are connected and make sure that's how you want them. You can use Mermaid graphs to do that.
+
+## Validation
+
+Validation happens when you connect pipeline components with `.connect()`, but before running the components to make it faster. The pipeline validates that:
+
+- The components exist in the pipeline.
+- The components' outputs and inputs match and are explicitly indicated. For example, if a component produces two outputs, when connecting it to another component, you must indicate which output connects to which input.
+- The components' types match.
+- For input types other than `Variadic`, checks if the input is already occupied by another connection.
+
+All of these checks produce detailed errors to help you quickly fix any issues identified.
+
+## Serialization
+
+Thanks to serialization, you can save and then load your pipelines. Serialization is converting a Haystack pipeline into a format you can store on disk or send over the wire. It's particularly useful for:
+
+- Editing, storing, and sharing pipelines.
+- Modifying existing pipelines in a format different than Python.
+
+Haystack pipelines delegate the serialization to its components, so serializing a pipeline simply means serializing each component in the pipeline one after the other, along with their connections. The pipeline is serialized into a dictionary format, which acts as an intermediate format that you can then convert into the final format you want.
+
+:::info[Serialization formats]
+
+Haystack only supports YAML format at this time. We'll be rolling out more formats gradually.
+:::
+
+For serialization to be possible, components must support conversion from and to Python dictionaries. All Haystack components have two methods that make them serializable: `from_dict` and `to_dict`. The `Pipeline` class, in turn, has its own `from_dict` and `to_dict` methods that take care of serializing components and connections.

docs-website/versioned_docs/version-2.29/concepts/pipelines/asyncpipeline.mdx

@@ -0,0 +1,158 @@
+---
+title: "AsyncPipeline"
+id: asyncpipeline
+slug: "/asyncpipeline"
+description: "Use AsyncPipeline to run multiple Haystack components at the same time for faster processing."
+---
+
+# AsyncPipeline
+
+Use AsyncPipeline to run multiple Haystack components at the same time for faster processing.
+
+The `AsyncPipeline` in Haystack introduces asynchronous execution capabilities, enabling concurrent component execution when dependencies allow. This optimizes performance, particularly in complex pipelines where multiple independent components can run in parallel.
+
+The `AsyncPipeline` provides significant performance improvements in scenarios such as:
+
+- Hybrid retrieval pipelines, where multiple Retrievers can run in parallel,
+- Multiple LLM calls that can be executed concurrently,
+- Complex pipelines with independent branches of execution,
+- I/O-bound operations that benefit from asynchronous execution.
+
+## Key Features
+
+### Concurrent Execution
+
+The `AsyncPipeline` schedules components based on input readiness and dependency resolution, ensuring efficient parallel execution when possible. For example, in a hybrid retrieval scenario, multiple Retrievers can run simultaneously if they do not depend on each other.
+
+### Execution Methods
+
+The `AsyncPipeline` offers three ways to run your pipeline:
+
+#### Synchronous Run (`run`)
+
+Executes the pipeline synchronously with the provided input data. This method is blocking, making it suitable for environments where asynchronous execution is not possible or desired. Although components execute concurrently internally, the method blocks until the pipeline completes.
+
+#### Asynchronous Run (`run_async`)
+
+Executes the pipeline in an asynchronous manner, allowing non-blocking execution. This method is ideal when integrating the pipeline into an async workflow, enabling smooth operation within larger async applications or services.
+
+#### Asynchronous Generator (`run_async_generator`)
+
+Allows step-by-step execution by yielding partial outputs as components complete their tasks. This is particularly useful for monitoring progress, debugging, and handling outputs incrementally. It differs from `run_async`, which executes the pipeline in a single async call.
+
+In an `AsyncPipeline`, components such as A and B will run in parallel _only if they have no shared dependencies_ and can process inputs independently.
+
+### Concurrency Control
+
+You can control the maximum number of components that run simultaneously using the `concurrency_limit` parameter to ensure controlled resource usage.
+
+You can find more details in our [API Reference](/reference/pipeline-api#asyncpipeline), or directly in the pipeline's [GitHub code](https://github.com/deepset-ai/haystack/blob/main/haystack/core/pipeline/async_pipeline.py).
+
+## Example
+
+```python
+import asyncio
+
+from haystack import AsyncPipeline, Document
+from haystack.components.builders import ChatPromptBuilder
+from haystack.components.embedders import (
+    SentenceTransformersDocumentEmbedder,
+    SentenceTransformersTextEmbedder,
+)
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.components.joiners import DocumentJoiner
+from haystack.components.retrievers import (
+    InMemoryBM25Retriever,
+    InMemoryEmbeddingRetriever,
+)
+from haystack.dataclasses import ChatMessage
+from haystack.document_stores.in_memory import InMemoryDocumentStore
+
+documents = [
+    Document(content="Khufu is the largest pyramid."),
+    Document(content="Khafre is the middle pyramid."),
+    Document(content="Menkaure is the smallest pyramid."),
+]
+
+docs_embedder = SentenceTransformersDocumentEmbedder()
+docs_embedder.warm_up()
+
+document_store = InMemoryDocumentStore()
+document_store.write_documents(docs_embedder.run(documents=documents)["documents"])
+
+prompt_template = [
+    ChatMessage.from_system(
+        """
+        You are a precise, factual QA assistant.
+        According to the following documents:
+        {% for document in documents %}
+        {{document.content}}
+        {% endfor %}
+
+        If an answer cannot be deduced from the documents, say "I don't know based on these documents".
+
+        When answering:
+        - be concise
+        - list the documents that support your answer
+
+        Answer the given question.
+        """,
+    ),
+    ChatMessage.from_user("{{query}}"),
+    ChatMessage.from_system("Answer:"),
+]
+
+hybrid_rag_retrieval = AsyncPipeline()
+hybrid_rag_retrieval.add_component("text_embedder", SentenceTransformersTextEmbedder())
+hybrid_rag_retrieval.add_component(
+    "embedding_retriever",
+    InMemoryEmbeddingRetriever(document_store=document_store, top_k=3),
+)
+hybrid_rag_retrieval.add_component(
+    "bm25_retriever",
+    InMemoryBM25Retriever(document_store=document_store, top_k=3),
+)
+hybrid_rag_retrieval.add_component("document_joiner", DocumentJoiner())
+hybrid_rag_retrieval.add_component(
+    "prompt_builder",
+    ChatPromptBuilder(template=prompt_template),
+)
+hybrid_rag_retrieval.add_component("llm", OpenAIChatGenerator())
+
+hybrid_rag_retrieval.connect(
+    "text_embedder.embedding",
+    "embedding_retriever.query_embedding",
+)
+hybrid_rag_retrieval.connect("bm25_retriever.documents", "document_joiner.documents")
+hybrid_rag_retrieval.connect(
+    "embedding_retriever.documents",
+    "document_joiner.documents",
+)
+hybrid_rag_retrieval.connect("document_joiner.documents", "prompt_builder.documents")
+hybrid_rag_retrieval.connect("prompt_builder.prompt", "llm.messages")
+
+question = "Which pyramid is neither the smallest nor the biggest?"
+
+data = {
+    "prompt_builder": {"query": question},
+    "text_embedder": {"text": question},
+    "bm25_retriever": {"query": question},
+}
+
+
+async def process_results():
+    async for partial_output in hybrid_rag_retrieval.run_async_generator(
+        data=data,
+        include_outputs_from={"document_joiner", "llm"},
+    ):
+        if "document_joiner" in partial_output:
+            print(
+                "Retrieved documents:",
+                len(partial_output["document_joiner"]["documents"]),
+            )
+        if "llm" in partial_output:
+            print("Generated answer:", partial_output["llm"]["replies"][0])
+
+
+asyncio.run(process_results())
+```

docs-website/versioned_docs/version-2.29/concepts/pipelines/creating-pipelines.mdx

@@ -0,0 +1,250 @@
+---
+title: "Creating Pipelines"
+id: creating-pipelines
+slug: "/creating-pipelines"
+description: "Learn the general principles of creating a pipeline."
+---
+
+import ClickableImage from "@site/src/components/ClickableImage";
+
+# Creating Pipelines
+
+Learn the general principles of creating a pipeline.
+
+You can use these instructions to create both indexing and query pipelines.
+
+This task uses an example of a semantic document search pipeline.
+
+## Prerequisites
+
+For each component you want to use in your pipeline, you must know the names of its input and output. You can check them on the documentation page for a specific component or in the component's `run()` method. For more information, see [Components: Input and Output](../components.mdx#input-and-output).
+
+## Steps to Create a Pipeline
+
+### 1\. Import dependencies
+
+Import all the dependencies, like pipeline, documents, Document Store, and all the components you want to use in your pipeline.
+For example, to create a semantic document search pipelines, you need the `Document` object, the pipeline, the Document Store, Embedders, and a Retriever:
+
+```python
+from haystack import Document, Pipeline
+from haystack.document_stores.in_memory import InMemoryDocumentStore
+from haystack.components.embedders import SentenceTransformersTextEmbedder
+from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
+```
+
+### 2\. Initialize components
+
+Initialize the components, passing any parameters you want to configure:
+
+```python
+document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
+text_embedder = SentenceTransformersTextEmbedder()
+retriever = InMemoryEmbeddingRetriever(document_store=document_store)
+```
+
+### 3\. Create the pipeline
+
+```python
+query_pipeline = Pipeline()
+```
+
+### 4\. Add components
+
+Add components to the pipeline one by one. The order in which you do this doesn't matter:
+
+```python
+query_pipeline.add_component("component_name", component_type)
+
+# Here is an example of how you'd add the components initialized in step 2 above:
+query_pipeline.add_component("text_embedder", text_embedder)
+query_pipeline.add_component("retriever", retriever)
+
+# You could also add components without initializing them before:
+query_pipeline.add_component("text_embedder", SentenceTransformersTextEmbedder())
+query_pipeline.add_component(
+    "retriever",
+    InMemoryEmbeddingRetriever(document_store=document_store),
+)
+```
+
+### 5\. Connect components
+
+Connect the components by indicating which output of a component should be connected to the input of the next component. If a component has only one input or output and the connection is obvious, you can just pass the component name without specifying the input or output.
+
+To understand what inputs are expected to run your pipeline, use an `.inputs()` pipeline function. See a detailed examples in the [Pipeline Inputs](#pipeline-inputs) section below.
+
+Here's a more visual explanation within the code:
+
+```python
+# This is the syntax to connect components. Here you're connecting output1 of component1 to input1 of component2:
+pipeline.connect("component1.output1", "component2.input1")
+
+# If both components have only one output and input, you can just pass their names:
+pipeline.connect("component1", "component2")
+
+# If one of the components has only one output but the other has multiple inputs,
+# you can pass just the name of the component with a single output, but for the component with
+# multiple inputs, you must specify which input you want to connect
+
+# Here, component1 has only one output, but component2 has multiple inputs:
+pipeline.connect("component1", "component2.input1")
+
+# And here's how it should look like for the semantic document search pipeline we're using as an example:
+pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
+# Because the InMemoryEmbeddingRetriever only has one input, this is also correct:
+pipeline.connect("text_embedder.embedding", "retriever")
+```
+
+You need to link all the components together, connecting them gradually in pairs. Here's an explicit example for the pipeline we're assembling:
+
+```python
+# Imagine this pipeline has four components: text_embedder, retriever, prompt_builder and llm.
+# Here's how you would connect them into a pipeline:
+
+query_pipeline.connect("text_embedder.embedding", "retriever")
+query_pipeline.connect("retriever", "prompt_builder.documents")
+query_pipeline.connect("prompt_builder", "llm")
+```
+
+### 6\. Run the pipeline
+
+Wait for the pipeline to validate the components and connections. If everything is OK, you can now run the pipeline. `Pipeline.run()` can be called in two ways, either passing a dictionary of the component names and their inputs, or by directly passing just the inputs. When passed directly, the pipeline resolves inputs to the correct components.
+
+```python
+# Here's one way of calling the run() method
+results = pipeline.run({"component1": {"input1_value": value1, "input2_value": value2}})
+
+# The inputs can also be passed directly without specifying component names
+results = pipeline.run({"input1_value": value1, "input2_value": value2})
+
+# This is how you'd run the semantic document search pipeline we're using as an example:
+query = "Here comes the query text"
+results = query_pipeline.run({"text_embedder": {"text": query}})
+```
+
+## Pipeline Inputs
+
+If you need to understand what component inputs are expected to run your pipeline, Haystack features a useful pipeline function `.inputs()` that lists all the required inputs for the components.
+
+This is how it works:
+
+```python
+# A short pipeline example that converts webpages into documents
+from haystack import Pipeline
+from haystack.document_stores.in_memory import InMemoryDocumentStore
+from haystack.components.fetchers import LinkContentFetcher
+from haystack.components.converters import HTMLToDocument
+from haystack.components.writers import DocumentWriter
+
+document_store = InMemoryDocumentStore()
+fetcher = LinkContentFetcher()
+converter = HTMLToDocument()
+writer = DocumentWriter(document_store=document_store)
+
+pipeline = Pipeline()
+pipeline.add_component(instance=fetcher, name="fetcher")
+pipeline.add_component(instance=converter, name="converter")
+pipeline.add_component(instance=writer, name="writer")
+
+pipeline.connect("fetcher.streams", "converter.sources")
+pipeline.connect("converter.documents", "writer.documents")
+
+# Requesting a list of required inputs
+pipeline.inputs()
+
+# {'fetcher': {'urls': {'type': typing.List[str], 'is_mandatory': True}},
+# 'converter': {'meta': {'type': typing.Union[typing.Dict[str, typing.Any], typing.List[typing.Dict[str, typing.Any]], NoneType],
+# 'is_mandatory': False,
+# 'default_value': None},
+# 'extraction_kwargs': {'type': typing.Optional[typing.Dict[str, typing.Any]],
+# 'is_mandatory': False,
+# 'default_value': None}},
+# 'writer': {'policy': {'type': typing.Optional[haystack.document_stores.types.policy.DuplicatePolicy],
+# 'is_mandatory': False,
+# 'default_value': None}}}
+```
+
+From the above response, you can see that the `urls` input is mandatory for `LinkContentFetcher`. This is how you would then run this pipeline:
+
+```python
+pipeline.run(
+    data={"fetcher": {"urls": ["https://docs.haystack.deepset.ai/docs/pipelines"]}},
+)
+```
+
+## Example
+
+The following example walks you through creating a RAG pipeline.
+
+```python
+# import necessary dependencies
+from haystack import Pipeline, Document
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.components.retrievers import InMemoryBM25Retriever
+from haystack.document_stores.in_memory import InMemoryDocumentStore
+from haystack.components.builders import ChatPromptBuilder
+from haystack.utils import Secret
+from haystack.dataclasses import ChatMessage
+
+# create a document store and write documents to it
+document_store = InMemoryDocumentStore()
+document_store.write_documents(
+    [
+        Document(content="My name is Jean and I live in Paris."),
+        Document(content="My name is Mark and I live in Berlin."),
+        Document(content="My name is Giorgio and I live in Rome."),
+    ],
+)
+
+# A prompt corresponds to an NLP task and contains instructions for the model. Here, the pipeline will go through each Document to figure out the answer.
+prompt_template = [
+    ChatMessage.from_system(
+        """
+        Given these documents, answer the question.
+        Documents:
+        {% for doc in documents %}
+            {{ doc.content }}
+        {% endfor %}
+        Question:
+        """,
+    ),
+    ChatMessage.from_user("{{question}}"),
+    ChatMessage.from_system("Answer:"),
+]
+
+# create the components adding the necessary parameters
+retriever = InMemoryBM25Retriever(document_store=document_store)
+prompt_builder = ChatPromptBuilder(template=prompt_template, required_variables="*")
+llm = OpenAIChatGenerator(
+    api_key=Secret.from_env_var("OPENAI_API_KEY"),
+    model="gpt-4o-mini",
+)
+
+# Create the pipeline and add the components to it. The order doesn't matter.
+# At this stage, the Pipeline validates the components without running them yet.
+rag_pipeline = Pipeline()
+rag_pipeline.add_component("retriever", retriever)
+rag_pipeline.add_component("prompt_builder", prompt_builder)
+rag_pipeline.add_component("llm", llm)
+
+# Arrange pipeline components in the order you need them. If a component has more than one inputs or outputs, indicate which input you want to connect to which output using the format ("component_name.output_name", "component_name, input_name").
+rag_pipeline.connect("retriever", "prompt_builder.documents")
+rag_pipeline.connect("prompt_builder", "llm")
+
+# Run the pipeline by specifying the first component in the pipeline and passing its mandatory inputs. Optionally, you can pass inputs to other components.
+question = "Who lives in paris?"
+results = rag_pipeline.run(
+    {
+        "retriever": {"query": question},
+        "prompt_builder": {"question": question},
+    },
+)
+
+print(results["llm"]["replies"])
+```
+
+Here's what a [visualized Mermaid graph](visualizing-pipelines.mdx) of this pipeline would look like:
+
+<br />
+<ClickableImage src="/img/vizualised-rag-pipeline.png" alt="RAG pipeline diagram with three connected components: InMemoryBM25Retriever receives a query string and outputs documents, ChatPromptBuilder combines the documents with a question input to create prompt messages, and OpenAIChatGenerator processes the messages to produce replies. Each component box displays its class name and optional input parameters." size="large" />

docs-website/versioned_docs/version-2.29/concepts/pipelines/debugging-pipelines.mdx

@@ -0,0 +1,125 @@
+---
+title: "Debugging Pipelines"
+id: debugging-pipelines
+slug: "/debugging-pipelines"
+description: "Learn how to debug and troubleshoot your Haystack pipelines."
+---
+
+import ClickableImage from "@site/src/components/ClickableImage";
+
+# Debugging Pipelines
+
+Learn how to debug and troubleshoot your Haystack pipelines.
+
+There are several options available to you to debug your pipelines:
+
+- [Inspect your components' outputs](#inspecting-component-outputs)
+- [Adjust logging](#logging)
+- [Set up tracing](#tracing)
+- [Try one of the monitoring tool integrations](#monitoring-tools)
+
+## Inspecting Component Outputs
+
+To view outputs from specific pipeline components, add the `include_outputs_from` parameter when executing your pipeline. Place it after the input dictionary and set it to the name of the component whose output you want included in the result.
+
+For example, here’s how you can print the output of `PromptBuilder` in this pipeline:
+
+```python
+from haystack import Pipeline, Document
+from haystack.utils import Secret
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
+from haystack.dataclasses import ChatMessage
+
+# Documents
+documents = [
+    Document(content="Joe lives in Berlin"),
+    Document(content="Joe is a software engineer"),
+]
+
+# Define prompt template
+prompt_template = [
+    ChatMessage.from_system("You are a helpful assistant."),
+    ChatMessage.from_user(
+        "Given these documents, answer the question.\nDocuments:\n"
+        "{% for doc in documents %}{{ doc.content }}{% endfor %}\n"
+        "Question: {{query}}\nAnswer:",
+    ),
+]
+
+# Define pipeline
+p = Pipeline()
+p.add_component(
+    instance=ChatPromptBuilder(
+        template=prompt_template,
+        required_variables={"query", "documents"},
+    ),
+    name="prompt_builder",
+)
+p.add_component(
+    instance=OpenAIChatGenerator(api_key=Secret.from_env_var("OPENAI_API_KEY")),
+    name="llm",
+)
+p.connect("prompt_builder", "llm.messages")
+
+# Define question
+question = "Where does Joe live?"
+
+# Execute pipeline
+result = p.run(
+    {"prompt_builder": {"documents": documents, "query": question}},
+    include_outputs_from="prompt_builder",
+)
+
+# Print result
+print(result)
+```
+
+## Logging
+
+Adjust the logging format according to your debugging needs. See our [Logging](../../development/logging.mdx) documentation for details.
+
+## Real-Time Pipeline Logging
+
+Use Haystack's [`LoggingTracer`](https://github.com/deepset-ai/haystack/blob/main/haystack/tracing/logging_tracer.py) logs to inspect the data that's flowing through your pipeline in real-time.
+
+This feature is particularly helpful during experimentation and prototyping, as you don’t need to set up any tracing backend beforehand.
+
+Here’s how you can enable this tracer. In this example, we are adding color tags (this is optional) to highlight the components' names and inputs:
+
+```python
+import logging
+from haystack import tracing
+from haystack.tracing.logging_tracer import LoggingTracer
+
+logging.basicConfig(
+    format="%(levelname)s - %(name)s -  %(message)s",
+    level=logging.WARNING,
+)
+logging.getLogger("haystack").setLevel(logging.DEBUG)
+
+tracing.tracer.is_content_tracing_enabled = (
+    True  # to enable tracing/logging content (inputs/outputs)
+)
+tracing.enable_tracing(
+    LoggingTracer(
+        tags_color_strings={
+            "haystack.component.input": "\x1b[1;31m",
+            "haystack.component.name": "\x1b[1;34m",
+        },
+    ),
+)
+```
+
+Here’s what the resulting log would look like when a pipeline is run:
+<ClickableImage src="/img/55c3d5c84282d726c95fb3350ec36be49a354edca8a6164f5dffdab7121cec58-image_2.png" alt="Console output showing Haystack pipeline execution with DEBUG level tracing logs including component names, types, and input/output specifications" />
+
+## Tracing
+
+To get a bigger picture of the pipeline’s performance, try tracing it with [Langfuse](../../development/tracing.mdx#langfuse).
+
+Our [Tracing](../../development/tracing.mdx) page has more about other tracing solutions for Haystack.
+
+## Monitoring Tools
+
+Take a look at available tracing and monitoring [integrations](https://haystack.deepset.ai/integrations?type=Monitoring+Tool&version=2.0) for Haystack pipelines, such as Arize AI or Arize Phoenix.

docs-website/versioned_docs/version-2.29/concepts/pipelines/pipeline-breakpoints.mdx

@@ -0,0 +1,214 @@
+---
+title: "Pipeline Breakpoints"
+id: pipeline-breakpoints
+slug: "/pipeline-breakpoints"
+description: "Learn how to pause and resume Haystack pipeline or Agent execution using breakpoints to debug, inspect, and continue workflows from saved snapshots."
+---
+
+# Pipeline Breakpoints
+
+Learn how to pause and resume Haystack pipeline or Agent execution using breakpoints to debug, inspect, and continue workflows from saved snapshots.
+
+## Introduction
+
+Haystack pipelines support breakpoints for debugging complex execution flows. A `Breakpoint` allows you to pause the execution at specific components, inspect the pipeline state, and resume execution from saved snapshots. This feature works for any regular component as well as an `Agent` component.
+
+You can set a `Breakpoint` on any component in a pipeline with a specific visit count. When triggered, the system stops the execution of the `Pipeline` and captures a snapshot of the current pipeline state. The state can be saved to a JSON file when snapshot file saving is enabled, see [Snapshot file saving](#snapshot-file-saving) below. You can inspect and modify the snapshot and use it to resume execution from the exact point where it stopped.
+
+You can also set breakpoints on an Agent, specifically on the `ChatGenerator` component or on any of the `Tool` specified in the `ToolInvoker` component .
+
+## Setting a `Breakpoint` on a Regular Component
+
+Create a `Breakpoint` by specifying the component name and the visit count at which to trigger it. This is useful for pipelines with loops. The default `visit_count` value is 0.
+
+```python
+from haystack.dataclasses.breakpoints import Breakpoint
+from haystack.core.errors import BreakpointException
+
+# Create a breakpoint that triggers on the first visit to the "llm" component
+break_point = Breakpoint(
+    component_name="llm",
+    visit_count=0,  # 0 = first visit, 1 = second visit, etc.
+    snapshot_file_path="/path/to/snapshots",  # Optional: save snapshot to file
+)
+
+# Run pipeline with breakpoint
+try:
+    result = pipeline.run(data=input_data, break_point=break_point)
+except BreakpointException as e:
+    print(f"Breakpoint triggered at component: {e.component}")
+    print(f"Component inputs: {e.inputs}")
+    print(f"Pipeline results so far: {e.results}")
+```
+
+A `BreakpointException` is raised containing the component inputs and the outputs of the pipeline up until the moment where the execution was interrupted, such as just before the execution of component associated with the breakpoint – the `llm` in the example above.
+
+If a `snapshot_file_path` is specified in the `Breakpoint` and snapshot file saving is enabled, the system saves a JSON snapshot with the same information as in the `BreakpointException`. Snapshot file saving to disk is disabled by default; see [Snapshot file saving](#snapshot-file-saving) below.
+
+To access the pipeline state during the breakpoint we can both catch the exception raised by the breakpoint as well as specify where the JSON file should be saved, note that file saving is enabled must be enabled.
+
+## Using a custom snapshot callback
+
+You can pass a `snapshot_callback` to `Pipeline.run()` or `Agent.run()` to handle snapshots yourself instead of saving to a file. When a breakpoint is triggered or a snapshot is created on error, the callback is invoked with the `PipelineSnapshot` object. This is useful for saving snapshots to a database, sending them to a remote service, or custom logging.
+
+```python
+from haystack.core.errors import BreakpointException
+from haystack.dataclasses.breakpoints import Breakpoint, PipelineSnapshot
+
+
+def my_snapshot_callback(snapshot: PipelineSnapshot) -> None:
+    # Custom handling: e.g. save to DB, send to API, or log
+    print(f"Snapshot at component: {snapshot.break_point}")
+
+
+break_point = Breakpoint(component_name="llm", visit_count=0)
+try:
+    result = pipeline.run(
+        data=input_data,
+        break_point=break_point,
+        snapshot_callback=my_snapshot_callback,
+    )
+except BreakpointException as e:
+    print(f"Breakpoint triggered: {e.component}")
+```
+
+When `snapshot_callback` is provided, file-saving is skipped and the callback is responsible for handling the snapshot. The same parameter is available on `Agent.run()` for agent breakpoints.
+
+## Snapshot file saving
+
+Snapshot file saving to disk is **disabled by default**. To save snapshots as JSON files when a breakpoint is triggered or on pipeline failure, set the environment variable `HAYSTACK_PIPELINE_SNAPSHOT_SAVE_ENABLED` to `"true"` or `"1"` (case-insensitive). When enabled, snapshots are written to the path given by `snapshot_file_path` on the breakpoint, or to the default directory in [Error Recovery with Snapshots](#error-recovery-with-snapshots) when a run fails.
+
+Custom `snapshot_callback` functions are always invoked when provided, regardless of this setting.
+
+```python
+import os
+
+# Enable saving snapshot files to disk
+os.environ["HAYSTACK_PIPELINE_SNAPSHOT_SAVE_ENABLED"] = "true"
+
+break_point = Breakpoint(
+    component_name="llm",
+    visit_count=0,
+    snapshot_file_path="/path/to/snapshots",
+)
+# When the breakpoint triggers, a JSON file will be written to /path/to/snapshots/
+```
+
+## Resuming a Pipeline Execution from a Breakpoint
+
+To resume the execution of a pipeline from the breakpoint, pass the path to the generated JSON file at the run time of the pipeline, using the `pipeline_snapshot`.
+
+Use the `load_pipeline_snapshot()` to first load the JSON and then pass it to the pipeline.
+
+```python
+from haystack.core.pipeline.breakpoint import load_pipeline_snapshot
+
+# Load the snapshot
+snapshot = load_pipeline_snapshot("llm_2025_05_03_11_23_23.json")
+
+# Resume execution from the snapshot
+result = pipeline.run(data={}, pipeline_snapshot=snapshot)
+print(result["llm"]["replies"])
+```
+
+## Setting a Breakpoint on an Agent
+
+You can also set breakpoints in an Agent component. An Agent supports two types of breakpoints:
+
+1. **Chat Generator Breakpoint**: Pauses before LLM calls.
+2. **Tool Invoker Breakpoint**: Pauses before any tool execution.
+
+A `ChatGenerator` breakpoint is defined as shown below. You need to define a `Breakpoint` as for a pipeline breakpoint and then an `AgentBreakpoint` where you pass the breakpoint defined before and the name of Agent component.
+
+```python
+from haystack.dataclasses.breakpoints import AgentBreakpoint, Breakpoint, ToolBreakpoint
+
+# Break at chat generator (LLM calls)
+chat_bp = Breakpoint(component_name="chat_generator", visit_count=0)
+agent_breakpoint = AgentBreakpoint(break_point=chat_bp, agent_name="my_agent")
+```
+
+To set a breakpoint on a Tool in an Agent, do the following:
+
+First, define a `ToolBreakpoint` specifying the `ToolInvoker` component whose name is `tool_invoker` and then the tool associated with the breakpoint, in this case – a `weather_tool` .
+
+Then, define an `AgentBreakpoint` passing the `ToolBreakpoint` defined before as the breakpoint.
+
+```python
+from haystack.dataclasses.breakpoints import AgentBreakpoint, Breakpoint, ToolBreakpoint
+
+# Break at tool invoker (tool calls)
+tool_bp = ToolBreakpoint(
+    component_name="tool_invoker",
+    visit_count=0,
+    tool_name="weather_tool",  # Specific tool, or None for any tool
+)
+agent_breakpoint = AgentBreakpoint(break_point=tool_bp, agent_name="my_agent")
+```
+
+### Resuming Agent Execution
+
+When an Agent breakpoint is triggered, you can resume execution using the saved snapshot. Similar to the regular component in a pipeline, pass the JSON file with the snapshot to the `run()` method of the pipeline.
+
+```python
+from haystack.core.pipeline.breakpoint import load_pipeline_snapshot
+
+# Load the snapshot
+snapshot_file = "./agent_debug/agent_chat_generator_2025_07_11_23_23.json"
+snapshot = load_pipeline_snapshot(snapshot_file)
+
+# Resume pipeline execution
+result = pipeline.run(data={}, pipeline_snapshot=snapshot)
+print("Pipeline resumed successfully")
+print(f"Final result: {result}")
+```
+
+## Error Recovery with Snapshots
+
+Pipelines automatically create a snapshot of the last valid state if a run fails. The snapshot contains inputs, visit counts, and intermediate outputs up to the failure. You can inspect it, fix the issue, and resume execution from that checkpoint instead of restarting the whole run.
+
+### Access the Snapshot on Failure
+
+Wrap `pipeline.run()` in a `try`/`except` block and retrieve the snapshot from the raised `PipelineRuntimeError`:
+
+```python
+from haystack.core.errors import PipelineRuntimeError
+
+try:
+    pipeline.run(data=input_data)
+except PipelineRuntimeError as e:
+    snapshot = e.pipeline_snapshot
+    if snapshot is not None:
+        intermediate_outputs = snapshot.pipeline_state.pipeline_outputs
+        # Inspect intermediate_outputs to diagnose the failure
+```
+
+When snapshot file saving is enabled (see [Snapshot file saving](#snapshot-file-saving)), Haystack also saves the same snapshot as a JSON file on disk.
+The directory is chosen automatically in this order:
+
+- `~/.haystack/pipeline_snapshot`
+- `/tmp/haystack/pipeline_snapshot`
+- `./.haystack/pipeline_snapshot`
+
+Filenames will have the following pattern: `{component_name}_{visit_nr}_{YYYY_MM_DD_HH_MM_SS}.json`.
+
+### Resume from a Snapshot
+
+You can resume directly from the in-memory snapshot or load it from disk.
+
+Resume from memory:
+
+```python
+result = pipeline.run(data={}, pipeline_snapshot=snapshot)
+```
+
+Resume from disk:
+
+```python
+from haystack.core.pipeline.breakpoint import load_pipeline_snapshot
+
+snapshot = load_pipeline_snapshot(
+    "/path/to/.haystack/pipeline_snapshot/reader_0_2025_09_20_12_33_10.json",
+)
+result = pipeline.run(data={}, pipeline_snapshot=snapshot)
+```

docs-website/versioned_docs/version-2.29/concepts/pipelines/pipeline-loops.mdx

@@ -0,0 +1,253 @@
+---
+title: "Pipeline Loops"
+id: pipeline-loops
+slug: "/pipeline-loops"
+description: "Understand how loops work in Haystack pipelines, how they terminate, and how to use them safely for feedback and self-correction."
+---
+
+# Pipeline Loops
+
+Learn how loops work in Haystack pipelines, how they terminate, and how to use them for feedback and self-correction.
+
+Haystack pipelines support **loops**: cycles in the component graph where the output of a later component is fed back into an earlier one.
+This enables feedback flows such as self-correction, validation, or iterative refinement, as well as more advanced [agentic behavior](../pipelines.mdx#agentic-pipelines).
+
+At runtime, the pipeline re-runs a component whenever all of its required inputs are ready again.
+You control when loops stop either by designing your graph and routing logic carefully or by using built-in [safety limits](#loop-termination-and-safety-limits).
+
+## Multiple Runs of the Same Component
+
+If a component participates in a loop, it can be run multiple times within a single `Pipeline.run()` call.
+The pipeline keeps an internal visit counter for each component:
+
+- Each time the component runs, its visit count increases by 1.
+- You can use this visit count in debugging tools like [breakpoints](./pipeline-breakpoints.mdx) to inspect specific iterations of a loop.
+
+In the final pipeline result:
+
+- For each component that ran, the pipeline returns **only the last-produced output**.
+- To capture outputs from intermediate components (for example, a validator or a router) in the final result dictionary, use the `include_outputs_from` argument of `Pipeline.run()`.
+
+## Loop Termination and Safety Limits
+
+Loops must eventually stop so that a pipeline run can complete.
+There are two main ways a loop ends:
+
+1. **Natural completion**: No more components are runnable  
+   The pipeline finishes when the work queue is empty and no component can run again (for example, the router stops feeding inputs back into the loop).
+
+2. **Reaching the maximum run count**  
+   Every pipeline has a per-component run limit, controlled by the `max_runs_per_component` parameter of the `Pipeline` (or `AsyncPipeline`) constructor, which is `100` by default. If any component exceeds this limit, Haystack raises a `PipelineMaxComponentRuns` error.
+
+   You can set this limit to a lower value:
+
+   ```python
+   from haystack import Pipeline
+
+   pipe = Pipeline(max_runs_per_component=5)
+   ```
+
+   The limit is checked before each execution, so a component with a limit of 3 will complete 3 runs successfully before the error is raised on the 4th attempt.
+
+   This safeguard is especially important when experimenting with new loops or complex routing logic.
+   If your loop condition is wrong or never satisfied, the error prevents the pipeline from running indefinitely.
+
+## Example: Feedback Loop for Self-Correction
+
+The following example shows a simple feedback loop where:
+
+- A `ChatPromptBuilder` creates a prompt that includes previous incorrect replies.
+- An `OpenAIChatGenerator` produces an answer.
+- A `ConditionalRouter` checks if the answer is correct:
+  - If correct, it sends the answer to `final_answer` and the loop ends.
+  - If incorrect, it sends the answer back to the `ChatPromptBuilder`, which triggers another iteration.
+
+```python
+from haystack import Pipeline
+from haystack.components.builders import ChatPromptBuilder
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.components.routers import ConditionalRouter
+from haystack.dataclasses import ChatMessage
+
+template = [
+    ChatMessage.from_system(
+        "Answer the following question concisely with just the answer, no punctuation.",
+    ),
+    ChatMessage.from_user(
+        "{% if previous_replies %}"
+        "Previously you replied incorrectly: {{ previous_replies[0].text }}\n"
+        "{% endif %}"
+        "Question: {{ query }}",
+    ),
+]
+
+prompt_builder = ChatPromptBuilder(template=template, required_variables=["query"])
+generator = OpenAIChatGenerator()
+
+router = ConditionalRouter(
+    routes=[
+        {
+            # End the loop when the answer is correct
+            "condition": "{{ 'Rome' in replies[0].text }}",
+            "output": "{{ replies }}",
+            "output_name": "final_answer",
+            "output_type": list[ChatMessage],
+        },
+        {
+            # Loop back when the answer is incorrect
+            "condition": "{{ 'Rome' not in replies[0].text }}",
+            "output": "{{ replies }}",
+            "output_name": "previous_replies",
+            "output_type": list[ChatMessage],
+        },
+    ],
+    unsafe=True,  # Required to handle ChatMessage objects
+)
+
+pipe = Pipeline(max_runs_per_component=3)
+
+pipe.add_component("prompt_builder", prompt_builder)
+pipe.add_component("generator", generator)
+pipe.add_component("router", router)
+
+pipe.connect("prompt_builder.prompt", "generator.messages")
+pipe.connect("generator.replies", "router.replies")
+pipe.connect("router.previous_replies", "prompt_builder.previous_replies")
+
+result = pipe.run(
+    {
+        "prompt_builder": {
+            "query": "What is the capital of Italy? If the statement 'Previously you replied incorrectly:' is missing "
+            "above then answer with Milan.",
+        },
+    },
+    include_outputs_from={"router", "prompt_builder"},
+)
+
+print(result["prompt_builder"]["prompt"][1].text)  # Shows the last prompt used
+print(result["router"]["final_answer"][0].text)  # Rome
+```
+
+### What Happens During This Loop
+
+1. **First iteration**
+   - `prompt_builder` runs with `query="What is the capital of Italy?"` and no previous replies.
+   - `generator` returns a `ChatMessage` with the LLM's answer.
+   - The router evaluates its conditions and checks if `"Rome"` is in the reply.
+   - If the answer is incorrect, `previous_replies` is fed back into `prompt_builder.previous_replies`.
+
+2. **Subsequent iterations** (if needed)
+   - `prompt_builder` runs again, now including the previous incorrect reply in the user message.
+   - `generator` produces a new answer with the additional context.
+   - The router checks again whether the answer contains `"Rome"`.
+
+3. **Termination**
+   - When the router routes to `final_answer`, no more inputs are fed back into the loop.
+   - The queue empties and the pipeline run finishes successfully.
+
+Because we used `max_runs_per_component=3`, any unexpected behavior that causes the loop to continue would raise a `PipelineMaxComponentRuns` error instead of looping forever.
+
+## Components for Building Loops
+
+Two components are particularly useful for building loops:
+
+- **[`ConditionalRouter`](../../pipeline-components/routers/conditionalrouter.mdx)**: Routes data to different outputs based on conditions. Use it to decide whether to exit the loop or continue iterating. The example above uses this pattern.
+
+- **[`BranchJoiner`](../../pipeline-components/joiners/branchjoiner.mdx)**: Merges inputs from multiple sources into a single output. Use it when a component inside the loop needs to receive both the initial input (on the first iteration) and looped-back values (on subsequent iterations). For example, you might use `BranchJoiner` to feed both user input and validation errors into the same Generator. See the [BranchJoiner documentation](../../pipeline-components/joiners/branchjoiner.mdx#enabling-loops) for a complete loop example.
+
+## Greedy vs. Lazy Variadic Sockets in Loops
+
+Some components support variadic inputs that can receive multiple values on a single socket.
+In loops, variadic behavior controls how inputs are consumed across iterations.
+
+- **Greedy variadic sockets**  
+  Consume exactly one value at a time and remove it after the component runs.
+  This includes user-provided inputs, which prevents them from retriggering the component indefinitely.
+  Most variadic sockets are greedy by default.
+
+- **Lazy variadic sockets**  
+  Accumulate all values received from predecessors across iterations.
+  Useful when you need to collect multiple partial results over time (for example, gathering outputs from several loop iterations before proceeding).
+
+For most loop scenarios it's sufficient to just connect components as usual and use `max_runs_per_component` to protect against mistakes.
+
+## Troubleshooting Loops
+
+If your pipeline seems stuck or runs longer than expected, here are common causes and how to debug them.
+
+### Common Causes of Infinite Loops
+
+1. **Condition never satisfied**: Your exit condition (for example, `"Rome" in reply`) might never be true due to LLM behavior or data issues. Always set a reasonable `max_runs_per_component` as a safety net.
+
+2. **Relying on optional outputs**: When a component has multiple output sockets but only returns some of them, the unreturned outputs don't trigger their downstream connections. This can cause confusion in loops.
+
+   For example, this pattern can be problematic:
+
+   ```python
+   @component
+   class Validator:
+       @component.output_types(valid=str, invalid=Optional[str])
+       def run(self, text: str):
+           if is_valid(text):
+               return {"valid": text}  # "invalid" is never returned
+           else:
+               return {"invalid": text}
+   ```
+
+   If you connect `invalid` back to an upstream component for retry, but also have other connections that keep the loop alive, you might get unexpected behavior.
+
+   Instead, use a `ConditionalRouter` with explicit, mutually exclusive conditions:
+
+   ```python
+   router = ConditionalRouter(
+       routes=[
+           {"condition": "{{ is_valid }}", "output": "{{ text }}", "output_name": "valid", ...},
+           {"condition": "{{ not is_valid }}", "output": "{{ text }}", "output_name": "invalid", ...},
+       ]
+   )
+   ```
+
+3. **User inputs retriggering the loop**: If a user-provided input is connected to a socket inside the loop, it might cause the loop to restart unexpectedly.
+
+   ```python
+   # Problematic: user input goes directly to a component inside the loop
+   result = pipe.run({
+       "generator": {"prompt": query},  # This input persists and may retrigger the loop
+   })
+
+   # Better: use an entry-point component outside the loop
+   result = pipe.run({
+       "prompt_builder": {"query": query},  # Entry point feeds into the loop once
+   })
+   ```
+
+   See [Greedy vs. Lazy Variadic Sockets](#greedy-vs-lazy-variadic-sockets-in-loops) for details on how inputs are consumed.
+
+4. **Multiple paths feeding the same component**: If a component inside the loop receives inputs from multiple sources, it runs whenever *any* path provides input.
+
+   ```python
+   # Component receives from two sources – runs when either provides input
+   pipe.connect("source_a.output", "processor.input")
+   pipe.connect("source_b.output", "processor.input")  # Variadic input
+   ```
+
+   Ensure you understand when each path produces output, or use `BranchJoiner` to explicitly control the merge point.
+
+### Debugging Tips
+
+1. **Start with a low limit**: When developing loops, set `max_runs_per_component=3` or similar. This helps you catch issues early with a clear error instead of waiting for a timeout.
+
+2. **Use `include_outputs_from`**: Add intermediate components (like your router) to see what's happening at each step:
+   ```python
+   result = pipe.run(data, include_outputs_from={"router", "validator"})
+   ```
+
+3. **Enable tracing**: Use tracing to see every component execution, including inputs and outputs. This makes it easy to follow each iteration of the loop. For quick debugging, use `LoggingTracer` ([setup instructions](./debugging-pipelines.mdx#real-time-pipeline-logging)). For deeper analysis, integrate with tools like Langfuse or other [tracing backends](../../development/tracing.mdx).
+
+4. **Visualize the pipeline**: Use `pipe.draw()` or `pipe.show()` to see the graph structure and verify your connections are correct. See the [Pipeline Visualization](./visualizing-pipelines.mdx) documentation for details.
+
+5. **Use breakpoints**: Set a `Breakpoint` on a specific component and visit count to inspect the state at that iteration. See [Pipeline Breakpoints](./pipeline-breakpoints.mdx) for details.
+
+6. **Check for blocked pipelines**: If you see a `PipelineComponentsBlockedError`, it means no components can run. This typically indicates a missing connection or a circular dependency. Check that all required inputs are provided.
+
+By combining careful graph design, per-component run limits, and these debugging tools, you can build robust feedback loops in your Haystack pipelines.

docs-website/versioned_docs/version-2.29/concepts/pipelines/serialization.mdx

@@ -0,0 +1,233 @@
+---
+title: "Serializing Pipelines"
+id: serialization
+slug: "/serialization"
+description: "Save your pipelines into a custom format and explore the serialization options."
+---
+
+# Serializing Pipelines
+
+Save your pipelines into a custom format and explore the serialization options.
+
+Serialization means converting a pipeline to a format that you can save on your disk and load later.
+
+Haystack supports YAML format for pipeline serialization.
+
+## Converting a Pipeline to YAML
+
+Use the `dumps()` method to convert a Pipeline object to YAML:
+
+```python
+from haystack import Pipeline
+
+pipe = Pipeline()
+print(pipe.dumps())
+
+# Prints:
+#
+# components: {}
+# connections: []
+# max_runs_per_component: 100
+# metadata: {}
+```
+
+You can also use `dump()` method to save the YAML representation of a pipeline in a file:
+
+```python
+with open("/content/test.yml", "w") as file:
+    pipe.dump(file)
+```
+
+## Converting a Pipeline Back to Python
+
+You can convert a YAML pipeline back into Python. Use the `loads()` method to convert a string representation of a pipeline (`str`, `bytes` or `bytearray`)  or the `load()` method to convert a pipeline represented in a file-like object into a corresponding Python object.
+
+Both loading methods support callbacks that let you modify components during the deserialization process. Therefore, loading a serialized pipeline or component assumes that the serialized definition originates from a trusted source and has been reviewed by the user.
+
+Here is an example script:
+
+```python
+from haystack import Pipeline
+from haystack.core.serialization import DeserializationCallbacks
+from typing import Type, Dict, Any
+
+# This is the YAML you want to convert to Python:
+pipeline_yaml = """
+components:
+  cleaner:
+    init_parameters:
+      remove_empty_lines: true
+      remove_extra_whitespaces: true
+      remove_regex: null
+      remove_repeated_substrings: false
+      remove_substrings: null
+    type: haystack.components.preprocessors.document_cleaner.DocumentCleaner
+  converter:
+    init_parameters:
+      encoding: utf-8
+    type: haystack.components.converters.txt.TextFileToDocument
+connections:
+- receiver: cleaner.documents
+  sender: converter.documents
+max_runs_per_component: 100
+metadata: {}
+"""
+
+
+def component_pre_init_callback(
+    component_name: str,
+    component_cls: Type,
+    init_params: Dict[str, Any],
+):
+    # This function gets called every time a component is deserialized.
+    if component_name == "cleaner":
+        assert "DocumentCleaner" in component_cls.__name__
+        # Modify the init parameters. The modified parameters are passed to
+        # the init method of the component during deserialization.
+        init_params["remove_empty_lines"] = False
+        print("Modified 'remove_empty_lines' to False in 'cleaner' component")
+    else:
+        print(f"Not modifying component {component_name} of class {component_cls}")
+
+
+pipe = Pipeline.loads(
+    pipeline_yaml,
+    callbacks=DeserializationCallbacks(component_pre_init_callback),
+)
+```
+
+## Default Serialization Behavior
+
+The serialization system uses `default_to_dict` and `default_from_dict` to handle many object types automatically. You typically do **not** need to implement custom `to_dict`/`from_dict` for:
+
+- **Secrets**: serialized and deserialized automatically so that sensitive values aren't stored in plain text.
+- **ComponentDevice**: device configuration is detected and restored automatically.
+- **Objects with their own `to_dict`/`from_dict`**: any init parameter whose type defines `to_dict()` is serialized by calling it; any dict in `init_parameters` with a `type` key pointing to a class with `from_dict()` is deserialized automatically.
+
+To serialize or deserialize a single component, you can use `component_to_dict` and `component_from_dict` from `haystack.core.serialization`. They use the default behavior above as a fallback when the component doesn't define custom `to_dict`/`from_dict`:
+
+```python
+from haystack import component
+from haystack.core.serialization import component_from_dict, component_to_dict
+
+
+@component
+class Greeter:
+    def __init__(self, message: str = "Hello"):
+        self.message = message
+
+    @component.output_types(greeting=str)
+    def run(self, name: str):
+        return {"greeting": f"{self.message}, {name}!"}
+
+
+# Serialize a component instance to a dictionary
+greeter = Greeter(message="Hi")
+data = component_to_dict(greeter, "my_greeter")
+
+# Deserialize back to a component instance
+restored = component_from_dict(Greeter, data, "my_greeter")
+assert restored.message == greeter.message
+```
+
+:::caution[Init parameters must be stored as instance attributes]
+
+Default serialization only works when there is a **1:1 mapping** between init parameter names and instance attributes. For every argument in `__init__`, the component must assign it to an attribute with the same name. For example, if you have `def __init__(self, prompt: str)`, you must have `self.prompt = prompt` in the class. Otherwise the serialization logic can't find the value to serialize and raises an error or uses the default value if the parameter has one.
+:::
+
+## Performing Custom Serialization
+
+Pipelines and components in Haystack can serialize simple components, including custom ones, out of the box. Code like this just works:
+
+```python
+from haystack import component
+
+
+@component
+class RepeatWordComponent:
+    def __init__(self, times: int):
+        self.times = times
+
+    @component.output_types(result=str)
+    def run(self, word: str):
+        return word * self.times
+```
+
+On the other hand, this code doesn't work if the final format is JSON, as the `set` type is not JSON-serializable:
+
+```python
+from haystack import component
+
+
+@component
+class SetIntersector:
+    def __init__(self, intersect_with: set):
+        self.intersect_with = intersect_with
+
+    @component.output_types(result=set)
+    def run(self, data: set):
+        return data.intersection(self.intersect_with)
+```
+
+In such cases, you can provide your own implementation  `from_dict` and `to_dict` to components:
+
+```python
+from haystack import component, default_from_dict, default_to_dict
+
+
+class SetIntersector:
+    def __init__(self, intersect_with: set):
+        self.intersect_with = intersect_with
+
+    @component.output_types(result=set)
+    def run(self, data: set):
+        return data.intersect(self.intersect_with)
+
+    def to_dict(self):
+        return default_to_dict(self, intersect_with=list(self.intersect_with))
+
+    @classmethod
+    def from_dict(cls, data):
+        # convert the set into a list for the dict representation,
+        # so it can be converted to JSON
+        data["intersect_with"] = set(data["intersect_with"])
+        return default_from_dict(cls, data)
+```
+
+## Saving a Pipeline to a Custom Format
+
+Once a pipeline is available in its dictionary format, the last step of serialization is to convert that dictionary into a format you can store or send over the wire. Haystack supports YAML out of the box, but if you need a different format, you can write a custom Marshaller.
+
+A `Marshaller` is a Python class responsible for converting text to a dictionary and a dictionary to text according to a certain format. Marshallers must respect the `Marshaller` [protocol](https://github.com/deepset-ai/haystack/blob/main/haystack/marshal/protocol.py), providing the methods `marshal` and `unmarshal`.
+
+This is the code for a custom TOML marshaller that relies on the `rtoml` library:
+
+```python
+# This code requires a `pip install rtoml`
+from typing import Dict, Any, Union
+import rtoml
+
+
+class TomlMarshaller:
+    def marshal(self, dict_: Dict[str, Any]) -> str:
+        return rtoml.dumps(dict_)
+
+    def unmarshal(self, data_: Union[str, bytes]) -> Dict[str, Any]:
+        return dict(rtoml.loads(data_))
+```
+
+You can then pass a Marshaller instance to the methods `dump`, `dumps`, `load`, and `loads`:
+
+```python
+from haystack import Pipeline
+from my_custom_marshallers import TomlMarshaller
+
+pipe = Pipeline()
+pipe.dumps(TomlMarshaller())
+# prints:
+# 'max_runs_per_component = 100\nconnections = []\n\n[metadata]\n\n[components]\n'
+```
+
+## Additional References
+
+:notebook: Tutorial: [Serializing LLM Pipelines](https://haystack.deepset.ai/tutorials/29_serializing_pipelines)

docs-website/versioned_docs/version-2.29/concepts/pipelines/smart-pipeline-connections.mdx

@@ -0,0 +1,148 @@
+---
+title: "Smart Pipeline Connections"
+id: smart-pipeline-connections
+slug: "/smart-pipeline-connections"
+description: "Learn how Haystack pipelines simplify connections through implicit joining and flexible type adaptation, reducing the need for glue components."
+---
+
+# Smart Pipeline Connections
+
+Haystack pipelines support smarter connection semantics that reduce boilerplate and make pipeline definitions easier to read and maintain.
+These features focus on simplifying how components are connected, without changing component behavior.
+
+Smart connections help eliminate common glue components such as `Joiners` and `OutputAdapters` in many pipelines.
+
+## Implicit List Joining
+
+Pipelines natively support connecting multiple component outputs directly to a single component input, without requiring an explicit `Joiner` component.
+
+This works when:
+
+* The target input is typed as `list`, `list | None`, or a union of list types (e.g. `list[int] | list[str]`).
+* All connected outputs are compatible list types.
+
+When multiple outputs are connected to the same input, the pipeline implicitly concatenates the lists from the outputs into a single list for the input.
+
+### Example
+
+Multiple converters can write directly into a single `DocumentWriter` without using a `DocumentJoiner`:
+
+<details>
+
+<summary>Expand to see the pipeline graph</summary>
+<ClickableImage src="/img/pipeline-illustration-auto-joiner.png" alt="Pipeline architecture diagram showing a DocumentWriter receiving inputs from multiple converters without a Joiner" size="large" />
+
+</details>
+
+```python
+from haystack import Pipeline
+from haystack.components.converters import HTMLToDocument, TextFileToDocument
+from haystack.components.routers import FileTypeRouter
+from haystack.components.writers import DocumentWriter
+from haystack.dataclasses import ByteStream
+from haystack.document_stores.in_memory import InMemoryDocumentStore
+
+sources = [
+    ByteStream.from_string(text="Text file content", mime_type="text/plain"),
+    ByteStream.from_string(
+        text="<html><body>Some content</body></html>",
+        mime_type="text/html",
+    ),
+]
+
+doc_store = InMemoryDocumentStore()
+
+pipe = Pipeline()
+pipe.add_component("router", FileTypeRouter(mime_types=["text/plain", "text/html"]))
+pipe.add_component("txt_converter", TextFileToDocument())
+pipe.add_component("html_converter", HTMLToDocument())
+pipe.add_component("writer", DocumentWriter(doc_store))
+pipe.connect("router.text/plain", "txt_converter.sources")
+pipe.connect("router.text/html", "html_converter.sources")
+pipe.connect("txt_converter.documents", "writer.documents")
+pipe.connect("html_converter.documents", "writer.documents")
+
+result = pipe.run({"router": {"sources": sources}})
+```
+
+This pattern is especially useful when routing files, documents, or results across multiple parallel branches.
+
+## Flexible Type Connections
+
+To further streamline pipeline definitions, Haystack pipelines support limited implicit type adaptation at connection time.
+This makes pipeline connections more flexible and reduces the need for `OutputAdapter` components.
+
+**Supported adaptations**
+
+| Source Type              | Target Type          | Behavior                                                      |
+|--------------------------|--------------------|---------------------------------------------------------------|
+| `str`                    | `ChatMessage`       | Wrapped into a `ChatMessage` with user role.                 |
+| `ChatMessage`            | `str`               | Extracts `ChatMessage.text`; raises `PipelineRuntimeError` if `None`. |
+| `T`                      | `list[T]`           | Wraps the item into a single-element list.                   |
+| `list[str] or list[ChatMessage]`| `str` or `ChatMessage` | Extracts the first item; raises `PipelineRuntimeError` if the list is empty. |
+
+
+All adaptations are checked at connection time to ensure type safety, but applied at runtime during pipeline execution.
+
+When multiple connections are possible, strict type matching is prioritized over implicit conversion.
+This preserves backward compatibility with earlier versions of Haystack, where flexible type connections were not supported.
+
+### Example
+
+Pipeline connecting the Chat Generator `messages` output (`list[ChatMessage]`) to the retriever `query` input (`str`)
+without using an `OutputAdapter`:
+
+```python
+from haystack.document_stores.in_memory import InMemoryDocumentStore
+from haystack.dataclasses import Document
+from haystack.components.retrievers import InMemoryBM25Retriever
+from haystack import Pipeline
+from haystack.components.builders import ChatPromptBuilder
+from haystack.components.generators.chat import OpenAIChatGenerator
+
+document_store = InMemoryDocumentStore()
+
+documents = [
+    Document(content="Bob lives in Paris."),
+    Document(content="Alice lives in London."),
+    Document(content="Ivy lives in Melbourne."),
+    Document(content="Kate lives in Brisbane."),
+    Document(content="Liam lives in Adelaide."),
+]
+
+document_store.write_documents(documents)
+
+template = """{% message role="user" %}
+Rewrite the following query to be used for keyword search.
+{{ query }}
+{% endmessage %}
+"""
+
+p = Pipeline()
+p.add_component("prompt_builder", ChatPromptBuilder(template=template))
+p.add_component("llm", OpenAIChatGenerator(model="gpt-4.1-mini"))
+p.add_component(
+    "retriever",
+    InMemoryBM25Retriever(document_store=document_store, top_k=3),
+)
+
+p.connect("prompt_builder", "llm")
+# implicitly converts list[ChatMessage] -> str
+p.connect("llm", "retriever")
+
+query = """Someday I'd love to visit Brisbane, but for now I just want
+to know the names of the people who live there."""
+
+result = p.run(data={"prompt_builder": {"query": query}})
+```
+
+## When You Still Need `Joiners` or `OutputAdapters`
+
+Explicit `Joiners` or `OutputAdapters` are still useful when you need:
+
+- Custom aggregation logic beyond simple list concatenation
+- Type conversions not covered by implicit adaptation
+- Explicit control over formatting or ordering
+
+Smart connections reduce the need for glue components, but they do not remove them entirely.
+When in doubt, explicit components provide clarity and more control.

docs-website/versioned_docs/version-2.29/concepts/pipelines/visualizing-pipelines.mdx

@@ -0,0 +1,88 @@
+---
+title: "Visualizing Haystack Pipelines"
+id: visualizing-pipelines
+slug: "/visualizing-pipelines"
+description: "You can visualize your Haystack AI pipelines as graphs to better understand how the components are connected."
+---
+
+import ClickableImage from "@site/src/components/ClickableImage";
+
+# Visualizing Haystack Pipelines
+
+You can visualize your pipelines as graphs to better understand how the components are connected.
+
+Haystack pipelines have  `draw()` and `show()` methods that enable you to visualize the pipeline as a graph using Mermaid graphs.
+
+:::note[Data Privacy Notice]
+
+Exercise caution with sensitive data when using pipeline visualization.
+
+This feature is based on Mermaid graphs web service that doesn't have clear terms of data retention or privacy policy.
+:::
+
+## Prerequisites
+
+To use Mermaid graphs, you must have an internet connection to reach the Mermaid graph renderer at https://mermaid.ink.
+
+## Displaying a Graph
+
+Use the pipeline's `show()` method to display the diagram in Jupyter notebooks.
+
+```python
+my_pipeline.show()
+```
+
+## Saving a Graph
+
+Use the pipeline's `draw()` method passing the path where you want to save the diagram and the diagram format. Possible formats are:  `mermaid-text` and `mermaid-image` (default).
+
+```python
+my_pipeline.draw(path=local_path)
+```
+
+## Visualizing SuperComponents
+
+To show the internal structure of [SuperComponents](../components/supercomponents.mdx) in your digram instead of a black box component, set the `super_component_expansion` parameter to `True`:
+
+```python
+my_pipeline.show(super_component_expansion=True)
+
+# or
+
+my_pipeline.draw(path=local_path, super_component_expansion=True)
+```
+
+## Visualizing Locally
+
+If you don't have an internet connection or don't want to send your pipeline data to the remote https://mermaid.ink, you can install a local mermaid.ink server and use it to render your pipeline.
+
+Let's run a local mermaid.ink server using their official Docker images from https://github.com/jihchi/mermaid.ink/pkgs/container/mermaid.ink.
+
+In this case, let's install one for a system running a MacOS M3 chip and expose it on port 3000:
+
+```dockerfile
+docker run --platform linux/amd64 --publish 3000:3000 --cap-add=SYS_ADMIN ghcr.io/jihchi/mermaid.ink
+```
+
+Check that the local mermaid.ink server is running by going to http://localhost:3000/.
+
+You should see a local server running, and now you can simply render the image using your local mermaid.ink server by specifying the URL when calling the`show()`or `draw()`  method:
+
+```python
+my_pipeline.show(server_url="http://localhost:3000")
+# or
+my_pipeline.draw("my_pipeline.png", server_url="http://localhost:3000")
+```
+
+## Example
+
+This is an example of what a pipeline graph may look like:
+<ClickableImage src="/img/46a8989-Untitled.png" alt="RAG pipeline flowchart showing the data flow from query through retriever, prompt builder, language model, and answer builder components" size="large" />
+
+<br />
+
+## Importing a Pipeline to Haystack Enterprise Platform
+
+You can import your Haystack pipeline into Haystack Enterprise Platform and continue visually building your pipeline
+
+To do that, follow the steps described in Haystack Enterprise Platform [documentation](https://docs.cloud.deepset.ai/docs/import-a-pipeline#import-your-pipeline).

docs-website/versioned_docs/version-2.29/concepts/secret-management.mdx

@@ -0,0 +1,202 @@
+---
+title: "Secret Management"
+id: secret-management
+slug: "/secret-management"
+description: "This page emphasizes secret management in Haystack components and introduces the `Secret` type for structured secret handling. It explains the drawbacks of hard-coding secrets in code and suggests using environment variables instead."
+---
+
+# Secret Management
+
+This page emphasizes secret management in Haystack components and introduces the `Secret` type for structured secret handling. It explains the drawbacks of hard-coding secrets in code and suggests using environment variables instead.
+
+Many Haystack components interact with third-party frameworks and service providers such as Azure, Google Vertex AI, and OpenAI. Their libraries often require the user to authenticate themselves to ensure they receive access to the underlying product. The authentication process usually works with a secret value that acts as an opaque identifier to the third-party backend.
+
+This page describes the two main types of secrets: token-based and environment variable-based, and how to handle them when using Haystack.
+
+You can find additional details for the `Secret` class in our [API reference](/reference/utils-api).
+
+<details>
+
+<summary>Example Use Case - Problem Statement</summary>
+
+### Problem Statement
+
+Let’s consider an example RAG pipeline that embeds a query, uses a Retriever component to locate documents relevant to the query, and then leverages an LLM to generate an answer based on the retrieved documents.
+
+The `OpenAIChatGenerator` component used in the pipeline below expects an API key to authenticate with OpenAI’s servers and perform the generation. Let’s assume that the component accepts a `str`  value for it:
+
+```python
+generator = OpenAIChatGenerator(api_key="sk-xxxxxxxxxxxxxxxxxx")
+pipeline.add_component("generator", generator)
+```
+
+This works in a pinch, but this is bad practice - we shouldn’t hard-code such secrets in the codebase. An alternative would be to store the key in an environment variable externally, read from it in Python, and pass that to the component:
+
+```python
+import os
+
+api_key = os.environ.get("OPENAI_API_KEY")
+generator = OpenAIChatGenerator(api_key=api_key)
+pipeline.add_component("generator", generator)
+```
+
+This is better – the pipeline works as intended, and we aren’t hard-coding any secrets in the code.
+
+Remember that pipelines are serializable. Since the API key is a secret, we should definitely avoid saving it to disk. Let’s modify the component’s `to_dict` method to exclude the key:
+
+```python
+def to_dict(self) -> Dict[str, Any]:
+    # Do not pass the `api_key` init parameter.
+    return default_to_dict(self, model=self.model)
+```
+
+But what happens when the pipeline is loaded from disk? In the best-case scenario, the component’s backend will automatically try to read the key from a hard-coded environment variable, and that key is the same as the one that was passed to the component before it was serialized. But in a worse case, the backend doesn’t look up the key in a hard-coded environment variable and fails when it gets called inside a `pipeline.run()`  invocation.
+
+</details>
+
+### Import
+
+To use Haystack secrets within the code, first import with:
+
+```python
+from haystack.utils import Secret
+```
+
+### Token-Based Secrets
+
+You can paste tokens directly as a string using the `from_token` method:
+
+```python
+llm = OpenAIChatGenerator(api_key=Secret.from_token("sk-randomAPIkeyasdsa32ekasd32e"))
+```
+
+Note that this type of code cannot be serialized, meaning you can't convert the above component to a dictionary or save a pipeline containing it to a YAML file. This is a security feature to prevent accidental exposure of sensitive data.
+
+### Environment Variable-Based Secrets
+
+Environment variable-based secrets are more flexible. They allow you to specify one or more environment variables that may contain your secret.
+
+Existing Haystack components that require an API Key (like OpenAIChatGenerator) have a default value for `Secret.from_env_var` (in this case, `OPENAI_API_KEY`). This means that the `OpenAIChatGenerator` will look for the value of the environment variable `OPENAI_API_KEY` (if it exists) and use it for authentication. And when pipelines are serialized to YAML, only the name of the environment variable is save to the YAML file. In doing so, this method ensures that there are no security leaks and is therefore strongly recommended.
+
+```bash
+## First, export an environment variable name `OPENAI_API_KEY` with its value
+export OPENAI_API_KEY=sk-randomAPIkeyasdsa32ekasd32e
+
+## or alternatively, using Python
+## import os
+## os.environ[”OPENAI_API_KEY”]=sk-randomAPIkeyasdsa32ekasd32e
+```
+
+```python
+llm_generator = (
+    OpenAIChatGenerator()
+)  # Uses the default value from the env var for the component
+```
+
+Alternatively, in components where a Secret is expected, you can customize the name of the environment variable from which the API Key is to be read.
+
+```python
+# Export an environment variable with custom name and its value
+llm_generator = OpenAIChatGenerator(api_key=Secret.from_env_var("YOUR_ENV_VAR"))
+```
+
+When `OpenAIChatGenerator` is serialized within a pipeline, this is what the YAML code will look like, using the custom variable name:
+
+```yaml
+components:
+  llm:
+    init_parameters:
+      api_base_url: null
+      api_key:
+        env_vars:
+        - YOUR_ENV_VAR
+        strict: true
+        type: env_var
+      generation_kwargs: {}
+      http_client_kwargs: null
+      max_retries: null
+      model: gpt-5-mini
+      organization: null
+      streaming_callback: null
+      timeout: null
+      tools: null
+      tools_strict: false
+    type: haystack.components.generators.chat.openai.OpenAIChatGenerator
+    ...
+```
+
+### Serialization
+
+While token-based secrets cannot be serialized, environment variable-based secrets can be converted to and from dictionaries:
+
+```python
+# Convert to dictionary
+env_secret_dict = env_secret.to_dict()
+
+# Create from dictionary
+new_env_secret = Secret.from_dict(env_secret_dict)
+```
+
+### Resolving Secrets
+
+Both types of secrets can be resolved to their actual values using the `resolve_value` method. This method returns the token or the value of the environment variable.
+
+```python
+# Resolve the token-based secret
+token_value = api_key_secret.resolve_value()
+
+# Resolve the environment variable-based secret
+env_value = env_secret.resolve_value()
+```
+
+### Custom Component Example
+
+Here is a complete example that shows how to create a component that uses the `Secret` class in Haystack, highlighting the differences between token-based and environment variable-based authentication, and showing that token-based secrets cannot be serialized:
+
+```python
+from haystack.utils import Secret, deserialize_secrets_inplace
+
+
+@component
+class MyComponent:
+    def __init__(self, api_key: Optional[Secret] = None, **kwargs):
+        self.api_key = api_key
+        self.backend = None
+
+    def warm_up(self):
+        # Call resolve_value to yield a single result. The semantics of the result is policy-dependent.
+        # Currently, all supported policies will return a single string token.
+        self.backend = SomeBackend(
+            api_key=self.api_key.resolve_value() if self.api_key else None,  # ...
+        )
+
+    def to_dict(self):
+        # Serialize the policy like any other (custom) data. If the policy is token-based, it will
+        # raise an error.
+        return default_to_dict(
+            self,
+            api_key=self.api_key.to_dict() if self.api_key else None,  # ...
+        )
+
+    @classmethod
+    def from_dict(cls, data):
+        # Deserialize the policy data before passing it to the generic from_dict function.
+        api_key_data = data["init_parameters"]["api_key"]
+        api_key = Secret.from_dict(api_key_data) if api_key_data is not None else None
+        data["init_parameters"]["api_key"] = api_key
+        # Alternatively, use the helper function.
+        # deserialize_secrets_inplace(data["init_parameters"], keys=["api_key"])
+        return default_from_dict(cls, data)
+
+
+# No authentication.
+component = MyComponent(api_key=None)
+
+# Token based authentication
+component = MyComponent(api_key=Secret.from_token("sk-randomAPIkeyasdsa32ekasd32e"))
+component.to_dict()  # Error! Can't serialize authentication tokens
+
+# Environment variable based authentication
+component = MyComponent(api_key=Secret.from_env_var("OPENAI_API_KEY"))
+component.to_dict()  # This is fine
+```