chore: import upstream snapshot with attribution

docs-website/reference_versioned_docs/version-2.22/haystack-api/agents_api.md

@@ -0,0 +1,418 @@
+---
+title: "Agents"
+id: agents-api
+description: "Tool-using agents with provider-agnostic chat model support."
+slug: "/agents-api"
+---
+
+<a id="agent"></a>
+
+## Module agent
+
+<a id="agent.Agent"></a>
+
+### Agent
+
+A Haystack component that implements a tool-using agent with provider-agnostic chat model support.
+
+The component processes messages and executes tools until an exit condition is met.
+The exit condition can be triggered either by a direct text response or by invoking a specific designated tool.
+Multiple exit conditions can be specified.
+
+When you call an Agent without tools, it acts as a ChatGenerator, produces one response, then exits.
+
+### Usage example
+```python
+from haystack.components.agents import Agent
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.dataclasses import ChatMessage
+from haystack.tools import Tool
+
+# Tool functions - in practice, these would have real implementations
+def search(query: str) -> str:
+    '''Search for information on the web.'''
+    # Placeholder: would call actual search API
+    return "In France, a 15% service charge is typically included, but leaving 5-10% extra is appreciated."
+
+def calculator(operation: str, a: float, b: float) -> float:
+    '''Perform mathematical calculations.'''
+    if operation == "multiply":
+        return a * b
+    elif operation == "percentage":
+        return (a / 100) * b
+    return 0
+
+# Define tools with JSON Schema
+tools = [
+    Tool(
+        name="search",
+        description="Searches for information on the web",
+        parameters={
+            "type": "object",
+            "properties": {
+                "query": {"type": "string", "description": "The search query"}
+            },
+            "required": ["query"]
+        },
+        function=search
+    ),
+    Tool(
+        name="calculator",
+        description="Performs mathematical calculations",
+        parameters={
+            "type": "object",
+            "properties": {
+                "operation": {"type": "string", "description": "Operation: multiply, percentage"},
+                "a": {"type": "number", "description": "First number"},
+                "b": {"type": "number", "description": "Second number"}
+            },
+            "required": ["operation", "a", "b"]
+        },
+        function=calculator
+    )
+]
+
+# Create and run the agent
+agent = Agent(
+    chat_generator=OpenAIChatGenerator(),
+    tools=tools
+)
+
+result = agent.run(
+    messages=[ChatMessage.from_user("Calculate the appropriate tip for an €85 meal in France")]
+)
+
+# The agent will:
+# 1. Search for tipping customs in France
+# 2. Use calculator to compute tip based on findings
+# 3. Return the final answer with context
+print(result["messages"][-1].text)
+```
+
+<a id="agent.Agent.__init__"></a>
+
+#### Agent.\_\_init\_\_
+
+```python
+def __init__(*,
+             chat_generator: ChatGenerator,
+             tools: ToolsType | None = None,
+             system_prompt: str | None = None,
+             exit_conditions: list[str] | None = None,
+             state_schema: dict[str, Any] | None = None,
+             max_agent_steps: int = 100,
+             streaming_callback: StreamingCallbackT | None = None,
+             raise_on_tool_invocation_failure: bool = False,
+             tool_invoker_kwargs: dict[str, Any] | None = None) -> None
+```
+
+Initialize the agent component.
+
+**Arguments**:
+
+- `chat_generator`: An instance of the chat generator that your agent should use. It must support tools.
+- `tools`: A list of Tool and/or Toolset objects, or a single Toolset that the agent can use.
+- `system_prompt`: System prompt for the agent.
+- `exit_conditions`: List of conditions that will cause the agent to return.
+Can include "text" if the agent should return when it generates a message without tool calls,
+or tool names that will cause the agent to return once the tool was executed. Defaults to ["text"].
+- `state_schema`: The schema for the runtime state used by the tools.
+- `max_agent_steps`: Maximum number of steps the agent will run before stopping. Defaults to 100.
+If the agent exceeds this number of steps, it will stop and return the current state.
+- `streaming_callback`: A callback that will be invoked when a response is streamed from the LLM.
+The same callback can be configured to emit tool results when a tool is called.
+- `raise_on_tool_invocation_failure`: Should the agent raise an exception when a tool invocation fails?
+If set to False, the exception will be turned into a chat message and passed to the LLM.
+- `tool_invoker_kwargs`: Additional keyword arguments to pass to the ToolInvoker.
+
+**Raises**:
+
+- `TypeError`: If the chat_generator does not support tools parameter in its run method.
+- `ValueError`: If the exit_conditions are not valid.
+
+<a id="agent.Agent.warm_up"></a>
+
+#### Agent.warm\_up
+
+```python
+def warm_up() -> None
+```
+
+Warm up the Agent.
+
+<a id="agent.Agent.to_dict"></a>
+
+#### Agent.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serialize the component to a dictionary.
+
+**Returns**:
+
+Dictionary with serialized data
+
+<a id="agent.Agent.from_dict"></a>
+
+#### Agent.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "Agent"
+```
+
+Deserialize the agent from a dictionary.
+
+**Arguments**:
+
+- `data`: Dictionary to deserialize from
+
+**Returns**:
+
+Deserialized agent
+
+<a id="agent.Agent.run"></a>
+
+#### Agent.run
+
+```python
+def run(messages: list[ChatMessage],
+        streaming_callback: StreamingCallbackT | None = None,
+        *,
+        generation_kwargs: dict[str, Any] | None = None,
+        break_point: AgentBreakpoint | None = None,
+        snapshot: AgentSnapshot | None = None,
+        system_prompt: str | None = None,
+        tools: ToolsType | list[str] | None = None,
+        **kwargs: Any) -> dict[str, Any]
+```
+
+Process messages and execute tools until an exit condition is met.
+
+**Arguments**:
+
+- `messages`: List of Haystack ChatMessage objects to process.
+- `streaming_callback`: A callback that will be invoked when a response is streamed from the LLM.
+The same callback can be configured to emit tool results when a tool is called.
+- `generation_kwargs`: Additional keyword arguments for LLM. These parameters will
+override the parameters passed during component initialization.
+- `break_point`: An AgentBreakpoint, can be a Breakpoint for the "chat_generator" or a ToolBreakpoint
+for "tool_invoker".
+- `snapshot`: A dictionary containing a snapshot of a previously saved agent execution. The snapshot contains
+the relevant information to restart the Agent execution from where it left off.
+- `system_prompt`: System prompt for the agent. If provided, it overrides the default system prompt.
+- `tools`: Optional list of Tool objects, a Toolset, or list of tool names to use for this run.
+When passing tool names, tools are selected from the Agent's originally configured tools.
+- `kwargs`: Additional data to pass to the State schema used by the Agent.
+The keys must match the schema defined in the Agent's `state_schema`.
+
+**Raises**:
+
+- `BreakpointException`: If an agent breakpoint is triggered.
+
+**Returns**:
+
+A dictionary with the following keys:
+- "messages": List of all messages exchanged during the agent's run.
+- "last_message": The last message exchanged during the agent's run.
+- Any additional keys defined in the `state_schema`.
+
+<a id="agent.Agent.run_async"></a>
+
+#### Agent.run\_async
+
+```python
+async def run_async(messages: list[ChatMessage],
+                    streaming_callback: StreamingCallbackT | None = None,
+                    *,
+                    generation_kwargs: dict[str, Any] | None = None,
+                    break_point: AgentBreakpoint | None = None,
+                    snapshot: AgentSnapshot | None = None,
+                    system_prompt: str | None = None,
+                    tools: ToolsType | list[str] | None = None,
+                    **kwargs: Any) -> dict[str, Any]
+```
+
+Asynchronously process messages and execute tools until the exit condition is met.
+
+This is the asynchronous version of the `run` method. It follows the same logic but uses
+asynchronous operations where possible, such as calling the `run_async` method of the ChatGenerator
+if available.
+
+**Arguments**:
+
+- `messages`: List of Haystack ChatMessage objects to process.
+- `streaming_callback`: An asynchronous callback that will be invoked when a response is streamed from the
+LLM. The same callback can be configured to emit tool results when a tool is called.
+- `generation_kwargs`: Additional keyword arguments for LLM. These parameters will
+override the parameters passed during component initialization.
+- `break_point`: An AgentBreakpoint, can be a Breakpoint for the "chat_generator" or a ToolBreakpoint
+for "tool_invoker".
+- `snapshot`: A dictionary containing a snapshot of a previously saved agent execution. The snapshot contains
+the relevant information to restart the Agent execution from where it left off.
+- `system_prompt`: System prompt for the agent. If provided, it overrides the default system prompt.
+- `tools`: Optional list of Tool objects, a Toolset, or list of tool names to use for this run.
+- `kwargs`: Additional data to pass to the State schema used by the Agent.
+The keys must match the schema defined in the Agent's `state_schema`.
+
+**Raises**:
+
+- `BreakpointException`: If an agent breakpoint is triggered.
+
+**Returns**:
+
+A dictionary with the following keys:
+- "messages": List of all messages exchanged during the agent's run.
+- "last_message": The last message exchanged during the agent's run.
+- Any additional keys defined in the `state_schema`.
+
+<a id="state/state"></a>
+
+## Module state/state
+
+<a id="state/state.State"></a>
+
+### State
+
+State is a container for storing shared information during the execution of an Agent and its tools.
+
+For instance, State can be used to store documents, context, and intermediate results.
+
+Internally it wraps a `_data` dictionary defined by a `schema`. Each schema entry has:
+```json
+  "parameter_name": {
+    "type": SomeType,  # expected type
+    "handler": Optional[Callable[[Any, Any], Any]]  # merge/update function
+  }
+  ```
+
+Handlers control how values are merged when using the `set()` method:
+- For list types: defaults to `merge_lists` (concatenates lists)
+- For other types: defaults to `replace_values` (overwrites existing value)
+
+A `messages` field with type `list[ChatMessage]` is automatically added to the schema.
+
+This makes it possible for the Agent to read from and write to the same context.
+
+### Usage example
+```python
+from haystack.components.agents.state import State
+
+my_state = State(
+    schema={"gh_repo_name": {"type": str}, "user_name": {"type": str}},
+    data={"gh_repo_name": "my_repo", "user_name": "my_user_name"}
+)
+```
+
+<a id="state/state.State.__init__"></a>
+
+#### State.\_\_init\_\_
+
+```python
+def __init__(schema: dict[str, Any], data: dict[str, Any] | None = None)
+```
+
+Initialize a State object with a schema and optional data.
+
+**Arguments**:
+
+- `schema`: Dictionary mapping parameter names to their type and handler configs.
+Type must be a valid Python type, and handler must be a callable function or None.
+If handler is None, the default handler for the type will be used. The default handlers are:
+    - For list types: `haystack.agents.state.state_utils.merge_lists`
+    - For all other types: `haystack.agents.state.state_utils.replace_values`
+- `data`: Optional dictionary of initial data to populate the state
+
+<a id="state/state.State.get"></a>
+
+#### State.get
+
+```python
+def get(key: str, default: Any = None) -> Any
+```
+
+Retrieve a value from the state by key.
+
+**Arguments**:
+
+- `key`: Key to look up in the state
+- `default`: Value to return if key is not found
+
+**Returns**:
+
+Value associated with key or default if not found
+
+<a id="state/state.State.set"></a>
+
+#### State.set
+
+```python
+def set(key: str,
+        value: Any,
+        handler_override: Callable[[Any, Any], Any] | None = None) -> None
+```
+
+Set or merge a value in the state according to schema rules.
+
+Value is merged or overwritten according to these rules:
+  - if handler_override is given, use that
+  - else use the handler defined in the schema for 'key'
+
+**Arguments**:
+
+- `key`: Key to store the value under
+- `value`: Value to store or merge
+- `handler_override`: Optional function to override the default merge behavior
+
+<a id="state/state.State.data"></a>
+
+#### State.data
+
+```python
+@property
+def data()
+```
+
+All current data of the state.
+
+<a id="state/state.State.has"></a>
+
+#### State.has
+
+```python
+def has(key: str) -> bool
+```
+
+Check if a key exists in the state.
+
+**Arguments**:
+
+- `key`: Key to check for existence
+
+**Returns**:
+
+True if key exists in state, False otherwise
+
+<a id="state/state.State.to_dict"></a>
+
+#### State.to\_dict
+
+```python
+def to_dict()
+```
+
+Convert the State object to a dictionary.
+
+<a id="state/state.State.from_dict"></a>
+
+#### State.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any])
+```
+
+Convert a dictionary back to a State object.
+

docs-website/reference_versioned_docs/version-2.22/haystack-api/audio_api.md

@@ -0,0 +1,267 @@
+---
+title: "Audio"
+id: audio-api
+description: "Transcribes audio files."
+slug: "/audio-api"
+---
+
+<a id="whisper_local"></a>
+
+## Module whisper\_local
+
+<a id="whisper_local.LocalWhisperTranscriber"></a>
+
+### LocalWhisperTranscriber
+
+Transcribes audio files using OpenAI's Whisper model on your local machine.
+
+For the supported audio formats, languages, and other parameters, see the
+[Whisper API documentation](https://platform.openai.com/docs/guides/speech-to-text) and the official Whisper
+[GitHub repository](https://github.com/openai/whisper).
+
+### Usage example
+
+```python
+from haystack.components.audio import LocalWhisperTranscriber
+
+whisper = LocalWhisperTranscriber(model="small")
+whisper.warm_up()
+transcription = whisper.run(sources=["test/test_files/audio/answer.wav"])
+```
+
+<a id="whisper_local.LocalWhisperTranscriber.__init__"></a>
+
+#### LocalWhisperTranscriber.\_\_init\_\_
+
+```python
+def __init__(model: WhisperLocalModel = "large",
+             device: ComponentDevice | None = None,
+             whisper_params: dict[str, Any] | None = None)
+```
+
+Creates an instance of the LocalWhisperTranscriber component.
+
+**Arguments**:
+
+- `model`: The name of the model to use. Set to one of the following models:
+"tiny", "base", "small", "medium", "large" (default).
+For details on the models and their modifications, see the
+[Whisper documentation](https://github.com/openai/whisper?tab=readme-ov-file#available-models-and-languages).
+- `device`: The device for loading the model. If `None`, automatically selects the default device.
+
+<a id="whisper_local.LocalWhisperTranscriber.warm_up"></a>
+
+#### LocalWhisperTranscriber.warm\_up
+
+```python
+def warm_up() -> None
+```
+
+Loads the model in memory.
+
+<a id="whisper_local.LocalWhisperTranscriber.to_dict"></a>
+
+#### LocalWhisperTranscriber.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+**Returns**:
+
+Dictionary with serialized data.
+
+<a id="whisper_local.LocalWhisperTranscriber.from_dict"></a>
+
+#### LocalWhisperTranscriber.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "LocalWhisperTranscriber"
+```
+
+Deserializes the component from a dictionary.
+
+**Arguments**:
+
+- `data`: The dictionary to deserialize from.
+
+**Returns**:
+
+The deserialized component.
+
+<a id="whisper_local.LocalWhisperTranscriber.run"></a>
+
+#### LocalWhisperTranscriber.run
+
+```python
+@component.output_types(documents=list[Document])
+def run(sources: list[str | Path | ByteStream],
+        whisper_params: dict[str, Any] | None = None)
+```
+
+Transcribes a list of audio files into a list of documents.
+
+**Arguments**:
+
+- `sources`: A list of paths or binary streams to transcribe.
+- `whisper_params`: For the supported audio formats, languages, and other parameters, see the
+[Whisper API documentation](https://platform.openai.com/docs/guides/speech-to-text) and the official Whisper
+[GitHup repo](https://github.com/openai/whisper).
+
+**Returns**:
+
+A dictionary with the following keys:
+- `documents`: A list of documents where each document is a transcribed audio file. The content of
+the document is the transcription text, and the document's metadata contains the values returned by
+the Whisper model, such as the alignment data and the path to the audio file used
+for the transcription.
+
+<a id="whisper_local.LocalWhisperTranscriber.transcribe"></a>
+
+#### LocalWhisperTranscriber.transcribe
+
+```python
+def transcribe(sources: list[str | Path | ByteStream],
+               **kwargs) -> list[Document]
+```
+
+Transcribes the audio files into a list of Documents, one for each input file.
+
+For the supported audio formats, languages, and other parameters, see the
+[Whisper API documentation](https://platform.openai.com/docs/guides/speech-to-text) and the official Whisper
+[github repo](https://github.com/openai/whisper).
+
+**Arguments**:
+
+- `sources`: A list of paths or binary streams to transcribe.
+
+**Returns**:
+
+A list of Documents, one for each file.
+
+<a id="whisper_remote"></a>
+
+## Module whisper\_remote
+
+<a id="whisper_remote.RemoteWhisperTranscriber"></a>
+
+### RemoteWhisperTranscriber
+
+Transcribes audio files using the OpenAI's Whisper API.
+
+The component requires an OpenAI API key, see the
+[OpenAI documentation](https://platform.openai.com/docs/api-reference/authentication) for more details.
+For the supported audio formats, languages, and other parameters, see the
+[Whisper API documentation](https://platform.openai.com/docs/guides/speech-to-text).
+
+### Usage example
+
+```python
+from haystack.components.audio import RemoteWhisperTranscriber
+
+whisper = RemoteWhisperTranscriber(model="whisper-1")
+transcription = whisper.run(sources=["test/test_files/audio/answer.wav"])
+```
+
+<a id="whisper_remote.RemoteWhisperTranscriber.__init__"></a>
+
+#### RemoteWhisperTranscriber.\_\_init\_\_
+
+```python
+def __init__(api_key: Secret = Secret.from_env_var("OPENAI_API_KEY"),
+             model: str = "whisper-1",
+             api_base_url: str | None = None,
+             organization: str | None = None,
+             http_client_kwargs: dict[str, Any] | None = None,
+             **kwargs)
+```
+
+Creates an instance of the RemoteWhisperTranscriber component.
+
+**Arguments**:
+
+- `api_key`: OpenAI API key.
+You can set it with an environment variable `OPENAI_API_KEY`, or pass with this parameter
+during initialization.
+- `model`: Name of the model to use. Currently accepts only `whisper-1`.
+- `organization`: Your OpenAI organization ID. See OpenAI's documentation on
+[Setting Up Your Organization](https://platform.openai.com/docs/guides/production-best-practices/setting-up-your-organization).
+- `api_base`: An optional URL to use as the API base. For details, see the
+OpenAI [documentation](https://platform.openai.com/docs/api-reference/audio).
+- `http_client_kwargs`: A dictionary of keyword arguments to configure a custom `httpx.Client`or `httpx.AsyncClient`.
+For more information, see the [HTTPX documentation](https://www.python-httpx.org/api/`client`).
+- `kwargs`: Other optional parameters for the model. These are sent directly to the OpenAI
+endpoint. See OpenAI [documentation](https://platform.openai.com/docs/api-reference/audio) for more details.
+Some of the supported parameters are:
+- `language`: The language of the input audio.
+  Provide the input language in ISO-639-1 format
+  to improve transcription accuracy and latency.
+- `prompt`: An optional text to guide the model's
+  style or continue a previous audio segment.
+  The prompt should match the audio language.
+- `response_format`: The format of the transcript
+  output. This component only supports `json`.
+- `temperature`: The sampling temperature, between 0
+and 1. Higher values like 0.8 make the output more
+random, while lower values like 0.2 make it more
+focused and deterministic. If set to 0, the model
+uses log probability to automatically increase the
+temperature until certain thresholds are hit.
+
+<a id="whisper_remote.RemoteWhisperTranscriber.to_dict"></a>
+
+#### RemoteWhisperTranscriber.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+**Returns**:
+
+Dictionary with serialized data.
+
+<a id="whisper_remote.RemoteWhisperTranscriber.from_dict"></a>
+
+#### RemoteWhisperTranscriber.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "RemoteWhisperTranscriber"
+```
+
+Deserializes the component from a dictionary.
+
+**Arguments**:
+
+- `data`: The dictionary to deserialize from.
+
+**Returns**:
+
+The deserialized component.
+
+<a id="whisper_remote.RemoteWhisperTranscriber.run"></a>
+
+#### RemoteWhisperTranscriber.run
+
+```python
+@component.output_types(documents=list[Document])
+def run(sources: list[str | Path | ByteStream])
+```
+
+Transcribes the list of audio files into a list of documents.
+
+**Arguments**:
+
+- `sources`: A list of file paths or `ByteStream` objects containing the audio files to transcribe.
+
+**Returns**:
+
+A dictionary with the following keys:
+- `documents`: A list of documents, one document for each file.
+The content of each document is the transcribed text.
+

docs-website/reference_versioned_docs/version-2.22/haystack-api/builders_api.md

@@ -0,0 +1,553 @@
+---
+title: "Builders"
+id: builders-api
+description: "Extract the output of a Generator to an Answer format, and build prompts."
+slug: "/builders-api"
+---
+
+<a id="answer_builder"></a>
+
+## Module answer\_builder
+
+<a id="answer_builder.AnswerBuilder"></a>
+
+### AnswerBuilder
+
+Converts a query and Generator replies into a `GeneratedAnswer` object.
+
+AnswerBuilder parses Generator replies using custom regular expressions.
+Check out the usage example below to see how it works.
+Optionally, it can also take documents and metadata from the Generator to add to the `GeneratedAnswer` object.
+AnswerBuilder works with both non-chat and chat Generators.
+
+### Usage example
+
+
+### Usage example with documents and reference pattern
+
+```python
+from haystack.components.builders import AnswerBuilder
+
+builder = AnswerBuilder(pattern="Answer: (.*)")
+builder.run(query="What's the answer?", replies=["This is an argument. Answer: This is the answer."])
+```
+```python
+from haystack import Document
+from haystack.components.builders import AnswerBuilder
+
+replies = ["The capital of France is Paris [2]."]
+
+docs = [
+    Document(content="Berlin is the capital of Germany."),
+    Document(content="Paris is the capital of France."),
+    Document(content="Rome is the capital of Italy."),
+]
+
+builder = AnswerBuilder(reference_pattern="\[(\d+)\]", return_only_referenced_documents=False)
+result = builder.run(query="What is the capital of France?", replies=replies, documents=docs)["answers"][0]
+
+print(f"Answer: {result.data}")
+print("References:")
+for doc in result.documents:
+    if doc.meta["referenced"]:
+        print(f"[{doc.meta['source_index']}] {doc.content}")
+print("Other sources:")
+for doc in result.documents:
+    if not doc.meta["referenced"]:
+        print(f"[{doc.meta['source_index']}] {doc.content}")
+
+# Answer: The capital of France is Paris
+# References:
+# [2] Paris is the capital of France.
+# Other sources:
+# [1] Berlin is the capital of Germany.
+# [3] Rome is the capital of Italy.
+```
+
+<a id="answer_builder.AnswerBuilder.__init__"></a>
+
+#### AnswerBuilder.\_\_init\_\_
+
+```python
+def __init__(pattern: str | None = None,
+             reference_pattern: str | None = None,
+             last_message_only: bool = False,
+             *,
+             return_only_referenced_documents: bool = True)
+```
+
+Creates an instance of the AnswerBuilder component.
+
+**Arguments**:
+
+- `pattern`: The regular expression pattern to extract the answer text from the Generator.
+If not specified, the entire response is used as the answer.
+The regular expression can have one capture group at most.
+If present, the capture group text
+is used as the answer. If no capture group is present, the whole match is used as the answer.
+Examples:
+    `[^\n]+$` finds "this is an answer" in a string "this is an argument.\nthis is an answer".
+    `Answer: (.*)` finds "this is an answer" in a string "this is an argument. Answer: this is an answer".
+- `reference_pattern`: The regular expression pattern used for parsing the document references.
+If not specified, no parsing is done, and all documents are returned.
+References need to be specified as indices of the input documents and start at [1].
+Example: `\[(\d+)\]` finds "1" in a string "this is an answer[1]".
+If this parameter is provided, documents metadata will contain a "referenced" key with a boolean value.
+- `last_message_only`: If False (default value), all messages are used as the answer.
+If True, only the last message is used as the answer.
+- `return_only_referenced_documents`: To be used in conjunction with `reference_pattern`.
+If True (default value), only the documents that were actually referenced in `replies` are returned.
+If False, all documents are returned.
+If `reference_pattern` is not provided, this parameter has no effect, and all documents are returned.
+
+<a id="answer_builder.AnswerBuilder.run"></a>
+
+#### AnswerBuilder.run
+
+```python
+@component.output_types(answers=list[GeneratedAnswer])
+def run(query: str,
+        replies: list[str] | list[ChatMessage],
+        meta: list[dict[str, Any]] | None = None,
+        documents: list[Document] | None = None,
+        pattern: str | None = None,
+        reference_pattern: str | None = None)
+```
+
+Turns the output of a Generator into `GeneratedAnswer` objects using regular expressions.
+
+**Arguments**:
+
+- `query`: The input query used as the Generator prompt.
+- `replies`: The output of the Generator. Can be a list of strings or a list of `ChatMessage` objects.
+- `meta`: The metadata returned by the Generator. If not specified, the generated answer will contain no metadata.
+- `documents`: The documents used as the Generator inputs. If specified, they are added to
+the `GeneratedAnswer` objects.
+Each Document.meta includes a "source_index" key, representing its 1-based position in the input list.
+When `reference_pattern` is provided:
+- "referenced" key is added to the Document.meta, indicating if the document was referenced in the output.
+- `return_only_referenced_documents` init parameter controls if all or only referenced documents are
+returned.
+- `pattern`: The regular expression pattern to extract the answer text from the Generator.
+If not specified, the entire response is used as the answer.
+The regular expression can have one capture group at most.
+If present, the capture group text
+is used as the answer. If no capture group is present, the whole match is used as the answer.
+    Examples:
+        `[^\n]+$` finds "this is an answer" in a string "this is an argument.\nthis is an answer".
+        `Answer: (.*)` finds "this is an answer" in a string
+        "this is an argument. Answer: this is an answer".
+- `reference_pattern`: The regular expression pattern used for parsing the document references.
+If not specified, no parsing is done, and all documents are returned.
+References need to be specified as indices of the input documents and start at [1].
+Example: `\[(\d+)\]` finds "1" in a string "this is an answer[1]".
+
+**Returns**:
+
+A dictionary with the following keys:
+- `answers`: The answers received from the output of the Generator.
+
+<a id="chat_prompt_builder"></a>
+
+## Module chat\_prompt\_builder
+
+<a id="chat_prompt_builder.ChatPromptBuilder"></a>
+
+### ChatPromptBuilder
+
+Renders a chat prompt from a template using Jinja2 syntax.
+
+A template can be a list of `ChatMessage` objects, or a special string, as shown in the usage examples.
+
+It constructs prompts using static or dynamic templates, which you can update for each pipeline run.
+
+Template variables in the template are optional unless specified otherwise.
+If an optional variable isn't provided, it defaults to an empty string. Use `variable` and `required_variables`
+to define input types and required variables.
+
+### Usage examples
+
+#### Static ChatMessage prompt template
+
+```python
+template = [ChatMessage.from_user("Translate to {{ target_language }}. Context: {{ snippet }}; Translation:")]
+builder = ChatPromptBuilder(template=template)
+builder.run(target_language="spanish", snippet="I can't speak spanish.")
+```
+
+#### Overriding static ChatMessage template at runtime
+
+```python
+template = [ChatMessage.from_user("Translate to {{ target_language }}. Context: {{ snippet }}; Translation:")]
+builder = ChatPromptBuilder(template=template)
+builder.run(target_language="spanish", snippet="I can't speak spanish.")
+
+msg = "Translate to {{ target_language }} and summarize. Context: {{ snippet }}; Summary:"
+summary_template = [ChatMessage.from_user(msg)]
+builder.run(target_language="spanish", snippet="I can't speak spanish.", template=summary_template)
+```
+
+#### Dynamic ChatMessage prompt template
+
+```python
+from haystack.components.builders import ChatPromptBuilder
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.dataclasses import ChatMessage
+from haystack import Pipeline
+
+# no parameter init, we don't use any runtime template variables
+prompt_builder = ChatPromptBuilder()
+llm = OpenAIChatGenerator(model="gpt-5-mini")
+
+pipe = Pipeline()
+pipe.add_component("prompt_builder", prompt_builder)
+pipe.add_component("llm", llm)
+pipe.connect("prompt_builder.prompt", "llm.messages")
+
+location = "Berlin"
+language = "English"
+system_message = ChatMessage.from_system("You are an assistant giving information to tourists in {{language}}")
+messages = [system_message, ChatMessage.from_user("Tell me about {{location}}")]
+
+res = pipe.run(data={"prompt_builder": {"template_variables": {"location": location, "language": language},
+                                    "template": messages}})
+print(res)
+# >> {'llm': {'replies': [ChatMessage(_role=<ChatRole.ASSISTANT: 'assistant'>, _content=[TextContent(text=
+# "Berlin is the capital city of Germany and one of the most vibrant
+# and diverse cities in Europe. Here are some key things to know...Enjoy your time exploring the vibrant and dynamic
+# capital of Germany!")], _name=None, _meta={'model': 'gpt-5-mini',
+# 'index': 0, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 27, 'completion_tokens': 681, 'total_tokens':
+# 708}})]}}
+
+messages = [system_message, ChatMessage.from_user("What's the weather forecast for {{location}} in the next
+{{day_count}} days?")]
+
+res = pipe.run(data={"prompt_builder": {"template_variables": {"location": location, "day_count": "5"},
+                                    "template": messages}})
+
+print(res)
+# >> {'llm': {'replies': [ChatMessage(_role=<ChatRole.ASSISTANT: 'assistant'>, _content=[TextContent(text=
+# "Here is the weather forecast for Berlin in the next 5
+# days:\n\nDay 1: Mostly cloudy with a high of 22°C (72°F) and...so it's always a good idea to check for updates
+# closer to your visit.")], _name=None, _meta={'model': 'gpt-5-mini',
+# 'index': 0, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 37, 'completion_tokens': 201,
+# 'total_tokens': 238}})]}}
+```
+
+#### String prompt template
+```python
+from haystack.components.builders import ChatPromptBuilder
+from haystack.dataclasses.image_content import ImageContent
+
+template = """
+{% message role="system" %}
+You are a helpful assistant.
+{% endmessage %}
+
+{% message role="user" %}
+Hello! I am {{user_name}}. What's the difference between the following images?
+{% for image in images %}
+{{ image | templatize_part }}
+{% endfor %}
+{% endmessage %}
+"""
+
+images = [ImageContent.from_file_path("test/test_files/images/apple.jpg"),
+          ImageContent.from_file_path("test/test_files/images/haystack-logo.png")]
+
+builder = ChatPromptBuilder(template=template)
+builder.run(user_name="John", images=images)
+```
+
+<a id="chat_prompt_builder.ChatPromptBuilder.__init__"></a>
+
+#### ChatPromptBuilder.\_\_init\_\_
+
+```python
+def __init__(template: list[ChatMessage] | str | None = None,
+             required_variables: list[str] | Literal["*"] | None = None,
+             variables: list[str] | None = None)
+```
+
+Constructs a ChatPromptBuilder component.
+
+**Arguments**:
+
+- `template`: A list of `ChatMessage` objects or a string template. The component looks for Jinja2 template syntax and
+renders the prompt with the provided variables. Provide the template in either
+the `init` method` or the `run` method.
+- `required_variables`: List variables that must be provided as input to ChatPromptBuilder.
+If a variable listed as required is not provided, an exception is raised.
+If set to "*", all variables found in the prompt are required. Optional.
+- `variables`: List input variables to use in prompt templates instead of the ones inferred from the
+`template` parameter. For example, to use more variables during prompt engineering than the ones present
+in the default template, you can provide them here.
+
+<a id="chat_prompt_builder.ChatPromptBuilder.run"></a>
+
+#### ChatPromptBuilder.run
+
+```python
+@component.output_types(prompt=list[ChatMessage])
+def run(template: list[ChatMessage] | str | None = None,
+        template_variables: dict[str, Any] | None = None,
+        **kwargs)
+```
+
+Renders the prompt template with the provided variables.
+
+It applies the template variables to render the final prompt. You can provide variables with pipeline kwargs.
+To overwrite the default template, you can set the `template` parameter.
+To overwrite pipeline kwargs, you can set the `template_variables` parameter.
+
+**Arguments**:
+
+- `template`: An optional list of `ChatMessage` objects or string template to overwrite ChatPromptBuilder's default
+template.
+If `None`, the default template provided at initialization is used.
+- `template_variables`: An optional dictionary of template variables to overwrite the pipeline variables.
+- `kwargs`: Pipeline variables used for rendering the prompt.
+
+**Raises**:
+
+- `ValueError`: If `chat_messages` is empty or contains elements that are not instances of `ChatMessage`.
+
+**Returns**:
+
+A dictionary with the following keys:
+- `prompt`: The updated list of `ChatMessage` objects after rendering the templates.
+
+<a id="chat_prompt_builder.ChatPromptBuilder.to_dict"></a>
+
+#### ChatPromptBuilder.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Returns a dictionary representation of the component.
+
+**Returns**:
+
+Serialized dictionary representation of the component.
+
+<a id="chat_prompt_builder.ChatPromptBuilder.from_dict"></a>
+
+#### ChatPromptBuilder.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "ChatPromptBuilder"
+```
+
+Deserialize this component from a dictionary.
+
+**Arguments**:
+
+- `data`: The dictionary to deserialize and create the component.
+
+**Returns**:
+
+The deserialized component.
+
+<a id="prompt_builder"></a>
+
+## Module prompt\_builder
+
+<a id="prompt_builder.PromptBuilder"></a>
+
+### PromptBuilder
+
+Renders a prompt filling in any variables so that it can send it to a Generator.
+
+The prompt uses Jinja2 template syntax.
+The variables in the default template are used as PromptBuilder's input and are all optional.
+If they're not provided, they're replaced with an empty string in the rendered prompt.
+To try out different prompts, you can replace the prompt template at runtime by
+providing a template for each pipeline run invocation.
+
+### Usage examples
+
+#### On its own
+
+This example uses PromptBuilder to render a prompt template and fill it with `target_language`
+and `snippet`. PromptBuilder returns a prompt with the string "Translate the following context to Spanish.
+Context: I can't speak Spanish.; Translation:".
+```python
+from haystack.components.builders import PromptBuilder
+
+template = "Translate the following context to {{ target_language }}. Context: {{ snippet }}; Translation:"
+builder = PromptBuilder(template=template)
+builder.run(target_language="spanish", snippet="I can't speak spanish.")
+```
+
+#### In a Pipeline
+
+This is an example of a RAG pipeline where PromptBuilder renders a custom prompt template and fills it
+with the contents of the retrieved documents and a query. The rendered prompt is then sent to a Generator.
+```python
+from haystack import Pipeline, Document
+from haystack.utils import Secret
+from haystack.components.generators import OpenAIGenerator
+from haystack.components.builders.prompt_builder import PromptBuilder
+
+# in a real world use case documents could come from a retriever, web, or any other source
+documents = [Document(content="Joe lives in Berlin"), Document(content="Joe is a software engineer")]
+prompt_template = """
+    Given these documents, answer the question.
+    Documents:
+    {% for doc in documents %}
+        {{ doc.content }}
+    {% endfor %}
+
+    Question: {{query}}
+    Answer:
+    """
+p = Pipeline()
+p.add_component(instance=PromptBuilder(template=prompt_template), name="prompt_builder")
+p.add_component(instance=OpenAIGenerator(api_key=Secret.from_env_var("OPENAI_API_KEY")), name="llm")
+p.connect("prompt_builder", "llm")
+
+question = "Where does Joe live?"
+result = p.run({"prompt_builder": {"documents": documents, "query": question}})
+print(result)
+```
+
+#### Changing the template at runtime (prompt engineering)
+
+You can change the prompt template of an existing pipeline, like in this example:
+```python
+documents = [
+    Document(content="Joe lives in Berlin", meta={"name": "doc1"}),
+    Document(content="Joe is a software engineer", meta={"name": "doc1"}),
+]
+new_template = """
+    You are a helpful assistant.
+    Given these documents, answer the question.
+    Documents:
+    {% for doc in documents %}
+        Document {{ loop.index }}:
+        Document name: {{ doc.meta['name'] }}
+        {{ doc.content }}
+    {% endfor %}
+
+    Question: {{ query }}
+    Answer:
+    """
+p.run({
+    "prompt_builder": {
+        "documents": documents,
+        "query": question,
+        "template": new_template,
+    },
+})
+```
+To replace the variables in the default template when testing your prompt,
+pass the new variables in the `variables` parameter.
+
+#### Overwriting variables at runtime
+
+To overwrite the values of variables, use `template_variables` during runtime:
+```python
+language_template = """
+You are a helpful assistant.
+Given these documents, answer the question.
+Documents:
+{% for doc in documents %}
+    Document {{ loop.index }}:
+    Document name: {{ doc.meta['name'] }}
+    {{ doc.content }}
+{% endfor %}
+
+Question: {{ query }}
+Please provide your answer in {{ answer_language | default('English') }}
+Answer:
+"""
+p.run({
+    "prompt_builder": {
+        "documents": documents,
+        "query": question,
+        "template": language_template,
+        "template_variables": {"answer_language": "German"},
+    },
+})
+```
+Note that `language_template` introduces variable `answer_language` which is not bound to any pipeline variable.
+If not set otherwise, it will use its default value 'English'.
+This example overwrites its value to 'German'.
+Use `template_variables` to overwrite pipeline variables (such as documents) as well.
+
+<a id="prompt_builder.PromptBuilder.__init__"></a>
+
+#### PromptBuilder.\_\_init\_\_
+
+```python
+def __init__(template: str,
+             required_variables: list[str] | Literal["*"] | None = None,
+             variables: list[str] | None = None)
+```
+
+Constructs a PromptBuilder component.
+
+**Arguments**:
+
+- `template`: A prompt template that uses Jinja2 syntax to add variables. For example:
+`"Summarize this document: {{ documents[0].content }}\nSummary:"`
+It's used to render the prompt.
+The variables in the default template are input for PromptBuilder and are all optional,
+unless explicitly specified.
+If an optional variable is not provided, it's replaced with an empty string in the rendered prompt.
+- `required_variables`: List variables that must be provided as input to PromptBuilder.
+If a variable listed as required is not provided, an exception is raised.
+If set to "*", all variables found in the prompt are required. Optional.
+- `variables`: List input variables to use in prompt templates instead of the ones inferred from the
+`template` parameter. For example, to use more variables during prompt engineering than the ones present
+in the default template, you can provide them here.
+
+<a id="prompt_builder.PromptBuilder.to_dict"></a>
+
+#### PromptBuilder.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Returns a dictionary representation of the component.
+
+**Returns**:
+
+Serialized dictionary representation of the component.
+
+<a id="prompt_builder.PromptBuilder.run"></a>
+
+#### PromptBuilder.run
+
+```python
+@component.output_types(prompt=str)
+def run(template: str | None = None,
+        template_variables: dict[str, Any] | None = None,
+        **kwargs)
+```
+
+Renders the prompt template with the provided variables.
+
+It applies the template variables to render the final prompt. You can provide variables via pipeline kwargs.
+In order to overwrite the default template, you can set the `template` parameter.
+In order to overwrite pipeline kwargs, you can set the `template_variables` parameter.
+
+**Arguments**:
+
+- `template`: An optional string template to overwrite PromptBuilder's default template. If None, the default template
+provided at initialization is used.
+- `template_variables`: An optional dictionary of template variables to overwrite the pipeline variables.
+- `kwargs`: Pipeline variables used for rendering the prompt.
+
+**Raises**:
+
+- `ValueError`: If any of the required template variables is not provided.
+
+**Returns**:
+
+A dictionary with the following keys:
+- `prompt`: The updated prompt text after rendering the prompt template.
+

docs-website/reference_versioned_docs/version-2.22/haystack-api/cachings_api.md

@@ -0,0 +1,110 @@
+---
+title: "Caching"
+id: caching-api
+description: "Checks if any document coming from the given URL is already present in the store."
+slug: "/caching-api"
+---
+
+<a id="cache_checker"></a>
+
+## Module cache\_checker
+
+<a id="cache_checker.CacheChecker"></a>
+
+### CacheChecker
+
+Checks for the presence of documents in a Document Store based on a specified field in each document's metadata.
+
+If matching documents are found, they are returned as "hits". If not found in the cache, the items
+are returned as "misses".
+
+### Usage example
+
+```python
+from haystack import Document
+from haystack.document_stores.in_memory import InMemoryDocumentStore
+from haystack.components.caching.cache_checker import CacheChecker
+
+docstore = InMemoryDocumentStore()
+documents = [
+    Document(content="doc1", meta={"url": "https://example.com/1"}),
+    Document(content="doc2", meta={"url": "https://example.com/2"}),
+    Document(content="doc3", meta={"url": "https://example.com/1"}),
+    Document(content="doc4", meta={"url": "https://example.com/2"}),
+]
+docstore.write_documents(documents)
+checker = CacheChecker(docstore, cache_field="url")
+results = checker.run(items=["https://example.com/1", "https://example.com/5"])
+assert results == {"hits": [documents[0], documents[2]], "misses": ["https://example.com/5"]}
+```
+
+<a id="cache_checker.CacheChecker.__init__"></a>
+
+#### CacheChecker.\_\_init\_\_
+
+```python
+def __init__(document_store: DocumentStore, cache_field: str)
+```
+
+Creates a CacheChecker component.
+
+**Arguments**:
+
+- `document_store`: Document Store to check for the presence of specific documents.
+- `cache_field`: Name of the document's metadata field
+to check for cache hits.
+
+<a id="cache_checker.CacheChecker.to_dict"></a>
+
+#### CacheChecker.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+**Returns**:
+
+Dictionary with serialized data.
+
+<a id="cache_checker.CacheChecker.from_dict"></a>
+
+#### CacheChecker.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "CacheChecker"
+```
+
+Deserializes the component from a dictionary.
+
+**Arguments**:
+
+- `data`: Dictionary to deserialize from.
+
+**Returns**:
+
+Deserialized component.
+
+<a id="cache_checker.CacheChecker.run"></a>
+
+#### CacheChecker.run
+
+```python
+@component.output_types(hits=list[Document], misses=list)
+def run(items: list[Any])
+```
+
+Checks if any document associated with the specified cache field is already present in the store.
+
+**Arguments**:
+
+- `items`: Values to be checked against the cache field.
+
+**Returns**:
+
+A dictionary with two keys:
+- `hits` - Documents that matched with at least one of the items.
+- `misses` - Items that were not present in any documents.
+

docs-website/reference_versioned_docs/version-2.22/haystack-api/classifiers_api.md

@@ -0,0 +1,270 @@
+---
+title: "Classifiers"
+id: classifiers-api
+description: "Classify documents based on the provided labels."
+slug: "/classifiers-api"
+---
+
+<a id="document_language_classifier"></a>
+
+## Module document\_language\_classifier
+
+<a id="document_language_classifier.DocumentLanguageClassifier"></a>
+
+### DocumentLanguageClassifier
+
+Classifies the language of each document and adds it to its metadata.
+
+Provide a list of languages during initialization. If the document's text doesn't match any of the
+specified languages, the metadata value is set to "unmatched".
+To route documents based on their language, use the MetadataRouter component after DocumentLanguageClassifier.
+For routing plain text, use the TextLanguageRouter component instead.
+
+### Usage example
+
+```python
+from haystack import Document, Pipeline
+from haystack.document_stores.in_memory import InMemoryDocumentStore
+from haystack.components.classifiers import DocumentLanguageClassifier
+from haystack.components.routers import MetadataRouter
+from haystack.components.writers import DocumentWriter
+
+docs = [Document(id="1", content="This is an English document"),
+        Document(id="2", content="Este es un documento en español")]
+
+document_store = InMemoryDocumentStore()
+
+p = Pipeline()
+p.add_component(instance=DocumentLanguageClassifier(languages=["en"]), name="language_classifier")
+p.add_component(
+instance=MetadataRouter(rules={
+    "en": {
+        "field": "meta.language",
+        "operator": "==",
+        "value": "en"
+    }
+}),
+name="router")
+p.add_component(instance=DocumentWriter(document_store=document_store), name="writer")
+p.connect("language_classifier.documents", "router.documents")
+p.connect("router.en", "writer.documents")
+
+p.run({"language_classifier": {"documents": docs}})
+
+written_docs = document_store.filter_documents()
+assert len(written_docs) == 1
+assert written_docs[0] == Document(id="1", content="This is an English document", meta={"language": "en"})
+```
+
+<a id="document_language_classifier.DocumentLanguageClassifier.__init__"></a>
+
+#### DocumentLanguageClassifier.\_\_init\_\_
+
+```python
+def __init__(languages: list[str] | None = None)
+```
+
+Initializes the DocumentLanguageClassifier component.
+
+**Arguments**:
+
+- `languages`: A list of ISO language codes.
+See the supported languages in [`langdetect` documentation](https://github.com/Mimino666/langdetect#languages).
+If not specified, defaults to ["en"].
+
+<a id="document_language_classifier.DocumentLanguageClassifier.run"></a>
+
+#### DocumentLanguageClassifier.run
+
+```python
+@component.output_types(documents=list[Document])
+def run(documents: list[Document])
+```
+
+Classifies the language of each document and adds it to its metadata.
+
+If the document's text doesn't match any of the languages specified at initialization,
+sets the metadata value to "unmatched".
+
+**Arguments**:
+
+- `documents`: A list of documents for language classification.
+
+**Raises**:
+
+- `TypeError`: if the input is not a list of Documents.
+
+**Returns**:
+
+A dictionary with the following key:
+- `documents`: A list of documents with an added `language` metadata field.
+
+<a id="zero_shot_document_classifier"></a>
+
+## Module zero\_shot\_document\_classifier
+
+<a id="zero_shot_document_classifier.TransformersZeroShotDocumentClassifier"></a>
+
+### TransformersZeroShotDocumentClassifier
+
+Performs zero-shot classification of documents based on given labels and adds the predicted label to their metadata.
+
+The component uses a Hugging Face pipeline for zero-shot classification.
+Provide the model and the set of labels to be used for categorization during initialization.
+Additionally, you can configure the component to allow multiple labels to be true.
+
+Classification is run on the document's content field by default. If you want it to run on another field, set the
+`classification_field` to one of the document's metadata fields.
+
+Available models for the task of zero-shot-classification include:
+    - `valhalla/distilbart-mnli-12-3`
+    - `cross-encoder/nli-distilroberta-base`
+    - `cross-encoder/nli-deberta-v3-xsmall`
+
+### Usage example
+
+The following is a pipeline that classifies documents based on predefined classification labels
+retrieved from a search pipeline:
+
+```python
+from haystack import Document
+from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
+from haystack.document_stores.in_memory import InMemoryDocumentStore
+from haystack.core.pipeline import Pipeline
+from haystack.components.classifiers import TransformersZeroShotDocumentClassifier
+
+documents = [Document(id="0", content="Today was a nice day!"),
+             Document(id="1", content="Yesterday was a bad day!")]
+
+document_store = InMemoryDocumentStore()
+retriever = InMemoryBM25Retriever(document_store=document_store)
+document_classifier = TransformersZeroShotDocumentClassifier(
+    model="cross-encoder/nli-deberta-v3-xsmall",
+    labels=["positive", "negative"],
+)
+
+document_store.write_documents(documents)
+
+pipeline = Pipeline()
+pipeline.add_component(instance=retriever, name="retriever")
+pipeline.add_component(instance=document_classifier, name="document_classifier")
+pipeline.connect("retriever", "document_classifier")
+
+queries = ["How was your day today?", "How was your day yesterday?"]
+expected_predictions = ["positive", "negative"]
+
+for idx, query in enumerate(queries):
+    result = pipeline.run({"retriever": {"query": query, "top_k": 1}})
+    assert result["document_classifier"]["documents"][0].to_dict()["id"] == str(idx)
+    assert (result["document_classifier"]["documents"][0].to_dict()["classification"]["label"]
+            == expected_predictions[idx])
+```
+
+<a id="zero_shot_document_classifier.TransformersZeroShotDocumentClassifier.__init__"></a>
+
+#### TransformersZeroShotDocumentClassifier.\_\_init\_\_
+
+```python
+def __init__(model: str,
+             labels: list[str],
+             multi_label: bool = False,
+             classification_field: str | None = None,
+             device: ComponentDevice | None = None,
+             token: Secret | None = Secret.from_env_var(
+                 ["HF_API_TOKEN", "HF_TOKEN"], strict=False),
+             huggingface_pipeline_kwargs: dict[str, Any] | None = None)
+```
+
+Initializes the TransformersZeroShotDocumentClassifier.
+
+See the Hugging Face [website](https://huggingface.co/models?pipeline_tag=zero-shot-classification&sort=downloads&search=nli)
+for the full list of zero-shot classification models (NLI) models.
+
+**Arguments**:
+
+- `model`: The name or path of a Hugging Face model for zero shot document classification.
+- `labels`: The set of possible class labels to classify each document into, for example,
+["positive", "negative"]. The labels depend on the selected model.
+- `multi_label`: Whether or not multiple candidate labels can be true.
+If `False`, the scores are normalized such that
+the sum of the label likelihoods for each sequence is 1. If `True`, the labels are considered
+independent and probabilities are normalized for each candidate by doing a softmax of the entailment
+score vs. the contradiction score.
+- `classification_field`: Name of document's meta field to be used for classification.
+If not set, `Document.content` is used by default.
+- `device`: The device on which the model is loaded. If `None`, the default device is automatically
+selected. If a device/device map is specified in `huggingface_pipeline_kwargs`, it overrides this parameter.
+- `token`: The Hugging Face token to use as HTTP bearer authorization.
+Check your HF token in your [account settings](https://huggingface.co/settings/tokens).
+- `huggingface_pipeline_kwargs`: Dictionary containing keyword arguments used to initialize the
+Hugging Face pipeline for text classification.
+
+<a id="zero_shot_document_classifier.TransformersZeroShotDocumentClassifier.warm_up"></a>
+
+#### TransformersZeroShotDocumentClassifier.warm\_up
+
+```python
+def warm_up()
+```
+
+Initializes the component.
+
+<a id="zero_shot_document_classifier.TransformersZeroShotDocumentClassifier.to_dict"></a>
+
+#### TransformersZeroShotDocumentClassifier.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+**Returns**:
+
+Dictionary with serialized data.
+
+<a id="zero_shot_document_classifier.TransformersZeroShotDocumentClassifier.from_dict"></a>
+
+#### TransformersZeroShotDocumentClassifier.from\_dict
+
+```python
+@classmethod
+def from_dict(
+        cls, data: dict[str, Any]) -> "TransformersZeroShotDocumentClassifier"
+```
+
+Deserializes the component from a dictionary.
+
+**Arguments**:
+
+- `data`: Dictionary to deserialize from.
+
+**Returns**:
+
+Deserialized component.
+
+<a id="zero_shot_document_classifier.TransformersZeroShotDocumentClassifier.run"></a>
+
+#### TransformersZeroShotDocumentClassifier.run
+
+```python
+@component.output_types(documents=list[Document])
+def run(documents: list[Document], batch_size: int = 1)
+```
+
+Classifies the documents based on the provided labels and adds them to their metadata.
+
+The classification results are stored in the `classification` dict within
+each document's metadata. If `multi_label` is set to `True`, the scores for each label are available under
+the `details` key within the `classification` dictionary.
+
+**Arguments**:
+
+- `documents`: Documents to process.
+- `batch_size`: Batch size used for processing the content in each document.
+
+**Returns**:
+
+A dictionary with the following key:
+- `documents`: A list of documents with an added metadata field called `classification`.
+

docs-website/reference_versioned_docs/version-2.22/haystack-api/connectors_api.md

@@ -0,0 +1,247 @@
+---
+title: "Connectors"
+id: connectors-api
+description: "Various connectors to integrate with external services."
+slug: "/connectors-api"
+---
+
+<a id="openapi"></a>
+
+## Module openapi
+
+<a id="openapi.OpenAPIConnector"></a>
+
+### OpenAPIConnector
+
+OpenAPIConnector enables direct invocation of REST endpoints defined in an OpenAPI specification.
+
+The OpenAPIConnector serves as a bridge between Haystack pipelines and any REST API that follows
+the OpenAPI(formerly Swagger) specification. It dynamically interprets the API specification and
+provides an interface for executing API operations. It is usually invoked by passing input
+arguments to it from a Haystack pipeline run method or by other components in a pipeline that
+pass input arguments to this component.
+
+**Example**:
+
+```python
+from haystack.utils import Secret
+from haystack.components.connectors.openapi import OpenAPIConnector
+
+connector = OpenAPIConnector(
+    openapi_spec="https://bit.ly/serperdev_openapi",
+    credentials=Secret.from_env_var("SERPERDEV_API_KEY"),
+    service_kwargs={"config_factory": my_custom_config_factory}
+)
+response = connector.run(
+    operation_id="search",
+    arguments={"q": "Who was Nikola Tesla?"}
+)
+```
+
+**Notes**:
+
+  - The `parameters` argument is required for this component.
+  - The `service_kwargs` argument is optional, it can be used to pass additional options to the OpenAPIClient.
+
+<a id="openapi.OpenAPIConnector.__init__"></a>
+
+#### OpenAPIConnector.\_\_init\_\_
+
+```python
+def __init__(openapi_spec: str,
+             credentials: Secret | None = None,
+             service_kwargs: dict[str, Any] | None = None)
+```
+
+Initialize the OpenAPIConnector with a specification and optional credentials.
+
+**Arguments**:
+
+- `openapi_spec`: URL, file path, or raw string of the OpenAPI specification
+- `credentials`: Optional API key or credentials for the service wrapped in a Secret
+- `service_kwargs`: Additional keyword arguments passed to OpenAPIClient.from_spec()
+For example, you can pass a custom config_factory or other configuration options.
+
+<a id="openapi.OpenAPIConnector.to_dict"></a>
+
+#### OpenAPIConnector.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serialize this component to a dictionary.
+
+<a id="openapi.OpenAPIConnector.from_dict"></a>
+
+#### OpenAPIConnector.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "OpenAPIConnector"
+```
+
+Deserialize this component from a dictionary.
+
+<a id="openapi.OpenAPIConnector.run"></a>
+
+#### OpenAPIConnector.run
+
+```python
+@component.output_types(response=dict[str, Any])
+def run(operation_id: str,
+        arguments: dict[str, Any] | None = None) -> dict[str, Any]
+```
+
+Invokes a REST endpoint specified in the OpenAPI specification.
+
+**Arguments**:
+
+- `operation_id`: The operationId from the OpenAPI spec to invoke
+- `arguments`: Optional parameters for the endpoint (query, path, or body parameters)
+
+**Returns**:
+
+Dictionary containing the service response
+
+<a id="openapi_service"></a>
+
+## Module openapi\_service
+
+<a id="openapi_service.OpenAPIServiceConnector"></a>
+
+### OpenAPIServiceConnector
+
+A component which connects the Haystack framework to OpenAPI services.
+
+The `OpenAPIServiceConnector` component connects the Haystack framework to OpenAPI services, enabling it to call
+operations as defined in the OpenAPI specification of the service.
+
+It integrates with `ChatMessage` dataclass, where the payload in messages is used to determine the method to be
+called and the parameters to be passed. The message payload should be an OpenAI JSON formatted function calling
+string consisting of the method name and the parameters to be passed to the method. The method name and parameters
+are then used to invoke the method on the OpenAPI service. The response from the service is returned as a
+`ChatMessage`.
+
+Before using this component, users usually resolve service endpoint parameters with a help of
+`OpenAPIServiceToFunctions` component.
+
+The example below demonstrates how to use the `OpenAPIServiceConnector` to invoke a method on a https://serper.dev/
+service specified via OpenAPI specification.
+
+Note, however, that `OpenAPIServiceConnector` is usually not meant to be used directly, but rather as part of a
+pipeline that includes the `OpenAPIServiceToFunctions` component and an `OpenAIChatGenerator` component using LLM
+with the function calling capabilities. In the example below we use the function calling payload directly, but in a
+real-world scenario, the function calling payload would usually be generated by the `OpenAIChatGenerator` component.
+
+Usage example:
+
+```python
+import json
+import requests
+
+from haystack.components.connectors import OpenAPIServiceConnector
+from haystack.dataclasses import ChatMessage
+
+
+fc_payload = [{'function': {'arguments': '{"q": "Why was Sam Altman ousted from OpenAI?"}', 'name': 'search'},
+               'id': 'call_PmEBYvZ7mGrQP5PUASA5m9wO', 'type': 'function'}]
+
+serper_token = <your_serper_dev_token>
+serperdev_openapi_spec = json.loads(requests.get("https://bit.ly/serper_dev_spec").text)
+service_connector = OpenAPIServiceConnector()
+result = service_connector.run(messages=[ChatMessage.from_assistant(json.dumps(fc_payload))],
+                               service_openapi_spec=serperdev_openapi_spec, service_credentials=serper_token)
+print(result)
+
+>> {'service_response': [ChatMessage(_role=<ChatRole.ASSISTANT: 'assistant'>, _content=[TextContent(text=
+>> '{"searchParameters": {"q": "Why was Sam Altman ousted from OpenAI?",
+>> "type": "search", "engine": "google"}, "answerBox": {"snippet": "Concerns over AI safety and OpenAI's role
+>> in protecting were at the center of Altman's brief ouster from the company."...
+```
+
+<a id="openapi_service.OpenAPIServiceConnector.__init__"></a>
+
+#### OpenAPIServiceConnector.\_\_init\_\_
+
+```python
+def __init__(ssl_verify: bool | str | None = None)
+```
+
+Initializes the OpenAPIServiceConnector instance
+
+**Arguments**:
+
+- `ssl_verify`: Decide if to use SSL verification to the requests or not,
+in case a string is passed, will be used as the CA.
+
+<a id="openapi_service.OpenAPIServiceConnector.run"></a>
+
+#### OpenAPIServiceConnector.run
+
+```python
+@component.output_types(service_response=dict[str, Any])
+def run(
+    messages: list[ChatMessage],
+    service_openapi_spec: dict[str, Any],
+    service_credentials: dict | str | None = None
+) -> dict[str, list[ChatMessage]]
+```
+
+Processes a list of chat messages to invoke a method on an OpenAPI service.
+
+It parses the last message in the list, expecting it to contain tool calls.
+
+**Arguments**:
+
+- `messages`: A list of `ChatMessage` objects containing the messages to be processed. The last message
+should contain the tool calls.
+- `service_openapi_spec`: The OpenAPI JSON specification object of the service to be invoked. All the refs
+should already be resolved.
+- `service_credentials`: The credentials to be used for authentication with the service.
+Currently, only the http and apiKey OpenAPI security schemes are supported.
+
+**Raises**:
+
+- `ValueError`: If the last message is not from the assistant or if it does not contain tool calls.
+
+**Returns**:
+
+A dictionary with the following keys:
+- `service_response`:  a list of `ChatMessage` objects, each containing the response from the service. The
+response is in JSON format, and the `content` attribute of the `ChatMessage` contains
+the JSON string.
+
+<a id="openapi_service.OpenAPIServiceConnector.to_dict"></a>
+
+#### OpenAPIServiceConnector.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+**Returns**:
+
+Dictionary with serialized data.
+
+<a id="openapi_service.OpenAPIServiceConnector.from_dict"></a>
+
+#### OpenAPIServiceConnector.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "OpenAPIServiceConnector"
+```
+
+Deserializes the component from a dictionary.
+
+**Arguments**:
+
+- `data`: The dictionary to deserialize from.
+
+**Returns**:
+
+The deserialized component.
+

docs-website/reference_versioned_docs/version-2.22/haystack-api/document_stores_api.md

@@ -0,0 +1,381 @@
+---
+title: "Document Stores"
+id: document-stores-api
+description: "Stores your texts and meta data and provides them to the Retriever at query time."
+slug: "/document-stores-api"
+---
+
+<a id="document_store"></a>
+
+## Module document\_store
+
+<a id="document_store.BM25DocumentStats"></a>
+
+### BM25DocumentStats
+
+A dataclass for managing document statistics for BM25 retrieval.
+
+**Arguments**:
+
+- `freq_token`: A Counter of token frequencies in the document.
+- `doc_len`: Number of tokens in the document.
+
+<a id="document_store.InMemoryDocumentStore"></a>
+
+### InMemoryDocumentStore
+
+Stores data in-memory. It's ephemeral and cannot be saved to disk.
+
+<a id="document_store.InMemoryDocumentStore.__init__"></a>
+
+#### InMemoryDocumentStore.\_\_init\_\_
+
+```python
+def __init__(bm25_tokenization_regex: str = r"(?u)\b\w\w+\b",
+             bm25_algorithm: Literal["BM25Okapi", "BM25L",
+                                     "BM25Plus"] = "BM25L",
+             bm25_parameters: dict | None = None,
+             embedding_similarity_function: Literal["dot_product",
+                                                    "cosine"] = "dot_product",
+             index: str | None = None,
+             async_executor: ThreadPoolExecutor | None = None,
+             return_embedding: bool = True)
+```
+
+Initializes the DocumentStore.
+
+**Arguments**:
+
+- `bm25_tokenization_regex`: The regular expression used to tokenize the text for BM25 retrieval.
+- `bm25_algorithm`: The BM25 algorithm to use. One of "BM25Okapi", "BM25L", or "BM25Plus".
+- `bm25_parameters`: Parameters for BM25 implementation in a dictionary format.
+For example: `{'k1':1.5, 'b':0.75, 'epsilon':0.25}`
+You can learn more about these parameters by visiting https://github.com/dorianbrown/rank_bm25.
+- `embedding_similarity_function`: The similarity function used to compare Documents embeddings.
+One of "dot_product" (default) or "cosine". To choose the most appropriate function, look for information
+about your embedding model.
+- `index`: A specific index to store the documents. If not specified, a random UUID is used.
+Using the same index allows you to store documents across multiple InMemoryDocumentStore instances.
+- `async_executor`: Optional ThreadPoolExecutor to use for async calls. If not provided, a single-threaded
+executor will be initialized and used.
+- `return_embedding`: Whether to return the embedding of the retrieved Documents. Default is True.
+
+<a id="document_store.InMemoryDocumentStore.__del__"></a>
+
+#### InMemoryDocumentStore.\_\_del\_\_
+
+```python
+def __del__()
+```
+
+Cleanup when the instance is being destroyed.
+
+<a id="document_store.InMemoryDocumentStore.shutdown"></a>
+
+#### InMemoryDocumentStore.shutdown
+
+```python
+def shutdown()
+```
+
+Explicitly shutdown the executor if we own it.
+
+<a id="document_store.InMemoryDocumentStore.storage"></a>
+
+#### InMemoryDocumentStore.storage
+
+```python
+@property
+def storage() -> dict[str, Document]
+```
+
+Utility property that returns the storage used by this instance of InMemoryDocumentStore.
+
+<a id="document_store.InMemoryDocumentStore.to_dict"></a>
+
+#### InMemoryDocumentStore.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+**Returns**:
+
+Dictionary with serialized data.
+
+<a id="document_store.InMemoryDocumentStore.from_dict"></a>
+
+#### InMemoryDocumentStore.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "InMemoryDocumentStore"
+```
+
+Deserializes the component from a dictionary.
+
+**Arguments**:
+
+- `data`: The dictionary to deserialize from.
+
+**Returns**:
+
+The deserialized component.
+
+<a id="document_store.InMemoryDocumentStore.save_to_disk"></a>
+
+#### InMemoryDocumentStore.save\_to\_disk
+
+```python
+def save_to_disk(path: str) -> None
+```
+
+Write the database and its' data to disk as a JSON file.
+
+**Arguments**:
+
+- `path`: The path to the JSON file.
+
+<a id="document_store.InMemoryDocumentStore.load_from_disk"></a>
+
+#### InMemoryDocumentStore.load\_from\_disk
+
+```python
+@classmethod
+def load_from_disk(cls, path: str) -> "InMemoryDocumentStore"
+```
+
+Load the database and its' data from disk as a JSON file.
+
+**Arguments**:
+
+- `path`: The path to the JSON file.
+
+**Returns**:
+
+The loaded InMemoryDocumentStore.
+
+<a id="document_store.InMemoryDocumentStore.count_documents"></a>
+
+#### InMemoryDocumentStore.count\_documents
+
+```python
+def count_documents() -> int
+```
+
+Returns the number of how many documents are present in the DocumentStore.
+
+<a id="document_store.InMemoryDocumentStore.filter_documents"></a>
+
+#### InMemoryDocumentStore.filter\_documents
+
+```python
+def filter_documents(filters: dict[str, Any] | None = None) -> list[Document]
+```
+
+Returns the documents that match the filters provided.
+
+For a detailed specification of the filters, refer to the DocumentStore.filter_documents() protocol
+documentation.
+
+**Arguments**:
+
+- `filters`: The filters to apply to the document list.
+
+**Returns**:
+
+A list of Documents that match the given filters.
+
+<a id="document_store.InMemoryDocumentStore.write_documents"></a>
+
+#### InMemoryDocumentStore.write\_documents
+
+```python
+def write_documents(documents: list[Document],
+                    policy: DuplicatePolicy = DuplicatePolicy.NONE) -> int
+```
+
+Refer to the DocumentStore.write_documents() protocol documentation.
+
+If `policy` is set to `DuplicatePolicy.NONE` defaults to `DuplicatePolicy.FAIL`.
+
+<a id="document_store.InMemoryDocumentStore.delete_documents"></a>
+
+#### InMemoryDocumentStore.delete\_documents
+
+```python
+def delete_documents(document_ids: list[str]) -> None
+```
+
+Deletes all documents with matching document_ids from the DocumentStore.
+
+**Arguments**:
+
+- `document_ids`: The object_ids to delete.
+
+<a id="document_store.InMemoryDocumentStore.bm25_retrieval"></a>
+
+#### InMemoryDocumentStore.bm25\_retrieval
+
+```python
+def bm25_retrieval(query: str,
+                   filters: dict[str, Any] | None = None,
+                   top_k: int = 10,
+                   scale_score: bool = False) -> list[Document]
+```
+
+Retrieves documents that are most relevant to the query using BM25 algorithm.
+
+**Arguments**:
+
+- `query`: The query string.
+- `filters`: A dictionary with filters to narrow down the search space.
+- `top_k`: The number of top documents to retrieve. Default is 10.
+- `scale_score`: Whether to scale the scores of the retrieved documents. Default is False.
+
+**Returns**:
+
+A list of the top_k documents most relevant to the query.
+
+<a id="document_store.InMemoryDocumentStore.embedding_retrieval"></a>
+
+#### InMemoryDocumentStore.embedding\_retrieval
+
+```python
+def embedding_retrieval(
+        query_embedding: list[float],
+        filters: dict[str, Any] | None = None,
+        top_k: int = 10,
+        scale_score: bool = False,
+        return_embedding: bool | None = False) -> list[Document]
+```
+
+Retrieves documents that are most similar to the query embedding using a vector similarity metric.
+
+**Arguments**:
+
+- `query_embedding`: Embedding of the query.
+- `filters`: A dictionary with filters to narrow down the search space.
+- `top_k`: The number of top documents to retrieve. Default is 10.
+- `scale_score`: Whether to scale the scores of the retrieved Documents. Default is False.
+- `return_embedding`: Whether to return the embedding of the retrieved Documents.
+If not provided, the value of the `return_embedding` parameter set at component
+initialization will be used. Default is False.
+
+**Returns**:
+
+A list of the top_k documents most relevant to the query.
+
+<a id="document_store.InMemoryDocumentStore.count_documents_async"></a>
+
+#### InMemoryDocumentStore.count\_documents\_async
+
+```python
+async def count_documents_async() -> int
+```
+
+Returns the number of how many documents are present in the DocumentStore.
+
+<a id="document_store.InMemoryDocumentStore.filter_documents_async"></a>
+
+#### InMemoryDocumentStore.filter\_documents\_async
+
+```python
+async def filter_documents_async(
+        filters: dict[str, Any] | None = None) -> list[Document]
+```
+
+Returns the documents that match the filters provided.
+
+For a detailed specification of the filters, refer to the DocumentStore.filter_documents() protocol
+documentation.
+
+**Arguments**:
+
+- `filters`: The filters to apply to the document list.
+
+**Returns**:
+
+A list of Documents that match the given filters.
+
+<a id="document_store.InMemoryDocumentStore.write_documents_async"></a>
+
+#### InMemoryDocumentStore.write\_documents\_async
+
+```python
+async def write_documents_async(
+        documents: list[Document],
+        policy: DuplicatePolicy = DuplicatePolicy.NONE) -> int
+```
+
+Refer to the DocumentStore.write_documents() protocol documentation.
+
+If `policy` is set to `DuplicatePolicy.NONE` defaults to `DuplicatePolicy.FAIL`.
+
+<a id="document_store.InMemoryDocumentStore.delete_documents_async"></a>
+
+#### InMemoryDocumentStore.delete\_documents\_async
+
+```python
+async def delete_documents_async(document_ids: list[str]) -> None
+```
+
+Deletes all documents with matching document_ids from the DocumentStore.
+
+**Arguments**:
+
+- `document_ids`: The object_ids to delete.
+
+<a id="document_store.InMemoryDocumentStore.bm25_retrieval_async"></a>
+
+#### InMemoryDocumentStore.bm25\_retrieval\_async
+
+```python
+async def bm25_retrieval_async(query: str,
+                               filters: dict[str, Any] | None = None,
+                               top_k: int = 10,
+                               scale_score: bool = False) -> list[Document]
+```
+
+Retrieves documents that are most relevant to the query using BM25 algorithm.
+
+**Arguments**:
+
+- `query`: The query string.
+- `filters`: A dictionary with filters to narrow down the search space.
+- `top_k`: The number of top documents to retrieve. Default is 10.
+- `scale_score`: Whether to scale the scores of the retrieved documents. Default is False.
+
+**Returns**:
+
+A list of the top_k documents most relevant to the query.
+
+<a id="document_store.InMemoryDocumentStore.embedding_retrieval_async"></a>
+
+#### InMemoryDocumentStore.embedding\_retrieval\_async
+
+```python
+async def embedding_retrieval_async(
+        query_embedding: list[float],
+        filters: dict[str, Any] | None = None,
+        top_k: int = 10,
+        scale_score: bool = False,
+        return_embedding: bool = False) -> list[Document]
+```
+
+Retrieves documents that are most similar to the query embedding using a vector similarity metric.
+
+**Arguments**:
+
+- `query_embedding`: Embedding of the query.
+- `filters`: A dictionary with filters to narrow down the search space.
+- `top_k`: The number of top documents to retrieve. Default is 10.
+- `scale_score`: Whether to scale the scores of the retrieved Documents. Default is False.
+- `return_embedding`: Whether to return the embedding of the retrieved Documents. Default is False.
+
+**Returns**:
+
+A list of the top_k documents most relevant to the query.
+

docs-website/reference_versioned_docs/version-2.22/haystack-api/document_writers_api.md

@@ -0,0 +1,140 @@
+---
+title: "Document Writers"
+id: document-writers-api
+description: "Writes Documents to a DocumentStore."
+slug: "/document-writers-api"
+---
+
+<a id="document_writer"></a>
+
+## Module document\_writer
+
+<a id="document_writer.DocumentWriter"></a>
+
+### DocumentWriter
+
+Writes documents to a DocumentStore.
+
+### Usage example
+```python
+from haystack import Document
+from haystack.components.writers import DocumentWriter
+from haystack.document_stores.in_memory import InMemoryDocumentStore
+docs = [
+    Document(content="Python is a popular programming language"),
+]
+doc_store = InMemoryDocumentStore()
+writer = DocumentWriter(document_store=doc_store)
+writer.run(docs)
+```
+
+<a id="document_writer.DocumentWriter.__init__"></a>
+
+#### DocumentWriter.\_\_init\_\_
+
+```python
+def __init__(document_store: DocumentStore,
+             policy: DuplicatePolicy = DuplicatePolicy.NONE)
+```
+
+Create a DocumentWriter component.
+
+**Arguments**:
+
+- `document_store`: The instance of the document store where you want to store your documents.
+- `policy`: The policy to apply when a Document with the same ID already exists in the DocumentStore.
+- `DuplicatePolicy.NONE`: Default policy, relies on the DocumentStore settings.
+- `DuplicatePolicy.SKIP`: Skips documents with the same ID and doesn't write them to the DocumentStore.
+- `DuplicatePolicy.OVERWRITE`: Overwrites documents with the same ID.
+- `DuplicatePolicy.FAIL`: Raises an error if a Document with the same ID is already in the DocumentStore.
+
+<a id="document_writer.DocumentWriter.to_dict"></a>
+
+#### DocumentWriter.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+**Returns**:
+
+Dictionary with serialized data.
+
+<a id="document_writer.DocumentWriter.from_dict"></a>
+
+#### DocumentWriter.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "DocumentWriter"
+```
+
+Deserializes the component from a dictionary.
+
+**Arguments**:
+
+- `data`: The dictionary to deserialize from.
+
+**Raises**:
+
+- `DeserializationError`: If the document store is not properly specified in the serialization data or its type cannot be imported.
+
+**Returns**:
+
+The deserialized component.
+
+<a id="document_writer.DocumentWriter.run"></a>
+
+#### DocumentWriter.run
+
+```python
+@component.output_types(documents_written=int)
+def run(documents: list[Document], policy: DuplicatePolicy | None = None)
+```
+
+Run the DocumentWriter on the given input data.
+
+**Arguments**:
+
+- `documents`: A list of documents to write to the document store.
+- `policy`: The policy to use when encountering duplicate documents.
+
+**Raises**:
+
+- `ValueError`: If the specified document store is not found.
+
+**Returns**:
+
+Number of documents written to the document store.
+
+<a id="document_writer.DocumentWriter.run_async"></a>
+
+#### DocumentWriter.run\_async
+
+```python
+@component.output_types(documents_written=int)
+async def run_async(documents: list[Document],
+                    policy: DuplicatePolicy | None = None)
+```
+
+Asynchronously run the DocumentWriter on the given input data.
+
+This is the asynchronous version of the `run` method. It has the same parameters and return values
+but can be used with `await` in async code.
+
+**Arguments**:
+
+- `documents`: A list of documents to write to the document store.
+- `policy`: The policy to use when encountering duplicate documents.
+
+**Raises**:
+
+- `ValueError`: If the specified document store is not found.
+- `TypeError`: If the specified document store does not implement `write_documents_async`.
+
+**Returns**:
+
+Number of documents written to the document store.
+

docs-website/reference_versioned_docs/version-2.22/haystack-api/evaluation_api.md

@@ -0,0 +1,110 @@
+---
+title: "Evaluation"
+id: evaluation-api
+description: "Represents the results of evaluation."
+slug: "/evaluation-api"
+---
+
+<a id="eval_run_result"></a>
+
+## Module eval\_run\_result
+
+<a id="eval_run_result.EvaluationRunResult"></a>
+
+### EvaluationRunResult
+
+Contains the inputs and the outputs of an evaluation pipeline and provides methods to inspect them.
+
+<a id="eval_run_result.EvaluationRunResult.__init__"></a>
+
+#### EvaluationRunResult.\_\_init\_\_
+
+```python
+def __init__(run_name: str, inputs: dict[str, list[Any]],
+             results: dict[str, dict[str, Any]])
+```
+
+Initialize a new evaluation run result.
+
+**Arguments**:
+
+- `run_name`: Name of the evaluation run.
+- `inputs`: Dictionary containing the inputs used for the run. Each key is the name of the input and its value is a list
+of input values. The length of the lists should be the same.
+- `results`: Dictionary containing the results of the evaluators used in the evaluation pipeline. Each key is the name
+of the metric and its value is dictionary with the following keys:
+- 'score': The aggregated score for the metric.
+- 'individual_scores': A list of scores for each input sample.
+
+<a id="eval_run_result.EvaluationRunResult.aggregated_report"></a>
+
+#### EvaluationRunResult.aggregated\_report
+
+```python
+def aggregated_report(
+    output_format: Literal["json", "csv", "df"] = "json",
+    csv_file: str | None = None
+) -> Union[dict[str, list[Any]], "DataFrame", str]
+```
+
+Generates a report with aggregated scores for each metric.
+
+**Arguments**:
+
+- `output_format`: The output format for the report, "json", "csv", or "df", default to "json".
+- `csv_file`: Filepath to save CSV output if `output_format` is "csv", must be provided.
+
+**Returns**:
+
+JSON or DataFrame with aggregated scores, in case the output is set to a CSV file, a message confirming the
+successful write or an error message.
+
+<a id="eval_run_result.EvaluationRunResult.detailed_report"></a>
+
+#### EvaluationRunResult.detailed\_report
+
+```python
+def detailed_report(
+    output_format: Literal["json", "csv", "df"] = "json",
+    csv_file: str | None = None
+) -> Union[dict[str, list[Any]], "DataFrame", str]
+```
+
+Generates a report with detailed scores for each metric.
+
+**Arguments**:
+
+- `output_format`: The output format for the report, "json", "csv", or "df", default to "json".
+- `csv_file`: Filepath to save CSV output if `output_format` is "csv", must be provided.
+
+**Returns**:
+
+JSON or DataFrame with the detailed scores, in case the output is set to a CSV file, a message confirming
+the successful write or an error message.
+
+<a id="eval_run_result.EvaluationRunResult.comparative_detailed_report"></a>
+
+#### EvaluationRunResult.comparative\_detailed\_report
+
+```python
+def comparative_detailed_report(
+        other: "EvaluationRunResult",
+        keep_columns: list[str] | None = None,
+        output_format: Literal["json", "csv", "df"] = "json",
+        csv_file: str | None = None) -> Union[str, "DataFrame", None]
+```
+
+Generates a report with detailed scores for each metric from two evaluation runs for comparison.
+
+**Arguments**:
+
+- `other`: Results of another evaluation run to compare with.
+- `keep_columns`: List of common column names to keep from the inputs of the evaluation runs to compare.
+- `output_format`: The output format for the report, "json", "csv", or "df", default to "json".
+- `csv_file`: Filepath to save CSV output if `output_format` is "csv", must be provided.
+
+**Returns**:
+
+JSON or DataFrame with a comparison of the detailed scores, in case the output is set to a CSV file,
+a message confirming the successful write or an error message.
+

docs-website/reference_versioned_docs/version-2.22/haystack-api/extractors_api.md

@@ -0,0 +1,647 @@
+---
+title: "Extractors"
+id: extractors-api
+description: "Components to extract specific elements from textual data."
+slug: "/extractors-api"
+---
+
+<a id="image/llm_document_content_extractor"></a>
+
+## Module image/llm\_document\_content\_extractor
+
+<a id="image/llm_document_content_extractor.LLMDocumentContentExtractor"></a>
+
+### LLMDocumentContentExtractor
+
+Extracts textual content from image-based documents using a vision-enabled LLM (Large Language Model).
+
+This component converts each input document into an image using the DocumentToImageContent component,
+uses a prompt to instruct the LLM on how to extract content, and uses a ChatGenerator to extract structured
+textual content based on the provided prompt.
+
+The prompt must not contain variables; it should only include instructions for the LLM. Image data and the prompt
+are passed together to the LLM as a chat message.
+
+Documents for which the LLM fails to extract content are returned in a separate `failed_documents` list. These
+failed documents will have a `content_extraction_error` entry in their metadata. This metadata can be used for
+debugging or for reprocessing the documents later.
+
+### Usage example
+```python
+from haystack import Document
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.components.extractors.image import LLMDocumentContentExtractor
+chat_generator = OpenAIChatGenerator()
+extractor = LLMDocumentContentExtractor(chat_generator=chat_generator)
+documents = [
+    Document(content="", meta={"file_path": "image.jpg"}),
+    Document(content="", meta={"file_path": "document.pdf", "page_number": 1}),
+]
+updated_documents = extractor.run(documents=documents)["documents"]
+print(updated_documents)
+# [Document(content='Extracted text from image.jpg',
+#           meta={'file_path': 'image.jpg'}),
+#  ...]
+```
+
+<a id="image/llm_document_content_extractor.LLMDocumentContentExtractor.__init__"></a>
+
+#### LLMDocumentContentExtractor.\_\_init\_\_
+
+```python
+def __init__(*,
+             chat_generator: ChatGenerator,
+             prompt: str = DEFAULT_PROMPT_TEMPLATE,
+             file_path_meta_field: str = "file_path",
+             root_path: str | None = None,
+             detail: Literal["auto", "high", "low"] | None = None,
+             size: tuple[int, int] | None = None,
+             raise_on_failure: bool = False,
+             max_workers: int = 3)
+```
+
+Initialize the LLMDocumentContentExtractor component.
+
+**Arguments**:
+
+- `chat_generator`: A ChatGenerator instance representing the LLM used to extract text. This generator must
+support vision-based input and return a plain text response.
+- `prompt`: Instructional text provided to the LLM. It must not contain Jinja variables.
+The prompt should only contain instructions on how to extract the content of the image-based document.
+- `file_path_meta_field`: The metadata field in the Document that contains the file path to the image or PDF.
+- `root_path`: The root directory path where document files are located. If provided, file paths in
+document metadata will be resolved relative to this path. If None, file paths are treated as absolute paths.
+- `detail`: Optional detail level of the image (only supported by OpenAI). Can be "auto", "high", or "low".
+This will be passed to chat_generator when processing the images.
+- `size`: If provided, resizes the image to fit within the specified dimensions (width, height) while
+maintaining aspect ratio. This reduces file size, memory usage, and processing time, which is beneficial
+when working with models that have resolution constraints or when transmitting images to remote services.
+- `raise_on_failure`: If True, exceptions from the LLM are raised. If False, failed documents are logged
+and returned.
+- `max_workers`: Maximum number of threads used to parallelize LLM calls across documents using a
+ThreadPoolExecutor.
+
+<a id="image/llm_document_content_extractor.LLMDocumentContentExtractor.warm_up"></a>
+
+#### LLMDocumentContentExtractor.warm\_up
+
+```python
+def warm_up()
+```
+
+Warm up the ChatGenerator if it has a warm_up method.
+
+<a id="image/llm_document_content_extractor.LLMDocumentContentExtractor.to_dict"></a>
+
+#### LLMDocumentContentExtractor.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+**Returns**:
+
+Dictionary with serialized data.
+
+<a id="image/llm_document_content_extractor.LLMDocumentContentExtractor.from_dict"></a>
+
+#### LLMDocumentContentExtractor.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "LLMDocumentContentExtractor"
+```
+
+Deserializes the component from a dictionary.
+
+**Arguments**:
+
+- `data`: Dictionary with serialized data.
+
+**Returns**:
+
+An instance of the component.
+
+<a id="image/llm_document_content_extractor.LLMDocumentContentExtractor.run"></a>
+
+#### LLMDocumentContentExtractor.run
+
+```python
+@component.output_types(documents=list[Document],
+                        failed_documents=list[Document])
+def run(documents: list[Document]) -> dict[str, list[Document]]
+```
+
+Run content extraction on a list of image-based documents using a vision-capable LLM.
+
+Each document is passed to the LLM along with a predefined prompt. The response is used to update the document's
+content. If the extraction fails, the document is returned in the `failed_documents` list with metadata
+describing the failure.
+
+**Arguments**:
+
+- `documents`: A list of image-based documents to process. Each must have a valid file path in its metadata.
+
+**Returns**:
+
+A dictionary with:
+- "documents": Successfully processed documents, updated with extracted content.
+- "failed_documents": Documents that failed processing, annotated with failure metadata.
+
+<a id="llm_metadata_extractor"></a>
+
+## Module llm\_metadata\_extractor
+
+<a id="llm_metadata_extractor.LLMMetadataExtractor"></a>
+
+### LLMMetadataExtractor
+
+Extracts metadata from documents using a Large Language Model (LLM).
+
+The metadata is extracted by providing a prompt to an LLM that generates the metadata.
+
+This component expects as input a list of documents and a prompt. The prompt should have a variable called
+`document` that will point to a single document in the list of documents. So to access the content of the document,
+you can use `{{ document.content }}` in the prompt.
+
+The component will run the LLM on each document in the list and extract metadata from the document. The metadata
+will be added to the document's metadata field. If the LLM fails to extract metadata from a document, the document
+will be added to the `failed_documents` list. The failed documents will have the keys `metadata_extraction_error` and
+`metadata_extraction_response` in their metadata. These documents can be re-run with another extractor to
+extract metadata by using the `metadata_extraction_response` and `metadata_extraction_error` in the prompt.
+
+```python
+from haystack import Document
+from haystack.components.extractors.llm_metadata_extractor import LLMMetadataExtractor
+from haystack.components.generators.chat import OpenAIChatGenerator
+
+NER_PROMPT = '''
+-Goal-
+Given text and a list of entity types, identify all entities of those types from the text.
+
+-Steps-
+1. Identify all entities. For each identified entity, extract the following information:
+- entity: Name of the entity
+- entity_type: One of the following types: [organization, product, service, industry]
+Format each entity as a JSON like: {"entity": <entity_name>, "entity_type": <entity_type>}
+
+2. Return output in a single list with all the entities identified in steps 1.
+
+-Examples-
+######################
+Example 1:
+entity_types: [organization, person, partnership, financial metric, product, service, industry, investment strategy, market trend]
+text: Another area of strength is our co-brand issuance. Visa is the primary network partner for eight of the top
+10 co-brand partnerships in the US today and we are pleased that Visa has finalized a multi-year extension of
+our successful credit co-branded partnership with Alaska Airlines, a portfolio that benefits from a loyal customer
+base and high cross-border usage.
+We have also had significant co-brand momentum in CEMEA. First, we launched a new co-brand card in partnership
+with Qatar Airways, British Airways and the National Bank of Kuwait. Second, we expanded our strong global
+Marriott relationship to launch Qatar's first hospitality co-branded card with Qatar Islamic Bank. Across the
+United Arab Emirates, we now have exclusive agreements with all the leading airlines marked by a recent
+agreement with Emirates Skywards.
+And we also signed an inaugural Airline co-brand agreement in Morocco with Royal Air Maroc. Now newer digital
+issuers are equally
+------------------------
+output:
+{"entities": [{"entity": "Visa", "entity_type": "company"}, {"entity": "Alaska Airlines", "entity_type": "company"}, {"entity": "Qatar Airways", "entity_type": "company"}, {"entity": "British Airways", "entity_type": "company"}, {"entity": "National Bank of Kuwait", "entity_type": "company"}, {"entity": "Marriott", "entity_type": "company"}, {"entity": "Qatar Islamic Bank", "entity_type": "company"}, {"entity": "Emirates Skywards", "entity_type": "company"}, {"entity": "Royal Air Maroc", "entity_type": "company"}]}
+#############################
+-Real Data-
+######################
+entity_types: [company, organization, person, country, product, service]
+text: {{ document.content }}
+######################
+output:
+'''
+
+docs = [
+    Document(content="deepset was founded in 2018 in Berlin, and is known for its Haystack framework"),
+    Document(content="Hugging Face is a company that was founded in New York, USA and is known for its Transformers library")
+]
+
+chat_generator = OpenAIChatGenerator(
+    generation_kwargs={
+        "max_completion_tokens": 500,
+        "temperature": 0.0,
+        "seed": 0,
+        "response_format": {"type": "json_object"},
+    },
+    max_retries=1,
+    timeout=60.0,
+)
+
+extractor = LLMMetadataExtractor(
+    prompt=NER_PROMPT,
+    chat_generator=generator,
+    expected_keys=["entities"],
+    raise_on_failure=False,
+)
+
+extractor.warm_up()
+extractor.run(documents=docs)
+>> {'documents': [
+    Document(id=.., content: 'deepset was founded in 2018 in Berlin, and is known for its Haystack framework',
+    meta: {'entities': [{'entity': 'deepset', 'entity_type': 'company'}, {'entity': 'Berlin', 'entity_type': 'city'},
+          {'entity': 'Haystack', 'entity_type': 'product'}]}),
+    Document(id=.., content: 'Hugging Face is a company that was founded in New York, USA and is known for its Transformers library',
+    meta: {'entities': [
+            {'entity': 'Hugging Face', 'entity_type': 'company'}, {'entity': 'New York', 'entity_type': 'city'},
+            {'entity': 'USA', 'entity_type': 'country'}, {'entity': 'Transformers', 'entity_type': 'product'}
+            ]})
+       ]
+    'failed_documents': []
+   }
+>>
+```
+
+<a id="llm_metadata_extractor.LLMMetadataExtractor.__init__"></a>
+
+#### LLMMetadataExtractor.\_\_init\_\_
+
+```python
+def __init__(prompt: str,
+             chat_generator: ChatGenerator,
+             expected_keys: list[str] | None = None,
+             page_range: list[str | int] | None = None,
+             raise_on_failure: bool = False,
+             max_workers: int = 3)
+```
+
+Initializes the LLMMetadataExtractor.
+
+**Arguments**:
+
+- `prompt`: The prompt to be used for the LLM.
+- `chat_generator`: a ChatGenerator instance which represents the LLM. In order for the component to work,
+the LLM should be configured to return a JSON object. For example, when using the OpenAIChatGenerator, you
+should pass `{"response_format": {"type": "json_object"}}` in the `generation_kwargs`.
+- `expected_keys`: The keys expected in the JSON output from the LLM.
+- `page_range`: A range of pages to extract metadata from. For example, page_range=['1', '3'] will extract
+metadata from the first and third pages of each document. It also accepts printable range strings, e.g.:
+['1-3', '5', '8', '10-12'] will extract metadata from pages 1, 2, 3, 5, 8, 10,11, 12.
+If None, metadata will be extracted from the entire document for each document in the documents list.
+This parameter is optional and can be overridden in the `run` method.
+- `raise_on_failure`: Whether to raise an error on failure during the execution of the Generator or
+validation of the JSON output.
+- `max_workers`: The maximum number of workers to use in the thread pool executor.
+
+<a id="llm_metadata_extractor.LLMMetadataExtractor.warm_up"></a>
+
+#### LLMMetadataExtractor.warm\_up
+
+```python
+def warm_up()
+```
+
+Warm up the LLM provider component.
+
+<a id="llm_metadata_extractor.LLMMetadataExtractor.to_dict"></a>
+
+#### LLMMetadataExtractor.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+**Returns**:
+
+Dictionary with serialized data.
+
+<a id="llm_metadata_extractor.LLMMetadataExtractor.from_dict"></a>
+
+#### LLMMetadataExtractor.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "LLMMetadataExtractor"
+```
+
+Deserializes the component from a dictionary.
+
+**Arguments**:
+
+- `data`: Dictionary with serialized data.
+
+**Returns**:
+
+An instance of the component.
+
+<a id="llm_metadata_extractor.LLMMetadataExtractor.run"></a>
+
+#### LLMMetadataExtractor.run
+
+```python
+@component.output_types(documents=list[Document],
+                        failed_documents=list[Document])
+def run(documents: list[Document], page_range: list[str | int] | None = None)
+```
+
+Extract metadata from documents using a Large Language Model.
+
+If `page_range` is provided, the metadata will be extracted from the specified range of pages. This component
+will split the documents into pages and extract metadata from the specified range of pages. The metadata will be
+extracted from the entire document if `page_range` is not provided.
+
+The original documents will be returned  updated with the extracted metadata.
+
+**Arguments**:
+
+- `documents`: List of documents to extract metadata from.
+- `page_range`: A range of pages to extract metadata from. For example, page_range=['1', '3'] will extract
+metadata from the first and third pages of each document. It also accepts printable range
+strings, e.g.: ['1-3', '5', '8', '10-12'] will extract metadata from pages 1, 2, 3, 5, 8, 10,
+11, 12.
+If None, metadata will be extracted from the entire document for each document in the
+documents list.
+
+**Returns**:
+
+A dictionary with the keys:
+- "documents": A list of documents that were successfully updated with the extracted metadata.
+- "failed_documents": A list of documents that failed to extract metadata. These documents will have
+"metadata_extraction_error" and "metadata_extraction_response" in their metadata. These documents can be
+re-run with the extractor to extract metadata.
+
+<a id="named_entity_extractor"></a>
+
+## Module named\_entity\_extractor
+
+<a id="named_entity_extractor.NamedEntityExtractorBackend"></a>
+
+### NamedEntityExtractorBackend
+
+NLP backend to use for Named Entity Recognition.
+
+<a id="named_entity_extractor.NamedEntityExtractorBackend.HUGGING_FACE"></a>
+
+#### HUGGING\_FACE
+
+Uses an Hugging Face model and pipeline.
+
+<a id="named_entity_extractor.NamedEntityExtractorBackend.SPACY"></a>
+
+#### SPACY
+
+Uses a spaCy model and pipeline.
+
+<a id="named_entity_extractor.NamedEntityExtractorBackend.from_str"></a>
+
+#### NamedEntityExtractorBackend.from\_str
+
+```python
+@staticmethod
+def from_str(string: str) -> "NamedEntityExtractorBackend"
+```
+
+Convert a string to a NamedEntityExtractorBackend enum.
+
+<a id="named_entity_extractor.NamedEntityAnnotation"></a>
+
+### NamedEntityAnnotation
+
+Describes a single NER annotation.
+
+**Arguments**:
+
+- `entity`: Entity label.
+- `start`: Start index of the entity in the document.
+- `end`: End index of the entity in the document.
+- `score`: Score calculated by the model.
+
+<a id="named_entity_extractor.NamedEntityExtractor"></a>
+
+### NamedEntityExtractor
+
+Annotates named entities in a collection of documents.
+
+The component supports two backends: Hugging Face and spaCy. The
+former can be used with any sequence classification model from the
+[Hugging Face model hub](https://huggingface.co/models), while the
+latter can be used with any [spaCy model](https://spacy.io/models)
+that contains an NER component. Annotations are stored as metadata
+in the documents.
+
+Usage example:
+```python
+from haystack import Document
+from haystack.components.extractors.named_entity_extractor import NamedEntityExtractor
+
+documents = [
+    Document(content="I'm Merlin, the happy pig!"),
+    Document(content="My name is Clara and I live in Berkeley, California."),
+]
+extractor = NamedEntityExtractor(backend="hugging_face", model="dslim/bert-base-NER")
+extractor.warm_up()
+results = extractor.run(documents=documents)["documents"]
+annotations = [NamedEntityExtractor.get_stored_annotations(doc) for doc in results]
+print(annotations)
+```
+
+<a id="named_entity_extractor.NamedEntityExtractor.__init__"></a>
+
+#### NamedEntityExtractor.\_\_init\_\_
+
+```python
+def __init__(
+    *,
+    backend: str | NamedEntityExtractorBackend,
+    model: str,
+    pipeline_kwargs: dict[str, Any] | None = None,
+    device: ComponentDevice | None = None,
+    token: Secret | None = Secret.from_env_var(["HF_API_TOKEN", "HF_TOKEN"],
+                                               strict=False)
+) -> None
+```
+
+Create a Named Entity extractor component.
+
+**Arguments**:
+
+- `backend`: Backend to use for NER.
+- `model`: Name of the model or a path to the model on
+the local disk. Dependent on the backend.
+- `pipeline_kwargs`: Keyword arguments passed to the pipeline. The
+pipeline can override these arguments. Dependent on the backend.
+- `device`: The device on which the model is loaded. If `None`,
+the default device is automatically selected. If a
+device/device map is specified in `pipeline_kwargs`,
+it overrides this parameter (only applicable to the
+HuggingFace backend).
+- `token`: The API token to download private models from Hugging Face.
+
+<a id="named_entity_extractor.NamedEntityExtractor.warm_up"></a>
+
+#### NamedEntityExtractor.warm\_up
+
+```python
+def warm_up()
+```
+
+Initialize the component.
+
+**Raises**:
+
+- `ComponentError`: If the backend fails to initialize successfully.
+
+<a id="named_entity_extractor.NamedEntityExtractor.run"></a>
+
+#### NamedEntityExtractor.run
+
+```python
+@component.output_types(documents=list[Document])
+def run(documents: list[Document], batch_size: int = 1) -> dict[str, Any]
+```
+
+Annotate named entities in each document and store the annotations in the document's metadata.
+
+**Arguments**:
+
+- `documents`: Documents to process.
+- `batch_size`: Batch size used for processing the documents.
+
+**Raises**:
+
+- `ComponentError`: If the backend fails to process a document.
+
+**Returns**:
+
+Processed documents.
+
+<a id="named_entity_extractor.NamedEntityExtractor.to_dict"></a>
+
+#### NamedEntityExtractor.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+**Returns**:
+
+Dictionary with serialized data.
+
+<a id="named_entity_extractor.NamedEntityExtractor.from_dict"></a>
+
+#### NamedEntityExtractor.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "NamedEntityExtractor"
+```
+
+Deserializes the component from a dictionary.
+
+**Arguments**:
+
+- `data`: Dictionary to deserialize from.
+
+**Returns**:
+
+Deserialized component.
+
+<a id="named_entity_extractor.NamedEntityExtractor.initialized"></a>
+
+#### NamedEntityExtractor.initialized
+
+```python
+@property
+def initialized() -> bool
+```
+
+Returns if the extractor is ready to annotate text.
+
+<a id="named_entity_extractor.NamedEntityExtractor.get_stored_annotations"></a>
+
+#### NamedEntityExtractor.get\_stored\_annotations
+
+```python
+@classmethod
+def get_stored_annotations(
+        cls, document: Document) -> list[NamedEntityAnnotation] | None
+```
+
+Returns the document's named entity annotations stored in its metadata, if any.
+
+**Arguments**:
+
+- `document`: Document whose annotations are to be fetched.
+
+**Returns**:
+
+The stored annotations.
+
+<a id="regex_text_extractor"></a>
+
+## Module regex\_text\_extractor
+
+<a id="regex_text_extractor.RegexTextExtractor"></a>
+
+### RegexTextExtractor
+
+Extracts text from chat message or string input using a regex pattern.
+
+RegexTextExtractor parses input text or ChatMessages using a provided regular expression pattern.
+It can be configured to search through all messages or only the last message in a list of ChatMessages.
+
+### Usage example
+
+```python
+from haystack.components.extractors import RegexTextExtractor
+from haystack.dataclasses import ChatMessage
+
+# Using with a string
+parser = RegexTextExtractor(regex_pattern='<issue url="(.+)">')
+result = parser.run(text_or_messages='<issue url="github.com/hahahaha">hahahah</issue>')
+# result: {"captured_text": "github.com/hahahaha"}
+
+# Using with ChatMessages
+messages = [ChatMessage.from_user('<issue url="github.com/hahahaha">hahahah</issue>')]
+result = parser.run(text_or_messages=messages)
+# result: {"captured_text": "github.com/hahahaha"}
+```
+
+<a id="regex_text_extractor.RegexTextExtractor.__init__"></a>
+
+#### RegexTextExtractor.\_\_init\_\_
+
+```python
+def __init__(regex_pattern: str)
+```
+
+Creates an instance of the RegexTextExtractor component.
+
+**Arguments**:
+
+- `regex_pattern`: The regular expression pattern used to extract text.
+The pattern should include a capture group to extract the desired text.
+Example: `'<issue url="(.+)">'` captures `'github.com/hahahaha'` from `'<issue url="github.com/hahahaha">'`.
+
+<a id="regex_text_extractor.RegexTextExtractor.run"></a>
+
+#### RegexTextExtractor.run
+
+```python
+@component.output_types(captured_text=str)
+def run(text_or_messages: str | list[ChatMessage]) -> dict[str, str]
+```
+
+Extracts text from input using the configured regex pattern.
+
+**Arguments**:
+
+- `text_or_messages`: Either a string or a list of ChatMessage objects to search through.
+
+**Raises**:
+
+- `None`: - ValueError: if receiving a list the last element is not a ChatMessage instance.
+
+**Returns**:
+
+- `{"captured_text": "matched text"}` if a match is found
+- `{"captured_text": ""}` if no match is found
+

docs-website/reference_versioned_docs/version-2.22/haystack-api/fetchers_api.md

@@ -0,0 +1,144 @@
+---
+title: "Fetchers"
+id: fetchers-api
+description: "Fetches content from a list of URLs and returns a list of extracted content streams."
+slug: "/fetchers-api"
+---
+
+<a id="link_content"></a>
+
+## Module link\_content
+
+<a id="link_content.LinkContentFetcher"></a>
+
+### LinkContentFetcher
+
+Fetches and extracts content from URLs.
+
+It supports various content types, retries on failures, and automatic user-agent rotation for failed web
+requests. Use it as the data-fetching step in your pipelines.
+
+You may need to convert LinkContentFetcher's output into a list of documents. Use HTMLToDocument
+converter to do this.
+
+### Usage example
+
+```python
+from haystack.components.fetchers.link_content import LinkContentFetcher
+
+fetcher = LinkContentFetcher()
+streams = fetcher.run(urls=["https://www.google.com"])["streams"]
+
+assert len(streams) == 1
+assert streams[0].meta == {'content_type': 'text/html', 'url': 'https://www.google.com'}
+assert streams[0].data
+```
+
+For async usage:
+
+```python
+import asyncio
+from haystack.components.fetchers import LinkContentFetcher
+
+async def fetch_async():
+    fetcher = LinkContentFetcher()
+    result = await fetcher.run_async(urls=["https://www.google.com"])
+    return result["streams"]
+
+streams = asyncio.run(fetch_async())
+```
+
+<a id="link_content.LinkContentFetcher.__init__"></a>
+
+#### LinkContentFetcher.\_\_init\_\_
+
+```python
+def __init__(raise_on_failure: bool = True,
+             user_agents: list[str] | None = None,
+             retry_attempts: int = 2,
+             timeout: int = 3,
+             http2: bool = False,
+             client_kwargs: dict | None = None,
+             request_headers: dict[str, str] | None = None)
+```
+
+Initializes the component.
+
+**Arguments**:
+
+- `raise_on_failure`: If `True`, raises an exception if it fails to fetch a single URL.
+For multiple URLs, it logs errors and returns the content it successfully fetched.
+- `user_agents`: [User agents](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent)
+for fetching content. If `None`, a default user agent is used.
+- `retry_attempts`: The number of times to retry to fetch the URL's content.
+- `timeout`: Timeout in seconds for the request.
+- `http2`: Whether to enable HTTP/2 support for requests. Defaults to False.
+Requires the 'h2' package to be installed (via `pip install httpx[http2]`).
+- `client_kwargs`: Additional keyword arguments to pass to the httpx client.
+If `None`, default values are used.
+
+<a id="link_content.LinkContentFetcher.__del__"></a>
+
+#### LinkContentFetcher.\_\_del\_\_
+
+```python
+def __del__()
+```
+
+Clean up resources when the component is deleted.
+
+Closes both the synchronous and asynchronous HTTP clients to prevent
+resource leaks.
+
+<a id="link_content.LinkContentFetcher.run"></a>
+
+#### LinkContentFetcher.run
+
+```python
+@component.output_types(streams=list[ByteStream])
+def run(urls: list[str])
+```
+
+Fetches content from a list of URLs and returns a list of extracted content streams.
+
+Each content stream is a `ByteStream` object containing the extracted content as binary data.
+Each ByteStream object in the returned list corresponds to the contents of a single URL.
+The content type of each stream is stored in the metadata of the ByteStream object under
+the key "content_type". The URL of the fetched content is stored under the key "url".
+
+**Arguments**:
+
+- `urls`: A list of URLs to fetch content from.
+
+**Raises**:
+
+- `Exception`: If the provided list of URLs contains only a single URL, and `raise_on_failure` is set to
+`True`, an exception will be raised in case of an error during content retrieval.
+In all other scenarios, any retrieval errors are logged, and a list of successfully retrieved `ByteStream`
+ objects is returned.
+
+**Returns**:
+
+`ByteStream` objects representing the extracted content.
+
+<a id="link_content.LinkContentFetcher.run_async"></a>
+
+#### LinkContentFetcher.run\_async
+
+```python
+@component.output_types(streams=list[ByteStream])
+async def run_async(urls: list[str])
+```
+
+Asynchronously fetches content from a list of URLs and returns a list of extracted content streams.
+
+This is the asynchronous version of the `run` method with the same parameters and return values.
+
+**Arguments**:
+
+- `urls`: A list of URLs to fetch content from.
+
+**Returns**:
+
+`ByteStream` objects representing the extracted content.
+

docs-website/reference_versioned_docs/version-2.22/haystack-api/image_converters_api.md

@@ -0,0 +1,374 @@
+---
+title: "Image Converters"
+id: image-converters-api
+description: "Various converters to transform image data from one format to another."
+slug: "/image-converters-api"
+---
+
+<a id="document_to_image"></a>
+
+## Module document\_to\_image
+
+<a id="document_to_image.DocumentToImageContent"></a>
+
+### DocumentToImageContent
+
+Converts documents sourced from PDF and image files into ImageContents.
+
+This component processes a list of documents and extracts visual content from supported file formats, converting
+them into ImageContents that can be used for multimodal AI tasks. It handles both direct image files and PDF
+documents by extracting specific pages as images.
+
+Documents are expected to have metadata containing:
+- The `file_path_meta_field` key with a valid file path that exists when combined with `root_path`
+- A supported image format (MIME type must be one of the supported image types)
+- For PDF files, a `page_number` key specifying which page to extract
+
+### Usage example
+    ```python
+    from haystack import Document
+    from haystack.components.converters.image.document_to_image import DocumentToImageContent
+
+    converter = DocumentToImageContent(
+        file_path_meta_field="file_path",
+        root_path="/data/files",
+        detail="high",
+        size=(800, 600)
+    )
+
+    documents = [
+        Document(content="Optional description of image.jpg", meta={"file_path": "image.jpg"}),
+        Document(content="Text content of page 1 of doc.pdf", meta={"file_path": "doc.pdf", "page_number": 1})
+    ]
+
+    result = converter.run(documents)
+    image_contents = result["image_contents"]
+    # [ImageContent(
+    #    base64_image='/9j/4A...', mime_type='image/jpeg', detail='high', meta={'file_path': 'image.jpg'}
+    #  ),
+    #  ImageContent(
+    #    base64_image='/9j/4A...', mime_type='image/jpeg', detail='high',
+    #    meta={'page_number': 1, 'file_path': 'doc.pdf'}
+    #  )]
+    ```
+
+<a id="document_to_image.DocumentToImageContent.__init__"></a>
+
+#### DocumentToImageContent.\_\_init\_\_
+
+```python
+def __init__(*,
+             file_path_meta_field: str = "file_path",
+             root_path: str | None = None,
+             detail: Literal["auto", "high", "low"] | None = None,
+             size: tuple[int, int] | None = None)
+```
+
+Initialize the DocumentToImageContent component.
+
+**Arguments**:
+
+- `file_path_meta_field`: The metadata field in the Document that contains the file path to the image or PDF.
+- `root_path`: The root directory path where document files are located. If provided, file paths in
+document metadata will be resolved relative to this path. If None, file paths are treated as absolute paths.
+- `detail`: Optional detail level of the image (only supported by OpenAI). Can be "auto", "high", or "low".
+This will be passed to the created ImageContent objects.
+- `size`: If provided, resizes the image to fit within the specified dimensions (width, height) while
+maintaining aspect ratio. This reduces file size, memory usage, and processing time, which is beneficial
+when working with models that have resolution constraints or when transmitting images to remote services.
+
+<a id="document_to_image.DocumentToImageContent.run"></a>
+
+#### DocumentToImageContent.run
+
+```python
+@component.output_types(image_contents=list[ImageContent | None])
+def run(documents: list[Document]) -> dict[str, list[ImageContent | None]]
+```
+
+Convert documents with image or PDF sources into ImageContent objects.
+
+This method processes the input documents, extracting images from supported file formats and converting them
+into ImageContent objects.
+
+**Arguments**:
+
+- `documents`: A list of documents to process. Each document should have metadata containing at minimum
+a 'file_path_meta_field' key. PDF documents additionally require a 'page_number' key to specify which
+page to convert.
+
+**Raises**:
+
+- `ValueError`: If any document is missing the required metadata keys, has an invalid file path, or has an unsupported
+MIME type. The error message will specify which document and what information is missing or incorrect.
+
+**Returns**:
+
+Dictionary containing one key:
+- "image_contents": ImageContents created from the processed documents. These contain base64-encoded image
+data and metadata. The order corresponds to order of input documents.
+
+<a id="file_to_document"></a>
+
+## Module file\_to\_document
+
+<a id="file_to_document.ImageFileToDocument"></a>
+
+### ImageFileToDocument
+
+Converts image file references into empty Document objects with associated metadata.
+
+This component is useful in pipelines where image file paths need to be wrapped in `Document` objects to be
+processed by downstream components such as the `SentenceTransformersImageDocumentEmbedder`.
+
+It does **not** extract any content from the image files, instead it creates `Document` objects with `None` as
+their content and attaches metadata such as file path and any user-provided values.
+
+### Usage example
+```python
+from haystack.components.converters.image import ImageFileToDocument
+
+converter = ImageFileToDocument()
+
+sources = ["image.jpg", "another_image.png"]
+
+result = converter.run(sources=sources)
+documents = result["documents"]
+
+print(documents)
+
+# [Document(id=..., meta: {'file_path': 'image.jpg'}),
+# Document(id=..., meta: {'file_path': 'another_image.png'})]
+```
+
+<a id="file_to_document.ImageFileToDocument.__init__"></a>
+
+#### ImageFileToDocument.\_\_init\_\_
+
+```python
+def __init__(*, store_full_path: bool = False)
+```
+
+Initialize the ImageFileToDocument component.
+
+**Arguments**:
+
+- `store_full_path`: If True, the full path of the file is stored in the metadata of the document.
+If False, only the file name is stored.
+
+<a id="file_to_document.ImageFileToDocument.run"></a>
+
+#### ImageFileToDocument.run
+
+```python
+@component.output_types(documents=list[Document])
+def run(
+    *,
+    sources: list[str | Path | ByteStream],
+    meta: dict[str, Any] | list[dict[str, Any]] | None = None
+) -> dict[str, list[Document]]
+```
+
+Convert image files into empty Document objects with metadata.
+
+This method accepts image file references (as file paths or ByteStreams) and creates `Document` objects
+without content. These documents are enriched with metadata derived from the input source and optional
+user-provided metadata.
+
+**Arguments**:
+
+- `sources`: List of file paths or ByteStream objects to convert.
+- `meta`: Optional metadata to attach to the documents.
+This value can be a list of dictionaries or a single dictionary.
+If it's a single dictionary, its content is added to the metadata of all produced documents.
+If it's a list, its length must match the number of sources, as they are zipped together.
+For ByteStream objects, their `meta` is added to the output documents.
+
+**Returns**:
+
+A dictionary containing:
+- `documents`: A list of `Document` objects with empty content and associated metadata.
+
+<a id="file_to_image"></a>
+
+## Module file\_to\_image
+
+<a id="file_to_image.ImageFileToImageContent"></a>
+
+### ImageFileToImageContent
+
+Converts image files to ImageContent objects.
+
+### Usage example
+```python
+from haystack.components.converters.image import ImageFileToImageContent
+
+converter = ImageFileToImageContent()
+
+sources = ["image.jpg", "another_image.png"]
+
+image_contents = converter.run(sources=sources)["image_contents"]
+print(image_contents)
+
+# [ImageContent(base64_image='...',
+#               mime_type='image/jpeg',
+#               detail=None,
+#               meta={'file_path': 'image.jpg'}),
+#  ...]
+```
+
+<a id="file_to_image.ImageFileToImageContent.__init__"></a>
+
+#### ImageFileToImageContent.\_\_init\_\_
+
+```python
+def __init__(*,
+             detail: Literal["auto", "high", "low"] | None = None,
+             size: tuple[int, int] | None = None)
+```
+
+Create the ImageFileToImageContent component.
+
+**Arguments**:
+
+- `detail`: Optional detail level of the image (only supported by OpenAI). One of "auto", "high", or "low".
+This will be passed to the created ImageContent objects.
+- `size`: If provided, resizes the image to fit within the specified dimensions (width, height) while
+maintaining aspect ratio. This reduces file size, memory usage, and processing time, which is beneficial
+when working with models that have resolution constraints or when transmitting images to remote services.
+
+<a id="file_to_image.ImageFileToImageContent.run"></a>
+
+#### ImageFileToImageContent.run
+
+```python
+@component.output_types(image_contents=list[ImageContent])
+def run(sources: list[str | Path | ByteStream],
+        meta: dict[str, Any] | list[dict[str, Any]] | None = None,
+        *,
+        detail: Literal["auto", "high", "low"] | None = None,
+        size: tuple[int, int] | None = None) -> dict[str, list[ImageContent]]
+```
+
+Converts files to ImageContent objects.
+
+**Arguments**:
+
+- `sources`: List of file paths or ByteStream objects to convert.
+- `meta`: Optional metadata to attach to the ImageContent objects.
+This value can be a list of dictionaries or a single dictionary.
+If it's a single dictionary, its content is added to the metadata of all produced ImageContent objects.
+If it's a list, its length must match the number of sources as they're zipped together.
+For ByteStream objects, their `meta` is added to the output ImageContent objects.
+- `detail`: Optional detail level of the image (only supported by OpenAI). One of "auto", "high", or "low".
+This will be passed to the created ImageContent objects.
+If not provided, the detail level will be the one set in the constructor.
+- `size`: If provided, resizes the image to fit within the specified dimensions (width, height) while
+maintaining aspect ratio. This reduces file size, memory usage, and processing time, which is beneficial
+when working with models that have resolution constraints or when transmitting images to remote services.
+If not provided, the size value will be the one set in the constructor.
+
+**Returns**:
+
+A dictionary with the following keys:
+- `image_contents`: A list of ImageContent objects.
+
+<a id="pdf_to_image"></a>
+
+## Module pdf\_to\_image
+
+<a id="pdf_to_image.PDFToImageContent"></a>
+
+### PDFToImageContent
+
+Converts PDF files to ImageContent objects.
+
+### Usage example
+```python
+from haystack.components.converters.image import PDFToImageContent
+
+converter = PDFToImageContent()
+
+sources = ["file.pdf", "another_file.pdf"]
+
+image_contents = converter.run(sources=sources)["image_contents"]
+print(image_contents)
+
+# [ImageContent(base64_image='...',
+#               mime_type='application/pdf',
+#               detail=None,
+#               meta={'file_path': 'file.pdf', 'page_number': 1}),
+#  ...]
+```
+
+<a id="pdf_to_image.PDFToImageContent.__init__"></a>
+
+#### PDFToImageContent.\_\_init\_\_
+
+```python
+def __init__(*,
+             detail: Literal["auto", "high", "low"] | None = None,
+             size: tuple[int, int] | None = None,
+             page_range: list[str | int] | None = None)
+```
+
+Create the PDFToImageContent component.
+
+**Arguments**:
+
+- `detail`: Optional detail level of the image (only supported by OpenAI). One of "auto", "high", or "low".
+This will be passed to the created ImageContent objects.
+- `size`: If provided, resizes the image to fit within the specified dimensions (width, height) while
+maintaining aspect ratio. This reduces file size, memory usage, and processing time, which is beneficial
+when working with models that have resolution constraints or when transmitting images to remote services.
+- `page_range`: List of page numbers and/or page ranges to convert to images. Page numbers start at 1.
+If None, all pages in the PDF will be converted. Pages outside the valid range (1 to number of pages)
+will be skipped with a warning. For example, page_range=[1, 3] will convert only the first and third
+pages of the document. It also accepts printable range strings, e.g.:  ['1-3', '5', '8', '10-12']
+will convert pages 1, 2, 3, 5, 8, 10, 11, 12.
+
+<a id="pdf_to_image.PDFToImageContent.run"></a>
+
+#### PDFToImageContent.run
+
+```python
+@component.output_types(image_contents=list[ImageContent])
+def run(
+    sources: list[str | Path | ByteStream],
+    meta: dict[str, Any] | list[dict[str, Any]] | None = None,
+    *,
+    detail: Literal["auto", "high", "low"] | None = None,
+    size: tuple[int, int] | None = None,
+    page_range: list[str | int] | None = None
+) -> dict[str, list[ImageContent]]
+```
+
+Converts files to ImageContent objects.
+
+**Arguments**:
+
+- `sources`: List of file paths or ByteStream objects to convert.
+- `meta`: Optional metadata to attach to the ImageContent objects.
+This value can be a list of dictionaries or a single dictionary.
+If it's a single dictionary, its content is added to the metadata of all produced ImageContent objects.
+If it's a list, its length must match the number of sources as they're zipped together.
+For ByteStream objects, their `meta` is added to the output ImageContent objects.
+- `detail`: Optional detail level of the image (only supported by OpenAI). One of "auto", "high", or "low".
+This will be passed to the created ImageContent objects.
+If not provided, the detail level will be the one set in the constructor.
+- `size`: If provided, resizes the image to fit within the specified dimensions (width, height) while
+maintaining aspect ratio. This reduces file size, memory usage, and processing time, which is beneficial
+when working with models that have resolution constraints or when transmitting images to remote services.
+If not provided, the size value will be the one set in the constructor.
+- `page_range`: List of page numbers and/or page ranges to convert to images. Page numbers start at 1.
+If None, all pages in the PDF will be converted. Pages outside the valid range (1 to number of pages)
+will be skipped with a warning. For example, page_range=[1, 3] will convert only the first and third
+pages of the document. It also accepts printable range strings, e.g.:  ['1-3', '5', '8', '10-12']
+will convert pages 1, 2, 3, 5, 8, 10, 11, 12.
+If not provided, the page_range value will be the one set in the constructor.
+
+**Returns**:
+
+A dictionary with the following keys:
+- `image_contents`: A list of ImageContent objects.
+

docs-website/reference_versioned_docs/version-2.22/haystack-api/joiners_api.md

@@ -0,0 +1,625 @@
+---
+title: "Joiners"
+id: joiners-api
+description: "Components that join list of different objects"
+slug: "/joiners-api"
+---
+
+<a id="answer_joiner"></a>
+
+## Module answer\_joiner
+
+<a id="answer_joiner.JoinMode"></a>
+
+### JoinMode
+
+Enum for AnswerJoiner join modes.
+
+<a id="answer_joiner.JoinMode.from_str"></a>
+
+#### JoinMode.from\_str
+
+```python
+@staticmethod
+def from_str(string: str) -> "JoinMode"
+```
+
+Convert a string to a JoinMode enum.
+
+<a id="answer_joiner.AnswerJoiner"></a>
+
+### AnswerJoiner
+
+Merges multiple lists of `Answer` objects into a single list.
+
+Use this component to combine answers from different Generators into a single list.
+Currently, the component supports only one join mode: `CONCATENATE`.
+This mode concatenates multiple lists of answers into a single list.
+
+### Usage example
+
+In this example, AnswerJoiner merges answers from two different Generators:
+
+```python
+from haystack.components.builders import AnswerBuilder
+from haystack.components.joiners import AnswerJoiner
+
+from haystack.core.pipeline import Pipeline
+
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.dataclasses import ChatMessage
+
+
+query = "What's Natural Language Processing?"
+messages = [ChatMessage.from_system("You are a helpful, respectful and honest assistant. Be super concise."),
+            ChatMessage.from_user(query)]
+
+pipe = Pipeline()
+pipe.add_component("llm_1", OpenAIChatGenerator()
+pipe.add_component("llm_2", OpenAIChatGenerator()
+pipe.add_component("aba", AnswerBuilder())
+pipe.add_component("abb", AnswerBuilder())
+pipe.add_component("joiner", AnswerJoiner())
+
+pipe.connect("llm_1.replies", "aba")
+pipe.connect("llm_2.replies", "abb")
+pipe.connect("aba.answers", "joiner")
+pipe.connect("abb.answers", "joiner")
+
+results = pipe.run(data={"llm_1": {"messages": messages},
+                            "llm_2": {"messages": messages},
+                            "aba": {"query": query},
+                            "abb": {"query": query}})
+```
+
+<a id="answer_joiner.AnswerJoiner.__init__"></a>
+
+#### AnswerJoiner.\_\_init\_\_
+
+```python
+def __init__(join_mode: str | JoinMode = JoinMode.CONCATENATE,
+             top_k: int | None = None,
+             sort_by_score: bool = False)
+```
+
+Creates an AnswerJoiner component.
+
+**Arguments**:
+
+- `join_mode`: Specifies the join mode to use. Available modes:
+- `concatenate`: Concatenates multiple lists of Answers into a single list.
+- `top_k`: The maximum number of Answers to return.
+- `sort_by_score`: If `True`, sorts the documents by score in descending order.
+If a document has no score, it is handled as if its score is -infinity.
+
+<a id="answer_joiner.AnswerJoiner.run"></a>
+
+#### AnswerJoiner.run
+
+```python
+@component.output_types(answers=list[AnswerType])
+def run(answers: Variadic[list[AnswerType]], top_k: int | None = None)
+```
+
+Joins multiple lists of Answers into a single list depending on the `join_mode` parameter.
+
+**Arguments**:
+
+- `answers`: Nested list of Answers to be merged.
+- `top_k`: The maximum number of Answers to return. Overrides the instance's `top_k` if provided.
+
+**Returns**:
+
+A dictionary with the following keys:
+- `answers`: Merged list of Answers
+
+<a id="answer_joiner.AnswerJoiner.to_dict"></a>
+
+#### AnswerJoiner.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+**Returns**:
+
+Dictionary with serialized data.
+
+<a id="answer_joiner.AnswerJoiner.from_dict"></a>
+
+#### AnswerJoiner.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "AnswerJoiner"
+```
+
+Deserializes the component from a dictionary.
+
+**Arguments**:
+
+- `data`: The dictionary to deserialize from.
+
+**Returns**:
+
+The deserialized component.
+
+<a id="branch"></a>
+
+## Module branch
+
+<a id="branch.BranchJoiner"></a>
+
+### BranchJoiner
+
+A component that merges multiple input branches of a pipeline into a single output stream.
+
+`BranchJoiner` receives multiple inputs of the same data type and forwards the first received value
+to its output. This is useful for scenarios where multiple branches need to converge before proceeding.
+
+### Common Use Cases:
+- **Loop Handling:** `BranchJoiner` helps close loops in pipelines. For example, if a pipeline component validates
+  or modifies incoming data and produces an error-handling branch, `BranchJoiner` can merge both branches and send
+  (or resend in the case of a loop) the data to the component that evaluates errors. See "Usage example" below.
+
+- **Decision-Based Merging:** `BranchJoiner` reconciles branches coming from Router components (such as
+  `ConditionalRouter`, `TextLanguageRouter`). Suppose a `TextLanguageRouter` directs user queries to different
+  Retrievers based on the detected language. Each Retriever processes its assigned query and passes the results
+  to `BranchJoiner`, which consolidates them into a single output before passing them to the next component, such
+  as a `PromptBuilder`.
+
+### Example Usage:
+```python
+import json
+
+from haystack import Pipeline
+from haystack.components.converters import OutputAdapter
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.components.joiners import BranchJoiner
+from haystack.components.validators import JsonSchemaValidator
+from haystack.dataclasses import ChatMessage
+
+# Define a schema for validation
+person_schema = {
+    "type": "object",
+    "properties": {
+        "first_name": {"type": "string", "pattern": "^[A-Z][a-z]+$"},
+        "last_name": {"type": "string", "pattern": "^[A-Z][a-z]+$"},
+        "nationality": {"type": "string", "enum": ["Italian", "Portuguese", "American"]},
+    },
+    "required": ["first_name", "last_name", "nationality"]
+}
+
+# Initialize a pipeline
+pipe = Pipeline()
+
+# Add components to the pipeline
+pipe.add_component('joiner', BranchJoiner(list[ChatMessage]))
+pipe.add_component('generator', OpenAIChatGenerator())
+pipe.add_component('validator', JsonSchemaValidator(json_schema=person_schema))
+pipe.add_component('adapter', OutputAdapter("{{chat_message}}", list[ChatMessage], unsafe=True))
+
+# And connect them
+pipe.connect("adapter", "joiner")
+pipe.connect("joiner", "generator")
+pipe.connect("generator.replies", "validator.messages")
+pipe.connect("validator.validation_error", "joiner")
+
+result = pipe.run(
+    data={
+    "generator": {"generation_kwargs": {"response_format": {"type": "json_object"}}},
+    "adapter": {"chat_message": [ChatMessage.from_user("Create json from Peter Parker")]}}
+)
+
+print(json.loads(result["validator"]["validated"][0].text))
+
+
+>> {'first_name': 'Peter', 'last_name': 'Parker', 'nationality': 'American', 'name': 'Spider-Man', 'occupation':
+>> 'Superhero', 'age': 23, 'location': 'New York City'}
+```
+
+Note that `BranchJoiner` can manage only one data type at a time. In this case, `BranchJoiner` is created for
+passing `list[ChatMessage]`. This determines the type of data that `BranchJoiner` will receive from the upstream
+connected components and also the type of data that `BranchJoiner` will send through its output.
+
+In the code example, `BranchJoiner` receives a looped back `list[ChatMessage]` from the `JsonSchemaValidator` and
+sends it down to the `OpenAIChatGenerator` for re-generation. We can have multiple loopback connections in the
+pipeline. In this instance, the downstream component is only one (the `OpenAIChatGenerator`), but the pipeline could
+have more than one downstream component.
+
+<a id="branch.BranchJoiner.__init__"></a>
+
+#### BranchJoiner.\_\_init\_\_
+
+```python
+def __init__(type_: type)
+```
+
+Creates a `BranchJoiner` component.
+
+**Arguments**:
+
+- `type_`: The expected data type of inputs and outputs.
+
+<a id="branch.BranchJoiner.to_dict"></a>
+
+#### BranchJoiner.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serializes the component into a dictionary.
+
+**Returns**:
+
+Dictionary with serialized data.
+
+<a id="branch.BranchJoiner.from_dict"></a>
+
+#### BranchJoiner.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "BranchJoiner"
+```
+
+Deserializes a `BranchJoiner` instance from a dictionary.
+
+**Arguments**:
+
+- `data`: The dictionary containing serialized component data.
+
+**Returns**:
+
+A deserialized `BranchJoiner` instance.
+
+<a id="branch.BranchJoiner.run"></a>
+
+#### BranchJoiner.run
+
+```python
+def run(**kwargs) -> dict[str, Any]
+```
+
+Executes the `BranchJoiner`, selecting the first available input value and passing it downstream.
+
+**Arguments**:
+
+- `**kwargs`: The input data. Must be of the type declared by `type_` during initialization.
+
+**Returns**:
+
+A dictionary with a single key `value`, containing the first input received.
+
+<a id="document_joiner"></a>
+
+## Module document\_joiner
+
+<a id="document_joiner.JoinMode"></a>
+
+### JoinMode
+
+Enum for join mode.
+
+<a id="document_joiner.JoinMode.from_str"></a>
+
+#### JoinMode.from\_str
+
+```python
+@staticmethod
+def from_str(string: str) -> "JoinMode"
+```
+
+Convert a string to a JoinMode enum.
+
+<a id="document_joiner.DocumentJoiner"></a>
+
+### DocumentJoiner
+
+Joins multiple lists of documents into a single list.
+
+It supports different join modes:
+- concatenate: Keeps the highest-scored document in case of duplicates.
+- merge: Calculates a weighted sum of scores for duplicates and merges them.
+- reciprocal_rank_fusion: Merges and assigns scores based on reciprocal rank fusion.
+- distribution_based_rank_fusion: Merges and assigns scores based on scores distribution in each Retriever.
+
+### Usage example:
+
+```python
+from haystack import Pipeline, Document
+from haystack.components.embedders import SentenceTransformersTextEmbedder, SentenceTransformersDocumentEmbedder
+from haystack.components.joiners import DocumentJoiner
+from haystack.components.retrievers import InMemoryBM25Retriever
+from haystack.components.retrievers import InMemoryEmbeddingRetriever
+from haystack.document_stores.in_memory import InMemoryDocumentStore
+
+document_store = InMemoryDocumentStore()
+docs = [Document(content="Paris"), Document(content="Berlin"), Document(content="London")]
+embedder = SentenceTransformersDocumentEmbedder(model="sentence-transformers/all-MiniLM-L6-v2")
+embedder.warm_up()
+docs_embeddings = embedder.run(docs)
+document_store.write_documents(docs_embeddings['documents'])
+
+p = Pipeline()
+p.add_component(instance=InMemoryBM25Retriever(document_store=document_store), name="bm25_retriever")
+p.add_component(
+        instance=SentenceTransformersTextEmbedder(model="sentence-transformers/all-MiniLM-L6-v2"),
+        name="text_embedder",
+    )
+p.add_component(instance=InMemoryEmbeddingRetriever(document_store=document_store), name="embedding_retriever")
+p.add_component(instance=DocumentJoiner(), name="joiner")
+p.connect("bm25_retriever", "joiner")
+p.connect("embedding_retriever", "joiner")
+p.connect("text_embedder", "embedding_retriever")
+query = "What is the capital of France?"
+p.run(data={"query": query, "text": query, "top_k": 1})
+```
+
+<a id="document_joiner.DocumentJoiner.__init__"></a>
+
+#### DocumentJoiner.\_\_init\_\_
+
+```python
+def __init__(join_mode: str | JoinMode = JoinMode.CONCATENATE,
+             weights: list[float] | None = None,
+             top_k: int | None = None,
+             sort_by_score: bool = True)
+```
+
+Creates a DocumentJoiner component.
+
+**Arguments**:
+
+- `join_mode`: Specifies the join mode to use. Available modes:
+- `concatenate`: Keeps the highest-scored document in case of duplicates.
+- `merge`: Calculates a weighted sum of scores for duplicates and merges them.
+- `reciprocal_rank_fusion`: Merges and assigns scores based on reciprocal rank fusion.
+- `distribution_based_rank_fusion`: Merges and assigns scores based on scores
+distribution in each Retriever.
+- `weights`: Assign importance to each list of documents to influence how they're joined.
+This parameter is ignored for
+`concatenate` or `distribution_based_rank_fusion` join modes.
+Weight for each list of documents must match the number of inputs.
+- `top_k`: The maximum number of documents to return.
+- `sort_by_score`: If `True`, sorts the documents by score in descending order.
+If a document has no score, it is handled as if its score is -infinity.
+
+<a id="document_joiner.DocumentJoiner.run"></a>
+
+#### DocumentJoiner.run
+
+```python
+@component.output_types(documents=list[Document])
+def run(documents: Variadic[list[Document]], top_k: int | None = None)
+```
+
+Joins multiple lists of Documents into a single list depending on the `join_mode` parameter.
+
+**Arguments**:
+
+- `documents`: List of list of documents to be merged.
+- `top_k`: The maximum number of documents to return. Overrides the instance's `top_k` if provided.
+
+**Returns**:
+
+A dictionary with the following keys:
+- `documents`: Merged list of Documents
+
+<a id="document_joiner.DocumentJoiner.to_dict"></a>
+
+#### DocumentJoiner.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+**Returns**:
+
+Dictionary with serialized data.
+
+<a id="document_joiner.DocumentJoiner.from_dict"></a>
+
+#### DocumentJoiner.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "DocumentJoiner"
+```
+
+Deserializes the component from a dictionary.
+
+**Arguments**:
+
+- `data`: The dictionary to deserialize from.
+
+**Returns**:
+
+The deserialized component.
+
+<a id="list_joiner"></a>
+
+## Module list\_joiner
+
+<a id="list_joiner.ListJoiner"></a>
+
+### ListJoiner
+
+A component that joins multiple lists into a single flat list.
+
+The ListJoiner receives multiple lists of the same type and concatenates them into a single flat list.
+The output order respects the pipeline's execution sequence, with earlier inputs being added first.
+
+Usage example:
+```python
+from haystack.components.builders import ChatPromptBuilder
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.dataclasses import ChatMessage
+from haystack import Pipeline
+from haystack.components.joiners import ListJoiner
+
+
+user_message = [ChatMessage.from_user("Give a brief answer the following question: {{query}}")]
+
+feedback_prompt = """
+    You are given a question and an answer.
+    Your task is to provide a score and a brief feedback on the answer.
+    Question: {{query}}
+    Answer: {{response}}
+    """
+feedback_message = [ChatMessage.from_system(feedback_prompt)]
+
+prompt_builder = ChatPromptBuilder(template=user_message)
+feedback_prompt_builder = ChatPromptBuilder(template=feedback_message)
+llm = OpenAIChatGenerator()
+feedback_llm = OpenAIChatGenerator()
+
+pipe = Pipeline()
+pipe.add_component("prompt_builder", prompt_builder)
+pipe.add_component("llm", llm)
+pipe.add_component("feedback_prompt_builder", feedback_prompt_builder)
+pipe.add_component("feedback_llm", feedback_llm)
+pipe.add_component("list_joiner", ListJoiner(list[ChatMessage]))
+
+pipe.connect("prompt_builder.prompt", "llm.messages")
+pipe.connect("prompt_builder.prompt", "list_joiner")
+pipe.connect("llm.replies", "list_joiner")
+pipe.connect("llm.replies", "feedback_prompt_builder.response")
+pipe.connect("feedback_prompt_builder.prompt", "feedback_llm.messages")
+pipe.connect("feedback_llm.replies", "list_joiner")
+
+query = "What is nuclear physics?"
+ans = pipe.run(data={"prompt_builder": {"template_variables":{"query": query}},
+    "feedback_prompt_builder": {"template_variables":{"query": query}}})
+
+print(ans["list_joiner"]["values"])
+```
+
+<a id="list_joiner.ListJoiner.__init__"></a>
+
+#### ListJoiner.\_\_init\_\_
+
+```python
+def __init__(list_type_: type | None = None)
+```
+
+Creates a ListJoiner component.
+
+**Arguments**:
+
+- `list_type_`: The expected type of the lists this component will join (e.g., list[ChatMessage]).
+If specified, all input lists must conform to this type. If None, the component defaults to handling
+lists of any type including mixed types.
+
+<a id="list_joiner.ListJoiner.to_dict"></a>
+
+#### ListJoiner.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+**Returns**:
+
+Dictionary with serialized data.
+
+<a id="list_joiner.ListJoiner.from_dict"></a>
+
+#### ListJoiner.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "ListJoiner"
+```
+
+Deserializes the component from a dictionary.
+
+**Arguments**:
+
+- `data`: Dictionary to deserialize from.
+
+**Returns**:
+
+Deserialized component.
+
+<a id="list_joiner.ListJoiner.run"></a>
+
+#### ListJoiner.run
+
+```python
+def run(values: Variadic[list[Any]]) -> dict[str, list[Any]]
+```
+
+Joins multiple lists into a single flat list.
+
+**Arguments**:
+
+- `values`: The list to be joined.
+
+**Returns**:
+
+Dictionary with 'values' key containing the joined list.
+
+<a id="string_joiner"></a>
+
+## Module string\_joiner
+
+<a id="string_joiner.StringJoiner"></a>
+
+### StringJoiner
+
+Component to join strings from different components to a list of strings.
+
+### Usage example
+
+```python
+from haystack.components.joiners import StringJoiner
+from haystack.components.builders import PromptBuilder
+from haystack.core.pipeline import Pipeline
+
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.dataclasses import ChatMessage
+
+string_1 = "What's Natural Language Processing?"
+string_2 = "What is life?"
+
+pipeline = Pipeline()
+pipeline.add_component("prompt_builder_1", PromptBuilder("Builder 1: {{query}}"))
+pipeline.add_component("prompt_builder_2", PromptBuilder("Builder 2: {{query}}"))
+pipeline.add_component("string_joiner", StringJoiner())
+
+pipeline.connect("prompt_builder_1.prompt", "string_joiner.strings")
+pipeline.connect("prompt_builder_2.prompt", "string_joiner.strings")
+
+print(pipeline.run(data={"prompt_builder_1": {"query": string_1}, "prompt_builder_2": {"query": string_2}}))
+
+>> {"string_joiner": {"strings": ["Builder 1: What's Natural Language Processing?", "Builder 2: What is life?"]}}
+```
+
+<a id="string_joiner.StringJoiner.run"></a>
+
+#### StringJoiner.run
+
+```python
+@component.output_types(strings=list[str])
+def run(strings: Variadic[str])
+```
+
+Joins strings into a list of strings
+
+**Arguments**:
+
+- `strings`: strings from different components
+
+**Returns**:
+
+A dictionary with the following keys:
+- `strings`: Merged list of strings
+

docs-website/reference_versioned_docs/version-2.22/haystack-api/preprocessors_api.md

@@ -0,0 +1,797 @@
+---
+title: "PreProcessors"
+id: preprocessors-api
+description: "Preprocess your Documents and texts. Clean, split, and more."
+slug: "/preprocessors-api"
+---
+
+<a id="csv_document_cleaner"></a>
+
+## Module csv\_document\_cleaner
+
+<a id="csv_document_cleaner.CSVDocumentCleaner"></a>
+
+### CSVDocumentCleaner
+
+A component for cleaning CSV documents by removing empty rows and columns.
+
+This component processes CSV content stored in Documents, allowing
+for the optional ignoring of a specified number of rows and columns before performing
+the cleaning operation. Additionally, it provides options to keep document IDs and
+control whether empty rows and columns should be removed.
+
+<a id="csv_document_cleaner.CSVDocumentCleaner.__init__"></a>
+
+#### CSVDocumentCleaner.\_\_init\_\_
+
+```python
+def __init__(*,
+             ignore_rows: int = 0,
+             ignore_columns: int = 0,
+             remove_empty_rows: bool = True,
+             remove_empty_columns: bool = True,
+             keep_id: bool = False) -> None
+```
+
+Initializes the CSVDocumentCleaner component.
+
+**Arguments**:
+
+- `ignore_rows`: Number of rows to ignore from the top of the CSV table before processing.
+- `ignore_columns`: Number of columns to ignore from the left of the CSV table before processing.
+- `remove_empty_rows`: Whether to remove rows that are entirely empty.
+- `remove_empty_columns`: Whether to remove columns that are entirely empty.
+- `keep_id`: Whether to retain the original document ID in the output document.
+Rows and columns ignored using these parameters are preserved in the final output, meaning
+they are not considered when removing empty rows and columns.
+
+<a id="csv_document_cleaner.CSVDocumentCleaner.run"></a>
+
+#### CSVDocumentCleaner.run
+
+```python
+@component.output_types(documents=list[Document])
+def run(documents: list[Document]) -> dict[str, list[Document]]
+```
+
+Cleans CSV documents by removing empty rows and columns while preserving specified ignored rows and columns.
+
+**Arguments**:
+
+- `documents`: List of Documents containing CSV-formatted content.
+
+**Returns**:
+
+A dictionary with a list of cleaned Documents under the key "documents".
+Processing steps:
+1. Reads each document's content as a CSV table.
+2. Retains the specified number of `ignore_rows` from the top and `ignore_columns` from the left.
+3. Drops any rows and columns that are entirely empty (if enabled by `remove_empty_rows` and
+    `remove_empty_columns`).
+4. Reattaches the ignored rows and columns to maintain their original positions.
+5. Returns the cleaned CSV content as a new `Document` object, with an option to retain the original
+    document ID.
+
+<a id="csv_document_splitter"></a>
+
+## Module csv\_document\_splitter
+
+<a id="csv_document_splitter.CSVDocumentSplitter"></a>
+
+### CSVDocumentSplitter
+
+A component for splitting CSV documents into sub-tables based on split arguments.
+
+The splitter supports two modes of operation:
+- identify consecutive empty rows or columns that exceed a given threshold
+and uses them as delimiters to segment the document into smaller tables.
+- split each row into a separate sub-table, represented as a Document.
+
+<a id="csv_document_splitter.CSVDocumentSplitter.__init__"></a>
+
+#### CSVDocumentSplitter.\_\_init\_\_
+
+```python
+def __init__(row_split_threshold: int | None = 2,
+             column_split_threshold: int | None = 2,
+             read_csv_kwargs: dict[str, Any] | None = None,
+             split_mode: SplitMode = "threshold") -> None
+```
+
+Initializes the CSVDocumentSplitter component.
+
+**Arguments**:
+
+- `row_split_threshold`: The minimum number of consecutive empty rows required to trigger a split.
+- `column_split_threshold`: The minimum number of consecutive empty columns required to trigger a split.
+- `read_csv_kwargs`: Additional keyword arguments to pass to `pandas.read_csv`.
+By default, the component with options:
+- `header=None`
+- `skip_blank_lines=False` to preserve blank lines
+- `dtype=object` to prevent type inference (e.g., converting numbers to floats).
+See https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html for more information.
+- `split_mode`: If `threshold`, the component will split the document based on the number of
+consecutive empty rows or columns that exceed the `row_split_threshold` or `column_split_threshold`.
+If `row-wise`, the component will split each row into a separate sub-table.
+
+<a id="csv_document_splitter.CSVDocumentSplitter.run"></a>
+
+#### CSVDocumentSplitter.run
+
+```python
+@component.output_types(documents=list[Document])
+def run(documents: list[Document]) -> dict[str, list[Document]]
+```
+
+Processes and splits a list of CSV documents into multiple sub-tables.
+
+**Splitting Process:**
+1. Applies a row-based split if `row_split_threshold` is provided.
+2. Applies a column-based split if `column_split_threshold` is provided.
+3. If both thresholds are specified, performs a recursive split by rows first, then columns, ensuring
+   further fragmentation of any sub-tables that still contain empty sections.
+4. Sorts the resulting sub-tables based on their original positions within the document.
+
+**Arguments**:
+
+- `documents`: A list of Documents containing CSV-formatted content.
+Each document is assumed to contain one or more tables separated by empty rows or columns.
+
+**Returns**:
+
+A dictionary with a key `"documents"`, mapping to a list of new `Document` objects,
+each representing an extracted sub-table from the original CSV.
+    The metadata of each document includes:
+        - A field `source_id` to track the original document.
+        - A field `row_idx_start` to indicate the starting row index of the sub-table in the original table.
+        - A field `col_idx_start` to indicate the starting column index of the sub-table in the original table.
+        - A field `split_id` to indicate the order of the split in the original document.
+        - All other metadata copied from the original document.
+
+- If a document cannot be processed, it is returned unchanged.
+- The `meta` field from the original document is preserved in the split documents.
+
+<a id="document_cleaner"></a>
+
+## Module document\_cleaner
+
+<a id="document_cleaner.DocumentCleaner"></a>
+
+### DocumentCleaner
+
+Cleans the text in the documents.
+
+It removes extra whitespaces,
+empty lines, specified substrings, regexes,
+page headers and footers (in this order).
+
+### Usage example:
+
+```python
+from haystack import Document
+from haystack.components.preprocessors import DocumentCleaner
+
+doc = Document(content="This   is  a  document  to  clean\n\n\nsubstring to remove")
+
+cleaner = DocumentCleaner(remove_substrings = ["substring to remove"])
+result = cleaner.run(documents=[doc])
+
+assert result["documents"][0].content == "This is a document to clean "
+```
+
+<a id="document_cleaner.DocumentCleaner.__init__"></a>
+
+#### DocumentCleaner.\_\_init\_\_
+
+```python
+def __init__(remove_empty_lines: bool = True,
+             remove_extra_whitespaces: bool = True,
+             remove_repeated_substrings: bool = False,
+             keep_id: bool = False,
+             remove_substrings: list[str] | None = None,
+             remove_regex: str | None = None,
+             unicode_normalization: Literal["NFC", "NFKC", "NFD", "NFKD"]
+             | None = None,
+             ascii_only: bool = False)
+```
+
+Initialize DocumentCleaner.
+
+**Arguments**:
+
+- `remove_empty_lines`: If `True`, removes empty lines.
+- `remove_extra_whitespaces`: If `True`, removes extra whitespaces.
+- `remove_repeated_substrings`: If `True`, removes repeated substrings (headers and footers) from pages.
+Pages must be separated by a form feed character "\f",
+which is supported by `TextFileToDocument` and `AzureOCRDocumentConverter`.
+- `remove_substrings`: List of substrings to remove from the text.
+- `remove_regex`: Regex to match and replace substrings by "".
+- `keep_id`: If `True`, keeps the IDs of the original documents.
+- `unicode_normalization`: Unicode normalization form to apply to the text.
+Note: This will run before any other steps.
+- `ascii_only`: Whether to convert the text to ASCII only.
+Will remove accents from characters and replace them with ASCII characters.
+Other non-ASCII characters will be removed.
+Note: This will run before any pattern matching or removal.
+
+<a id="document_cleaner.DocumentCleaner.run"></a>
+
+#### DocumentCleaner.run
+
+```python
+@component.output_types(documents=list[Document])
+def run(documents: list[Document])
+```
+
+Cleans up the documents.
+
+**Arguments**:
+
+- `documents`: List of Documents to clean.
+
+**Raises**:
+
+- `TypeError`: if documents is not a list of Documents.
+
+**Returns**:
+
+A dictionary with the following key:
+- `documents`: List of cleaned Documents.
+
+<a id="document_preprocessor"></a>
+
+## Module document\_preprocessor
+
+<a id="document_preprocessor.DocumentPreprocessor"></a>
+
+### DocumentPreprocessor
+
+A SuperComponent that first splits and then cleans documents.
+
+This component consists of a DocumentSplitter followed by a DocumentCleaner in a single pipeline.
+It takes a list of documents as input and returns a processed list of documents.
+
+Usage example:
+```python
+from haystack import Document
+from haystack.components.preprocessors import DocumentPreprocessor
+
+doc = Document(content="I love pizza!")
+preprocessor = DocumentPreprocessor()
+result = preprocessor.run(documents=[doc])
+print(result["documents"])
+```
+
+<a id="document_preprocessor.DocumentPreprocessor.__init__"></a>
+
+#### DocumentPreprocessor.\_\_init\_\_
+
+```python
+def __init__(*,
+             split_by: Literal["function", "page", "passage", "period", "word",
+                               "line", "sentence"] = "word",
+             split_length: int = 250,
+             split_overlap: int = 0,
+             split_threshold: int = 0,
+             splitting_function: Callable[[str], list[str]] | None = None,
+             respect_sentence_boundary: bool = False,
+             language: Language = "en",
+             use_split_rules: bool = True,
+             extend_abbreviations: bool = True,
+             remove_empty_lines: bool = True,
+             remove_extra_whitespaces: bool = True,
+             remove_repeated_substrings: bool = False,
+             keep_id: bool = False,
+             remove_substrings: list[str] | None = None,
+             remove_regex: str | None = None,
+             unicode_normalization: Literal["NFC", "NFKC", "NFD", "NFKD"]
+             | None = None,
+             ascii_only: bool = False) -> None
+```
+
+Initialize a DocumentPreProcessor that first splits and then cleans documents.
+
+**Splitter Parameters**:
+
+**Arguments**:
+
+- `split_by`: The unit of splitting: "function", "page", "passage", "period", "word", "line", or "sentence".
+- `split_length`: The maximum number of units (words, lines, pages, and so on) in each split.
+- `split_overlap`: The number of overlapping units between consecutive splits.
+- `split_threshold`: The minimum number of units per split. If a split is smaller than this, it's merged
+with the previous split.
+- `splitting_function`: A custom function for splitting if `split_by="function"`.
+- `respect_sentence_boundary`: If `True`, splits by words but tries not to break inside a sentence.
+- `language`: Language used by the sentence tokenizer if `split_by="sentence"` or
+`respect_sentence_boundary=True`.
+- `use_split_rules`: Whether to apply additional splitting heuristics for the sentence splitter.
+- `extend_abbreviations`: Whether to extend the sentence splitter with curated abbreviations for certain
+languages.
+
+**Cleaner Parameters**:
+- `remove_empty_lines`: If `True`, removes empty lines.
+- `remove_extra_whitespaces`: If `True`, removes extra whitespaces.
+- `remove_repeated_substrings`: If `True`, removes repeated substrings like headers/footers across pages.
+- `keep_id`: If `True`, keeps the original document IDs.
+- `remove_substrings`: A list of strings to remove from the document content.
+- `remove_regex`: A regex pattern whose matches will be removed from the document content.
+- `unicode_normalization`: Unicode normalization form to apply to the text, for example `"NFC"`.
+- `ascii_only`: If `True`, converts text to ASCII only.
+
+<a id="document_preprocessor.DocumentPreprocessor.to_dict"></a>
+
+#### DocumentPreprocessor.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serialize SuperComponent to a dictionary.
+
+**Returns**:
+
+Dictionary with serialized data.
+
+<a id="document_preprocessor.DocumentPreprocessor.from_dict"></a>
+
+#### DocumentPreprocessor.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "DocumentPreprocessor"
+```
+
+Deserializes the SuperComponent from a dictionary.
+
+**Arguments**:
+
+- `data`: Dictionary to deserialize from.
+
+**Returns**:
+
+Deserialized SuperComponent.
+
+<a id="document_splitter"></a>
+
+## Module document\_splitter
+
+<a id="document_splitter.DocumentSplitter"></a>
+
+### DocumentSplitter
+
+Splits long documents into smaller chunks.
+
+This is a common preprocessing step during indexing. It helps Embedders create meaningful semantic representations
+and prevents exceeding language model context limits.
+
+The DocumentSplitter is compatible with the following DocumentStores:
+- [Astra](https://docs.haystack.deepset.ai/docs/astradocumentstore)
+- [Chroma](https://docs.haystack.deepset.ai/docs/chromadocumentstore) limited support, overlapping information is
+  not stored
+- [Elasticsearch](https://docs.haystack.deepset.ai/docs/elasticsearch-document-store)
+- [OpenSearch](https://docs.haystack.deepset.ai/docs/opensearch-document-store)
+- [Pgvector](https://docs.haystack.deepset.ai/docs/pgvectordocumentstore)
+- [Pinecone](https://docs.haystack.deepset.ai/docs/pinecone-document-store) limited support, overlapping
+   information is not stored
+- [Qdrant](https://docs.haystack.deepset.ai/docs/qdrant-document-store)
+- [Weaviate](https://docs.haystack.deepset.ai/docs/weaviatedocumentstore)
+
+### Usage example
+
+```python
+from haystack import Document
+from haystack.components.preprocessors import DocumentSplitter
+
+doc = Document(content="Moonlight shimmered softly, wolves howled nearby, night enveloped everything.")
+
+splitter = DocumentSplitter(split_by="word", split_length=3, split_overlap=0)
+result = splitter.run(documents=[doc])
+```
+
+<a id="document_splitter.DocumentSplitter.__init__"></a>
+
+#### DocumentSplitter.\_\_init\_\_
+
+```python
+def __init__(split_by: Literal["function", "page", "passage", "period", "word",
+                               "line", "sentence"] = "word",
+             split_length: int = 200,
+             split_overlap: int = 0,
+             split_threshold: int = 0,
+             splitting_function: Callable[[str], list[str]] | None = None,
+             respect_sentence_boundary: bool = False,
+             language: Language = "en",
+             use_split_rules: bool = True,
+             extend_abbreviations: bool = True,
+             *,
+             skip_empty_documents: bool = True)
+```
+
+Initialize DocumentSplitter.
+
+**Arguments**:
+
+- `split_by`: The unit for splitting your documents. Choose from:
+- `word` for splitting by spaces (" ")
+- `period` for splitting by periods (".")
+- `page` for splitting by form feed ("\f")
+- `passage` for splitting by double line breaks ("\n\n")
+- `line` for splitting each line ("\n")
+- `sentence` for splitting by NLTK sentence tokenizer
+- `split_length`: The maximum number of units in each split.
+- `split_overlap`: The number of overlapping units for each split.
+- `split_threshold`: The minimum number of units per split. If a split has fewer units
+than the threshold, it's attached to the previous split.
+- `splitting_function`: Necessary when `split_by` is set to "function".
+This is a function which must accept a single `str` as input and return a `list` of `str` as output,
+representing the chunks after splitting.
+- `respect_sentence_boundary`: Choose whether to respect sentence boundaries when splitting by "word".
+If True, uses NLTK to detect sentence boundaries, ensuring splits occur only between sentences.
+- `language`: Choose the language for the NLTK tokenizer. The default is English ("en").
+- `use_split_rules`: Choose whether to use additional split rules when splitting by `sentence`.
+- `extend_abbreviations`: Choose whether to extend NLTK's PunktTokenizer abbreviations with a list
+of curated abbreviations, if available. This is currently supported for English ("en") and German ("de").
+- `skip_empty_documents`: Choose whether to skip documents with empty content. Default is True.
+Set to False when downstream components in the Pipeline (like LLMDocumentContentExtractor) can extract text
+from non-textual documents.
+
+<a id="document_splitter.DocumentSplitter.warm_up"></a>
+
+#### DocumentSplitter.warm\_up
+
+```python
+def warm_up()
+```
+
+Warm up the DocumentSplitter by loading the sentence tokenizer.
+
+<a id="document_splitter.DocumentSplitter.run"></a>
+
+#### DocumentSplitter.run
+
+```python
+@component.output_types(documents=list[Document])
+def run(documents: list[Document])
+```
+
+Split documents into smaller parts.
+
+Splits documents by the unit expressed in `split_by`, with a length of `split_length`
+and an overlap of `split_overlap`.
+
+**Arguments**:
+
+- `documents`: The documents to split.
+
+**Raises**:
+
+- `TypeError`: if the input is not a list of Documents.
+- `ValueError`: if the content of a document is None.
+
+**Returns**:
+
+A dictionary with the following key:
+- `documents`: List of documents with the split texts. Each document includes:
+- A metadata field `source_id` to track the original document.
+- A metadata field `page_number` to track the original page number.
+- All other metadata copied from the original document.
+
+<a id="document_splitter.DocumentSplitter.to_dict"></a>
+
+#### DocumentSplitter.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+<a id="document_splitter.DocumentSplitter.from_dict"></a>
+
+#### DocumentSplitter.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "DocumentSplitter"
+```
+
+Deserializes the component from a dictionary.
+
+<a id="hierarchical_document_splitter"></a>
+
+## Module hierarchical\_document\_splitter
+
+<a id="hierarchical_document_splitter.HierarchicalDocumentSplitter"></a>
+
+### HierarchicalDocumentSplitter
+
+Splits a documents into different block sizes building a hierarchical tree structure of blocks of different sizes.
+
+The root node of the tree is the original document, the leaf nodes are the smallest blocks. The blocks in between
+are connected such that the smaller blocks are children of the parent-larger blocks.
+
+## Usage example
+```python
+from haystack import Document
+from haystack.components.preprocessors import HierarchicalDocumentSplitter
+
+doc = Document(content="This is a simple test document")
+splitter = HierarchicalDocumentSplitter(block_sizes={3, 2}, split_overlap=0, split_by="word")
+splitter.run([doc])
+>> {'documents': [Document(id=3f7..., content: 'This is a simple test document', meta: {'block_size': 0, 'parent_id': None, 'children_ids': ['5ff..', '8dc..'], 'level': 0}),
+>> Document(id=5ff.., content: 'This is a ', meta: {'block_size': 3, 'parent_id': '3f7..', 'children_ids': ['f19..', '52c..'], 'level': 1, 'source_id': '3f7..', 'page_number': 1, 'split_id': 0, 'split_idx_start': 0}),
+>> Document(id=8dc.., content: 'simple test document', meta: {'block_size': 3, 'parent_id': '3f7..', 'children_ids': ['39d..', 'e23..'], 'level': 1, 'source_id': '3f7..', 'page_number': 1, 'split_id': 1, 'split_idx_start': 10}),
+>> Document(id=f19.., content: 'This is ', meta: {'block_size': 2, 'parent_id': '5ff..', 'children_ids': [], 'level': 2, 'source_id': '5ff..', 'page_number': 1, 'split_id': 0, 'split_idx_start': 0}),
+>> Document(id=52c.., content: 'a ', meta: {'block_size': 2, 'parent_id': '5ff..', 'children_ids': [], 'level': 2, 'source_id': '5ff..', 'page_number': 1, 'split_id': 1, 'split_idx_start': 8}),
+>> Document(id=39d.., content: 'simple test ', meta: {'block_size': 2, 'parent_id': '8dc..', 'children_ids': [], 'level': 2, 'source_id': '8dc..', 'page_number': 1, 'split_id': 0, 'split_idx_start': 0}),
+>> Document(id=e23.., content: 'document', meta: {'block_size': 2, 'parent_id': '8dc..', 'children_ids': [], 'level': 2, 'source_id': '8dc..', 'page_number': 1, 'split_id': 1, 'split_idx_start': 12})]}
+```
+
+<a id="hierarchical_document_splitter.HierarchicalDocumentSplitter.__init__"></a>
+
+#### HierarchicalDocumentSplitter.\_\_init\_\_
+
+```python
+def __init__(block_sizes: set[int],
+             split_overlap: int = 0,
+             split_by: Literal["word", "sentence", "page",
+                               "passage"] = "word")
+```
+
+Initialize HierarchicalDocumentSplitter.
+
+**Arguments**:
+
+- `block_sizes`: Set of block sizes to split the document into. The blocks are split in descending order.
+- `split_overlap`: The number of overlapping units for each split.
+- `split_by`: The unit for splitting your documents.
+
+<a id="hierarchical_document_splitter.HierarchicalDocumentSplitter.run"></a>
+
+#### HierarchicalDocumentSplitter.run
+
+```python
+@component.output_types(documents=list[Document])
+def run(documents: list[Document])
+```
+
+Builds a hierarchical document structure for each document in a list of documents.
+
+**Arguments**:
+
+- `documents`: List of Documents to split into hierarchical blocks.
+
+**Returns**:
+
+List of HierarchicalDocument
+
+<a id="hierarchical_document_splitter.HierarchicalDocumentSplitter.build_hierarchy_from_doc"></a>
+
+#### HierarchicalDocumentSplitter.build\_hierarchy\_from\_doc
+
+```python
+def build_hierarchy_from_doc(document: Document) -> list[Document]
+```
+
+Build a hierarchical tree document structure from a single document.
+
+Given a document, this function splits the document into hierarchical blocks of different sizes represented
+as HierarchicalDocument objects.
+
+**Arguments**:
+
+- `document`: Document to split into hierarchical blocks.
+
+**Returns**:
+
+List of HierarchicalDocument
+
+<a id="hierarchical_document_splitter.HierarchicalDocumentSplitter.to_dict"></a>
+
+#### HierarchicalDocumentSplitter.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Returns a dictionary representation of the component.
+
+**Returns**:
+
+Serialized dictionary representation of the component.
+
+<a id="hierarchical_document_splitter.HierarchicalDocumentSplitter.from_dict"></a>
+
+#### HierarchicalDocumentSplitter.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "HierarchicalDocumentSplitter"
+```
+
+Deserialize this component from a dictionary.
+
+**Arguments**:
+
+- `data`: The dictionary to deserialize and create the component.
+
+**Returns**:
+
+The deserialized component.
+
+<a id="recursive_splitter"></a>
+
+## Module recursive\_splitter
+
+<a id="recursive_splitter.RecursiveDocumentSplitter"></a>
+
+### RecursiveDocumentSplitter
+
+Recursively chunk text into smaller chunks.
+
+This component is used to split text into smaller chunks, it does so by recursively applying a list of separators
+to the text.
+
+The separators are applied in the order they are provided, typically this is a list of separators that are
+applied in a specific order, being the last separator the most specific one.
+
+Each separator is applied to the text, it then checks each of the resulting chunks, it keeps the chunks that
+are within the split_length, for the ones that are larger than the split_length, it applies the next separator in the
+list to the remaining text.
+
+This is done until all chunks are smaller than the split_length parameter.
+
+**Example**:
+
+  
+```python
+from haystack import Document
+from haystack.components.preprocessors import RecursiveDocumentSplitter
+
+chunker = RecursiveDocumentSplitter(split_length=260, split_overlap=0, separators=["\n\n", "\n", ".", " "])
+text = ('''Artificial intelligence (AI) - Introduction
+
+AI, in its broadest sense, is intelligence exhibited by machines, particularly computer systems.
+AI technology is widely used throughout industry, government, and science. Some high-profile applications include advanced web search engines; recommendation systems; interacting via human speech; autonomous vehicles; generative and creative tools; and superhuman play and analysis in strategy games.''')
+chunker.warm_up()
+doc = Document(content=text)
+doc_chunks = chunker.run([doc])
+print(doc_chunks["documents"])
+>[
+>Document(id=..., content: 'Artificial intelligence (AI) - Introduction\n\n', meta: {'original_id': '...', 'split_id': 0, 'split_idx_start': 0, '_split_overlap': []})
+>Document(id=..., content: 'AI, in its broadest sense, is intelligence exhibited by machines, particularly computer systems.\n', meta: {'original_id': '...', 'split_id': 1, 'split_idx_start': 45, '_split_overlap': []})
+>Document(id=..., content: 'AI technology is widely used throughout industry, government, and science.', meta: {'original_id': '...', 'split_id': 2, 'split_idx_start': 142, '_split_overlap': []})
+>Document(id=..., content: ' Some high-profile applications include advanced web search engines; recommendation systems; interac...', meta: {'original_id': '...', 'split_id': 3, 'split_idx_start': 216, '_split_overlap': []})
+>]
+```
+
+<a id="recursive_splitter.RecursiveDocumentSplitter.__init__"></a>
+
+#### RecursiveDocumentSplitter.\_\_init\_\_
+
+```python
+def __init__(*,
+             split_length: int = 200,
+             split_overlap: int = 0,
+             split_unit: Literal["word", "char", "token"] = "word",
+             separators: list[str] | None = None,
+             sentence_splitter_params: dict[str, Any] | None = None)
+```
+
+Initializes a RecursiveDocumentSplitter.
+
+**Arguments**:
+
+- `split_length`: The maximum length of each chunk by default in words, but can be in characters or tokens.
+See the `split_units` parameter.
+- `split_overlap`: The number of characters to overlap between consecutive chunks.
+- `split_unit`: The unit of the split_length parameter. It can be either "word", "char", or "token".
+If "token" is selected, the text will be split into tokens using the tiktoken tokenizer (o200k_base).
+- `separators`: An optional list of separator strings to use for splitting the text. The string
+separators will be treated as regular expressions unless the separator is "sentence", in that case the
+text will be split into sentences using a custom sentence tokenizer based on NLTK.
+See: haystack.components.preprocessors.sentence_tokenizer.SentenceSplitter.
+If no separators are provided, the default separators ["\n\n", "sentence", "\n", " "] are used.
+- `sentence_splitter_params`: Optional parameters to pass to the sentence tokenizer.
+See: haystack.components.preprocessors.sentence_tokenizer.SentenceSplitter for more information.
+
+**Raises**:
+
+- `ValueError`: If the overlap is greater than or equal to the chunk size or if the overlap is negative, or
+if any separator is not a string.
+
+<a id="recursive_splitter.RecursiveDocumentSplitter.warm_up"></a>
+
+#### RecursiveDocumentSplitter.warm\_up
+
+```python
+def warm_up() -> None
+```
+
+Warm up the sentence tokenizer and tiktoken tokenizer if needed.
+
+<a id="recursive_splitter.RecursiveDocumentSplitter.run"></a>
+
+#### RecursiveDocumentSplitter.run
+
+```python
+@component.output_types(documents=list[Document])
+def run(documents: list[Document]) -> dict[str, list[Document]]
+```
+
+Split a list of documents into documents with smaller chunks of text.
+
+**Arguments**:
+
+- `documents`: List of Documents to split.
+
+**Returns**:
+
+A dictionary containing a key "documents" with a List of Documents with smaller chunks of text corresponding
+to the input documents.
+
+<a id="text_cleaner"></a>
+
+## Module text\_cleaner
+
+<a id="text_cleaner.TextCleaner"></a>
+
+### TextCleaner
+
+Cleans text strings.
+
+It can remove substrings matching a list of regular expressions, convert text to lowercase,
+remove punctuation, and remove numbers.
+Use it to clean up text data before evaluation.
+
+### Usage example
+
+```python
+from haystack.components.preprocessors import TextCleaner
+
+text_to_clean = "1Moonlight shimmered softly, 300 Wolves howled nearby, Night enveloped everything."
+
+cleaner = TextCleaner(convert_to_lowercase=True, remove_punctuation=False, remove_numbers=True)
+result = cleaner.run(texts=[text_to_clean])
+```
+
+<a id="text_cleaner.TextCleaner.__init__"></a>
+
+#### TextCleaner.\_\_init\_\_
+
+```python
+def __init__(remove_regexps: list[str] | None = None,
+             convert_to_lowercase: bool = False,
+             remove_punctuation: bool = False,
+             remove_numbers: bool = False)
+```
+
+Initializes the TextCleaner component.
+
+**Arguments**:
+
+- `remove_regexps`: A list of regex patterns to remove matching substrings from the text.
+- `convert_to_lowercase`: If `True`, converts all characters to lowercase.
+- `remove_punctuation`: If `True`, removes punctuation from the text.
+- `remove_numbers`: If `True`, removes numerical digits from the text.
+
+<a id="text_cleaner.TextCleaner.run"></a>
+
+#### TextCleaner.run
+
+```python
+@component.output_types(texts=list[str])
+def run(texts: list[str]) -> dict[str, Any]
+```
+
+Cleans up the given list of strings.
+
+**Arguments**:
+
+- `texts`: List of strings to clean.
+
+**Returns**:
+
+A dictionary with the following key:
+- `texts`:  the cleaned list of strings.
+

docs-website/reference_versioned_docs/version-2.22/haystack-api/query_api.md

@@ -0,0 +1,142 @@
+---
+title: "Query"
+id: query-api
+description: "Components for query processing and expansion."
+slug: "/query-api"
+---
+
+<a id="query_expander"></a>
+
+## Module query\_expander
+
+<a id="query_expander.QueryExpander"></a>
+
+### QueryExpander
+
+A component that returns a list of semantically similar queries to improve retrieval recall in RAG systems.
+
+The component uses a chat generator to expand queries. The chat generator is expected to return a JSON response
+with the following structure:
+
+### Usage example
+
+```json
+{"queries": ["expanded query 1", "expanded query 2", "expanded query 3"]}
+```
+```python
+from haystack.components.generators.chat.openai import OpenAIChatGenerator
+from haystack.components.query import QueryExpander
+
+expander = QueryExpander(
+    chat_generator=OpenAIChatGenerator(model="gpt-4.1-mini"),
+    n_expansions=3
+)
+
+result = expander.run(query="green energy sources")
+print(result["queries"])
+# Output: ['alternative query 1', 'alternative query 2', 'alternative query 3', 'green energy sources']
+# Note: Up to 3 additional queries + 1 original query (if include_original_query=True)
+
+# To control total number of queries:
+expander = QueryExpander(n_expansions=2, include_original_query=True)  # Up to 3 total
+# or
+expander = QueryExpander(n_expansions=3, include_original_query=False)  # Exactly 3 total
+```
+
+<a id="query_expander.QueryExpander.__init__"></a>
+
+#### QueryExpander.\_\_init\_\_
+
+```python
+def __init__(*,
+             chat_generator: ChatGenerator | None = None,
+             prompt_template: str | None = None,
+             n_expansions: int = 4,
+             include_original_query: bool = True) -> None
+```
+
+Initialize the QueryExpander component.
+
+**Arguments**:
+
+- `chat_generator`: The chat generator component to use for query expansion.
+If None, a default OpenAIChatGenerator with gpt-4.1-mini model is used.
+- `prompt_template`: Custom [PromptBuilder](https://docs.haystack.deepset.ai/docs/promptbuilder)
+template for query expansion. The template should instruct the LLM to return a JSON response with the
+structure: `{"queries": ["query1", "query2", "query3"]}`. The template should include 'query' and
+'n_expansions' variables.
+- `n_expansions`: Number of alternative queries to generate (default: 4).
+- `include_original_query`: Whether to include the original query in the output.
+
+<a id="query_expander.QueryExpander.to_dict"></a>
+
+#### QueryExpander.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+**Returns**:
+
+Dictionary with serialized data.
+
+<a id="query_expander.QueryExpander.from_dict"></a>
+
+#### QueryExpander.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "QueryExpander"
+```
+
+Deserializes the component from a dictionary.
+
+**Arguments**:
+
+- `data`: Dictionary with serialized data.
+
+**Returns**:
+
+Deserialized component.
+
+<a id="query_expander.QueryExpander.run"></a>
+
+#### QueryExpander.run
+
+```python
+@component.output_types(queries=list[str])
+def run(query: str, n_expansions: int | None = None) -> dict[str, list[str]]
+```
+
+Expand the input query into multiple semantically similar queries.
+
+The language of the original query is preserved in the expanded queries.
+
+**Arguments**:
+
+- `query`: The original query to expand.
+- `n_expansions`: Number of additional queries to generate (not including the original).
+If None, uses the value from initialization. Can be 0 to generate no additional queries.
+
+**Raises**:
+
+- `ValueError`: If n_expansions is not positive (less than or equal to 0).
+
+**Returns**:
+
+Dictionary with "queries" key containing the list of expanded queries.
+If include_original_query=True, the original query will be included in addition
+to the n_expansions alternative queries.
+
+<a id="query_expander.QueryExpander.warm_up"></a>
+
+#### QueryExpander.warm\_up
+
+```python
+def warm_up()
+```
+
+Warm up the LLM provider component.
+

docs-website/reference_versioned_docs/version-2.22/haystack-api/readers_api.md

@@ -0,0 +1,206 @@
+---
+title: "Readers"
+id: readers-api
+description: "Takes a query and a set of Documents as input and returns ExtractedAnswers by selecting a text span within the Documents."
+slug: "/readers-api"
+---
+
+<a id="extractive"></a>
+
+## Module extractive
+
+<a id="extractive.ExtractiveReader"></a>
+
+### ExtractiveReader
+
+Locates and extracts answers to a given query from Documents.
+
+The ExtractiveReader component performs extractive question answering.
+It assigns a score to every possible answer span independently of other answer spans.
+This fixes a common issue of other implementations which make comparisons across documents harder by normalizing
+each document's answers independently.
+
+Example usage:
+```python
+from haystack import Document
+from haystack.components.readers import ExtractiveReader
+
+docs = [
+    Document(content="Python is a popular programming language"),
+    Document(content="python ist eine beliebte Programmiersprache"),
+]
+
+reader = ExtractiveReader()
+reader.warm_up()
+
+question = "What is a popular programming language?"
+result = reader.run(query=question, documents=docs)
+assert "Python" in result["answers"][0].data
+```
+
+<a id="extractive.ExtractiveReader.__init__"></a>
+
+#### ExtractiveReader.\_\_init\_\_
+
+```python
+def __init__(model: Path | str = "deepset/roberta-base-squad2-distilled",
+             device: ComponentDevice | None = None,
+             token: Secret | None = Secret.from_env_var(
+                 ["HF_API_TOKEN", "HF_TOKEN"], strict=False),
+             top_k: int = 20,
+             score_threshold: float | None = None,
+             max_seq_length: int = 384,
+             stride: int = 128,
+             max_batch_size: int | None = None,
+             answers_per_seq: int | None = None,
+             no_answer: bool = True,
+             calibration_factor: float = 0.1,
+             overlap_threshold: float | None = 0.01,
+             model_kwargs: dict[str, Any] | None = None) -> None
+```
+
+Creates an instance of ExtractiveReader.
+
+**Arguments**:
+
+- `model`: A Hugging Face transformers question answering model.
+Can either be a path to a folder containing the model files or an identifier for the Hugging Face hub.
+- `device`: The device on which the model is loaded. If `None`, the default device is automatically selected.
+- `token`: The API token used to download private models from Hugging Face.
+- `top_k`: Number of answers to return per query. It is required even if score_threshold is set.
+An additional answer with no text is returned if no_answer is set to True (default).
+- `score_threshold`: Returns only answers with the probability score above this threshold.
+- `max_seq_length`: Maximum number of tokens. If a sequence exceeds it, the sequence is split.
+- `stride`: Number of tokens that overlap when sequence is split because it exceeds max_seq_length.
+- `max_batch_size`: Maximum number of samples that are fed through the model at the same time.
+- `answers_per_seq`: Number of answer candidates to consider per sequence.
+This is relevant when a Document was split into multiple sequences because of max_seq_length.
+- `no_answer`: Whether to return an additional `no answer` with an empty text and a score representing the
+probability that the other top_k answers are incorrect.
+- `calibration_factor`: Factor used for calibrating probabilities.
+- `overlap_threshold`: If set this will remove duplicate answers if they have an overlap larger than the
+supplied threshold. For example, for the answers "in the river in Maine" and "the river" we would remove
+one of these answers since the second answer has a 100% (1.0) overlap with the first answer.
+However, for the answers "the river in" and "in Maine" there is only a max overlap percentage of 25% so
+both of these answers could be kept if this variable is set to 0.24 or lower.
+If None is provided then all answers are kept.
+- `model_kwargs`: Additional keyword arguments passed to `AutoModelForQuestionAnswering.from_pretrained`
+when loading the model specified in `model`. For details on what kwargs you can pass,
+see the model's documentation.
+
+<a id="extractive.ExtractiveReader.to_dict"></a>
+
+#### ExtractiveReader.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+**Returns**:
+
+Dictionary with serialized data.
+
+<a id="extractive.ExtractiveReader.from_dict"></a>
+
+#### ExtractiveReader.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "ExtractiveReader"
+```
+
+Deserializes the component from a dictionary.
+
+**Arguments**:
+
+- `data`: Dictionary to deserialize from.
+
+**Returns**:
+
+Deserialized component.
+
+<a id="extractive.ExtractiveReader.warm_up"></a>
+
+#### ExtractiveReader.warm\_up
+
+```python
+def warm_up()
+```
+
+Initializes the component.
+
+<a id="extractive.ExtractiveReader.deduplicate_by_overlap"></a>
+
+#### ExtractiveReader.deduplicate\_by\_overlap
+
+```python
+def deduplicate_by_overlap(
+        answers: list[ExtractedAnswer],
+        overlap_threshold: float | None) -> list[ExtractedAnswer]
+```
+
+De-duplicates overlapping Extractive Answers.
+
+De-duplicates overlapping Extractive Answers from the same document based on how much the spans of the
+answers overlap.
+
+**Arguments**:
+
+- `answers`: List of answers to be deduplicated.
+- `overlap_threshold`: If set this will remove duplicate answers if they have an overlap larger than the
+supplied threshold. For example, for the answers "in the river in Maine" and "the river" we would remove
+one of these answers since the second answer has a 100% (1.0) overlap with the first answer.
+However, for the answers "the river in" and "in Maine" there is only a max overlap percentage of 25% so
+both of these answers could be kept if this variable is set to 0.24 or lower.
+If None is provided then all answers are kept.
+
+**Returns**:
+
+List of deduplicated answers.
+
+<a id="extractive.ExtractiveReader.run"></a>
+
+#### ExtractiveReader.run
+
+```python
+@component.output_types(answers=list[ExtractedAnswer])
+def run(query: str,
+        documents: list[Document],
+        top_k: int | None = None,
+        score_threshold: float | None = None,
+        max_seq_length: int | None = None,
+        stride: int | None = None,
+        max_batch_size: int | None = None,
+        answers_per_seq: int | None = None,
+        no_answer: bool | None = None,
+        overlap_threshold: float | None = None)
+```
+
+Locates and extracts answers from the given Documents using the given query.
+
+**Arguments**:
+
+- `query`: Query string.
+- `documents`: List of Documents in which you want to search for an answer to the query.
+- `top_k`: The maximum number of answers to return.
+An additional answer is returned if no_answer is set to True (default).
+- `score_threshold`: Returns only answers with the score above this threshold.
+- `max_seq_length`: Maximum number of tokens. If a sequence exceeds it, the sequence is split.
+- `stride`: Number of tokens that overlap when sequence is split because it exceeds max_seq_length.
+- `max_batch_size`: Maximum number of samples that are fed through the model at the same time.
+- `answers_per_seq`: Number of answer candidates to consider per sequence.
+This is relevant when a Document was split into multiple sequences because of max_seq_length.
+- `no_answer`: Whether to return no answer scores.
+- `overlap_threshold`: If set this will remove duplicate answers if they have an overlap larger than the
+supplied threshold. For example, for the answers "in the river in Maine" and "the river" we would remove
+one of these answers since the second answer has a 100% (1.0) overlap with the first answer.
+However, for the answers "the river in" and "in Maine" there is only a max overlap percentage of 25% so
+both of these answers could be kept if this variable is set to 0.24 or lower.
+If None is provided then all answers are kept.
+
+**Returns**:
+
+List of answers sorted by (desc.) answer score.
+

docs-website/reference_versioned_docs/version-2.22/haystack-api/samplers_api.md

@@ -0,0 +1,88 @@
+---
+title: "Samplers"
+id: samplers-api
+description: "Filters documents based on their similarity scores using top-p sampling."
+slug: "/samplers-api"
+---
+
+<a id="top_p"></a>
+
+## Module top\_p
+
+<a id="top_p.TopPSampler"></a>
+
+### TopPSampler
+
+Implements top-p (nucleus) sampling for document filtering based on cumulative probability scores.
+
+This component provides functionality to filter a list of documents by selecting those whose scores fall
+within the top 'p' percent of the cumulative distribution. It is useful for focusing on high-probability
+documents while filtering out less relevant ones based on their assigned scores.
+
+Usage example:
+
+```python
+from haystack import Document
+from haystack.components.samplers import TopPSampler
+
+sampler = TopPSampler(top_p=0.95, score_field="similarity_score")
+docs = [
+    Document(content="Berlin", meta={"similarity_score": -10.6}),
+    Document(content="Belgrade", meta={"similarity_score": -8.9}),
+    Document(content="Sarajevo", meta={"similarity_score": -4.6}),
+]
+output = sampler.run(documents=docs)
+docs = output["documents"]
+assert len(docs) == 1
+assert docs[0].content == "Sarajevo"
+```
+
+<a id="top_p.TopPSampler.__init__"></a>
+
+#### TopPSampler.\_\_init\_\_
+
+```python
+def __init__(top_p: float = 1.0,
+             score_field: str | None = None,
+             min_top_k: int | None = None)
+```
+
+Creates an instance of TopPSampler.
+
+**Arguments**:
+
+- `top_p`: Float between 0 and 1 representing the cumulative probability threshold for document selection.
+A value of 1.0 indicates no filtering (all documents are retained).
+- `score_field`: Name of the field in each document's metadata that contains the score. If None, the default
+document score field is used.
+- `min_top_k`: If specified, the minimum number of documents to return. If the top_p selects
+fewer documents, additional ones with the next highest scores are added to the selection.
+
+<a id="top_p.TopPSampler.run"></a>
+
+#### TopPSampler.run
+
+```python
+@component.output_types(documents=list[Document])
+def run(documents: list[Document], top_p: float | None = None)
+```
+
+Filters documents using top-p sampling based on their scores.
+
+If the specified top_p results in no documents being selected (especially in cases of a low top_p value), the
+method returns the document with the highest score.
+
+**Arguments**:
+
+- `documents`: List of Document objects to be filtered.
+- `top_p`: If specified, a float to override the cumulative probability threshold set during initialization.
+
+**Raises**:
+
+- `ValueError`: If the top_p value is not within the range [0, 1].
+
+**Returns**:
+
+A dictionary with the following key:
+- `documents`: List of Document objects that have been selected based on the top-p sampling.
+

docs-website/reference_versioned_docs/version-2.22/haystack-api/tool_components_api.md

@@ -0,0 +1,323 @@
+---
+title: "Tool Components"
+id: tool-components-api
+description: "Components related to Tool Calling."
+slug: "/tool-components-api"
+---
+
+<a id="tool_invoker"></a>
+
+## Module tool\_invoker
+
+<a id="tool_invoker.ToolInvokerError"></a>
+
+### ToolInvokerError
+
+Base exception class for ToolInvoker errors.
+
+<a id="tool_invoker.ToolNotFoundException"></a>
+
+### ToolNotFoundException
+
+Exception raised when a tool is not found in the list of available tools.
+
+<a id="tool_invoker.StringConversionError"></a>
+
+### StringConversionError
+
+Exception raised when the conversion of a tool result to a string fails.
+
+<a id="tool_invoker.ToolOutputMergeError"></a>
+
+### ToolOutputMergeError
+
+Exception raised when merging tool outputs into state fails.
+
+<a id="tool_invoker.ToolOutputMergeError.from_exception"></a>
+
+#### ToolOutputMergeError.from\_exception
+
+```python
+@classmethod
+def from_exception(cls, tool_name: str,
+                   error: Exception) -> "ToolOutputMergeError"
+```
+
+Create a ToolOutputMergeError from an exception.
+
+<a id="tool_invoker.ToolInvoker"></a>
+
+### ToolInvoker
+
+Invokes tools based on prepared tool calls and returns the results as a list of ChatMessage objects.
+
+Also handles reading/writing from a shared `State`.
+At initialization, the ToolInvoker component is provided with a list of available tools.
+At runtime, the component processes a list of ChatMessage object containing tool calls
+and invokes the corresponding tools.
+The results of the tool invocations are returned as a list of ChatMessage objects with tool role.
+
+Usage example:
+```python
+from haystack.dataclasses import ChatMessage, ToolCall
+from haystack.tools import Tool
+from haystack.components.tools import ToolInvoker
+
+# Tool definition
+def dummy_weather_function(city: str):
+    return f"The weather in {city} is 20 degrees."
+
+parameters = {"type": "object",
+            "properties": {"city": {"type": "string"}},
+            "required": ["city"]}
+
+tool = Tool(name="weather_tool",
+            description="A tool to get the weather",
+            function=dummy_weather_function,
+            parameters=parameters)
+
+# Usually, the ChatMessage with tool_calls is generated by a Language Model
+# Here, we create it manually for demonstration purposes
+tool_call = ToolCall(
+    tool_name="weather_tool",
+    arguments={"city": "Berlin"}
+)
+message = ChatMessage.from_assistant(tool_calls=[tool_call])
+
+# ToolInvoker initialization and run
+invoker = ToolInvoker(tools=[tool])
+result = invoker.run(messages=[message])
+
+print(result)
+```
+
+```
+>>  {
+>>      'tool_messages': [
+>>          ChatMessage(
+>>              _role=<ChatRole.TOOL: 'tool'>,
+>>              _content=[
+>>                  ToolCallResult(
+>>                      result='"The weather in Berlin is 20 degrees."',
+>>                      origin=ToolCall(
+>>                          tool_name='weather_tool',
+>>                          arguments={'city': 'Berlin'},
+>>                          id=None
+>>                      )
+>>                  )
+>>              ],
+>>              _meta={}
+>>          )
+>>      ]
+>>  }
+```
+
+Usage example with a Toolset:
+```python
+from haystack.dataclasses import ChatMessage, ToolCall
+from haystack.tools import Tool, Toolset
+from haystack.components.tools import ToolInvoker
+
+# Tool definition
+def dummy_weather_function(city: str):
+    return f"The weather in {city} is 20 degrees."
+
+parameters = {"type": "object",
+            "properties": {"city": {"type": "string"}},
+            "required": ["city"]}
+
+tool = Tool(name="weather_tool",
+            description="A tool to get the weather",
+            function=dummy_weather_function,
+            parameters=parameters)
+
+# Create a Toolset
+toolset = Toolset([tool])
+
+# Usually, the ChatMessage with tool_calls is generated by a Language Model
+# Here, we create it manually for demonstration purposes
+tool_call = ToolCall(
+    tool_name="weather_tool",
+    arguments={"city": "Berlin"}
+)
+message = ChatMessage.from_assistant(tool_calls=[tool_call])
+
+# ToolInvoker initialization and run with Toolset
+invoker = ToolInvoker(tools=toolset)
+result = invoker.run(messages=[message])
+
+print(result)
+
+<a id="tool_invoker.ToolInvoker.__init__"></a>
+
+#### ToolInvoker.\_\_init\_\_
+
+```python
+def __init__(tools: ToolsType,
+             raise_on_failure: bool = True,
+             convert_result_to_json_string: bool = False,
+             streaming_callback: StreamingCallbackT | None = None,
+             *,
+             enable_streaming_callback_passthrough: bool = False,
+             max_workers: int = 4)
+```
+
+Initialize the ToolInvoker component.
+
+**Arguments**:
+
+- `tools`: A list of Tool and/or Toolset objects, or a Toolset instance that can resolve tools.
+- `raise_on_failure`: If True, the component will raise an exception in case of errors
+(tool not found, tool invocation errors, tool result conversion errors).
+If False, the component will return a ChatMessage object with `error=True`
+and a description of the error in `result`.
+- `convert_result_to_json_string`: If True, the tool invocation result will be converted to a string using `json.dumps`.
+If False, the tool invocation result will be converted to a string using `str`.
+- `streaming_callback`: A callback function that will be called to emit tool results.
+Note that the result is only emitted once it becomes available — it is not
+streamed incrementally in real time.
+- `enable_streaming_callback_passthrough`: If True, the `streaming_callback` will be passed to the tool invocation if the tool supports it.
+This allows tools to stream their results back to the client.
+Note that this requires the tool to have a `streaming_callback` parameter in its `invoke` method signature.
+If False, the `streaming_callback` will not be passed to the tool invocation.
+- `max_workers`: The maximum number of workers to use in the thread pool executor.
+This also decides the maximum number of concurrent tool invocations.
+
+**Raises**:
+
+- `ValueError`: If no tools are provided or if duplicate tool names are found.
+
+<a id="tool_invoker.ToolInvoker.warm_up"></a>
+
+#### ToolInvoker.warm\_up
+
+```python
+def warm_up()
+```
+
+Warm up the tool invoker.
+
+This will warm up the tools registered in the tool invoker.
+This method is idempotent and will only warm up the tools once.
+
+<a id="tool_invoker.ToolInvoker.run"></a>
+
+#### ToolInvoker.run
+
+```python
+@component.output_types(tool_messages=list[ChatMessage], state=State)
+def run(messages: list[ChatMessage],
+        state: State | None = None,
+        streaming_callback: StreamingCallbackT | None = None,
+        *,
+        enable_streaming_callback_passthrough: bool | None = None,
+        tools: ToolsType | None = None) -> dict[str, Any]
+```
+
+Processes ChatMessage objects containing tool calls and invokes the corresponding tools, if available.
+
+**Arguments**:
+
+- `messages`: A list of ChatMessage objects.
+- `state`: The runtime state that should be used by the tools.
+- `streaming_callback`: A callback function that will be called to emit tool results.
+Note that the result is only emitted once it becomes available — it is not
+streamed incrementally in real time.
+- `enable_streaming_callback_passthrough`: If True, the `streaming_callback` will be passed to the tool invocation if the tool supports it.
+This allows tools to stream their results back to the client.
+Note that this requires the tool to have a `streaming_callback` parameter in its `invoke` method signature.
+If False, the `streaming_callback` will not be passed to the tool invocation.
+If None, the value from the constructor will be used.
+- `tools`: A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
+If set, it will override the `tools` parameter provided during initialization.
+
+**Raises**:
+
+- `ToolNotFoundException`: If the tool is not found in the list of available tools and `raise_on_failure` is True.
+- `ToolInvocationError`: If the tool invocation fails and `raise_on_failure` is True.
+- `StringConversionError`: If the conversion of the tool result to a string fails and `raise_on_failure` is True.
+- `ToolOutputMergeError`: If merging tool outputs into state fails and `raise_on_failure` is True.
+
+**Returns**:
+
+A dictionary with the key `tool_messages` containing a list of ChatMessage objects with tool role.
+Each ChatMessage objects wraps the result of a tool invocation.
+
+<a id="tool_invoker.ToolInvoker.run_async"></a>
+
+#### ToolInvoker.run\_async
+
+```python
+@component.output_types(tool_messages=list[ChatMessage], state=State)
+async def run_async(messages: list[ChatMessage],
+                    state: State | None = None,
+                    streaming_callback: StreamingCallbackT | None = None,
+                    *,
+                    enable_streaming_callback_passthrough: bool | None = None,
+                    tools: ToolsType | None = None) -> dict[str, Any]
+```
+
+Asynchronously processes ChatMessage objects containing tool calls.
+
+Multiple tool calls are performed concurrently.
+
+**Arguments**:
+
+- `messages`: A list of ChatMessage objects.
+- `state`: The runtime state that should be used by the tools.
+- `streaming_callback`: An asynchronous callback function that will be called to emit tool results.
+Note that the result is only emitted once it becomes available — it is not
+streamed incrementally in real time.
+- `enable_streaming_callback_passthrough`: If True, the `streaming_callback` will be passed to the tool invocation if the tool supports it.
+This allows tools to stream their results back to the client.
+Note that this requires the tool to have a `streaming_callback` parameter in its `invoke` method signature.
+If False, the `streaming_callback` will not be passed to the tool invocation.
+If None, the value from the constructor will be used.
+- `tools`: A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
+If set, it will override the `tools` parameter provided during initialization.
+
+**Raises**:
+
+- `ToolNotFoundException`: If the tool is not found in the list of available tools and `raise_on_failure` is True.
+- `ToolInvocationError`: If the tool invocation fails and `raise_on_failure` is True.
+- `StringConversionError`: If the conversion of the tool result to a string fails and `raise_on_failure` is True.
+- `ToolOutputMergeError`: If merging tool outputs into state fails and `raise_on_failure` is True.
+
+**Returns**:
+
+A dictionary with the key `tool_messages` containing a list of ChatMessage objects with tool role.
+Each ChatMessage objects wraps the result of a tool invocation.
+
+<a id="tool_invoker.ToolInvoker.to_dict"></a>
+
+#### ToolInvoker.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+**Returns**:
+
+Dictionary with serialized data.
+
+<a id="tool_invoker.ToolInvoker.from_dict"></a>
+
+#### ToolInvoker.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "ToolInvoker"
+```
+
+Deserializes the component from a dictionary.
+
+**Arguments**:
+
+- `data`: The dictionary to deserialize from.
+
+**Returns**:
+
+The deserialized component.
+

docs-website/reference_versioned_docs/version-2.22/haystack-api/tools_api.md

@@ -0,0 +1,953 @@
+---
+title: "Tools"
+id: tools-api
+description: "Unified abstractions to represent tools across the framework."
+slug: "/tools-api"
+---
+
+<a id="component_tool"></a>
+
+## Module component\_tool
+
+<a id="component_tool.ComponentTool"></a>
+
+### ComponentTool
+
+A Tool that wraps Haystack components, allowing them to be used as tools by LLMs.
+
+ComponentTool automatically generates LLM-compatible tool schemas from component input sockets,
+which are derived from the component's `run` method signature and type hints.
+
+
+Key features:
+- Automatic LLM tool calling schema generation from component input sockets
+- Type conversion and validation for component inputs
+- Support for types:
+- Dataclasses
+- Lists of dataclasses
+- Basic types (str, int, float, bool, dict)
+- Lists of basic types
+- Automatic name generation from component class name
+- Description extraction from component docstrings
+
+To use ComponentTool, you first need a Haystack component - either an existing one or a new one you create.
+You can create a ComponentTool from the component by passing the component to the ComponentTool constructor.
+Below is an example of creating a ComponentTool from an existing SerperDevWebSearch component.
+
+## Usage Example:
+
+```python
+from haystack import component, Pipeline
+from haystack.tools import ComponentTool
+from haystack.components.websearch import SerperDevWebSearch
+from haystack.utils import Secret
+from haystack.components.tools.tool_invoker import ToolInvoker
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.dataclasses import ChatMessage
+
+# Create a SerperDev search component
+search = SerperDevWebSearch(api_key=Secret.from_env_var("SERPERDEV_API_KEY"), top_k=3)
+
+# Create a tool from the component
+tool = ComponentTool(
+    component=search,
+    name="web_search",  # Optional: defaults to "serper_dev_web_search"
+    description="Search the web for current information on any topic"  # Optional: defaults to component docstring
+)
+
+# Create pipeline with OpenAIChatGenerator and ToolInvoker
+pipeline = Pipeline()
+pipeline.add_component("llm", OpenAIChatGenerator(tools=[tool]))
+pipeline.add_component("tool_invoker", ToolInvoker(tools=[tool]))
+
+# Connect components
+pipeline.connect("llm.replies", "tool_invoker.messages")
+
+message = ChatMessage.from_user("Use the web search tool to find information about Nikola Tesla")
+
+# Run pipeline
+result = pipeline.run({"llm": {"messages": [message]}})
+
+print(result)
+```
+
+<a id="component_tool.ComponentTool.__init__"></a>
+
+#### ComponentTool.\_\_init\_\_
+
+```python
+def __init__(
+    component: Component,
+    name: str | None = None,
+    description: str | None = None,
+    parameters: dict[str, Any] | None = None,
+    *,
+    outputs_to_string: dict[str, str | Callable[[Any], str]] | None = None,
+    inputs_from_state: dict[str, str] | None = None,
+    outputs_to_state: dict[str, dict[str, str | Callable]] | None = None
+) -> None
+```
+
+Create a Tool instance from a Haystack component.
+
+**Arguments**:
+
+- `component`: The Haystack component to wrap as a tool.
+- `name`: Optional name for the tool (defaults to snake_case of component class name).
+- `description`: Optional description (defaults to component's docstring).
+- `parameters`: A JSON schema defining the parameters expected by the Tool.
+Will fall back to the parameters defined in the component's run method signature if not provided.
+- `outputs_to_string`: Optional dictionary defining how tool outputs should be converted into string(s).
+Supports two formats:
+
+1. Single output format - use "source" and/or "handler" at the root level:
+    ```python
+    {
+        "source": "docs", "handler": format_documents
+    }
+    ```
+    If the source is provided, only the specified output key is sent to the handler.
+    If the source is omitted, the whole tool result is sent to the handler.
+
+2. Multiple output format - map keys to individual configurations:
+    ```python
+    {
+        "formatted_docs": {"source": "docs", "handler": format_documents},
+        "summary": {"source": "summary_text", "handler": str.upper}
+    }
+    ```
+    Each key maps to a dictionary that can contain "source" and/or "handler".
+- `inputs_from_state`: Optional dictionary mapping state keys to tool parameter names.
+Example: `{"repository": "repo"}` maps state's "repository" to tool's "repo" parameter.
+- `outputs_to_state`: Optional dictionary defining how tool outputs map to keys within state as well as optional handlers.
+If the source is provided only the specified output key is sent to the handler.
+Example:
+```python
+{
+    "documents": {"source": "docs", "handler": custom_handler}
+}
+```
+If the source is omitted the whole tool result is sent to the handler.
+Example:
+```python
+{
+    "documents": {"handler": custom_handler}
+}
+```
+
+**Raises**:
+
+- `ValueError`: If the component is invalid or schema generation fails.
+
+<a id="component_tool.ComponentTool.warm_up"></a>
+
+#### ComponentTool.warm\_up
+
+```python
+def warm_up()
+```
+
+Prepare the ComponentTool for use.
+
+<a id="component_tool.ComponentTool.to_dict"></a>
+
+#### ComponentTool.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serializes the ComponentTool to a dictionary.
+
+<a id="component_tool.ComponentTool.from_dict"></a>
+
+#### ComponentTool.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "ComponentTool"
+```
+
+Deserializes the ComponentTool from a dictionary.
+
+<a id="component_tool.ComponentTool.tool_spec"></a>
+
+#### ComponentTool.tool\_spec
+
+```python
+@property
+def tool_spec() -> dict[str, Any]
+```
+
+Return the Tool specification to be used by the Language Model.
+
+<a id="component_tool.ComponentTool.invoke"></a>
+
+#### ComponentTool.invoke
+
+```python
+def invoke(**kwargs: Any) -> Any
+```
+
+Invoke the Tool with the provided keyword arguments.
+
+<a id="from_function"></a>
+
+## Module from\_function
+
+<a id="from_function.create_tool_from_function"></a>
+
+#### create\_tool\_from\_function
+
+```python
+def create_tool_from_function(
+        function: Callable,
+        name: str | None = None,
+        description: str | None = None,
+        inputs_from_state: dict[str, str] | None = None,
+        outputs_to_state: dict[str, dict[str, Any]] | None = None) -> "Tool"
+```
+
+Create a Tool instance from a function.
+
+Allows customizing the Tool name and description.
+For simpler use cases, consider using the `@tool` decorator.
+
+### Usage example
+
+```python
+from typing import Annotated, Literal
+from haystack.tools import create_tool_from_function
+
+def get_weather(
+    city: Annotated[str, "the city for which to get the weather"] = "Munich",
+    unit: Annotated[Literal["Celsius", "Fahrenheit"], "the unit for the temperature"] = "Celsius"):
+    '''A simple function to get the current weather for a location.'''
+    return f"Weather report for {city}: 20 {unit}, sunny"
+
+tool = create_tool_from_function(get_weather)
+
+print(tool)
+>>> Tool(name='get_weather', description='A simple function to get the current weather for a location.',
+>>> parameters={
+>>> 'type': 'object',
+>>> 'properties': {
+>>>     'city': {'type': 'string', 'description': 'the city for which to get the weather', 'default': 'Munich'},
+>>>     'unit': {
+>>>         'type': 'string',
+>>>         'enum': ['Celsius', 'Fahrenheit'],
+>>>         'description': 'the unit for the temperature',
+>>>         'default': 'Celsius',
+>>>     },
+>>>     }
+>>> },
+>>> function=<function get_weather at 0x7f7b3a8a9b80>)
+```
+
+**Arguments**:
+
+- `function`: The function to be converted into a Tool.
+The function must include type hints for all parameters.
+The function is expected to have basic python input types (str, int, float, bool, list, dict, tuple).
+Other input types may work but are not guaranteed.
+If a parameter is annotated using `typing.Annotated`, its metadata will be used as parameter description.
+- `name`: The name of the Tool. If not provided, the name of the function will be used.
+- `description`: The description of the Tool. If not provided, the docstring of the function will be used.
+To intentionally leave the description empty, pass an empty string.
+- `inputs_from_state`: Optional dictionary mapping state keys to tool parameter names.
+Example: `{"repository": "repo"}` maps state's "repository" to tool's "repo" parameter.
+- `outputs_to_state`: Optional dictionary defining how tool outputs map to state and message handling.
+Example:
+```python
+{
+    "documents": {"source": "docs", "handler": custom_handler},
+    "message": {"source": "summary", "handler": format_summary}
+}
+```
+
+**Raises**:
+
+- `ValueError`: If any parameter of the function lacks a type hint.
+- `SchemaGenerationError`: If there is an error generating the JSON schema for the Tool.
+
+**Returns**:
+
+The Tool created from the function.
+
+<a id="from_function.tool"></a>
+
+#### tool
+
+```python
+def tool(
+    function: Callable | None = None,
+    *,
+    name: str | None = None,
+    description: str | None = None,
+    inputs_from_state: dict[str, str] | None = None,
+    outputs_to_state: dict[str, dict[str, Any]] | None = None
+) -> Tool | Callable[[Callable], Tool]
+```
+
+Decorator to convert a function into a Tool.
+
+Can be used with or without parameters:
+@tool  # without parameters
+def my_function(): ...
+
+@tool(name="custom_name")  # with parameters
+def my_function(): ...
+
+### Usage example
+```python
+from typing import Annotated, Literal
+from haystack.tools import tool
+
+@tool
+def get_weather(
+    city: Annotated[str, "the city for which to get the weather"] = "Munich",
+    unit: Annotated[Literal["Celsius", "Fahrenheit"], "the unit for the temperature"] = "Celsius"):
+    '''A simple function to get the current weather for a location.'''
+    return f"Weather report for {city}: 20 {unit}, sunny"
+
+print(get_weather)
+>>> Tool(name='get_weather', description='A simple function to get the current weather for a location.',
+>>> parameters={
+>>> 'type': 'object',
+>>> 'properties': {
+>>>     'city': {'type': 'string', 'description': 'the city for which to get the weather', 'default': 'Munich'},
+>>>     'unit': {
+>>>         'type': 'string',
+>>>         'enum': ['Celsius', 'Fahrenheit'],
+>>>         'description': 'the unit for the temperature',
+>>>         'default': 'Celsius',
+>>>     },
+>>>     }
+>>> },
+>>> function=<function get_weather at 0x7f7b3a8a9b80>)
+```
+
+**Arguments**:
+
+- `function`: The function to decorate (when used without parameters)
+- `name`: Optional custom name for the tool
+- `description`: Optional custom description
+- `inputs_from_state`: Optional dictionary mapping state keys to tool parameter names
+- `outputs_to_state`: Optional dictionary defining how tool outputs map to state and message handling
+
+**Returns**:
+
+Either a Tool instance or a decorator function that will create one
+
+<a id="tool"></a>
+
+## Module tool
+
+<a id="tool.Tool"></a>
+
+### Tool
+
+Data class representing a Tool that Language Models can prepare a call for.
+
+Accurate definitions of the textual attributes such as `name` and `description`
+are important for the Language Model to correctly prepare the call.
+
+For resource-intensive operations like establishing connections to remote services or
+loading models, override the `warm_up()` method. This method is called before the Tool
+is used and should be idempotent, as it may be called multiple times during
+pipeline/agent setup.
+
+**Arguments**:
+
+- `name`: Name of the Tool.
+- `description`: Description of the Tool.
+- `parameters`: A JSON schema defining the parameters expected by the Tool.
+- `function`: The function that will be invoked when the Tool is called.
+Must be a synchronous function; async functions are not supported.
+- `outputs_to_string`: Optional dictionary defining how tool outputs should be converted into string(s).
+Supports two formats:
+
+1. Single output format - use "source" and/or "handler" at the root level:
+   ```python
+   {
+       "source": "docs", "handler": format_documents
+   }
+   ```
+   If the source is provided, only the specified output key is sent to the handler.
+   If the source is omitted, the whole tool result is sent to the handler.
+
+2. Multiple output format - map keys to individual configurations:
+   ```python
+   {
+       "formatted_docs": {"source": "docs", "handler": format_documents},
+       "summary": {"source": "summary_text", "handler": str.upper}
+   }
+   ```
+   Each key maps to a dictionary that can contain "source" and/or "handler".
+- `inputs_from_state`: Optional dictionary mapping state keys to tool parameter names.
+Example: `{"repository": "repo"}` maps state's "repository" to tool's "repo" parameter.
+- `outputs_to_state`: Optional dictionary defining how tool outputs map to keys within state as well as optional handlers.
+If the source is provided only the specified output key is sent to the handler.
+Example:
+```python
+{
+    "documents": {"source": "docs", "handler": custom_handler}
+}
+```
+If the source is omitted the whole tool result is sent to the handler.
+Example:
+```python
+{
+    "documents": {"handler": custom_handler}
+}
+```
+
+<a id="tool.Tool.tool_spec"></a>
+
+#### Tool.tool\_spec
+
+```python
+@property
+def tool_spec() -> dict[str, Any]
+```
+
+Return the Tool specification to be used by the Language Model.
+
+<a id="tool.Tool.warm_up"></a>
+
+#### Tool.warm\_up
+
+```python
+def warm_up() -> None
+```
+
+Prepare the Tool for use.
+
+Override this method to establish connections to remote services, load models,
+or perform other resource-intensive initialization. This method should be idempotent,
+as it may be called multiple times.
+
+<a id="tool.Tool.invoke"></a>
+
+#### Tool.invoke
+
+```python
+def invoke(**kwargs: Any) -> Any
+```
+
+Invoke the Tool with the provided keyword arguments.
+
+<a id="tool.Tool.to_dict"></a>
+
+#### Tool.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serializes the Tool to a dictionary.
+
+**Returns**:
+
+Dictionary with serialized data.
+
+<a id="tool.Tool.from_dict"></a>
+
+#### Tool.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "Tool"
+```
+
+Deserializes the Tool from a dictionary.
+
+**Arguments**:
+
+- `data`: Dictionary to deserialize from.
+
+**Returns**:
+
+Deserialized Tool.
+
+<a id="toolset"></a>
+
+## Module toolset
+
+<a id="toolset.Toolset"></a>
+
+### Toolset
+
+A collection of related Tools that can be used and managed as a cohesive unit.
+
+Toolset serves two main purposes:
+
+1. Group related tools together:
+Toolset allows you to organize related tools into a single collection, making it easier
+to manage and use them as a unit in Haystack pipelines.
+
+**Example**:
+
+   ```python
+   from haystack.tools import Tool, Toolset
+   from haystack.components.tools import ToolInvoker
+
+   # Define math functions
+   def add_numbers(a: int, b: int) -> int:
+       return a + b
+
+   def subtract_numbers(a: int, b: int) -> int:
+       return a - b
+
+   # Create tools with proper schemas
+   add_tool = Tool(
+       name="add",
+       description="Add two numbers",
+       parameters={
+           "type": "object",
+           "properties": {
+               "a": {"type": "integer"},
+               "b": {"type": "integer"}
+           },
+           "required": ["a", "b"]
+       },
+       function=add_numbers
+   )
+
+   subtract_tool = Tool(
+       name="subtract",
+       description="Subtract b from a",
+       parameters={
+           "type": "object",
+           "properties": {
+               "a": {"type": "integer"},
+               "b": {"type": "integer"}
+           },
+           "required": ["a", "b"]
+       },
+       function=subtract_numbers
+   )
+
+   # Create a toolset with the math tools
+   math_toolset = Toolset([add_tool, subtract_tool])
+
+   # Use the toolset with a ToolInvoker or ChatGenerator component
+   invoker = ToolInvoker(tools=math_toolset)
+   ```
+  
+  2. Base class for dynamic tool loading:
+  By subclassing Toolset, you can create implementations that dynamically load tools
+  from external sources like OpenAPI URLs, MCP servers, or other resources.
+  
+
+**Example**:
+
+   ```python
+   from haystack.core.serialization import generate_qualified_class_name
+   from haystack.tools import Tool, Toolset
+   from haystack.components.tools import ToolInvoker
+
+   class CalculatorToolset(Toolset):
+       '''A toolset for calculator operations.'''
+
+       def __init__(self):
+           tools = self._create_tools()
+           super().__init__(tools)
+
+       def _create_tools(self):
+           # These Tool instances are obviously defined statically and for illustration purposes only.
+           # In a real-world scenario, you would dynamically load tools from an external source here.
+           tools = []
+           add_tool = Tool(
+               name="add",
+               description="Add two numbers",
+               parameters={
+                   "type": "object",
+                   "properties": {"a": {"type": "integer"}, "b": {"type": "integer"}},
+                   "required": ["a", "b"],
+               },
+               function=lambda a, b: a + b,
+           )
+
+           multiply_tool = Tool(
+               name="multiply",
+               description="Multiply two numbers",
+               parameters={
+                   "type": "object",
+                   "properties": {"a": {"type": "integer"}, "b": {"type": "integer"}},
+                   "required": ["a", "b"],
+               },
+               function=lambda a, b: a * b,
+           )
+
+           tools.append(add_tool)
+           tools.append(multiply_tool)
+
+           return tools
+
+       def to_dict(self):
+           return {
+               "type": generate_qualified_class_name(type(self)),
+               "data": {},  # no data to serialize as we define the tools dynamically
+           }
+
+       @classmethod
+       def from_dict(cls, data):
+           return cls()  # Recreate the tools dynamically during deserialization
+
+   # Create the dynamic toolset and use it with ToolInvoker
+   calculator_toolset = CalculatorToolset()
+   invoker = ToolInvoker(tools=calculator_toolset)
+   ```
+  
+  Toolset implements the collection interface (__iter__, __contains__, __len__, __getitem__),
+  making it behave like a list of Tools. This makes it compatible with components that expect
+  iterable tools, such as ToolInvoker or Haystack chat generators.
+  
+  When implementing a custom Toolset subclass for dynamic tool loading:
+  - Perform the dynamic loading in the __init__ method
+  - Override to_dict() and from_dict() methods if your tools are defined dynamically
+  - Serialize endpoint descriptors rather than tool instances if your tools
+  are loaded from external sources
+
+<a id="toolset.Toolset.__post_init__"></a>
+
+#### Toolset.\_\_post\_init\_\_
+
+```python
+def __post_init__()
+```
+
+Validate and set up the toolset after initialization.
+
+This handles the case when tools are provided during initialization.
+
+<a id="toolset.Toolset.__iter__"></a>
+
+#### Toolset.\_\_iter\_\_
+
+```python
+def __iter__() -> Iterator[Tool]
+```
+
+Return an iterator over the Tools in this Toolset.
+
+This allows the Toolset to be used wherever a list of Tools is expected.
+
+**Returns**:
+
+An iterator yielding Tool instances
+
+<a id="toolset.Toolset.__contains__"></a>
+
+#### Toolset.\_\_contains\_\_
+
+```python
+def __contains__(item: Any) -> bool
+```
+
+Check if a tool is in this Toolset.
+
+Supports checking by:
+- Tool instance: tool in toolset
+- Tool name: "tool_name" in toolset
+
+**Arguments**:
+
+- `item`: Tool instance or tool name string
+
+**Returns**:
+
+True if contained, False otherwise
+
+<a id="toolset.Toolset.warm_up"></a>
+
+#### Toolset.warm\_up
+
+```python
+def warm_up() -> None
+```
+
+Prepare the Toolset for use.
+
+By default, this method iterates through and warms up all tools in the Toolset.
+Subclasses can override this method to customize initialization behavior, such as:
+
+- Setting up shared resources (database connections, HTTP sessions) instead of
+  warming individual tools
+- Implementing custom initialization logic for dynamically loaded tools
+- Controlling when and how tools are initialized
+
+For example, a Toolset that manages tools from an external service (like MCPToolset)
+might override this to initialize a shared connection rather than warming up
+individual tools:
+
+```python
+class MCPToolset(Toolset):
+    def warm_up(self) -> None:
+        # Only warm up the shared MCP connection, not individual tools
+        self.mcp_connection = establish_connection(self.server_url)
+```
+
+This method should be idempotent, as it may be called multiple times.
+
+<a id="toolset.Toolset.add"></a>
+
+#### Toolset.add
+
+```python
+def add(tool: Union[Tool, "Toolset"]) -> None
+```
+
+Add a new Tool or merge another Toolset.
+
+**Arguments**:
+
+- `tool`: A Tool instance or another Toolset to add
+
+**Raises**:
+
+- `ValueError`: If adding the tool would result in duplicate tool names
+- `TypeError`: If the provided object is not a Tool or Toolset
+
+<a id="toolset.Toolset.to_dict"></a>
+
+#### Toolset.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serialize the Toolset to a dictionary.
+
+**Returns**:
+
+A dictionary representation of the Toolset
+Note for subclass implementers:
+The default implementation is ideal for scenarios where Tool resolution is static. However, if your subclass
+of Toolset dynamically resolves Tool instances from external sources—such as an MCP server, OpenAPI URL, or
+a local OpenAPI specification—you should consider serializing the endpoint descriptor instead of the Tool
+instances themselves. This strategy preserves the dynamic nature of your Toolset and minimizes the overhead
+associated with serializing potentially large collections of Tool objects. Moreover, by serializing the
+descriptor, you ensure that the deserialization process can accurately reconstruct the Tool instances, even
+if they have been modified or removed since the last serialization. Failing to serialize the descriptor may
+lead to issues where outdated or incorrect Tool configurations are loaded, potentially causing errors or
+unexpected behavior.
+
+<a id="toolset.Toolset.from_dict"></a>
+
+#### Toolset.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "Toolset"
+```
+
+Deserialize a Toolset from a dictionary.
+
+**Arguments**:
+
+- `data`: Dictionary representation of the Toolset
+
+**Returns**:
+
+A new Toolset instance
+
+<a id="toolset.Toolset.__add__"></a>
+
+#### Toolset.\_\_add\_\_
+
+```python
+def __add__(other: Union[Tool, "Toolset", list[Tool]]) -> "Toolset"
+```
+
+Concatenate this Toolset with another Tool, Toolset, or list of Tools.
+
+**Arguments**:
+
+- `other`: Another Tool, Toolset, or list of Tools to concatenate
+
+**Raises**:
+
+- `TypeError`: If the other parameter is not a Tool, Toolset, or list of Tools
+- `ValueError`: If the combination would result in duplicate tool names
+
+**Returns**:
+
+A new Toolset containing all tools
+
+<a id="toolset.Toolset.__len__"></a>
+
+#### Toolset.\_\_len\_\_
+
+```python
+def __len__() -> int
+```
+
+Return the number of Tools in this Toolset.
+
+**Returns**:
+
+Number of Tools
+
+<a id="toolset.Toolset.__getitem__"></a>
+
+#### Toolset.\_\_getitem\_\_
+
+```python
+def __getitem__(index)
+```
+
+Get a Tool by index.
+
+**Arguments**:
+
+- `index`: Index of the Tool to get
+
+**Returns**:
+
+The Tool at the specified index
+
+<a id="toolset._ToolsetWrapper"></a>
+
+### \_ToolsetWrapper
+
+A wrapper that holds multiple toolsets and provides a unified interface.
+
+This is used internally when combining different types of toolsets to preserve
+their individual configurations while still being usable with ToolInvoker.
+
+<a id="toolset._ToolsetWrapper.__iter__"></a>
+
+#### \_ToolsetWrapper.\_\_iter\_\_
+
+```python
+def __iter__()
+```
+
+Iterate over all tools from all toolsets.
+
+<a id="toolset._ToolsetWrapper.__contains__"></a>
+
+#### \_ToolsetWrapper.\_\_contains\_\_
+
+```python
+def __contains__(item)
+```
+
+Check if a tool is in any of the toolsets.
+
+<a id="toolset._ToolsetWrapper.warm_up"></a>
+
+#### \_ToolsetWrapper.warm\_up
+
+```python
+def warm_up()
+```
+
+Warm up all toolsets.
+
+<a id="toolset._ToolsetWrapper.__len__"></a>
+
+#### \_ToolsetWrapper.\_\_len\_\_
+
+```python
+def __len__()
+```
+
+Return total number of tools across all toolsets.
+
+<a id="toolset._ToolsetWrapper.__getitem__"></a>
+
+#### \_ToolsetWrapper.\_\_getitem\_\_
+
+```python
+def __getitem__(index)
+```
+
+Get a tool by index across all toolsets.
+
+<a id="toolset._ToolsetWrapper.__add__"></a>
+
+#### \_ToolsetWrapper.\_\_add\_\_
+
+```python
+def __add__(other)
+```
+
+Add another toolset or tool to this wrapper.
+
+<a id="toolset._ToolsetWrapper.__post_init__"></a>
+
+#### \_ToolsetWrapper.\_\_post\_init\_\_
+
+```python
+def __post_init__()
+```
+
+Validate and set up the toolset after initialization.
+
+This handles the case when tools are provided during initialization.
+
+<a id="toolset._ToolsetWrapper.add"></a>
+
+#### \_ToolsetWrapper.add
+
+```python
+def add(tool: Union[Tool, "Toolset"]) -> None
+```
+
+Add a new Tool or merge another Toolset.
+
+**Arguments**:
+
+- `tool`: A Tool instance or another Toolset to add
+
+**Raises**:
+
+- `ValueError`: If adding the tool would result in duplicate tool names
+- `TypeError`: If the provided object is not a Tool or Toolset
+
+<a id="toolset._ToolsetWrapper.to_dict"></a>
+
+#### \_ToolsetWrapper.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serialize the Toolset to a dictionary.
+
+**Returns**:
+
+A dictionary representation of the Toolset
+Note for subclass implementers:
+The default implementation is ideal for scenarios where Tool resolution is static. However, if your subclass
+of Toolset dynamically resolves Tool instances from external sources—such as an MCP server, OpenAPI URL, or
+a local OpenAPI specification—you should consider serializing the endpoint descriptor instead of the Tool
+instances themselves. This strategy preserves the dynamic nature of your Toolset and minimizes the overhead
+associated with serializing potentially large collections of Tool objects. Moreover, by serializing the
+descriptor, you ensure that the deserialization process can accurately reconstruct the Tool instances, even
+if they have been modified or removed since the last serialization. Failing to serialize the descriptor may
+lead to issues where outdated or incorrect Tool configurations are loaded, potentially causing errors or
+unexpected behavior.
+
+<a id="toolset._ToolsetWrapper.from_dict"></a>
+
+#### \_ToolsetWrapper.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "Toolset"
+```
+
+Deserialize a Toolset from a dictionary.
+
+**Arguments**:
+
+- `data`: Dictionary representation of the Toolset
+
+**Returns**:
+
+A new Toolset instance
+

docs-website/reference_versioned_docs/version-2.22/haystack-api/validators_api.md

@@ -0,0 +1,145 @@
+---
+title: "Validators"
+id: validators-api
+description: "Validators validate LLM outputs"
+slug: "/validators-api"
+---
+
+<a id="json_schema"></a>
+
+## Module json\_schema
+
+<a id="json_schema.is_valid_json"></a>
+
+#### is\_valid\_json
+
+```python
+def is_valid_json(s: str) -> bool
+```
+
+Check if the provided string is a valid JSON.
+
+**Arguments**:
+
+- `s`: The string to be checked.
+
+**Returns**:
+
+`True` if the string is a valid JSON; otherwise, `False`.
+
+<a id="json_schema.JsonSchemaValidator"></a>
+
+### JsonSchemaValidator
+
+Validates JSON content of `ChatMessage` against a specified [JSON Schema](https://json-schema.org/).
+
+If JSON content of a message conforms to the provided schema, the message is passed along the "validated" output.
+If the JSON content does not conform to the schema, the message is passed along the "validation_error" output.
+In the latter case, the error message is constructed using the provided `error_template` or a default template.
+These error ChatMessages can be used by LLMs in Haystack 2.x recovery loops.
+
+Usage example:
+
+```python
+from haystack import Pipeline
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.components.joiners import BranchJoiner
+from haystack.components.validators import JsonSchemaValidator
+from haystack import component
+from haystack.dataclasses import ChatMessage
+
+
+@component
+class MessageProducer:
+
+    @component.output_types(messages=list[ChatMessage])
+    def run(self, messages: list[ChatMessage]) -> dict:
+        return {"messages": messages}
+
+
+p = Pipeline()
+p.add_component("llm", OpenAIChatGenerator(model="gpt-4-1106-preview",
+                                           generation_kwargs={"response_format": {"type": "json_object"}}))
+p.add_component("schema_validator", JsonSchemaValidator())
+p.add_component("joiner_for_llm", BranchJoiner(list[ChatMessage]))
+p.add_component("message_producer", MessageProducer())
+
+p.connect("message_producer.messages", "joiner_for_llm")
+p.connect("joiner_for_llm", "llm")
+p.connect("llm.replies", "schema_validator.messages")
+p.connect("schema_validator.validation_error", "joiner_for_llm")
+
+result = p.run(data={
+    "message_producer": {
+        "messages":[ChatMessage.from_user("Generate JSON for person with name 'John' and age 30")]},
+        "schema_validator": {
+            "json_schema": {
+                "type": "object",
+                "properties": {"name": {"type": "string"},
+                "age": {"type": "integer"}
+            }
+        }
+    }
+})
+print(result)
+>> {'schema_validator': {'validated': [ChatMessage(_role=<ChatRole.ASSISTANT: 'assistant'>,
+_content=[TextContent(text="\n{\n  "name": "John",\n  "age": 30\n}")],
+_name=None, _meta={'model': 'gpt-4-1106-preview', 'index': 0,
+'finish_reason': 'stop', 'usage': {'completion_tokens': 17, 'prompt_tokens': 20, 'total_tokens': 37}})]}}
+```
+
+<a id="json_schema.JsonSchemaValidator.__init__"></a>
+
+#### JsonSchemaValidator.\_\_init\_\_
+
+```python
+def __init__(json_schema: dict[str, Any] | None = None,
+             error_template: str | None = None)
+```
+
+Initialize the JsonSchemaValidator component.
+
+**Arguments**:
+
+- `json_schema`: A dictionary representing the [JSON schema](https://json-schema.org/) against which
+the messages' content is validated.
+- `error_template`: A custom template string for formatting the error message in case of validation failure.
+
+<a id="json_schema.JsonSchemaValidator.run"></a>
+
+#### JsonSchemaValidator.run
+
+```python
+@component.output_types(validated=list[ChatMessage],
+                        validation_error=list[ChatMessage])
+def run(messages: list[ChatMessage],
+        json_schema: dict[str, Any] | None = None,
+        error_template: str | None = None) -> dict[str, list[ChatMessage]]
+```
+
+Validates the last of the provided messages against the specified json schema.
+
+If it does, the message is passed along the "validated" output. If it does not, the message is passed along
+the "validation_error" output.
+
+**Arguments**:
+
+- `messages`: A list of ChatMessage instances to be validated. The last message in this list is the one
+that is validated.
+- `json_schema`: A dictionary representing the [JSON schema](https://json-schema.org/)
+against which the messages' content is validated. If not provided, the schema from the component init
+is used.
+- `error_template`: A custom template string for formatting the error message in case of validation. If not
+provided, the `error_template` from the component init is used.
+
+**Raises**:
+
+- `ValueError`: If no JSON schema is provided or if the message content is not a dictionary or a list of
+dictionaries.
+
+**Returns**:
+
+A dictionary with the following keys:
+- "validated": A list of messages if the last message is valid.
+- "validation_error": A list of messages if the last message is invalid.
+

docs-website/reference_versioned_docs/version-2.22/haystack-api/websearch_api.md

@@ -0,0 +1,228 @@
+---
+title: "Websearch"
+id: websearch-api
+description: "Web search engine for Haystack."
+slug: "/websearch-api"
+---
+
+<a id="searchapi"></a>
+
+## Module searchapi
+
+<a id="searchapi.SearchApiWebSearch"></a>
+
+### SearchApiWebSearch
+
+Uses [SearchApi](https://www.searchapi.io/) to search the web for relevant documents.
+
+Usage example:
+```python
+from haystack.components.websearch import SearchApiWebSearch
+from haystack.utils import Secret
+
+websearch = SearchApiWebSearch(top_k=10, api_key=Secret.from_token("test-api-key"))
+results = websearch.run(query="Who is the boyfriend of Olivia Wilde?")
+
+assert results["documents"]
+assert results["links"]
+```
+
+<a id="searchapi.SearchApiWebSearch.__init__"></a>
+
+#### SearchApiWebSearch.\_\_init\_\_
+
+```python
+def __init__(api_key: Secret = Secret.from_env_var("SEARCHAPI_API_KEY"),
+             top_k: int | None = 10,
+             allowed_domains: list[str] | None = None,
+             search_params: dict[str, Any] | None = None)
+```
+
+Initialize the SearchApiWebSearch component.
+
+**Arguments**:
+
+- `api_key`: API key for the SearchApi API
+- `top_k`: Number of documents to return.
+- `allowed_domains`: List of domains to limit the search to.
+- `search_params`: Additional parameters passed to the SearchApi API.
+For example, you can set 'num' to 100 to increase the number of search results.
+See the [SearchApi website](https://www.searchapi.io/) for more details.
+
+The default search engine is Google, however, users can change it by setting the `engine`
+parameter in the `search_params`.
+
+<a id="searchapi.SearchApiWebSearch.to_dict"></a>
+
+#### SearchApiWebSearch.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+**Returns**:
+
+Dictionary with serialized data.
+
+<a id="searchapi.SearchApiWebSearch.from_dict"></a>
+
+#### SearchApiWebSearch.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "SearchApiWebSearch"
+```
+
+Deserializes the component from a dictionary.
+
+**Arguments**:
+
+- `data`: The dictionary to deserialize from.
+
+**Returns**:
+
+The deserialized component.
+
+<a id="searchapi.SearchApiWebSearch.run"></a>
+
+#### SearchApiWebSearch.run
+
+```python
+@component.output_types(documents=list[Document], links=list[str])
+def run(query: str) -> dict[str, list[Document] | list[str]]
+```
+
+Uses [SearchApi](https://www.searchapi.io/) to search the web.
+
+**Arguments**:
+
+- `query`: Search query.
+
+**Raises**:
+
+- `TimeoutError`: If the request to the SearchApi API times out.
+- `SearchApiError`: If an error occurs while querying the SearchApi API.
+
+**Returns**:
+
+A dictionary with the following keys:
+- "documents": List of documents returned by the search engine.
+- "links": List of links returned by the search engine.
+
+<a id="serper_dev"></a>
+
+## Module serper\_dev
+
+<a id="serper_dev.SerperDevWebSearch"></a>
+
+### SerperDevWebSearch
+
+Uses [Serper](https://serper.dev/) to search the web for relevant documents.
+
+See the [Serper Dev website](https://serper.dev/) for more details.
+
+Usage example:
+```python
+from haystack.components.websearch import SerperDevWebSearch
+from haystack.utils import Secret
+
+websearch = SerperDevWebSearch(top_k=10, api_key=Secret.from_token("test-api-key"))
+results = websearch.run(query="Who is the boyfriend of Olivia Wilde?")
+
+assert results["documents"]
+assert results["links"]
+
+# Example with domain filtering - exclude subdomains
+websearch_filtered = SerperDevWebSearch(
+    top_k=10,
+    allowed_domains=["example.com"],
+    exclude_subdomains=True,  # Only results from example.com, not blog.example.com
+    api_key=Secret.from_token("test-api-key")
+)
+results_filtered = websearch_filtered.run(query="search query")
+```
+
+<a id="serper_dev.SerperDevWebSearch.__init__"></a>
+
+#### SerperDevWebSearch.\_\_init\_\_
+
+```python
+def __init__(api_key: Secret = Secret.from_env_var("SERPERDEV_API_KEY"),
+             top_k: int | None = 10,
+             allowed_domains: list[str] | None = None,
+             search_params: dict[str, Any] | None = None,
+             *,
+             exclude_subdomains: bool = False)
+```
+
+Initialize the SerperDevWebSearch component.
+
+**Arguments**:
+
+- `api_key`: API key for the Serper API.
+- `top_k`: Number of documents to return.
+- `allowed_domains`: List of domains to limit the search to.
+- `exclude_subdomains`: Whether to exclude subdomains when filtering by allowed_domains.
+If True, only results from the exact domains in allowed_domains will be returned.
+If False, results from subdomains will also be included. Defaults to False.
+- `search_params`: Additional parameters passed to the Serper API.
+For example, you can set 'num' to 20 to increase the number of search results.
+See the [Serper website](https://serper.dev/) for more details.
+
+<a id="serper_dev.SerperDevWebSearch.to_dict"></a>
+
+#### SerperDevWebSearch.to\_dict
+
+```python
+def to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+**Returns**:
+
+Dictionary with serialized data.
+
+<a id="serper_dev.SerperDevWebSearch.from_dict"></a>
+
+#### SerperDevWebSearch.from\_dict
+
+```python
+@classmethod
+def from_dict(cls, data: dict[str, Any]) -> "SerperDevWebSearch"
+```
+
+Serializes the component to a dictionary.
+
+**Returns**:
+
+Dictionary with serialized data.
+
+<a id="serper_dev.SerperDevWebSearch.run"></a>
+
+#### SerperDevWebSearch.run
+
+```python
+@component.output_types(documents=list[Document], links=list[str])
+def run(query: str) -> dict[str, list[Document] | list[str]]
+```
+
+Use [Serper](https://serper.dev/) to search the web.
+
+**Arguments**:
+
+- `query`: Search query.
+
+**Raises**:
+
+- `SerperDevError`: If an error occurs while querying the SerperDev API.
+- `TimeoutError`: If the request to the SerperDev API times out.
+
+**Returns**:
+
+A dictionary with the following keys:
+- "documents": List of documents returned by the search engine.
+- "links": List of links returned by the search engine.
+