chore: import upstream snapshot with attribution
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,492 @@
|
||||
---
|
||||
title: "Agent"
|
||||
id: agent
|
||||
slug: "/agent"
|
||||
description: "The `Agent` component is a tool-using agent that interacts with chat-based LLMs and tools to solve complex queries iteratively. It can execute external tools, manage state across multiple LLM calls, and stop execution based on configurable `exit_conditions`."
|
||||
---
|
||||
|
||||
# Agent
|
||||
|
||||
The `Agent` component is a tool-using agent that interacts with chat-based LLMs and tools to solve complex queries iteratively. It can execute external tools, manage state across multiple LLM calls, and stop execution based on configurable `exit_conditions`.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`ChatPromptBuilder`](../builders/chatpromptbuilder.mdx) or user input |
|
||||
| **Mandatory init variables** | `chat_generator`: An instance of a Chat Generator that supports tools |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx)s |
|
||||
| **Output variables** | `messages`: Chat history with tool and model responses |
|
||||
| **API reference** | [Agents](/reference/agents-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/agents/agent.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `Agent` component is a loop-based system that uses a chat-based large language model (LLM) and external tools to solve complex user queries.
|
||||
It works iteratively—calling tools, updating state, and generating prompts—until one of the configurable `exit_conditions` is met.
|
||||
|
||||
It can:
|
||||
|
||||
- Dynamically select tools based on user input,
|
||||
- Maintain and validate runtime state using a schema,
|
||||
- Stream token-level outputs from the LLM.
|
||||
|
||||
The `Agent` returns a dictionary containing:
|
||||
|
||||
- `messages`: the full conversation history,
|
||||
- `last_message`: the final `ChatMessage` from the agent,
|
||||
- `step_count`: the number of steps the agent ran,
|
||||
- `token_usage`: aggregated token usage summed across every LLM call in the run,
|
||||
- `tool_call_counts`: how many times each tool was invoked, keyed by tool name,
|
||||
- Additional dynamic keys based on `state_schema`.
|
||||
|
||||
### Run Metadata
|
||||
|
||||
The `step_count`, `token_usage`, and `tool_call_counts` outputs are populated automatically during a run. They are added to the agent's `state_schema` behind the scenes, so tools registered with `inputs_from_state` can read them mid-run. They are outputs only — they cannot be passed as inputs to `run()` or `run_async()`, and using them as keys in your own `state_schema` raises a `ValueError`. See [State](./state.mdx#schema-definition) for details.
|
||||
|
||||
```python
|
||||
response = agent.run(messages=[ChatMessage.from_user("What is 7 * (4 + 2)?")])
|
||||
|
||||
print(response["step_count"]) # 2
|
||||
print(
|
||||
response["token_usage"],
|
||||
) # {"prompt_tokens": 512, "completion_tokens": 86, ...}
|
||||
print(response["tool_call_counts"]) # {"calculator": 1}
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
`chat_generator` is the only mandatory parameter — an instance of a Chat Generator that supports tools. All other parameters are optional.
|
||||
|
||||
- `tools`: A list of tool or toolset instances the agent can call. Supported types: [`Tool`](../../tools/tool.mdx), [`ComponentTool`](../../tools/componenttool.mdx), [`PipelineTool`](../../tools/pipelinetool.mdx), [`MCPTool`](../../tools/mcptool.mdx), [`Toolset`](../../tools/toolset.mdx), [`MCPToolset`](../../tools/mcptoolset.mdx), [`SearchableToolset`](../../tools/searchabletoolset.mdx). Tool names must be unique; duplicate names are detected at the start of each agent step, before the chat generator is called.
|
||||
- `system_prompt`: A plain string or Jinja2 template used as the system message for every run. If the template contains Jinja2 variables, those variables become additional inputs to `run()`.
|
||||
- `user_prompt`: A Jinja2 template appended to the user-provided messages on each run. Template variables become additional inputs to `run()`. Use `required_variables` to enforce which variables must be provided.
|
||||
- `exit_conditions`: List of conditions that cause the agent to stop. Use `”text”` to stop when the LLM replies without a tool call, or a tool name to stop once that tool has been executed. Defaults to `[“text”]`. Exit conditions are evaluated at runtime rather than validated at initialization, so a condition can name a tool that is only loaded later — for example, a tool passed at runtime via `run(tools=...)` or one discovered by a [`SearchableToolset`](../../tools/searchabletoolset.mdx).
|
||||
- `state_schema`: Defines the agent's runtime state — a dict mapping key names to type configs (e.g. `{“docs”: {“type”: list[Document]}}`). Tools can read from and write to state keys via `inputs_from_state` and `outputs_to_state`. See [State](./state.mdx) for full details.
|
||||
- `streaming_callback`: A callback invoked for each streamed token. Use the built-in `print_streaming_chunk` for console output.
|
||||
- `max_agent_steps`: Maximum number of LLM + tool call iterations before the agent stops. Defaults to `100`.
|
||||
- `raise_on_tool_invocation_failure`: If `True`, raises an exception when a tool call fails. If `False` (default), the error is passed back to the LLM as a message so it can recover.
|
||||
- `hooks`: A dict mapping a hook point (`"before_llm"`, `"before_tool"`, `"after_tool"`, `"on_exit"`) to a list of hooks the agent runs at that point. Hooks receive the live `State` and influence the run by mutating it — for example, to build run-time context or require human confirmation of tool calls. See [Hooks](./hooks.mdx) and [Human in the Loop](./human-in-the-loop.mdx).
|
||||
- `tool_concurrency_limit`: Maximum number of tool calls to execute at the same time. Defaults to `4`; set to `1` to disable parallel tool execution.
|
||||
- `tool_streaming_callback_passthrough`: If `True`, passes the streaming callback to tools that accept it.
|
||||
|
||||
### Runtime overrides
|
||||
|
||||
`run()` also accepts parameters that override the init-time configuration for a single call:
|
||||
|
||||
- `tools`: Pass a list of `Tool`/`Toolset` objects, or a list of tool name strings to select a subset of the agent's configured tools for this run.
|
||||
- `generation_kwargs`: Additional keyword arguments forwarded to the LLM, overriding any set at init time (e.g. `{“temperature”: 0.2}`).
|
||||
- `hook_context`: A dict of request-scoped resources made available to [hooks](./hooks.mdx) via `state.data["hook_context"]` — for example, a user ID or a WebSocket connection.
|
||||
|
||||
:::info
|
||||
For the full parameter reference, see the [Agents API Documentation](/reference/agents-api).
|
||||
:::
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.tools import tool
|
||||
from haystack.components.agents import Agent
|
||||
from typing import Annotated
|
||||
|
||||
|
||||
@tool(outputs_to_state={"calc_result": {"source": "result"}})
|
||||
def calculator(
|
||||
expression: Annotated[str, "Math expression to evaluate, e.g. '7 * (4 + 2)'"],
|
||||
) -> dict:
|
||||
"""Evaluate basic math expressions."""
|
||||
try:
|
||||
result = eval(expression, {"__builtins__": {}})
|
||||
return {"result": result}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
tools=[calculator],
|
||||
system_prompt="You are a helpful assistant. Always use the calculator tool to evaluate math expressions.",
|
||||
state_schema={"calc_result": {"type": int}},
|
||||
)
|
||||
|
||||
response = agent.run(messages=[ChatMessage.from_user("What is 7 * (4 + 2)?")])
|
||||
|
||||
print(response["last_message"].text)
|
||||
print("Calc Result:", response.get("calc_result"))
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
The example pipeline below creates a database assistant using `OpenAIChatGenerator`, `LinkContentFetcher`, and custom database tool.
|
||||
It reads the given URL and processes the page content, then builds a prompt for the AI.
|
||||
The assistant uses this information to write people's names and titles from the given page to the database.
|
||||
|
||||
```python
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
|
||||
from haystack.components.converters.html import HTMLToDocument
|
||||
from haystack.components.fetchers.link_content import LinkContentFetcher
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.tools import tool
|
||||
from typing import Annotated, Optional
|
||||
|
||||
document_store = InMemoryDocumentStore() # create a document store or an SQL database
|
||||
|
||||
|
||||
@tool
|
||||
def add_database_tool(
|
||||
name: Annotated[str, "First name of the person"],
|
||||
surname: Annotated[str, "Last name of the person"],
|
||||
job_title: Annotated[Optional[str], "Job title or role of the person"] = None,
|
||||
other: Annotated[Optional[str], "Any other relevant information"] = None,
|
||||
) -> str:
|
||||
"""Add a person to the database with information about them."""
|
||||
document_store.write_documents(
|
||||
[
|
||||
Document(
|
||||
content=name + " " + surname + " " + (job_title or ""),
|
||||
meta={"other": other},
|
||||
),
|
||||
],
|
||||
)
|
||||
# Returning a confirmation lets the agent know the tool call succeeded
|
||||
return f"Successfully added {name} {surname} to the database."
|
||||
|
||||
|
||||
database_assistant = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
tools=[add_database_tool],
|
||||
system_prompt="""
|
||||
You are a database assistant.
|
||||
Your task is to extract the names of people mentioned in the given context and add them to a knowledge base,
|
||||
along with additional relevant information about them that can be extracted from the context.
|
||||
Do not use your own knowledge, stay grounded to the given context.
|
||||
Do not ask the user for confirmation.
|
||||
Instead, automatically update the knowledge base and return a brief summary of the people added,
|
||||
including the information stored for each.
|
||||
""",
|
||||
)
|
||||
|
||||
extraction_agent = Pipeline()
|
||||
extraction_agent.add_component("fetcher", LinkContentFetcher())
|
||||
extraction_agent.add_component("converter", HTMLToDocument())
|
||||
extraction_agent.add_component(
|
||||
"builder",
|
||||
ChatPromptBuilder(
|
||||
template=[
|
||||
ChatMessage.from_user("""
|
||||
{% for doc in docs %}
|
||||
{{ doc.content|default|truncate(25000) }}
|
||||
{% endfor %}
|
||||
"""),
|
||||
],
|
||||
required_variables=["docs"],
|
||||
),
|
||||
)
|
||||
|
||||
extraction_agent.add_component("database_agent", database_assistant)
|
||||
extraction_agent.connect("fetcher.streams", "converter.sources")
|
||||
extraction_agent.connect("converter.documents", "builder.docs")
|
||||
extraction_agent.connect("builder", "database_agent")
|
||||
|
||||
agent_output = extraction_agent.run(
|
||||
{
|
||||
"fetcher": {
|
||||
"urls": ["https://github.com/deepset-ai/haystack/releases/tag/v2.27.0"],
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
print(agent_output["database_agent"]["last_message"].text)
|
||||
|
||||
# Inspect what was written to the document store
|
||||
written_docs = document_store.filter_documents()
|
||||
print(f"\n{len(written_docs)} people added to the database:")
|
||||
for doc in written_docs:
|
||||
print(f" - {doc.content}")
|
||||
```
|
||||
|
||||
### In YAML
|
||||
The example pipeline below fetches a webpage, converts its HTML to text, and builds a chat prompt combining the page content with a user query.
|
||||
The `Agent` then answers the question based on the provided content and can use its web search tool to find additional information if needed.
|
||||
|
||||
<details>
|
||||
<summary>View YAML</summary>
|
||||
|
||||
```yaml
|
||||
components:
|
||||
agent:
|
||||
init_parameters:
|
||||
chat_generator:
|
||||
init_parameters:
|
||||
api_base_url: null
|
||||
api_key:
|
||||
env_vars:
|
||||
- OPENAI_API_KEY
|
||||
strict: true
|
||||
type: env_var
|
||||
generation_kwargs: {}
|
||||
http_client_kwargs: null
|
||||
max_retries: null
|
||||
model: gpt-5.4-nano
|
||||
organization: null
|
||||
streaming_callback: null
|
||||
timeout: null
|
||||
tools: null
|
||||
tools_strict: false
|
||||
type: haystack.components.generators.chat.openai.OpenAIChatGenerator
|
||||
exit_conditions:
|
||||
- text
|
||||
hooks: null
|
||||
max_agent_steps: 5
|
||||
raise_on_tool_invocation_failure: false
|
||||
required_variables: null
|
||||
state_schema: {}
|
||||
streaming_callback: null
|
||||
system_prompt: You are a helpful assistant. Use the web search tool to find
|
||||
information when needed.
|
||||
tool_concurrency_limit: 4
|
||||
tool_streaming_callback_passthrough: false
|
||||
tools:
|
||||
- data:
|
||||
component:
|
||||
init_parameters:
|
||||
allowed_domains: null
|
||||
api_key:
|
||||
env_vars:
|
||||
- SERPERDEV_API_KEY
|
||||
strict: true
|
||||
type: env_var
|
||||
exclude_subdomains: false
|
||||
search_params: {}
|
||||
top_k: 3
|
||||
type: haystack_integrations.components.websearch.serperdev.websearch.SerperDevWebSearch
|
||||
description: Search the web for current information on any topic
|
||||
inputs_from_state: null
|
||||
name: web_search
|
||||
outputs_to_state: null
|
||||
outputs_to_string: null
|
||||
parameters: null
|
||||
type: haystack.tools.component_tool.ComponentTool
|
||||
user_prompt: null
|
||||
type: haystack.components.agents.agent.Agent
|
||||
converter:
|
||||
init_parameters:
|
||||
extraction_kwargs: {}
|
||||
store_full_path: false
|
||||
type: haystack.components.converters.html.HTMLToDocument
|
||||
fetcher:
|
||||
init_parameters:
|
||||
client_kwargs:
|
||||
follow_redirects: true
|
||||
timeout: 3
|
||||
http2: false
|
||||
raise_on_failure: true
|
||||
request_headers: {}
|
||||
retry_attempts: 2
|
||||
timeout: 3
|
||||
user_agents:
|
||||
- haystack/LinkContentFetcher/2.27.0rc0
|
||||
type: haystack.components.fetchers.link_content.LinkContentFetcher
|
||||
prompt_builder:
|
||||
init_parameters:
|
||||
required_variables:
|
||||
- docs
|
||||
- query
|
||||
template:
|
||||
- content:
|
||||
- text: 'Based on the following content:
|
||||
|
||||
{% for doc in docs %}
|
||||
|
||||
{{ doc.content }}
|
||||
|
||||
{% endfor %}
|
||||
|
||||
Answer this question: {{ query }}'
|
||||
meta: {}
|
||||
name: null
|
||||
role: user
|
||||
variables: null
|
||||
type: haystack.components.builders.chat_prompt_builder.ChatPromptBuilder
|
||||
connection_type_validation: true
|
||||
connections:
|
||||
- receiver: converter.sources
|
||||
sender: fetcher.streams
|
||||
- receiver: prompt_builder.docs
|
||||
sender: converter.documents
|
||||
- receiver: agent.messages
|
||||
sender: prompt_builder.prompt
|
||||
max_runs_per_component: 100
|
||||
metadata: {}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
## Streaming
|
||||
|
||||
You can stream output as it's generated. Pass a callback to `streaming_callback`.
|
||||
Use the built-in `print_streaming_chunk` to print text tokens and tool events (tool calls and tool results).
|
||||
|
||||
```python
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
tools=[...],
|
||||
system_prompt="...",
|
||||
streaming_callback=print_streaming_chunk,
|
||||
)
|
||||
```
|
||||
|
||||
See our [Streaming Support](../generators/guides-to-generators/choosing-the-right-generator.mdx#streaming-support) docs to learn more how `StreamingChunk` works and how to write a custom callback.
|
||||
|
||||
Give preference to `print_streaming_chunk` by default.
|
||||
Write a custom callback only if you need a specific transport (for example, SSE/WebSocket) or custom UI formatting.
|
||||
|
||||
## Multimodal Inputs
|
||||
|
||||
Agents support multimodal inputs when paired with a vision-capable model such as `gpt-5` (OpenAI) or `gemini-2.5-flash` (Google).
|
||||
Pass images alongside text by including `ImageContent` objects in the `content_parts` of a `ChatMessage`:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage, ImageContent
|
||||
|
||||
image = ImageContent.from_url("https://example.com/chart.png")
|
||||
result = agent.run(
|
||||
messages=[
|
||||
ChatMessage.from_user(content_parts=["What does this chart show?", image]),
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
Tools can also return `ImageContent` directly, letting the agent fetch and reason about images dynamically during its loop.
|
||||
Two things are required: set `outputs_to_string={"raw_result": True}` so the `ToolInvoker` skips string conversion, and return a `list[ImageContent]` (the tool result type is `str | Sequence[TextContent | ImageContent]`).
|
||||
|
||||
The standard Chat Completions API doesn't support images in tool results — use `OpenAIResponsesChatGenerator` (OpenAI's Responses API) instead:
|
||||
|
||||
```python
|
||||
from typing import Annotated
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.generators.chat import OpenAIResponsesChatGenerator
|
||||
from haystack.dataclasses import ChatMessage, ImageContent
|
||||
from haystack.tools import tool
|
||||
|
||||
|
||||
@tool(outputs_to_string={"raw_result": True})
|
||||
def fetch_image(
|
||||
url: Annotated[str, "URL of the image to fetch and analyze"],
|
||||
) -> list[ImageContent]:
|
||||
"""Fetch an image from a URL so the agent can analyze its contents."""
|
||||
return [ImageContent.from_url(url)]
|
||||
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIResponsesChatGenerator(model="gpt-5"),
|
||||
tools=[fetch_image],
|
||||
system_prompt="You are a helpful assistant that can fetch and analyze images from URLs.",
|
||||
)
|
||||
|
||||
result = agent.run(
|
||||
messages=[
|
||||
ChatMessage.from_user(
|
||||
"Fetch the image at https://picsum.photos/seed/haystack/640/480 and describe what you see.",
|
||||
),
|
||||
],
|
||||
)
|
||||
print(result["last_message"].text)
|
||||
```
|
||||
|
||||
`ImageContent` can be created from a URL, a local file path, or a PDF page using the `PDFToImageContent` converter.
|
||||
|
||||
### In a pipeline
|
||||
|
||||
When an `Agent` sits inside a pipeline, use `ChatPromptBuilder` with its string template format and the `| templatize_part` filter to pass images as structured content parts:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ImageContent
|
||||
|
||||
template = """
|
||||
{% message role="user" %}
|
||||
{{ question }}
|
||||
{{ image | templatize_part }}
|
||||
{% endmessage %}
|
||||
"""
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5"),
|
||||
system_prompt="You are a helpful assistant that can analyze images.",
|
||||
)
|
||||
prompt_builder = ChatPromptBuilder(
|
||||
template=template,
|
||||
required_variables=["question", "image"],
|
||||
)
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("prompt_builder", prompt_builder)
|
||||
pipeline.add_component("agent", agent)
|
||||
pipeline.connect("prompt_builder.prompt", "agent.messages")
|
||||
|
||||
# Download or provide your own chart image as "chart.png"
|
||||
image = ImageContent.from_file_path("chart.png")
|
||||
result = pipeline.run(
|
||||
{
|
||||
"prompt_builder": {"question": "What does this chart show?", "image": image},
|
||||
},
|
||||
)
|
||||
print(result["agent"]["last_message"].text)
|
||||
```
|
||||
|
||||
:::tip
|
||||
See these cookbooks for complete multimodal agent examples:
|
||||
- [Multimodal Agents](https://haystack.deepset.ai/cookbook/multimodal_intro#multimodal-agent) — image inputs and tool use with agents
|
||||
- [Gemma Chat RAG](https://haystack.deepset.ai/cookbook/gemma_chat_rag) — vision model in a RAG pipeline
|
||||
:::
|
||||
|
||||
## Multi-Agent Systems
|
||||
|
||||
You can wrap an `Agent` as a tool to build multi-agent systems where specialist agents handle focused subtasks and a coordinator agent plans and delegates.
|
||||
|
||||
See [Multi-Agent Systems](../../concepts/agents/multi-agent-systems.mdx) for a full guide, including the recommended `@tool` decorator approach for full interface control and `ComponentTool` for declarative configuration.
|
||||
|
||||
## MCP Integration
|
||||
|
||||
Agents work with MCP in two directions:
|
||||
|
||||
- **Consuming MCP tools**: Pass `MCPTool` or `MCPToolset` instances in the `tools` list to call tools on any MCP-compatible server (filesystem, browser, databases, and more). See [MCPTool](../../tools/mcptool.mdx) and [MCPToolset](../../tools/mcptoolset.mdx).
|
||||
- **Exposing as an MCP server**: Use [Hayhooks](../../development/hayhooks.mdx) to deploy your agent and expose it as an MCP server, making it callable from any MCP-compatible client such as Claude Desktop or Cursor.
|
||||
|
||||
## Additional References
|
||||
|
||||
📖 Related docs:
|
||||
|
||||
- [State](./state.mdx) — managing shared data between tools
|
||||
- [Hooks](./hooks.mdx) — running custom logic at defined points of the run loop
|
||||
- [Human in the Loop](./human-in-the-loop.mdx) — intercepting tool calls for human review
|
||||
- [Tool Result Offloading](./tool-result-offloading.mdx) — keeping large tool results out of the context window
|
||||
|
||||
📚 Tutorials:
|
||||
|
||||
- [Build a Tool-Calling Agent](https://haystack.deepset.ai/tutorials/43_building_a_tool_calling_agent)
|
||||
- [Creating a Multi-Agent System](https://haystack.deepset.ai/tutorials/45_creating_a_multi_agent_system)
|
||||
- [Human-in-the-Loop with Haystack Agents](https://haystack.deepset.ai/tutorials/47_human_in_the_loop_agent/)
|
||||
|
||||
🧑🍳 Cookbook:
|
||||
|
||||
- [Build a GitHub Issue Resolver Agent](https://haystack.deepset.ai/cookbook/github_issue_resolver_agent)
|
||||
- [Multimodal Agents](https://haystack.deepset.ai/cookbook/multimodal_intro#multimodal-agent)
|
||||
- [Gemma Chat RAG](https://haystack.deepset.ai/cookbook/gemma_chat_rag)
|
||||
@@ -0,0 +1,202 @@
|
||||
---
|
||||
title: "Hooks"
|
||||
id: hooks
|
||||
slug: "/hooks"
|
||||
description: "Hooks let you run custom logic at defined points of an Agent's run loop — before each LLM call, before and after tool execution, and on exit."
|
||||
---
|
||||
|
||||
# Hooks
|
||||
|
||||
Hooks let you run custom logic at defined points of an [`Agent`](./agent.mdx)'s run loop — before each LLM call, before and after tool execution, and on exit.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Configured on** | The [`Agent`](./agent.mdx) component via the `hooks` parameter |
|
||||
| **Key classes** | `hook` (decorator), `FunctionHook`, `Hook` (protocol) |
|
||||
| **Import path** | `haystack.hooks` |
|
||||
| **API reference** | [Hooks](/reference/hooks-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/hooks/ |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
Pass `hooks` to the `Agent` as a dictionary mapping a *hook point* to a list of hooks the Agent runs at that point. Each hook receives the live [`State`](./state.mdx) and influences the run by mutating it in place. Hooks for a hook point run in list order, and the same hook can be registered under multiple hook points.
|
||||
|
||||
This enables patterns such as building run-time system context, retrieving memories before the first LLM call, auditing or intercepting tool calls, and requiring a condition to hold before the Agent is allowed to finish.
|
||||
|
||||
### Hook points
|
||||
|
||||
- `before_llm`: Runs before each chat-generator call.
|
||||
- `before_tool`: Runs after the model requests tool calls, before any tools run. After these hooks run, the Agent re-reads the current last message from `state.data["messages"]`. If that message contains tool calls, those calls are executed. If it does not, no tools run for that step, no tool-based exit condition is triggered, and the Agent loops back to the next LLM call unless `max_agent_steps` has been reached.
|
||||
- `after_tool`: Runs after tools execute, once their result messages are in `state.data["messages"]`, before the exit-condition check and the next LLM call. Use it to rewrite the freshly produced tool-result messages — for example, to offload, redact, truncate, or summarize results. It does not run on the plain-text exit step. It does still run when a `before_tool` hook removed the pending tool calls: no tools executed on that step, so don't assume the last message is a fresh tool result.
|
||||
- `on_exit`: Runs when the Agent is about to stop on an exit condition. An `on_exit` hook can keep the Agent running by setting the `continue_run` control flag (`state.set("continue_run", True)`), usually alongside a message telling the model what to do next. `on_exit` hooks run when the Agent stops on an exit condition, but not when it stops because `max_agent_steps` is reached.
|
||||
|
||||
Registering a hook under an unknown hook point raises a `ValueError` at construction. A hook class can declare an `allowed_hook_points` attribute listing the hook points it supports; the Agent validates it and fails fast if the hook is registered somewhere it doesn't belong.
|
||||
|
||||
### State keys for hooks
|
||||
|
||||
The Agent manages a few state keys that hooks interact with. Like the run-metadata keys (`step_count`, `token_usage`, `tool_call_counts`), they are reserved — using any of them in your own `state_schema` raises a `ValueError`. See [State](./state.mdx#schema-definition) for the full list:
|
||||
|
||||
- `continue_run`: Set by an `on_exit` hook to keep the Agent running.
|
||||
- `tools`: The tools available in the current step, for hooks to inspect.
|
||||
- `hook_context`: Request-scoped resources passed to `Agent.run(hook_context={...})` / `run_async(hook_context={...})`. Hooks read it with `state.data["hook_context"]` or `state.data.get("hook_context")` — use it for per-request resources such as a user ID, a WebSocket, or a database client. Avoid the plain `state.get("hook_context")` here: `State.get` returns a deep copy of the value, which often fails for the kinds of resources stored in this dict (such as a WebSocket or a database client).
|
||||
|
||||
Hooks can also read the automatically tracked run metadata: `step_count`, `token_usage`, and `tool_call_counts`.
|
||||
|
||||
## Creating hooks
|
||||
|
||||
### With the `@hook` decorator
|
||||
|
||||
The `@hook` decorator wraps a function taking a single `State` argument into a hook. A regular function becomes the hook's sync path, a coroutine function its async path. To give a single hook both paths, construct a `FunctionHook` directly with both `function` and `async_function`.
|
||||
|
||||
The example below registers a hook at each of `before_llm`, `before_tool`, and `on_exit` to show what hooks can do:
|
||||
|
||||
```python
|
||||
from datetime import datetime, timezone
|
||||
from typing import Annotated
|
||||
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.agents.state import State, replace_values
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.hooks import hook
|
||||
from haystack.tools import tool
|
||||
|
||||
|
||||
@tool
|
||||
def search(query: Annotated[str, "The search query"]) -> str:
|
||||
"""Search the web."""
|
||||
# Placeholder: would call a real search API
|
||||
return "Fusion startups reported net-energy-gain milestones this year."
|
||||
|
||||
|
||||
@hook
|
||||
def build_context(state: State) -> None:
|
||||
# before_llm: build run-time system context once, before the first model call.
|
||||
if state.get("step_count") == 0:
|
||||
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
|
||||
system = ChatMessage.from_system(
|
||||
f"You are a research assistant. The current time is {now}.",
|
||||
)
|
||||
state.set(
|
||||
"messages",
|
||||
[system, *state.data["messages"]],
|
||||
handler_override=replace_values,
|
||||
)
|
||||
|
||||
|
||||
@hook
|
||||
def audit_tool_calls(state: State) -> None:
|
||||
# before_tool: see which tools the model is about to run.
|
||||
pending = state.data["messages"][-1].tool_calls
|
||||
print(f"about to run: {[tc.tool_name for tc in pending]}")
|
||||
|
||||
|
||||
@hook
|
||||
def require_search(state: State) -> None:
|
||||
# on_exit: keep going until the agent has actually searched.
|
||||
if state.get("tool_call_counts", {}).get("search", 0) == 0:
|
||||
state.set("messages", [ChatMessage.from_system("Search before answering.")])
|
||||
state.set("continue_run", True)
|
||||
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
tools=[search],
|
||||
hooks={
|
||||
"before_llm": [build_context],
|
||||
"before_tool": [audit_tool_calls],
|
||||
"on_exit": [require_search],
|
||||
},
|
||||
)
|
||||
|
||||
result = agent.run(
|
||||
messages=[
|
||||
ChatMessage.from_user("What are the latest developments in fusion energy?"),
|
||||
],
|
||||
)
|
||||
print(result["last_message"].text)
|
||||
```
|
||||
|
||||
### Class-based hooks
|
||||
|
||||
A hook is any object with a `run(state)` method; it may additionally define `run_async(state)` for true async behavior. Class-based hooks may also implement the optional lifecycle methods `warm_up` / `warm_up_async` and `close` / `close_async`. The Agent calls them from its own `warm_up` / `close`, so a hook can defer opening clients or reading credentials until warm-up and release them on close.
|
||||
|
||||
When a class-based hook should be serializable (so an Agent using it can be serialized), implement `to_dict` / `from_dict`: store serializable constructor arguments on the hook and rebuild runtime clients from those values.
|
||||
|
||||
The example below is an `on_exit` hook that grades the Agent's answer with its own LLM and asks the Agent to improve a weak answer before finishing:
|
||||
|
||||
```python
|
||||
from typing import Any
|
||||
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.agents.state import State
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.core.serialization import default_from_dict, default_to_dict
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
|
||||
class GradeFinalAnswer:
|
||||
"""Grade the Agent's answer with an LLM and ask it to improve a weak answer before finishing."""
|
||||
|
||||
def __init__(self, model: str = "gpt-5.4-nano"):
|
||||
self.model = model
|
||||
self._judge = OpenAIChatGenerator(model=self.model)
|
||||
|
||||
def warm_up(self) -> None:
|
||||
# Warm up the judge's own client during the Agent's warm-up.
|
||||
self._judge.warm_up()
|
||||
|
||||
def close(self) -> None:
|
||||
# Release the judge's client during the Agent's close.
|
||||
self._judge.close()
|
||||
|
||||
def run(self, state: State) -> None:
|
||||
answer = state.data["messages"][-1].text or ""
|
||||
verdict = (
|
||||
self._judge.run(
|
||||
messages=[
|
||||
ChatMessage.from_user(
|
||||
f"Reply with only PASS or FAIL. Is this answer complete?\n\n{answer}",
|
||||
),
|
||||
],
|
||||
)["replies"][0].text
|
||||
or ""
|
||||
)
|
||||
if "FAIL" in verdict.upper():
|
||||
state.set(
|
||||
"messages",
|
||||
[
|
||||
ChatMessage.from_user(
|
||||
"Your answer was incomplete. Please improve it.",
|
||||
),
|
||||
],
|
||||
)
|
||||
state.set("continue_run", True)
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return default_to_dict(self, model=self.model)
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: dict[str, Any]) -> "GradeFinalAnswer":
|
||||
return default_from_dict(cls, data)
|
||||
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
hooks={"on_exit": [GradeFinalAnswer()]},
|
||||
)
|
||||
result = agent.run(messages=[ChatMessage.from_user("Explain how vaccines work.")])
|
||||
print(result["last_message"].text)
|
||||
```
|
||||
|
||||
## Ready-made hooks
|
||||
|
||||
Haystack ships two ready-made hooks:
|
||||
|
||||
- `ConfirmationHook`: A `before_tool` hook that applies Human-in-the-Loop confirmation strategies to pending tool calls — a human can confirm, modify, or reject the tool calls the model requested before they run. See [Human in the Loop](./human-in-the-loop.mdx).
|
||||
- `ToolResultOffloadHook`: An `after_tool` hook that offloads tool results to a `ToolResultStore` (such as `FileSystemToolResultStore`) and replaces them in the conversation with a compact pointer, so the next LLM call sees a reference instead of the full result. Per-tool policies (`AlwaysOffload`, `NeverOffload`, `OffloadOverChars`) control which results are offloaded. See [Tool Result Offloading](./tool-result-offloading.mdx).
|
||||
@@ -0,0 +1,321 @@
|
||||
---
|
||||
title: "Human in the Loop"
|
||||
id: human-in-the-loop
|
||||
slug: "/human-in-the-loop"
|
||||
description: "Human-in-the-loop allows you to intercept agent tool calls before execution, letting a human confirm, reject, or modify the tool parameters."
|
||||
---
|
||||
|
||||
# Human in the Loop
|
||||
|
||||
Human-in-the-loop (HITL) lets you intercept an agent's tool calls before they are executed.
|
||||
A human can **confirm**, **reject**, or **modify** the parameters of each tool call in real time.
|
||||
This is useful for high-stakes operations - such as sending emails, modifying databases, or making API calls - where you want a human to review the action first.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Configured on** | The [`Agent`](./agent.mdx) component, as a `ConfirmationHook` registered under the `before_tool` [hook point](./hooks.mdx) |
|
||||
| **Key classes** | `ConfirmationHook`, `BlockingConfirmationStrategy`, `AlwaysAskPolicy`, `AskOncePolicy`, `NeverAskPolicy`, `RichConsoleUI`, `SimpleConsoleUI` |
|
||||
| **Import path** | `haystack.human_in_the_loop` |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/human_in_the_loop/ |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
HITL is one application of the Agent's general [hooks](./hooks.mdx) mechanism: a `ConfirmationHook` registered under the `before_tool` hook point intercepts the tool calls the model requested before they run, and confirms, modifies, or rejects them by rewriting the conversation in the Agent's `State`.
|
||||
|
||||
The HITL system is composed of these layers:
|
||||
|
||||
- **`ConfirmationHook`** - the `before_tool` hook that applies your confirmation strategies to pending tool calls. Its `confirmation_strategies` mapping accepts a single tool name, a tuple of tool names, or the wildcard `"*"` that applies to any tool without a more specific entry.
|
||||
- **Strategy** - decides what to do when a tool is about to be called. The built-in `BlockingConfirmationStrategy` pauses execution and asks a human.
|
||||
- **Policy** - decides *when* to ask. Built-in policies: `AlwaysAskPolicy`, `NeverAskPolicy`, `AskOncePolicy`.
|
||||
- **UI** - the interface used to ask the human. Built-in UIs: `RichConsoleUI` (requires `rich`) and `SimpleConsoleUI` (stdlib only).
|
||||
|
||||
When the agent is about to invoke a tool, the strategy checks the policy.
|
||||
If the policy says to ask, the UI prompts the human with the tool name, description, and parameters. The human can:
|
||||
|
||||
- **Confirm** (`y`) - execute as-is
|
||||
- **Reject** (`n`) - skip execution and feed rejection feedback back to the LLM
|
||||
- **Modify** (`m`) - edit the parameters before execution
|
||||
|
||||
The agent then continues with the human's decision.
|
||||
|
||||
:::info
|
||||
Strategies see only the arguments the model produced for a tool call. Values injected from [`State`](./state.mdx) via a tool's `inputs_from_state` mapping are not included in what is presented for confirmation — that injection happens at tool execution time.
|
||||
:::
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic setup
|
||||
|
||||
```python
|
||||
from typing import Annotated
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.human_in_the_loop import (
|
||||
AlwaysAskPolicy,
|
||||
BlockingConfirmationStrategy,
|
||||
ConfirmationHook,
|
||||
SimpleConsoleUI,
|
||||
)
|
||||
from haystack.tools import tool
|
||||
|
||||
|
||||
@tool
|
||||
def send_email(
|
||||
to: Annotated[str, "The recipient email address"],
|
||||
subject: Annotated[str, "The email subject line"],
|
||||
body: Annotated[str, "The email body"],
|
||||
) -> str:
|
||||
"""Send an email to a recipient."""
|
||||
return f"Email sent to {to}."
|
||||
|
||||
|
||||
strategy = BlockingConfirmationStrategy(
|
||||
confirmation_policy=AlwaysAskPolicy(),
|
||||
confirmation_ui=SimpleConsoleUI(),
|
||||
)
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-mini"),
|
||||
tools=[send_email],
|
||||
hooks={
|
||||
"before_tool": [
|
||||
ConfirmationHook(confirmation_strategies={"send_email": strategy}),
|
||||
],
|
||||
},
|
||||
)
|
||||
|
||||
result = agent.run(
|
||||
messages=[ChatMessage.from_user("Send a welcome email to alice@example.com")],
|
||||
)
|
||||
```
|
||||
|
||||
When the agent calls `send_email`, the terminal will pause and show:
|
||||
|
||||
```
|
||||
--- Tool Execution Request ---
|
||||
Tool: send_email
|
||||
Description: Send an email to a recipient.
|
||||
Arguments:
|
||||
to: alice@example.com
|
||||
subject: Welcome!
|
||||
body: Hi Alice, welcome aboard!
|
||||
------------------------------
|
||||
Confirm execution? (y=confirm / n=reject / m=modify):
|
||||
```
|
||||
|
||||
### Using RichConsoleUI
|
||||
|
||||
`RichConsoleUI` provides a styled terminal prompt using the [`rich`](https://github.com/Textualize/rich) library:
|
||||
|
||||
```shell
|
||||
pip install rich
|
||||
```
|
||||
|
||||
```python
|
||||
from haystack.human_in_the_loop import RichConsoleUI
|
||||
|
||||
strategy = BlockingConfirmationStrategy(
|
||||
confirmation_policy=AlwaysAskPolicy(),
|
||||
confirmation_ui=RichConsoleUI(),
|
||||
)
|
||||
```
|
||||
|
||||
### Applying strategies to multiple tools
|
||||
|
||||
You can configure different strategies per tool, share one strategy across a group of tools using a tuple key, or set a default for all tools with the wildcard `"*"` (applied to any tool without a more specific entry):
|
||||
|
||||
```python
|
||||
@tool
|
||||
def delete_record(record_id: Annotated[str, "The ID of the record to delete"]) -> str:
|
||||
"""Delete a record from the database."""
|
||||
return f"Record {record_id} deleted."
|
||||
|
||||
|
||||
@tool
|
||||
def update_record(
|
||||
record_id: Annotated[str, "The ID of the record to update"],
|
||||
data: Annotated[str, "The new data as a JSON string"],
|
||||
) -> str:
|
||||
"""Update a record in the database."""
|
||||
return f"Record {record_id} updated."
|
||||
|
||||
|
||||
@tool
|
||||
def search(query: Annotated[str, "The search query"]) -> str:
|
||||
"""Search the knowledge base."""
|
||||
return f"Results for: {query}"
|
||||
|
||||
|
||||
ask_strategy = BlockingConfirmationStrategy(
|
||||
confirmation_policy=AlwaysAskPolicy(),
|
||||
confirmation_ui=SimpleConsoleUI(),
|
||||
)
|
||||
|
||||
confirmation_hook = ConfirmationHook(
|
||||
confirmation_strategies={
|
||||
# Share one strategy across multiple sensitive tools using a tuple key
|
||||
("send_email", "delete_record", "update_record"): ask_strategy,
|
||||
# search has no strategy - always executes without asking
|
||||
},
|
||||
)
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-mini"),
|
||||
tools=[send_email, delete_record, update_record, search],
|
||||
hooks={"before_tool": [confirmation_hook]},
|
||||
)
|
||||
```
|
||||
|
||||
### Customizing feedback messages
|
||||
|
||||
When a tool call is rejected or modified, `BlockingConfirmationStrategy` sends a message back to the LLM explaining what happened. Three optional template parameters control these messages — each has a sensible default, so you only need to set them if you want different wording:
|
||||
|
||||
- `reject_template`: Sent to the LLM when the user rejects a tool call. Must include a `{tool_name}` placeholder. Default: `"Tool execution for '{tool_name}' was rejected by the user."`
|
||||
- `modify_template`: Sent when the user modifies the parameters. Must include `{tool_name}` and `{final_tool_params}` placeholders. Default: `"The parameters for tool '{tool_name}' were updated by the user to:\n{final_tool_params}"`
|
||||
- `user_feedback_template`: Appends the user's optional free-text feedback to either message. Must include a `{feedback}` placeholder. Default: `"With user feedback: {feedback}"`
|
||||
|
||||
```python
|
||||
strategy = BlockingConfirmationStrategy(
|
||||
confirmation_policy=AlwaysAskPolicy(),
|
||||
confirmation_ui=SimpleConsoleUI(),
|
||||
reject_template="Skipping '{tool_name}' — rejected by operator.",
|
||||
modify_template="Updated parameters for '{tool_name}': {final_tool_params}",
|
||||
user_feedback_template="Reason: {feedback}",
|
||||
)
|
||||
```
|
||||
|
||||
## Policies
|
||||
|
||||
Policies control *when* the human is asked.
|
||||
|
||||
| Policy | Behavior |
|
||||
| --- | --- |
|
||||
| `AlwaysAskPolicy` | Ask every time the tool is called |
|
||||
| `NeverAskPolicy` | Never ask - always proceed (useful for toggling HITL off without removing the strategy) |
|
||||
| `AskOncePolicy` | Ask once per unique `(tool_name, parameters)` combination. Remembers confirmed calls and skips asking on repeats. |
|
||||
|
||||
### Custom policy
|
||||
|
||||
You can implement your own policy by subclassing `ConfirmationPolicy` from `haystack.human_in_the_loop.types`:
|
||||
|
||||
```python
|
||||
from haystack.human_in_the_loop.types import ConfirmationPolicy, ConfirmationUIResult
|
||||
from typing import Any
|
||||
|
||||
|
||||
class AskForSensitiveParamsPolicy(ConfirmationPolicy):
|
||||
"""Only ask when the 'to' parameter looks like an external email domain."""
|
||||
|
||||
def should_ask(
|
||||
self,
|
||||
tool_name: str,
|
||||
tool_description: str,
|
||||
tool_params: dict[str, Any],
|
||||
) -> bool:
|
||||
to = tool_params.get("to", "")
|
||||
return not to.endswith("@mycompany.com")
|
||||
```
|
||||
|
||||
For stateful policies, also implement `update_after_confirmation`.
|
||||
It is called after the user responds and receives the full `ConfirmationUIResult`, letting you update internal state based on the outcome.
|
||||
The following policy asks once per tool name and skips re-asking for any tool the user has already confirmed:
|
||||
|
||||
```python
|
||||
from haystack.human_in_the_loop.types import ConfirmationPolicy
|
||||
from haystack.human_in_the_loop import ConfirmationUIResult
|
||||
from typing import Any
|
||||
|
||||
|
||||
class AskOncePerToolPolicy(ConfirmationPolicy):
|
||||
"""Ask once per tool name, regardless of parameters. Skip on repeat confirmed calls."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._confirmed_tools: set[str] = set()
|
||||
|
||||
def should_ask(
|
||||
self,
|
||||
tool_name: str,
|
||||
tool_description: str,
|
||||
tool_params: dict[str, Any],
|
||||
) -> bool:
|
||||
return tool_name not in self._confirmed_tools
|
||||
|
||||
def update_after_confirmation(
|
||||
self,
|
||||
tool_name: str,
|
||||
tool_description: str,
|
||||
tool_params: dict[str, Any],
|
||||
confirmation_result: ConfirmationUIResult,
|
||||
) -> None:
|
||||
if confirmation_result.action == "confirm":
|
||||
self._confirmed_tools.add(tool_name)
|
||||
```
|
||||
|
||||
## Dataclasses
|
||||
|
||||
### `ConfirmationUIResult`
|
||||
|
||||
Returned by the UI after the human responds.
|
||||
|
||||
| Field | Type | Description |
|
||||
| --- | --- | --- |
|
||||
| `action` | `str` | `"confirm"`, `"reject"`, or `"modify"` |
|
||||
| `feedback` | `str \| None` | Optional free-text feedback from the human |
|
||||
| `new_tool_params` | `dict \| None` | Replacement parameters when action is `"modify"` |
|
||||
|
||||
### `ToolExecutionDecision`
|
||||
|
||||
Returned by the strategy to the agent.
|
||||
|
||||
| Field | Type | Description |
|
||||
| --- | --- | --- |
|
||||
| `tool_name` | `str` | Name of the tool |
|
||||
| `execute` | `bool` | Whether to execute the tool |
|
||||
| `tool_call_id` | `str \| None` | ID of the tool call |
|
||||
| `feedback` | `str \| None` | Feedback message passed back to the LLM on rejection or modification |
|
||||
| `final_tool_params` | `dict \| None` | Final parameters to use for execution |
|
||||
|
||||
## Example: HITL with Hayhooks and Open WebUI
|
||||
|
||||
The [hitl-hayhooks-redis-openwebui](https://github.com/deepset-ai/hitl-hayhooks-redis-openwebui) repository shows a full production-style HITL setup using a Haystack Agent served via [Hayhooks](https://github.com/deepset-ai/hayhooks) with approval dialogs rendered in [Open WebUI](https://github.com/open-webui/open-webui).
|
||||
|
||||
The key pattern it demonstrates is a custom `RedisConfirmationStrategy` that receives per-request resources - a Redis client and an async event queue - at runtime. Pass such resources via the generic `hook_context` run argument (`agent.run(messages=[...], hook_context={"redis": client})`). `ConfirmationHook` reads this dict from state with `state.data["hook_context"]` (not `state.get`, which returns a deep copy that fails for live resources like clients and queues - see [Hooks](./hooks.mdx)) and passes it to each strategy's `run()` as the `confirmation_strategy_context` keyword argument, which is how a custom strategy receives the Redis client and event queue:
|
||||
|
||||
- When a tool call is about to execute, the strategy emits a `tool_call_start` SSE event and blocks on `Redis BLPOP` waiting for an approval decision.
|
||||
- The Open WebUI Pipe function receives the SSE event, shows the user a confirmation dialog, then writes `approved` or `rejected` to Redis via `LPUSH`.
|
||||
- Once Redis unblocks, the strategy returns a `ToolExecutionDecision` and the agent continues.
|
||||
|
||||
This is a good reference if you need non-blocking HITL in a web or server environment where `SimpleConsoleUI` and `RichConsoleUI` are not suitable.
|
||||
|
||||
## Custom UI
|
||||
|
||||
Implement `ConfirmationUI` from `haystack.human_in_the_loop.types` to build your own interface - for example, a web-based approval queue:
|
||||
|
||||
```python
|
||||
from haystack.human_in_the_loop.types import ConfirmationUI
|
||||
from haystack.human_in_the_loop import ConfirmationUIResult
|
||||
from typing import Any
|
||||
|
||||
|
||||
class WebhookApprovalUI(ConfirmationUI):
|
||||
"""Sends a webhook and waits for an async approval response."""
|
||||
|
||||
def get_user_confirmation(
|
||||
self,
|
||||
tool_name: str,
|
||||
tool_description: str,
|
||||
tool_params: dict[str, Any],
|
||||
) -> ConfirmationUIResult:
|
||||
# Send approval request to your system and wait for response
|
||||
response = send_approval_request_and_wait(tool_name, tool_params)
|
||||
return ConfirmationUIResult(
|
||||
action=response["action"],
|
||||
feedback=response.get("feedback"),
|
||||
)
|
||||
```
|
||||
@@ -0,0 +1,424 @@
|
||||
---
|
||||
title: "State"
|
||||
id: state
|
||||
slug: "/state"
|
||||
description: "`State` is a container for storing shared information during Agent and Tool execution. It provides a structured way to share data between tools, accumulate results across multiple tool calls, and surface them alongside the agent's final answer."
|
||||
---
|
||||
|
||||
# State
|
||||
|
||||
`State` is a container for storing shared information during Agent and Tool execution.
|
||||
It provides a structured way to share data between tools, accumulate results across multiple tool calls, and surface them alongside the agent's final answer.
|
||||
|
||||
## Overview
|
||||
|
||||
When building agents that use multiple tools, you often need tools to share information or accumulate results across iterations.
|
||||
State provides centralized storage that all tools can read from and write to.
|
||||
For example, a search tool called multiple times can append its results to a shared `documents` list, which is then returned alongside the agent's final answer for source inspection.
|
||||
|
||||
State uses a schema-based approach where you define:
|
||||
|
||||
- What data can be stored,
|
||||
- The type of each piece of data,
|
||||
- How values are merged when updated.
|
||||
|
||||
The Agent creates and manages the `State` object internally. You shouldn't need to instantiate it directly.
|
||||
You interact with it through tool definitions (`inputs_from_state`, `outputs_to_state`, or a `state: State` parameter) and read results from the agent's output dict.
|
||||
|
||||
### Supported Types
|
||||
|
||||
State supports standard Python types:
|
||||
|
||||
- Basic types: `str`, `int`, `float`, `bool`, `dict`
|
||||
- List types: `list`, `list[str]`, `list[int]`, `list[Document]`
|
||||
- Union types: `str | int`, `str | None`
|
||||
- Custom classes and data classes.
|
||||
|
||||
### Automatic Message Handling
|
||||
|
||||
State automatically includes a `messages` field that stores the full conversation history during execution.
|
||||
You don't need to define this in your schema.
|
||||
It uses `list[ChatMessage]` type with the `merge_lists` handler, so new messages are appended on each iteration.
|
||||
|
||||
### State API
|
||||
|
||||
| Method | Description |
|
||||
| --- | --- |
|
||||
| `state.get(key, default=None)` | Read a value; returns `default` if the key doesn't exist |
|
||||
| `state.set(key, value)` | Write a value, merged using the schema's handler |
|
||||
| `state.has(key)` | Returns `True` if the key exists in state |
|
||||
| `state.data` | Returns a snapshot of all current state as a `dict` |
|
||||
|
||||
## Schema Definition
|
||||
|
||||
The schema defines what data can be stored and how values are updated. Each schema entry consists of:
|
||||
|
||||
- `type` (required): The Python type for this field (for example, `str`, `int`, `list`)
|
||||
- `handler` (optional): A callable that determines how new values are merged when `set()` is called
|
||||
|
||||
```python
|
||||
{
|
||||
"parameter_name": {
|
||||
"type": SomeType, # Required: expected Python type
|
||||
"handler": some_func, # Optional: merge function
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
If you don't specify a handler, State automatically assigns a default based on the type.
|
||||
|
||||
:::info Reserved keys
|
||||
The `Agent` manages some state keys itself and rejects them in a user-provided `state_schema` with a `ValueError`:
|
||||
|
||||
- The run-metadata keys `step_count`, `token_usage`, and `tool_call_counts`, which the Agent populates automatically during a run: tools can read them mid-run via `inputs_from_state`, and they are returned in the result dictionary.
|
||||
- The hook-facing keys `continue_run` (set by an `on_exit` hook to keep the Agent running), `tools` (the tools available in the current step, for hooks to inspect), and `hook_context` (the request-scoped resources passed to `Agent.run(hook_context={...})`).
|
||||
|
||||
If one of your state keys clashes, rename it (for example, `my_token_usage`).
|
||||
:::
|
||||
|
||||
### Default Handlers
|
||||
|
||||
State provides two built-in merge behaviors (importable from `haystack.components.agents.state`):
|
||||
|
||||
- **`merge_lists`**: Appends to the existing list (default for list types)
|
||||
- **`replace_values`**: Overwrites the existing value (default for non-list types)
|
||||
|
||||
```python
|
||||
from haystack.components.agents import State
|
||||
|
||||
schema = {
|
||||
"documents": {"type": list}, # uses merge_lists by default
|
||||
"user_name": {"type": str}, # uses replace_values by default
|
||||
}
|
||||
|
||||
state = State(schema=schema)
|
||||
|
||||
state.set("documents", [1, 2])
|
||||
state.set("documents", [3, 4])
|
||||
print(state.get("documents")) # [1, 2, 3, 4]
|
||||
|
||||
state.set("user_name", "Alice")
|
||||
state.set("user_name", "Bob")
|
||||
print(state.get("user_name")) # "Bob"
|
||||
```
|
||||
|
||||
### Custom Handlers
|
||||
|
||||
Custom handlers are useful when the default `merge_lists` or `replace_values` behaviors don't fit your needs.
|
||||
A handler takes the current state value and the new value and returns the merged result.
|
||||
|
||||
The example below uses a deduplication handler, useful when multiple tool calls might return overlapping results and you want to avoid accumulating duplicates in state:
|
||||
|
||||
```python
|
||||
def deduplicate(current_value: list | None, new_value: list) -> list:
|
||||
"""Append new items, skipping any already in the list."""
|
||||
existing = set(current_value or [])
|
||||
return (current_value or []) + [item for item in new_value if item not in existing]
|
||||
|
||||
|
||||
schema = {"doc_ids": {"type": list, "handler": deduplicate}}
|
||||
|
||||
state = State(schema=schema)
|
||||
state.set("doc_ids", ["doc-1", "doc-2"])
|
||||
state.set("doc_ids", ["doc-2", "doc-3"])
|
||||
print(state.get("doc_ids")) # ["doc-1", "doc-2", "doc-3"]
|
||||
```
|
||||
|
||||
You can also override the handler for a single `set()` call:
|
||||
|
||||
```python
|
||||
from haystack.components.agents import State
|
||||
|
||||
|
||||
def concatenate_strings(current: str | None, new: str) -> str:
|
||||
return f"{current}-{new}" if current else new
|
||||
|
||||
|
||||
state = State(schema={"user_name": {"type": str}})
|
||||
state.set("user_name", "Alice")
|
||||
state.set("user_name", "Bob", handler_override=concatenate_strings)
|
||||
print(state.get("user_name")) # "Alice-Bob"
|
||||
```
|
||||
|
||||
## Using State
|
||||
|
||||
Define a `state_schema` when creating the Agent.
|
||||
State keys declared in `state_schema` are exposed as output keys on the agent's result dict alongside `messages` and `last_message`.
|
||||
|
||||
Tools interact with State through three mechanisms:
|
||||
|
||||
- **`outputs_to_state`**: Write tool results to state keys after the tool runs.
|
||||
- **`inputs_from_state`**: Inject state values into tool parameters before the tool runs.
|
||||
- **Direct `State` injection**: Add a `state: State` parameter to your tool function's signature. The Agent detects the `State` annotation and injects the live `State` object automatically, so you can read or write any key defined in the schema. The `State` object is never exposed to the LLM's parameter schema.
|
||||
|
||||
### Reading from State: `inputs_from_state`
|
||||
|
||||
`inputs_from_state` maps state keys to function parameter names using the format `{"state_key": "param_name"}`.
|
||||
The value is injected from state before the tool runs, so the LLM never needs to provide it.
|
||||
|
||||
Parameters mapped via `inputs_from_state` are automatically excluded from the LLM's parameter schema.
|
||||
The model never sees or provides them:
|
||||
|
||||
```python
|
||||
from typing import Annotated
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.tools import tool
|
||||
|
||||
|
||||
@tool(inputs_from_state={"user_name": "user_context"})
|
||||
def search_documents(
|
||||
query: Annotated[str, "The search query"],
|
||||
user_context: str, # injected from state; excluded from LLM schema
|
||||
) -> dict:
|
||||
"""Search documents using query and user context."""
|
||||
return {"results": [f"Found results for '{query}' (user: {user_context})"]}
|
||||
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
tools=[search_documents],
|
||||
system_prompt="Use the search_documents tool to find information.",
|
||||
streaming_callback=print_streaming_chunk,
|
||||
state_schema={"user_name": {"type": str}},
|
||||
)
|
||||
|
||||
result = agent.run(
|
||||
messages=[ChatMessage.from_user("Search for Python tutorials")],
|
||||
user_name="Alice", # state key "user_name" is pre-populated by passing user_name= to agent.run()
|
||||
)
|
||||
|
||||
print(result["last_message"].text)
|
||||
```
|
||||
|
||||
### Writing to State: `outputs_to_state`
|
||||
|
||||
The `outputs_to_state` parameter maps tool output keys to state keys. Each entry supports two optional fields:
|
||||
|
||||
```python
|
||||
{
|
||||
"state_key": {
|
||||
"source": "tool_output_key", # which key to read from the tool's return dict; omit to store the entire dict
|
||||
"handler": some_func, # override the schema's merge handler for this mapping only
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
```python
|
||||
from typing import Annotated
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.tools import tool
|
||||
|
||||
|
||||
@tool(
|
||||
outputs_to_state={
|
||||
"documents": {"source": "documents"},
|
||||
"result_count": {"source": "count"},
|
||||
"last_query": {"source": "query"},
|
||||
},
|
||||
)
|
||||
def retrieve_documents(
|
||||
query: Annotated[str, "The search query"],
|
||||
) -> dict:
|
||||
"""Retrieve relevant documents."""
|
||||
return {
|
||||
"documents": [
|
||||
{"title": "Doc 1", "content": "Content about Python"},
|
||||
{"title": "Doc 2", "content": "More about Python"},
|
||||
],
|
||||
"count": 2,
|
||||
"query": query,
|
||||
}
|
||||
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
tools=[retrieve_documents],
|
||||
system_prompt="Use the retrieve_documents tool to find information.",
|
||||
streaming_callback=print_streaming_chunk,
|
||||
state_schema={
|
||||
"documents": {"type": list},
|
||||
"result_count": {"type": int},
|
||||
"last_query": {"type": str},
|
||||
},
|
||||
)
|
||||
|
||||
result = agent.run(messages=[ChatMessage.from_user("Find information about Python")])
|
||||
|
||||
print(f"Documents: {result['documents']}")
|
||||
print(f"Result count: {result['result_count']}")
|
||||
print(f"Last query: {result['last_query']}")
|
||||
```
|
||||
|
||||
If you omit `source`, the entire tool result dict is stored under the state key:
|
||||
|
||||
```python
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.tools import tool
|
||||
|
||||
|
||||
@tool(outputs_to_state={"user_info": {}})
|
||||
def get_user_info() -> dict:
|
||||
"""Get user information."""
|
||||
return {"name": "Alice", "email": "alice@example.com", "role": "admin"}
|
||||
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
tools=[get_user_info],
|
||||
system_prompt="Use the get_user_info tool to look up user details.",
|
||||
streaming_callback=print_streaming_chunk,
|
||||
state_schema={"user_info": {"type": dict}},
|
||||
)
|
||||
|
||||
result = agent.run(messages=[ChatMessage.from_user("What are the user's details?")])
|
||||
|
||||
print(result["last_message"].text)
|
||||
print(f"User info: {result['user_info']}")
|
||||
```
|
||||
|
||||
### Combining Inputs and Outputs
|
||||
|
||||
Tools can both read from and write to State, enabling tool chaining across iterations.
|
||||
This example builds on `retrieve_documents` from the previous section:
|
||||
|
||||
```python
|
||||
@tool(
|
||||
inputs_from_state={"documents": "documents"},
|
||||
outputs_to_state={
|
||||
"final_docs": {"source": "processed_docs"},
|
||||
"final_count": {"source": "processed_count"},
|
||||
},
|
||||
)
|
||||
def process_documents(
|
||||
max_results: Annotated[int, "Maximum number of documents to return"],
|
||||
documents: list = None, # injected from state; LLM does not provide this
|
||||
) -> dict:
|
||||
"""Process retrieved documents and return a filtered subset."""
|
||||
processed = (documents or [])[:max_results]
|
||||
return {"processed_docs": processed, "processed_count": len(processed)}
|
||||
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
tools=[retrieve_documents, process_documents], # chained through state
|
||||
system_prompt="Use the available tools to retrieve and process documents.",
|
||||
streaming_callback=print_streaming_chunk,
|
||||
state_schema={
|
||||
"documents": {"type": list},
|
||||
"result_count": {"type": int},
|
||||
"last_query": {"type": str},
|
||||
"final_docs": {"type": list},
|
||||
"final_count": {"type": int},
|
||||
},
|
||||
)
|
||||
|
||||
result = agent.run(
|
||||
messages=[ChatMessage.from_user("Find and process 3 documents about Python")],
|
||||
)
|
||||
print(f"Processed {result['final_count']} documents")
|
||||
```
|
||||
|
||||
### Injecting State Directly into Tools
|
||||
|
||||
As an alternative to `inputs_from_state` and `outputs_to_state`, a tool can declare a `state: State` parameter to receive the live `State` object at invocation time.
|
||||
This lets the tool read from and write to any number of state keys without declaring mappings upfront.
|
||||
|
||||
The ToolInvoker detects the `State` annotation and injects the object automatically.
|
||||
It is excluded from the LLM-facing schema. The model is never asked to supply it.
|
||||
Both `State` and `State | None` annotations are supported.
|
||||
|
||||
For function-based tools, add the `state` parameter and use the `@tool` decorator:
|
||||
|
||||
```python
|
||||
from typing import Annotated
|
||||
from haystack.components.agents import Agent, State
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
from haystack.dataclasses import ChatMessage, Document
|
||||
from haystack.tools import tool
|
||||
|
||||
|
||||
@tool
|
||||
def retrieve_and_store(
|
||||
query: Annotated[str, "The search query"],
|
||||
state: State,
|
||||
) -> str:
|
||||
"""Retrieve documents and store them directly in state."""
|
||||
documents = [Document(content=f"Result for '{query}'")]
|
||||
state.set("documents", documents)
|
||||
user_name = state.get("user_name", "unknown")
|
||||
return f"Retrieved {len(documents)} document(s) for {user_name}"
|
||||
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
tools=[retrieve_and_store],
|
||||
system_prompt="Use the retrieve_and_store tool to find documents.",
|
||||
streaming_callback=print_streaming_chunk,
|
||||
state_schema={"documents": {"type": list[Document]}, "user_name": {"type": str}},
|
||||
)
|
||||
|
||||
result = agent.run(
|
||||
messages=[ChatMessage.from_user("Find documents about Python")],
|
||||
user_name="Alice",
|
||||
)
|
||||
|
||||
print(result["last_message"].text)
|
||||
print(result["documents"])
|
||||
```
|
||||
|
||||
For component-based tools, declare a `State` input socket on the `run` method and wrap it with `ComponentTool`:
|
||||
|
||||
```python
|
||||
from haystack import component
|
||||
from haystack.components.agents import Agent, State
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
from haystack.dataclasses import ChatMessage, Document
|
||||
from haystack.tools import ComponentTool
|
||||
|
||||
|
||||
@component
|
||||
class DocumentRetriever:
|
||||
"""Retrieve documents and store them in state."""
|
||||
|
||||
@component.output_types(reply=str)
|
||||
def run(self, query: str, state: State) -> dict[str, str]:
|
||||
"""
|
||||
Retrieve documents based on query and store them in state.
|
||||
|
||||
:param query: The search query
|
||||
"""
|
||||
documents = [Document(content=f"Result for '{query}'")]
|
||||
state.set("documents", documents)
|
||||
return {"reply": f"Retrieved {len(documents)} document(s)"}
|
||||
|
||||
|
||||
retriever_tool = ComponentTool(
|
||||
component=DocumentRetriever(),
|
||||
name="retrieve",
|
||||
description="Retrieve documents based on a search query",
|
||||
)
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
tools=[retriever_tool],
|
||||
system_prompt="Use the retrieve tool to find documents.",
|
||||
streaming_callback=print_streaming_chunk,
|
||||
state_schema={"documents": {"type": list[Document]}},
|
||||
)
|
||||
|
||||
result = agent.run(messages=[ChatMessage.from_user("Find documents about Python")])
|
||||
|
||||
print(result["last_message"].text)
|
||||
print(result["documents"])
|
||||
```
|
||||
@@ -0,0 +1,224 @@
|
||||
---
|
||||
title: "Tool Result Offloading"
|
||||
id: tool-result-offloading
|
||||
slug: "/tool-result-offloading"
|
||||
description: "Tool result offloading writes large tool results to a store and replaces them in the conversation with a compact pointer, keeping the Agent's context window small."
|
||||
---
|
||||
|
||||
# Tool Result Offloading
|
||||
|
||||
Tool result offloading writes selected tool results to a store and replaces them in the conversation with a compact pointer — a reference plus a short preview — so the next LLM call sees a reference instead of the full result.
|
||||
This keeps the context window small when tools return large outputs (web pages, file contents, query results), and it is a step towards letting an Agent operate on offloaded results with follow-up tools, such as a file-reading tool that opens the referenced files.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Configured on** | The [`Agent`](./agent.mdx) component, as a `ToolResultOffloadHook` registered under the `after_tool` [hook point](./hooks.mdx) |
|
||||
| **Key classes** | `ToolResultOffloadHook`, `FileSystemToolResultStore`, `AlwaysOffload`, `NeverOffload`, `OffloadOverChars` |
|
||||
| **Import path** | `haystack.hooks.tool_result_offloading` |
|
||||
| **API reference** | [Hooks](/reference/hooks-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/hooks/tool_result_offloading/ |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
Tool result offloading is one application of the Agent's general [hooks](./hooks.mdx) mechanism: a `ToolResultOffloadHook` registered under the `after_tool` hook point runs after each step's tools execute and rewrites the freshly produced tool-result messages in the Agent's [`State`](./state.mdx). It only considers the current step's results; earlier conversation history is left untouched.
|
||||
|
||||
The system is composed of these layers:
|
||||
|
||||
- **`ToolResultOffloadHook`** - the `after_tool` hook that applies your offload strategies to fresh tool results. Its `offload_strategies` mapping accepts a single tool name, a tuple of tool names, or the wildcard `"*"` that applies to any tool without a more specific entry.
|
||||
- **Policy** - decides *whether* a given result is offloaded. Built-in policies: `AlwaysOffload`, `NeverOffload`, `OffloadOverChars`.
|
||||
- **Store** - decides *where* the full result lives. The built-in `FileSystemToolResultStore` writes results to the local file system.
|
||||
|
||||
When a result is offloaded, the hook writes the full text to the store and rebuilds the message with a one-line pointer in its place:
|
||||
|
||||
```
|
||||
Tool result offloaded to '/abs/path/tool_results/2_search_call-123.txt' (18234 characters). Preview: Fusion startups reported...
|
||||
```
|
||||
|
||||
The pointer carries the store reference, the original length, and a preview of the first `preview_chars` characters (200 by default, configurable on the hook), so the model knows roughly what was offloaded and where to find it.
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic setup
|
||||
|
||||
The example below offloads any tool result longer than 4,000 characters to files under a local `tool_results` directory:
|
||||
|
||||
```python
|
||||
from typing import Annotated
|
||||
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.hooks.tool_result_offloading import (
|
||||
FileSystemToolResultStore,
|
||||
OffloadOverChars,
|
||||
ToolResultOffloadHook,
|
||||
)
|
||||
from haystack.tools import tool
|
||||
|
||||
|
||||
@tool
|
||||
def search(query: Annotated[str, "The search query"]) -> str:
|
||||
"""Search the web and return the (potentially large) results."""
|
||||
# Placeholder: would call a real search API
|
||||
return f"... large result for {query} ..."
|
||||
|
||||
|
||||
offload_hook = ToolResultOffloadHook(
|
||||
store=FileSystemToolResultStore(root="tool_results"),
|
||||
offload_strategies={"*": OffloadOverChars(4000)},
|
||||
)
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
tools=[search],
|
||||
hooks={"after_tool": [offload_hook]},
|
||||
)
|
||||
|
||||
result = agent.run(messages=[ChatMessage.from_user("Summarize today's tech news")])
|
||||
```
|
||||
|
||||
### Configuring what gets offloaded per tool
|
||||
|
||||
Each key in `offload_strategies` may be a single tool name, a tuple of tool names sharing one policy, or the wildcard `"*"`. More specific keys win over `"*"`, and a tool with no matching key (and no `"*"`) is never offloaded:
|
||||
|
||||
```python
|
||||
from haystack.hooks.tool_result_offloading import (
|
||||
AlwaysOffload,
|
||||
FileSystemToolResultStore,
|
||||
NeverOffload,
|
||||
OffloadOverChars,
|
||||
ToolResultOffloadHook,
|
||||
)
|
||||
|
||||
offload_hook = ToolResultOffloadHook(
|
||||
store=FileSystemToolResultStore(root="tool_results"),
|
||||
offload_strategies={
|
||||
"web_search": AlwaysOffload(), # force offload
|
||||
"get_time": NeverOffload(), # opt out of the wildcard default
|
||||
("read_file", "list_dir"): OffloadOverChars(4000), # tuple key: shared policy
|
||||
"*": OffloadOverChars(8000), # default for any unlisted tool
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
### What is offloaded
|
||||
|
||||
The hook only offloads **successful, text** tool results:
|
||||
|
||||
- Error results — including rejections produced by a `before_tool` [Human-in-the-Loop](./human-in-the-loop.mdx) hook — are always left in context, so the model sees what went wrong.
|
||||
- Non-text results (image or file content) are left in context; supporting only text is a deliberate choice for now. A warning is logged when a non-text result has a matching offload policy.
|
||||
- Each result is offloaded at most once, even though the hook runs on every tool step. This also means two offload hooks registered under `after_tool` won't offload each other's pointers.
|
||||
|
||||
## Policies
|
||||
|
||||
Policies control *whether* a result is offloaded.
|
||||
|
||||
| Policy | Behavior |
|
||||
| --- | --- |
|
||||
| `AlwaysOffload` | Offload every result of the tool it is assigned to |
|
||||
| `NeverOffload` | Never offload - keep the full result in context (useful to opt a tool out of a wildcard default) |
|
||||
| `OffloadOverChars(threshold)` | Offload only when the result is longer than `threshold` characters |
|
||||
|
||||
### Custom policy
|
||||
|
||||
Subclass the `OffloadPolicy` protocol from `haystack.hooks.tool_result_offloading` for custom conditions. A policy needs a `should_offload` method, which receives the tool name, the result text, and the Agent's live [`State`](./state.mdx), so it can also decide based on run context:
|
||||
|
||||
```python
|
||||
from haystack.components.agents.state import State
|
||||
from haystack.hooks.tool_result_offloading import OffloadPolicy
|
||||
|
||||
|
||||
class OffloadLateSteps(OffloadPolicy):
|
||||
"""Offload results only once the run is several steps deep and context pressure builds up."""
|
||||
|
||||
def should_offload(self, tool_name: str, result: str, state: State) -> bool:
|
||||
return state.data.get("step_count", 0) >= 3 and len(result) > 1000
|
||||
```
|
||||
|
||||
The protocol provides default `to_dict` / `from_dict` implementations, so a policy like this one, whose constructor takes no arguments, is serializable as-is. A policy with constructor arguments should implement both methods itself, following `OffloadOverChars` as an example.
|
||||
|
||||
## Stores
|
||||
|
||||
### `FileSystemToolResultStore`
|
||||
|
||||
`FileSystemToolResultStore(root=...)` writes each offloaded result to a file under its root directory and returns the absolute file path as the reference. The directory is created on first write. Store keys are derived from the step count, tool name, and tool call ID (for example `2_search_call-123.txt`), so results from different tools and steps do not collide. A key that would resolve outside the root directory is rejected.
|
||||
|
||||
### Custom store
|
||||
|
||||
Subclass the `ToolResultStore` protocol to target other backends, such as object storage or an isolated sandbox file system. A store needs two methods: `write(key=..., content=...)` persists the content and returns an opaque reference string, and `read(reference)` resolves that reference back to the content:
|
||||
|
||||
```python
|
||||
from haystack.hooks.tool_result_offloading import ToolResultStore
|
||||
|
||||
|
||||
class InMemoryToolResultStore(ToolResultStore):
|
||||
"""Keep offloaded results in a dict - useful for tests."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._data: dict[str, str] = {}
|
||||
|
||||
def write(self, *, key: str, content: str) -> str:
|
||||
self._data[key] = content
|
||||
return key
|
||||
|
||||
def read(self, reference: str) -> str:
|
||||
return self._data[reference]
|
||||
```
|
||||
|
||||
Like `OffloadPolicy`, the protocol provides default `to_dict` / `from_dict` implementations covering stores whose constructor takes no arguments; implement both methods for stores with constructor arguments.
|
||||
|
||||
### Per-run stores via `hook_context`
|
||||
|
||||
The constructor `store` is shared by every run - fine for single-user or local use. In a multi-user server, give each run its own isolated store (for example, a per-session directory) by passing it in the Agent's generic `hook_context` run argument under the key `RESULT_STORE_CONTEXT_KEY`. It overrides the constructor store for that run:
|
||||
|
||||
```python
|
||||
from haystack.hooks.tool_result_offloading import (
|
||||
RESULT_STORE_CONTEXT_KEY,
|
||||
FileSystemToolResultStore,
|
||||
)
|
||||
|
||||
per_request_store = FileSystemToolResultStore(root=f"tool_results/{session_id}")
|
||||
result = agent.run(
|
||||
messages=[ChatMessage.from_user("...")],
|
||||
hook_context={RESULT_STORE_CONTEXT_KEY: per_request_store},
|
||||
)
|
||||
```
|
||||
|
||||
Isolating the store per run keeps concurrent users from colliding on store keys or reading each other's offloaded results — especially important when a file-reading tool is scoped to the store. The hook itself keeps no mutable state, so a single instance is safe to share across concurrent runs.
|
||||
|
||||
## Letting the Agent read offloaded results back
|
||||
|
||||
The pointer left in the conversation tells the model where the full result lives, but the model can only act on it if the Agent has a tool that can read from the store. With `FileSystemToolResultStore`, that can be a simple file-reading tool:
|
||||
|
||||
```python
|
||||
from typing import Annotated
|
||||
|
||||
from haystack.tools import tool
|
||||
|
||||
|
||||
@tool
|
||||
def read_offloaded_result(
|
||||
path: Annotated[str, "Absolute path of an offloaded tool result"],
|
||||
) -> str:
|
||||
"""Read back the full content of an offloaded tool result."""
|
||||
return FileSystemToolResultStore(root="tool_results").read(path)
|
||||
```
|
||||
|
||||
With this tool available, the Agent can work with a compact conversation and selectively re-read only the offloaded results it actually needs — instead of carrying every full result in context on every LLM call.
|
||||
|
||||
## Serialization
|
||||
|
||||
`ToolResultOffloadHook` implements `to_dict` / `from_dict`, so an Agent using it can be serialized as long as the configured store and policies are serializable too. The built-in store and policies all are; for custom ones, see the notes in [Policies](#custom-policy) and [Stores](#custom-store) above.
|
||||
|
||||
## Additional References
|
||||
|
||||
📖 Related docs:
|
||||
|
||||
- [Hooks](./hooks.mdx) — the general mechanism behind this feature, including the `after_tool` hook point
|
||||
- [Human in the Loop](./human-in-the-loop.mdx) — another ready-made hook, intercepting tool calls for human review
|
||||
- [State](./state.mdx) — the live run state hooks and policies receive
|
||||
Reference in New Issue
Block a user