chore: import upstream snapshot with attribution
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,492 @@
|
||||
---
|
||||
title: "Agent"
|
||||
id: agent
|
||||
slug: "/agent"
|
||||
description: "The `Agent` component is a tool-using agent that interacts with chat-based LLMs and tools to solve complex queries iteratively. It can execute external tools, manage state across multiple LLM calls, and stop execution based on configurable `exit_conditions`."
|
||||
---
|
||||
|
||||
# Agent
|
||||
|
||||
The `Agent` component is a tool-using agent that interacts with chat-based LLMs and tools to solve complex queries iteratively. It can execute external tools, manage state across multiple LLM calls, and stop execution based on configurable `exit_conditions`.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`ChatPromptBuilder`](../builders/chatpromptbuilder.mdx) or user input |
|
||||
| **Mandatory init variables** | `chat_generator`: An instance of a Chat Generator that supports tools |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx)s |
|
||||
| **Output variables** | `messages`: Chat history with tool and model responses |
|
||||
| **API reference** | [Agents](/reference/agents-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/agents/agent.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `Agent` component is a loop-based system that uses a chat-based large language model (LLM) and external tools to solve complex user queries.
|
||||
It works iteratively—calling tools, updating state, and generating prompts—until one of the configurable `exit_conditions` is met.
|
||||
|
||||
It can:
|
||||
|
||||
- Dynamically select tools based on user input,
|
||||
- Maintain and validate runtime state using a schema,
|
||||
- Stream token-level outputs from the LLM.
|
||||
|
||||
The `Agent` returns a dictionary containing:
|
||||
|
||||
- `messages`: the full conversation history,
|
||||
- `last_message`: the final `ChatMessage` from the agent,
|
||||
- `step_count`: the number of steps the agent ran,
|
||||
- `token_usage`: aggregated token usage summed across every LLM call in the run,
|
||||
- `tool_call_counts`: how many times each tool was invoked, keyed by tool name,
|
||||
- Additional dynamic keys based on `state_schema`.
|
||||
|
||||
### Run Metadata
|
||||
|
||||
The `step_count`, `token_usage`, and `tool_call_counts` outputs are populated automatically during a run. They are added to the agent's `state_schema` behind the scenes, so tools registered with `inputs_from_state` can read them mid-run. They are outputs only — they cannot be passed as inputs to `run()` or `run_async()`, and using them as keys in your own `state_schema` raises a `ValueError`. See [State](./state.mdx#schema-definition) for details.
|
||||
|
||||
```python
|
||||
response = agent.run(messages=[ChatMessage.from_user("What is 7 * (4 + 2)?")])
|
||||
|
||||
print(response["step_count"]) # 2
|
||||
print(
|
||||
response["token_usage"],
|
||||
) # {"prompt_tokens": 512, "completion_tokens": 86, ...}
|
||||
print(response["tool_call_counts"]) # {"calculator": 1}
|
||||
```
|
||||
|
||||
## Parameters
|
||||
|
||||
`chat_generator` is the only mandatory parameter — an instance of a Chat Generator that supports tools. All other parameters are optional.
|
||||
|
||||
- `tools`: A list of tool or toolset instances the agent can call. Supported types: [`Tool`](../../tools/tool.mdx), [`ComponentTool`](../../tools/componenttool.mdx), [`PipelineTool`](../../tools/pipelinetool.mdx), [`MCPTool`](../../tools/mcptool.mdx), [`Toolset`](../../tools/toolset.mdx), [`MCPToolset`](../../tools/mcptoolset.mdx), [`SearchableToolset`](../../tools/searchabletoolset.mdx). Tool names must be unique; duplicate names are detected at the start of each agent step, before the chat generator is called.
|
||||
- `system_prompt`: A plain string or Jinja2 template used as the system message for every run. If the template contains Jinja2 variables, those variables become additional inputs to `run()`.
|
||||
- `user_prompt`: A Jinja2 template appended to the user-provided messages on each run. Template variables become additional inputs to `run()`. Use `required_variables` to enforce which variables must be provided.
|
||||
- `exit_conditions`: List of conditions that cause the agent to stop. Use `”text”` to stop when the LLM replies without a tool call, or a tool name to stop once that tool has been executed. Defaults to `[“text”]`. Exit conditions are evaluated at runtime rather than validated at initialization, so a condition can name a tool that is only loaded later — for example, a tool passed at runtime via `run(tools=...)` or one discovered by a [`SearchableToolset`](../../tools/searchabletoolset.mdx).
|
||||
- `state_schema`: Defines the agent's runtime state — a dict mapping key names to type configs (e.g. `{“docs”: {“type”: list[Document]}}`). Tools can read from and write to state keys via `inputs_from_state` and `outputs_to_state`. See [State](./state.mdx) for full details.
|
||||
- `streaming_callback`: A callback invoked for each streamed token. Use the built-in `print_streaming_chunk` for console output.
|
||||
- `max_agent_steps`: Maximum number of LLM + tool call iterations before the agent stops. Defaults to `100`.
|
||||
- `raise_on_tool_invocation_failure`: If `True`, raises an exception when a tool call fails. If `False` (default), the error is passed back to the LLM as a message so it can recover.
|
||||
- `hooks`: A dict mapping a hook point (`"before_llm"`, `"before_tool"`, `"after_tool"`, `"on_exit"`) to a list of hooks the agent runs at that point. Hooks receive the live `State` and influence the run by mutating it — for example, to build run-time context or require human confirmation of tool calls. See [Hooks](./hooks.mdx) and [Human in the Loop](./human-in-the-loop.mdx).
|
||||
- `tool_concurrency_limit`: Maximum number of tool calls to execute at the same time. Defaults to `4`; set to `1` to disable parallel tool execution.
|
||||
- `tool_streaming_callback_passthrough`: If `True`, passes the streaming callback to tools that accept it.
|
||||
|
||||
### Runtime overrides
|
||||
|
||||
`run()` also accepts parameters that override the init-time configuration for a single call:
|
||||
|
||||
- `tools`: Pass a list of `Tool`/`Toolset` objects, or a list of tool name strings to select a subset of the agent's configured tools for this run.
|
||||
- `generation_kwargs`: Additional keyword arguments forwarded to the LLM, overriding any set at init time (e.g. `{“temperature”: 0.2}`).
|
||||
- `hook_context`: A dict of request-scoped resources made available to [hooks](./hooks.mdx) via `state.data["hook_context"]` — for example, a user ID or a WebSocket connection.
|
||||
|
||||
:::info
|
||||
For the full parameter reference, see the [Agents API Documentation](/reference/agents-api).
|
||||
:::
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.tools import tool
|
||||
from haystack.components.agents import Agent
|
||||
from typing import Annotated
|
||||
|
||||
|
||||
@tool(outputs_to_state={"calc_result": {"source": "result"}})
|
||||
def calculator(
|
||||
expression: Annotated[str, "Math expression to evaluate, e.g. '7 * (4 + 2)'"],
|
||||
) -> dict:
|
||||
"""Evaluate basic math expressions."""
|
||||
try:
|
||||
result = eval(expression, {"__builtins__": {}})
|
||||
return {"result": result}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
tools=[calculator],
|
||||
system_prompt="You are a helpful assistant. Always use the calculator tool to evaluate math expressions.",
|
||||
state_schema={"calc_result": {"type": int}},
|
||||
)
|
||||
|
||||
response = agent.run(messages=[ChatMessage.from_user("What is 7 * (4 + 2)?")])
|
||||
|
||||
print(response["last_message"].text)
|
||||
print("Calc Result:", response.get("calc_result"))
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
The example pipeline below creates a database assistant using `OpenAIChatGenerator`, `LinkContentFetcher`, and custom database tool.
|
||||
It reads the given URL and processes the page content, then builds a prompt for the AI.
|
||||
The assistant uses this information to write people's names and titles from the given page to the database.
|
||||
|
||||
```python
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
|
||||
from haystack.components.converters.html import HTMLToDocument
|
||||
from haystack.components.fetchers.link_content import LinkContentFetcher
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.tools import tool
|
||||
from typing import Annotated, Optional
|
||||
|
||||
document_store = InMemoryDocumentStore() # create a document store or an SQL database
|
||||
|
||||
|
||||
@tool
|
||||
def add_database_tool(
|
||||
name: Annotated[str, "First name of the person"],
|
||||
surname: Annotated[str, "Last name of the person"],
|
||||
job_title: Annotated[Optional[str], "Job title or role of the person"] = None,
|
||||
other: Annotated[Optional[str], "Any other relevant information"] = None,
|
||||
) -> str:
|
||||
"""Add a person to the database with information about them."""
|
||||
document_store.write_documents(
|
||||
[
|
||||
Document(
|
||||
content=name + " " + surname + " " + (job_title or ""),
|
||||
meta={"other": other},
|
||||
),
|
||||
],
|
||||
)
|
||||
# Returning a confirmation lets the agent know the tool call succeeded
|
||||
return f"Successfully added {name} {surname} to the database."
|
||||
|
||||
|
||||
database_assistant = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
tools=[add_database_tool],
|
||||
system_prompt="""
|
||||
You are a database assistant.
|
||||
Your task is to extract the names of people mentioned in the given context and add them to a knowledge base,
|
||||
along with additional relevant information about them that can be extracted from the context.
|
||||
Do not use your own knowledge, stay grounded to the given context.
|
||||
Do not ask the user for confirmation.
|
||||
Instead, automatically update the knowledge base and return a brief summary of the people added,
|
||||
including the information stored for each.
|
||||
""",
|
||||
)
|
||||
|
||||
extraction_agent = Pipeline()
|
||||
extraction_agent.add_component("fetcher", LinkContentFetcher())
|
||||
extraction_agent.add_component("converter", HTMLToDocument())
|
||||
extraction_agent.add_component(
|
||||
"builder",
|
||||
ChatPromptBuilder(
|
||||
template=[
|
||||
ChatMessage.from_user("""
|
||||
{% for doc in docs %}
|
||||
{{ doc.content|default|truncate(25000) }}
|
||||
{% endfor %}
|
||||
"""),
|
||||
],
|
||||
required_variables=["docs"],
|
||||
),
|
||||
)
|
||||
|
||||
extraction_agent.add_component("database_agent", database_assistant)
|
||||
extraction_agent.connect("fetcher.streams", "converter.sources")
|
||||
extraction_agent.connect("converter.documents", "builder.docs")
|
||||
extraction_agent.connect("builder", "database_agent")
|
||||
|
||||
agent_output = extraction_agent.run(
|
||||
{
|
||||
"fetcher": {
|
||||
"urls": ["https://github.com/deepset-ai/haystack/releases/tag/v2.27.0"],
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
print(agent_output["database_agent"]["last_message"].text)
|
||||
|
||||
# Inspect what was written to the document store
|
||||
written_docs = document_store.filter_documents()
|
||||
print(f"\n{len(written_docs)} people added to the database:")
|
||||
for doc in written_docs:
|
||||
print(f" - {doc.content}")
|
||||
```
|
||||
|
||||
### In YAML
|
||||
The example pipeline below fetches a webpage, converts its HTML to text, and builds a chat prompt combining the page content with a user query.
|
||||
The `Agent` then answers the question based on the provided content and can use its web search tool to find additional information if needed.
|
||||
|
||||
<details>
|
||||
<summary>View YAML</summary>
|
||||
|
||||
```yaml
|
||||
components:
|
||||
agent:
|
||||
init_parameters:
|
||||
chat_generator:
|
||||
init_parameters:
|
||||
api_base_url: null
|
||||
api_key:
|
||||
env_vars:
|
||||
- OPENAI_API_KEY
|
||||
strict: true
|
||||
type: env_var
|
||||
generation_kwargs: {}
|
||||
http_client_kwargs: null
|
||||
max_retries: null
|
||||
model: gpt-5.4-nano
|
||||
organization: null
|
||||
streaming_callback: null
|
||||
timeout: null
|
||||
tools: null
|
||||
tools_strict: false
|
||||
type: haystack.components.generators.chat.openai.OpenAIChatGenerator
|
||||
exit_conditions:
|
||||
- text
|
||||
hooks: null
|
||||
max_agent_steps: 5
|
||||
raise_on_tool_invocation_failure: false
|
||||
required_variables: null
|
||||
state_schema: {}
|
||||
streaming_callback: null
|
||||
system_prompt: You are a helpful assistant. Use the web search tool to find
|
||||
information when needed.
|
||||
tool_concurrency_limit: 4
|
||||
tool_streaming_callback_passthrough: false
|
||||
tools:
|
||||
- data:
|
||||
component:
|
||||
init_parameters:
|
||||
allowed_domains: null
|
||||
api_key:
|
||||
env_vars:
|
||||
- SERPERDEV_API_KEY
|
||||
strict: true
|
||||
type: env_var
|
||||
exclude_subdomains: false
|
||||
search_params: {}
|
||||
top_k: 3
|
||||
type: haystack_integrations.components.websearch.serperdev.websearch.SerperDevWebSearch
|
||||
description: Search the web for current information on any topic
|
||||
inputs_from_state: null
|
||||
name: web_search
|
||||
outputs_to_state: null
|
||||
outputs_to_string: null
|
||||
parameters: null
|
||||
type: haystack.tools.component_tool.ComponentTool
|
||||
user_prompt: null
|
||||
type: haystack.components.agents.agent.Agent
|
||||
converter:
|
||||
init_parameters:
|
||||
extraction_kwargs: {}
|
||||
store_full_path: false
|
||||
type: haystack.components.converters.html.HTMLToDocument
|
||||
fetcher:
|
||||
init_parameters:
|
||||
client_kwargs:
|
||||
follow_redirects: true
|
||||
timeout: 3
|
||||
http2: false
|
||||
raise_on_failure: true
|
||||
request_headers: {}
|
||||
retry_attempts: 2
|
||||
timeout: 3
|
||||
user_agents:
|
||||
- haystack/LinkContentFetcher/2.27.0rc0
|
||||
type: haystack.components.fetchers.link_content.LinkContentFetcher
|
||||
prompt_builder:
|
||||
init_parameters:
|
||||
required_variables:
|
||||
- docs
|
||||
- query
|
||||
template:
|
||||
- content:
|
||||
- text: 'Based on the following content:
|
||||
|
||||
{% for doc in docs %}
|
||||
|
||||
{{ doc.content }}
|
||||
|
||||
{% endfor %}
|
||||
|
||||
Answer this question: {{ query }}'
|
||||
meta: {}
|
||||
name: null
|
||||
role: user
|
||||
variables: null
|
||||
type: haystack.components.builders.chat_prompt_builder.ChatPromptBuilder
|
||||
connection_type_validation: true
|
||||
connections:
|
||||
- receiver: converter.sources
|
||||
sender: fetcher.streams
|
||||
- receiver: prompt_builder.docs
|
||||
sender: converter.documents
|
||||
- receiver: agent.messages
|
||||
sender: prompt_builder.prompt
|
||||
max_runs_per_component: 100
|
||||
metadata: {}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
## Streaming
|
||||
|
||||
You can stream output as it's generated. Pass a callback to `streaming_callback`.
|
||||
Use the built-in `print_streaming_chunk` to print text tokens and tool events (tool calls and tool results).
|
||||
|
||||
```python
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
tools=[...],
|
||||
system_prompt="...",
|
||||
streaming_callback=print_streaming_chunk,
|
||||
)
|
||||
```
|
||||
|
||||
See our [Streaming Support](../generators/guides-to-generators/choosing-the-right-generator.mdx#streaming-support) docs to learn more how `StreamingChunk` works and how to write a custom callback.
|
||||
|
||||
Give preference to `print_streaming_chunk` by default.
|
||||
Write a custom callback only if you need a specific transport (for example, SSE/WebSocket) or custom UI formatting.
|
||||
|
||||
## Multimodal Inputs
|
||||
|
||||
Agents support multimodal inputs when paired with a vision-capable model such as `gpt-5` (OpenAI) or `gemini-2.5-flash` (Google).
|
||||
Pass images alongside text by including `ImageContent` objects in the `content_parts` of a `ChatMessage`:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage, ImageContent
|
||||
|
||||
image = ImageContent.from_url("https://example.com/chart.png")
|
||||
result = agent.run(
|
||||
messages=[
|
||||
ChatMessage.from_user(content_parts=["What does this chart show?", image]),
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
Tools can also return `ImageContent` directly, letting the agent fetch and reason about images dynamically during its loop.
|
||||
Two things are required: set `outputs_to_string={"raw_result": True}` so the `ToolInvoker` skips string conversion, and return a `list[ImageContent]` (the tool result type is `str | Sequence[TextContent | ImageContent]`).
|
||||
|
||||
The standard Chat Completions API doesn't support images in tool results — use `OpenAIResponsesChatGenerator` (OpenAI's Responses API) instead:
|
||||
|
||||
```python
|
||||
from typing import Annotated
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.generators.chat import OpenAIResponsesChatGenerator
|
||||
from haystack.dataclasses import ChatMessage, ImageContent
|
||||
from haystack.tools import tool
|
||||
|
||||
|
||||
@tool(outputs_to_string={"raw_result": True})
|
||||
def fetch_image(
|
||||
url: Annotated[str, "URL of the image to fetch and analyze"],
|
||||
) -> list[ImageContent]:
|
||||
"""Fetch an image from a URL so the agent can analyze its contents."""
|
||||
return [ImageContent.from_url(url)]
|
||||
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIResponsesChatGenerator(model="gpt-5"),
|
||||
tools=[fetch_image],
|
||||
system_prompt="You are a helpful assistant that can fetch and analyze images from URLs.",
|
||||
)
|
||||
|
||||
result = agent.run(
|
||||
messages=[
|
||||
ChatMessage.from_user(
|
||||
"Fetch the image at https://picsum.photos/seed/haystack/640/480 and describe what you see.",
|
||||
),
|
||||
],
|
||||
)
|
||||
print(result["last_message"].text)
|
||||
```
|
||||
|
||||
`ImageContent` can be created from a URL, a local file path, or a PDF page using the `PDFToImageContent` converter.
|
||||
|
||||
### In a pipeline
|
||||
|
||||
When an `Agent` sits inside a pipeline, use `ChatPromptBuilder` with its string template format and the `| templatize_part` filter to pass images as structured content parts:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ImageContent
|
||||
|
||||
template = """
|
||||
{% message role="user" %}
|
||||
{{ question }}
|
||||
{{ image | templatize_part }}
|
||||
{% endmessage %}
|
||||
"""
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5"),
|
||||
system_prompt="You are a helpful assistant that can analyze images.",
|
||||
)
|
||||
prompt_builder = ChatPromptBuilder(
|
||||
template=template,
|
||||
required_variables=["question", "image"],
|
||||
)
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("prompt_builder", prompt_builder)
|
||||
pipeline.add_component("agent", agent)
|
||||
pipeline.connect("prompt_builder.prompt", "agent.messages")
|
||||
|
||||
# Download or provide your own chart image as "chart.png"
|
||||
image = ImageContent.from_file_path("chart.png")
|
||||
result = pipeline.run(
|
||||
{
|
||||
"prompt_builder": {"question": "What does this chart show?", "image": image},
|
||||
},
|
||||
)
|
||||
print(result["agent"]["last_message"].text)
|
||||
```
|
||||
|
||||
:::tip
|
||||
See these cookbooks for complete multimodal agent examples:
|
||||
- [Multimodal Agents](https://haystack.deepset.ai/cookbook/multimodal_intro#multimodal-agent) — image inputs and tool use with agents
|
||||
- [Gemma Chat RAG](https://haystack.deepset.ai/cookbook/gemma_chat_rag) — vision model in a RAG pipeline
|
||||
:::
|
||||
|
||||
## Multi-Agent Systems
|
||||
|
||||
You can wrap an `Agent` as a tool to build multi-agent systems where specialist agents handle focused subtasks and a coordinator agent plans and delegates.
|
||||
|
||||
See [Multi-Agent Systems](../../concepts/agents/multi-agent-systems.mdx) for a full guide, including the recommended `@tool` decorator approach for full interface control and `ComponentTool` for declarative configuration.
|
||||
|
||||
## MCP Integration
|
||||
|
||||
Agents work with MCP in two directions:
|
||||
|
||||
- **Consuming MCP tools**: Pass `MCPTool` or `MCPToolset` instances in the `tools` list to call tools on any MCP-compatible server (filesystem, browser, databases, and more). See [MCPTool](../../tools/mcptool.mdx) and [MCPToolset](../../tools/mcptoolset.mdx).
|
||||
- **Exposing as an MCP server**: Use [Hayhooks](../../development/hayhooks.mdx) to deploy your agent and expose it as an MCP server, making it callable from any MCP-compatible client such as Claude Desktop or Cursor.
|
||||
|
||||
## Additional References
|
||||
|
||||
📖 Related docs:
|
||||
|
||||
- [State](./state.mdx) — managing shared data between tools
|
||||
- [Hooks](./hooks.mdx) — running custom logic at defined points of the run loop
|
||||
- [Human in the Loop](./human-in-the-loop.mdx) — intercepting tool calls for human review
|
||||
- [Tool Result Offloading](./tool-result-offloading.mdx) — keeping large tool results out of the context window
|
||||
|
||||
📚 Tutorials:
|
||||
|
||||
- [Build a Tool-Calling Agent](https://haystack.deepset.ai/tutorials/43_building_a_tool_calling_agent)
|
||||
- [Creating a Multi-Agent System](https://haystack.deepset.ai/tutorials/45_creating_a_multi_agent_system)
|
||||
- [Human-in-the-Loop with Haystack Agents](https://haystack.deepset.ai/tutorials/47_human_in_the_loop_agent/)
|
||||
|
||||
🧑🍳 Cookbook:
|
||||
|
||||
- [Build a GitHub Issue Resolver Agent](https://haystack.deepset.ai/cookbook/github_issue_resolver_agent)
|
||||
- [Multimodal Agents](https://haystack.deepset.ai/cookbook/multimodal_intro#multimodal-agent)
|
||||
- [Gemma Chat RAG](https://haystack.deepset.ai/cookbook/gemma_chat_rag)
|
||||
@@ -0,0 +1,202 @@
|
||||
---
|
||||
title: "Hooks"
|
||||
id: hooks
|
||||
slug: "/hooks"
|
||||
description: "Hooks let you run custom logic at defined points of an Agent's run loop — before each LLM call, before and after tool execution, and on exit."
|
||||
---
|
||||
|
||||
# Hooks
|
||||
|
||||
Hooks let you run custom logic at defined points of an [`Agent`](./agent.mdx)'s run loop — before each LLM call, before and after tool execution, and on exit.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Configured on** | The [`Agent`](./agent.mdx) component via the `hooks` parameter |
|
||||
| **Key classes** | `hook` (decorator), `FunctionHook`, `Hook` (protocol) |
|
||||
| **Import path** | `haystack.hooks` |
|
||||
| **API reference** | [Hooks](/reference/hooks-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/hooks/ |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
Pass `hooks` to the `Agent` as a dictionary mapping a *hook point* to a list of hooks the Agent runs at that point. Each hook receives the live [`State`](./state.mdx) and influences the run by mutating it in place. Hooks for a hook point run in list order, and the same hook can be registered under multiple hook points.
|
||||
|
||||
This enables patterns such as building run-time system context, retrieving memories before the first LLM call, auditing or intercepting tool calls, and requiring a condition to hold before the Agent is allowed to finish.
|
||||
|
||||
### Hook points
|
||||
|
||||
- `before_llm`: Runs before each chat-generator call.
|
||||
- `before_tool`: Runs after the model requests tool calls, before any tools run. After these hooks run, the Agent re-reads the current last message from `state.data["messages"]`. If that message contains tool calls, those calls are executed. If it does not, no tools run for that step, no tool-based exit condition is triggered, and the Agent loops back to the next LLM call unless `max_agent_steps` has been reached.
|
||||
- `after_tool`: Runs after tools execute, once their result messages are in `state.data["messages"]`, before the exit-condition check and the next LLM call. Use it to rewrite the freshly produced tool-result messages — for example, to offload, redact, truncate, or summarize results. It does not run on the plain-text exit step. It does still run when a `before_tool` hook removed the pending tool calls: no tools executed on that step, so don't assume the last message is a fresh tool result.
|
||||
- `on_exit`: Runs when the Agent is about to stop on an exit condition. An `on_exit` hook can keep the Agent running by setting the `continue_run` control flag (`state.set("continue_run", True)`), usually alongside a message telling the model what to do next. `on_exit` hooks run when the Agent stops on an exit condition, but not when it stops because `max_agent_steps` is reached.
|
||||
|
||||
Registering a hook under an unknown hook point raises a `ValueError` at construction. A hook class can declare an `allowed_hook_points` attribute listing the hook points it supports; the Agent validates it and fails fast if the hook is registered somewhere it doesn't belong.
|
||||
|
||||
### State keys for hooks
|
||||
|
||||
The Agent manages a few state keys that hooks interact with. Like the run-metadata keys (`step_count`, `token_usage`, `tool_call_counts`), they are reserved — using any of them in your own `state_schema` raises a `ValueError`. See [State](./state.mdx#schema-definition) for the full list:
|
||||
|
||||
- `continue_run`: Set by an `on_exit` hook to keep the Agent running.
|
||||
- `tools`: The tools available in the current step, for hooks to inspect.
|
||||
- `hook_context`: Request-scoped resources passed to `Agent.run(hook_context={...})` / `run_async(hook_context={...})`. Hooks read it with `state.data["hook_context"]` or `state.data.get("hook_context")` — use it for per-request resources such as a user ID, a WebSocket, or a database client. Avoid the plain `state.get("hook_context")` here: `State.get` returns a deep copy of the value, which often fails for the kinds of resources stored in this dict (such as a WebSocket or a database client).
|
||||
|
||||
Hooks can also read the automatically tracked run metadata: `step_count`, `token_usage`, and `tool_call_counts`.
|
||||
|
||||
## Creating hooks
|
||||
|
||||
### With the `@hook` decorator
|
||||
|
||||
The `@hook` decorator wraps a function taking a single `State` argument into a hook. A regular function becomes the hook's sync path, a coroutine function its async path. To give a single hook both paths, construct a `FunctionHook` directly with both `function` and `async_function`.
|
||||
|
||||
The example below registers a hook at each of `before_llm`, `before_tool`, and `on_exit` to show what hooks can do:
|
||||
|
||||
```python
|
||||
from datetime import datetime, timezone
|
||||
from typing import Annotated
|
||||
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.agents.state import State, replace_values
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.hooks import hook
|
||||
from haystack.tools import tool
|
||||
|
||||
|
||||
@tool
|
||||
def search(query: Annotated[str, "The search query"]) -> str:
|
||||
"""Search the web."""
|
||||
# Placeholder: would call a real search API
|
||||
return "Fusion startups reported net-energy-gain milestones this year."
|
||||
|
||||
|
||||
@hook
|
||||
def build_context(state: State) -> None:
|
||||
# before_llm: build run-time system context once, before the first model call.
|
||||
if state.get("step_count") == 0:
|
||||
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
|
||||
system = ChatMessage.from_system(
|
||||
f"You are a research assistant. The current time is {now}.",
|
||||
)
|
||||
state.set(
|
||||
"messages",
|
||||
[system, *state.data["messages"]],
|
||||
handler_override=replace_values,
|
||||
)
|
||||
|
||||
|
||||
@hook
|
||||
def audit_tool_calls(state: State) -> None:
|
||||
# before_tool: see which tools the model is about to run.
|
||||
pending = state.data["messages"][-1].tool_calls
|
||||
print(f"about to run: {[tc.tool_name for tc in pending]}")
|
||||
|
||||
|
||||
@hook
|
||||
def require_search(state: State) -> None:
|
||||
# on_exit: keep going until the agent has actually searched.
|
||||
if state.get("tool_call_counts", {}).get("search", 0) == 0:
|
||||
state.set("messages", [ChatMessage.from_system("Search before answering.")])
|
||||
state.set("continue_run", True)
|
||||
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
tools=[search],
|
||||
hooks={
|
||||
"before_llm": [build_context],
|
||||
"before_tool": [audit_tool_calls],
|
||||
"on_exit": [require_search],
|
||||
},
|
||||
)
|
||||
|
||||
result = agent.run(
|
||||
messages=[
|
||||
ChatMessage.from_user("What are the latest developments in fusion energy?"),
|
||||
],
|
||||
)
|
||||
print(result["last_message"].text)
|
||||
```
|
||||
|
||||
### Class-based hooks
|
||||
|
||||
A hook is any object with a `run(state)` method; it may additionally define `run_async(state)` for true async behavior. Class-based hooks may also implement the optional lifecycle methods `warm_up` / `warm_up_async` and `close` / `close_async`. The Agent calls them from its own `warm_up` / `close`, so a hook can defer opening clients or reading credentials until warm-up and release them on close.
|
||||
|
||||
When a class-based hook should be serializable (so an Agent using it can be serialized), implement `to_dict` / `from_dict`: store serializable constructor arguments on the hook and rebuild runtime clients from those values.
|
||||
|
||||
The example below is an `on_exit` hook that grades the Agent's answer with its own LLM and asks the Agent to improve a weak answer before finishing:
|
||||
|
||||
```python
|
||||
from typing import Any
|
||||
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.agents.state import State
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.core.serialization import default_from_dict, default_to_dict
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
|
||||
class GradeFinalAnswer:
|
||||
"""Grade the Agent's answer with an LLM and ask it to improve a weak answer before finishing."""
|
||||
|
||||
def __init__(self, model: str = "gpt-5.4-nano"):
|
||||
self.model = model
|
||||
self._judge = OpenAIChatGenerator(model=self.model)
|
||||
|
||||
def warm_up(self) -> None:
|
||||
# Warm up the judge's own client during the Agent's warm-up.
|
||||
self._judge.warm_up()
|
||||
|
||||
def close(self) -> None:
|
||||
# Release the judge's client during the Agent's close.
|
||||
self._judge.close()
|
||||
|
||||
def run(self, state: State) -> None:
|
||||
answer = state.data["messages"][-1].text or ""
|
||||
verdict = (
|
||||
self._judge.run(
|
||||
messages=[
|
||||
ChatMessage.from_user(
|
||||
f"Reply with only PASS or FAIL. Is this answer complete?\n\n{answer}",
|
||||
),
|
||||
],
|
||||
)["replies"][0].text
|
||||
or ""
|
||||
)
|
||||
if "FAIL" in verdict.upper():
|
||||
state.set(
|
||||
"messages",
|
||||
[
|
||||
ChatMessage.from_user(
|
||||
"Your answer was incomplete. Please improve it.",
|
||||
),
|
||||
],
|
||||
)
|
||||
state.set("continue_run", True)
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return default_to_dict(self, model=self.model)
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: dict[str, Any]) -> "GradeFinalAnswer":
|
||||
return default_from_dict(cls, data)
|
||||
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
hooks={"on_exit": [GradeFinalAnswer()]},
|
||||
)
|
||||
result = agent.run(messages=[ChatMessage.from_user("Explain how vaccines work.")])
|
||||
print(result["last_message"].text)
|
||||
```
|
||||
|
||||
## Ready-made hooks
|
||||
|
||||
Haystack ships two ready-made hooks:
|
||||
|
||||
- `ConfirmationHook`: A `before_tool` hook that applies Human-in-the-Loop confirmation strategies to pending tool calls — a human can confirm, modify, or reject the tool calls the model requested before they run. See [Human in the Loop](./human-in-the-loop.mdx).
|
||||
- `ToolResultOffloadHook`: An `after_tool` hook that offloads tool results to a `ToolResultStore` (such as `FileSystemToolResultStore`) and replaces them in the conversation with a compact pointer, so the next LLM call sees a reference instead of the full result. Per-tool policies (`AlwaysOffload`, `NeverOffload`, `OffloadOverChars`) control which results are offloaded. See [Tool Result Offloading](./tool-result-offloading.mdx).
|
||||
@@ -0,0 +1,321 @@
|
||||
---
|
||||
title: "Human in the Loop"
|
||||
id: human-in-the-loop
|
||||
slug: "/human-in-the-loop"
|
||||
description: "Human-in-the-loop allows you to intercept agent tool calls before execution, letting a human confirm, reject, or modify the tool parameters."
|
||||
---
|
||||
|
||||
# Human in the Loop
|
||||
|
||||
Human-in-the-loop (HITL) lets you intercept an agent's tool calls before they are executed.
|
||||
A human can **confirm**, **reject**, or **modify** the parameters of each tool call in real time.
|
||||
This is useful for high-stakes operations - such as sending emails, modifying databases, or making API calls - where you want a human to review the action first.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Configured on** | The [`Agent`](./agent.mdx) component, as a `ConfirmationHook` registered under the `before_tool` [hook point](./hooks.mdx) |
|
||||
| **Key classes** | `ConfirmationHook`, `BlockingConfirmationStrategy`, `AlwaysAskPolicy`, `AskOncePolicy`, `NeverAskPolicy`, `RichConsoleUI`, `SimpleConsoleUI` |
|
||||
| **Import path** | `haystack.human_in_the_loop` |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/human_in_the_loop/ |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
HITL is one application of the Agent's general [hooks](./hooks.mdx) mechanism: a `ConfirmationHook` registered under the `before_tool` hook point intercepts the tool calls the model requested before they run, and confirms, modifies, or rejects them by rewriting the conversation in the Agent's `State`.
|
||||
|
||||
The HITL system is composed of these layers:
|
||||
|
||||
- **`ConfirmationHook`** - the `before_tool` hook that applies your confirmation strategies to pending tool calls. Its `confirmation_strategies` mapping accepts a single tool name, a tuple of tool names, or the wildcard `"*"` that applies to any tool without a more specific entry.
|
||||
- **Strategy** - decides what to do when a tool is about to be called. The built-in `BlockingConfirmationStrategy` pauses execution and asks a human.
|
||||
- **Policy** - decides *when* to ask. Built-in policies: `AlwaysAskPolicy`, `NeverAskPolicy`, `AskOncePolicy`.
|
||||
- **UI** - the interface used to ask the human. Built-in UIs: `RichConsoleUI` (requires `rich`) and `SimpleConsoleUI` (stdlib only).
|
||||
|
||||
When the agent is about to invoke a tool, the strategy checks the policy.
|
||||
If the policy says to ask, the UI prompts the human with the tool name, description, and parameters. The human can:
|
||||
|
||||
- **Confirm** (`y`) - execute as-is
|
||||
- **Reject** (`n`) - skip execution and feed rejection feedback back to the LLM
|
||||
- **Modify** (`m`) - edit the parameters before execution
|
||||
|
||||
The agent then continues with the human's decision.
|
||||
|
||||
:::info
|
||||
Strategies see only the arguments the model produced for a tool call. Values injected from [`State`](./state.mdx) via a tool's `inputs_from_state` mapping are not included in what is presented for confirmation — that injection happens at tool execution time.
|
||||
:::
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic setup
|
||||
|
||||
```python
|
||||
from typing import Annotated
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.human_in_the_loop import (
|
||||
AlwaysAskPolicy,
|
||||
BlockingConfirmationStrategy,
|
||||
ConfirmationHook,
|
||||
SimpleConsoleUI,
|
||||
)
|
||||
from haystack.tools import tool
|
||||
|
||||
|
||||
@tool
|
||||
def send_email(
|
||||
to: Annotated[str, "The recipient email address"],
|
||||
subject: Annotated[str, "The email subject line"],
|
||||
body: Annotated[str, "The email body"],
|
||||
) -> str:
|
||||
"""Send an email to a recipient."""
|
||||
return f"Email sent to {to}."
|
||||
|
||||
|
||||
strategy = BlockingConfirmationStrategy(
|
||||
confirmation_policy=AlwaysAskPolicy(),
|
||||
confirmation_ui=SimpleConsoleUI(),
|
||||
)
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-mini"),
|
||||
tools=[send_email],
|
||||
hooks={
|
||||
"before_tool": [
|
||||
ConfirmationHook(confirmation_strategies={"send_email": strategy}),
|
||||
],
|
||||
},
|
||||
)
|
||||
|
||||
result = agent.run(
|
||||
messages=[ChatMessage.from_user("Send a welcome email to alice@example.com")],
|
||||
)
|
||||
```
|
||||
|
||||
When the agent calls `send_email`, the terminal will pause and show:
|
||||
|
||||
```
|
||||
--- Tool Execution Request ---
|
||||
Tool: send_email
|
||||
Description: Send an email to a recipient.
|
||||
Arguments:
|
||||
to: alice@example.com
|
||||
subject: Welcome!
|
||||
body: Hi Alice, welcome aboard!
|
||||
------------------------------
|
||||
Confirm execution? (y=confirm / n=reject / m=modify):
|
||||
```
|
||||
|
||||
### Using RichConsoleUI
|
||||
|
||||
`RichConsoleUI` provides a styled terminal prompt using the [`rich`](https://github.com/Textualize/rich) library:
|
||||
|
||||
```shell
|
||||
pip install rich
|
||||
```
|
||||
|
||||
```python
|
||||
from haystack.human_in_the_loop import RichConsoleUI
|
||||
|
||||
strategy = BlockingConfirmationStrategy(
|
||||
confirmation_policy=AlwaysAskPolicy(),
|
||||
confirmation_ui=RichConsoleUI(),
|
||||
)
|
||||
```
|
||||
|
||||
### Applying strategies to multiple tools
|
||||
|
||||
You can configure different strategies per tool, share one strategy across a group of tools using a tuple key, or set a default for all tools with the wildcard `"*"` (applied to any tool without a more specific entry):
|
||||
|
||||
```python
|
||||
@tool
|
||||
def delete_record(record_id: Annotated[str, "The ID of the record to delete"]) -> str:
|
||||
"""Delete a record from the database."""
|
||||
return f"Record {record_id} deleted."
|
||||
|
||||
|
||||
@tool
|
||||
def update_record(
|
||||
record_id: Annotated[str, "The ID of the record to update"],
|
||||
data: Annotated[str, "The new data as a JSON string"],
|
||||
) -> str:
|
||||
"""Update a record in the database."""
|
||||
return f"Record {record_id} updated."
|
||||
|
||||
|
||||
@tool
|
||||
def search(query: Annotated[str, "The search query"]) -> str:
|
||||
"""Search the knowledge base."""
|
||||
return f"Results for: {query}"
|
||||
|
||||
|
||||
ask_strategy = BlockingConfirmationStrategy(
|
||||
confirmation_policy=AlwaysAskPolicy(),
|
||||
confirmation_ui=SimpleConsoleUI(),
|
||||
)
|
||||
|
||||
confirmation_hook = ConfirmationHook(
|
||||
confirmation_strategies={
|
||||
# Share one strategy across multiple sensitive tools using a tuple key
|
||||
("send_email", "delete_record", "update_record"): ask_strategy,
|
||||
# search has no strategy - always executes without asking
|
||||
},
|
||||
)
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-mini"),
|
||||
tools=[send_email, delete_record, update_record, search],
|
||||
hooks={"before_tool": [confirmation_hook]},
|
||||
)
|
||||
```
|
||||
|
||||
### Customizing feedback messages
|
||||
|
||||
When a tool call is rejected or modified, `BlockingConfirmationStrategy` sends a message back to the LLM explaining what happened. Three optional template parameters control these messages — each has a sensible default, so you only need to set them if you want different wording:
|
||||
|
||||
- `reject_template`: Sent to the LLM when the user rejects a tool call. Must include a `{tool_name}` placeholder. Default: `"Tool execution for '{tool_name}' was rejected by the user."`
|
||||
- `modify_template`: Sent when the user modifies the parameters. Must include `{tool_name}` and `{final_tool_params}` placeholders. Default: `"The parameters for tool '{tool_name}' were updated by the user to:\n{final_tool_params}"`
|
||||
- `user_feedback_template`: Appends the user's optional free-text feedback to either message. Must include a `{feedback}` placeholder. Default: `"With user feedback: {feedback}"`
|
||||
|
||||
```python
|
||||
strategy = BlockingConfirmationStrategy(
|
||||
confirmation_policy=AlwaysAskPolicy(),
|
||||
confirmation_ui=SimpleConsoleUI(),
|
||||
reject_template="Skipping '{tool_name}' — rejected by operator.",
|
||||
modify_template="Updated parameters for '{tool_name}': {final_tool_params}",
|
||||
user_feedback_template="Reason: {feedback}",
|
||||
)
|
||||
```
|
||||
|
||||
## Policies
|
||||
|
||||
Policies control *when* the human is asked.
|
||||
|
||||
| Policy | Behavior |
|
||||
| --- | --- |
|
||||
| `AlwaysAskPolicy` | Ask every time the tool is called |
|
||||
| `NeverAskPolicy` | Never ask - always proceed (useful for toggling HITL off without removing the strategy) |
|
||||
| `AskOncePolicy` | Ask once per unique `(tool_name, parameters)` combination. Remembers confirmed calls and skips asking on repeats. |
|
||||
|
||||
### Custom policy
|
||||
|
||||
You can implement your own policy by subclassing `ConfirmationPolicy` from `haystack.human_in_the_loop.types`:
|
||||
|
||||
```python
|
||||
from haystack.human_in_the_loop.types import ConfirmationPolicy, ConfirmationUIResult
|
||||
from typing import Any
|
||||
|
||||
|
||||
class AskForSensitiveParamsPolicy(ConfirmationPolicy):
|
||||
"""Only ask when the 'to' parameter looks like an external email domain."""
|
||||
|
||||
def should_ask(
|
||||
self,
|
||||
tool_name: str,
|
||||
tool_description: str,
|
||||
tool_params: dict[str, Any],
|
||||
) -> bool:
|
||||
to = tool_params.get("to", "")
|
||||
return not to.endswith("@mycompany.com")
|
||||
```
|
||||
|
||||
For stateful policies, also implement `update_after_confirmation`.
|
||||
It is called after the user responds and receives the full `ConfirmationUIResult`, letting you update internal state based on the outcome.
|
||||
The following policy asks once per tool name and skips re-asking for any tool the user has already confirmed:
|
||||
|
||||
```python
|
||||
from haystack.human_in_the_loop.types import ConfirmationPolicy
|
||||
from haystack.human_in_the_loop import ConfirmationUIResult
|
||||
from typing import Any
|
||||
|
||||
|
||||
class AskOncePerToolPolicy(ConfirmationPolicy):
|
||||
"""Ask once per tool name, regardless of parameters. Skip on repeat confirmed calls."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._confirmed_tools: set[str] = set()
|
||||
|
||||
def should_ask(
|
||||
self,
|
||||
tool_name: str,
|
||||
tool_description: str,
|
||||
tool_params: dict[str, Any],
|
||||
) -> bool:
|
||||
return tool_name not in self._confirmed_tools
|
||||
|
||||
def update_after_confirmation(
|
||||
self,
|
||||
tool_name: str,
|
||||
tool_description: str,
|
||||
tool_params: dict[str, Any],
|
||||
confirmation_result: ConfirmationUIResult,
|
||||
) -> None:
|
||||
if confirmation_result.action == "confirm":
|
||||
self._confirmed_tools.add(tool_name)
|
||||
```
|
||||
|
||||
## Dataclasses
|
||||
|
||||
### `ConfirmationUIResult`
|
||||
|
||||
Returned by the UI after the human responds.
|
||||
|
||||
| Field | Type | Description |
|
||||
| --- | --- | --- |
|
||||
| `action` | `str` | `"confirm"`, `"reject"`, or `"modify"` |
|
||||
| `feedback` | `str \| None` | Optional free-text feedback from the human |
|
||||
| `new_tool_params` | `dict \| None` | Replacement parameters when action is `"modify"` |
|
||||
|
||||
### `ToolExecutionDecision`
|
||||
|
||||
Returned by the strategy to the agent.
|
||||
|
||||
| Field | Type | Description |
|
||||
| --- | --- | --- |
|
||||
| `tool_name` | `str` | Name of the tool |
|
||||
| `execute` | `bool` | Whether to execute the tool |
|
||||
| `tool_call_id` | `str \| None` | ID of the tool call |
|
||||
| `feedback` | `str \| None` | Feedback message passed back to the LLM on rejection or modification |
|
||||
| `final_tool_params` | `dict \| None` | Final parameters to use for execution |
|
||||
|
||||
## Example: HITL with Hayhooks and Open WebUI
|
||||
|
||||
The [hitl-hayhooks-redis-openwebui](https://github.com/deepset-ai/hitl-hayhooks-redis-openwebui) repository shows a full production-style HITL setup using a Haystack Agent served via [Hayhooks](https://github.com/deepset-ai/hayhooks) with approval dialogs rendered in [Open WebUI](https://github.com/open-webui/open-webui).
|
||||
|
||||
The key pattern it demonstrates is a custom `RedisConfirmationStrategy` that receives per-request resources - a Redis client and an async event queue - at runtime. Pass such resources via the generic `hook_context` run argument (`agent.run(messages=[...], hook_context={"redis": client})`). `ConfirmationHook` reads this dict from state with `state.data["hook_context"]` (not `state.get`, which returns a deep copy that fails for live resources like clients and queues - see [Hooks](./hooks.mdx)) and passes it to each strategy's `run()` as the `confirmation_strategy_context` keyword argument, which is how a custom strategy receives the Redis client and event queue:
|
||||
|
||||
- When a tool call is about to execute, the strategy emits a `tool_call_start` SSE event and blocks on `Redis BLPOP` waiting for an approval decision.
|
||||
- The Open WebUI Pipe function receives the SSE event, shows the user a confirmation dialog, then writes `approved` or `rejected` to Redis via `LPUSH`.
|
||||
- Once Redis unblocks, the strategy returns a `ToolExecutionDecision` and the agent continues.
|
||||
|
||||
This is a good reference if you need non-blocking HITL in a web or server environment where `SimpleConsoleUI` and `RichConsoleUI` are not suitable.
|
||||
|
||||
## Custom UI
|
||||
|
||||
Implement `ConfirmationUI` from `haystack.human_in_the_loop.types` to build your own interface - for example, a web-based approval queue:
|
||||
|
||||
```python
|
||||
from haystack.human_in_the_loop.types import ConfirmationUI
|
||||
from haystack.human_in_the_loop import ConfirmationUIResult
|
||||
from typing import Any
|
||||
|
||||
|
||||
class WebhookApprovalUI(ConfirmationUI):
|
||||
"""Sends a webhook and waits for an async approval response."""
|
||||
|
||||
def get_user_confirmation(
|
||||
self,
|
||||
tool_name: str,
|
||||
tool_description: str,
|
||||
tool_params: dict[str, Any],
|
||||
) -> ConfirmationUIResult:
|
||||
# Send approval request to your system and wait for response
|
||||
response = send_approval_request_and_wait(tool_name, tool_params)
|
||||
return ConfirmationUIResult(
|
||||
action=response["action"],
|
||||
feedback=response.get("feedback"),
|
||||
)
|
||||
```
|
||||
@@ -0,0 +1,424 @@
|
||||
---
|
||||
title: "State"
|
||||
id: state
|
||||
slug: "/state"
|
||||
description: "`State` is a container for storing shared information during Agent and Tool execution. It provides a structured way to share data between tools, accumulate results across multiple tool calls, and surface them alongside the agent's final answer."
|
||||
---
|
||||
|
||||
# State
|
||||
|
||||
`State` is a container for storing shared information during Agent and Tool execution.
|
||||
It provides a structured way to share data between tools, accumulate results across multiple tool calls, and surface them alongside the agent's final answer.
|
||||
|
||||
## Overview
|
||||
|
||||
When building agents that use multiple tools, you often need tools to share information or accumulate results across iterations.
|
||||
State provides centralized storage that all tools can read from and write to.
|
||||
For example, a search tool called multiple times can append its results to a shared `documents` list, which is then returned alongside the agent's final answer for source inspection.
|
||||
|
||||
State uses a schema-based approach where you define:
|
||||
|
||||
- What data can be stored,
|
||||
- The type of each piece of data,
|
||||
- How values are merged when updated.
|
||||
|
||||
The Agent creates and manages the `State` object internally. You shouldn't need to instantiate it directly.
|
||||
You interact with it through tool definitions (`inputs_from_state`, `outputs_to_state`, or a `state: State` parameter) and read results from the agent's output dict.
|
||||
|
||||
### Supported Types
|
||||
|
||||
State supports standard Python types:
|
||||
|
||||
- Basic types: `str`, `int`, `float`, `bool`, `dict`
|
||||
- List types: `list`, `list[str]`, `list[int]`, `list[Document]`
|
||||
- Union types: `str | int`, `str | None`
|
||||
- Custom classes and data classes.
|
||||
|
||||
### Automatic Message Handling
|
||||
|
||||
State automatically includes a `messages` field that stores the full conversation history during execution.
|
||||
You don't need to define this in your schema.
|
||||
It uses `list[ChatMessage]` type with the `merge_lists` handler, so new messages are appended on each iteration.
|
||||
|
||||
### State API
|
||||
|
||||
| Method | Description |
|
||||
| --- | --- |
|
||||
| `state.get(key, default=None)` | Read a value; returns `default` if the key doesn't exist |
|
||||
| `state.set(key, value)` | Write a value, merged using the schema's handler |
|
||||
| `state.has(key)` | Returns `True` if the key exists in state |
|
||||
| `state.data` | Returns a snapshot of all current state as a `dict` |
|
||||
|
||||
## Schema Definition
|
||||
|
||||
The schema defines what data can be stored and how values are updated. Each schema entry consists of:
|
||||
|
||||
- `type` (required): The Python type for this field (for example, `str`, `int`, `list`)
|
||||
- `handler` (optional): A callable that determines how new values are merged when `set()` is called
|
||||
|
||||
```python
|
||||
{
|
||||
"parameter_name": {
|
||||
"type": SomeType, # Required: expected Python type
|
||||
"handler": some_func, # Optional: merge function
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
If you don't specify a handler, State automatically assigns a default based on the type.
|
||||
|
||||
:::info Reserved keys
|
||||
The `Agent` manages some state keys itself and rejects them in a user-provided `state_schema` with a `ValueError`:
|
||||
|
||||
- The run-metadata keys `step_count`, `token_usage`, and `tool_call_counts`, which the Agent populates automatically during a run: tools can read them mid-run via `inputs_from_state`, and they are returned in the result dictionary.
|
||||
- The hook-facing keys `continue_run` (set by an `on_exit` hook to keep the Agent running), `tools` (the tools available in the current step, for hooks to inspect), and `hook_context` (the request-scoped resources passed to `Agent.run(hook_context={...})`).
|
||||
|
||||
If one of your state keys clashes, rename it (for example, `my_token_usage`).
|
||||
:::
|
||||
|
||||
### Default Handlers
|
||||
|
||||
State provides two built-in merge behaviors (importable from `haystack.components.agents.state`):
|
||||
|
||||
- **`merge_lists`**: Appends to the existing list (default for list types)
|
||||
- **`replace_values`**: Overwrites the existing value (default for non-list types)
|
||||
|
||||
```python
|
||||
from haystack.components.agents import State
|
||||
|
||||
schema = {
|
||||
"documents": {"type": list}, # uses merge_lists by default
|
||||
"user_name": {"type": str}, # uses replace_values by default
|
||||
}
|
||||
|
||||
state = State(schema=schema)
|
||||
|
||||
state.set("documents", [1, 2])
|
||||
state.set("documents", [3, 4])
|
||||
print(state.get("documents")) # [1, 2, 3, 4]
|
||||
|
||||
state.set("user_name", "Alice")
|
||||
state.set("user_name", "Bob")
|
||||
print(state.get("user_name")) # "Bob"
|
||||
```
|
||||
|
||||
### Custom Handlers
|
||||
|
||||
Custom handlers are useful when the default `merge_lists` or `replace_values` behaviors don't fit your needs.
|
||||
A handler takes the current state value and the new value and returns the merged result.
|
||||
|
||||
The example below uses a deduplication handler, useful when multiple tool calls might return overlapping results and you want to avoid accumulating duplicates in state:
|
||||
|
||||
```python
|
||||
def deduplicate(current_value: list | None, new_value: list) -> list:
|
||||
"""Append new items, skipping any already in the list."""
|
||||
existing = set(current_value or [])
|
||||
return (current_value or []) + [item for item in new_value if item not in existing]
|
||||
|
||||
|
||||
schema = {"doc_ids": {"type": list, "handler": deduplicate}}
|
||||
|
||||
state = State(schema=schema)
|
||||
state.set("doc_ids", ["doc-1", "doc-2"])
|
||||
state.set("doc_ids", ["doc-2", "doc-3"])
|
||||
print(state.get("doc_ids")) # ["doc-1", "doc-2", "doc-3"]
|
||||
```
|
||||
|
||||
You can also override the handler for a single `set()` call:
|
||||
|
||||
```python
|
||||
from haystack.components.agents import State
|
||||
|
||||
|
||||
def concatenate_strings(current: str | None, new: str) -> str:
|
||||
return f"{current}-{new}" if current else new
|
||||
|
||||
|
||||
state = State(schema={"user_name": {"type": str}})
|
||||
state.set("user_name", "Alice")
|
||||
state.set("user_name", "Bob", handler_override=concatenate_strings)
|
||||
print(state.get("user_name")) # "Alice-Bob"
|
||||
```
|
||||
|
||||
## Using State
|
||||
|
||||
Define a `state_schema` when creating the Agent.
|
||||
State keys declared in `state_schema` are exposed as output keys on the agent's result dict alongside `messages` and `last_message`.
|
||||
|
||||
Tools interact with State through three mechanisms:
|
||||
|
||||
- **`outputs_to_state`**: Write tool results to state keys after the tool runs.
|
||||
- **`inputs_from_state`**: Inject state values into tool parameters before the tool runs.
|
||||
- **Direct `State` injection**: Add a `state: State` parameter to your tool function's signature. The Agent detects the `State` annotation and injects the live `State` object automatically, so you can read or write any key defined in the schema. The `State` object is never exposed to the LLM's parameter schema.
|
||||
|
||||
### Reading from State: `inputs_from_state`
|
||||
|
||||
`inputs_from_state` maps state keys to function parameter names using the format `{"state_key": "param_name"}`.
|
||||
The value is injected from state before the tool runs, so the LLM never needs to provide it.
|
||||
|
||||
Parameters mapped via `inputs_from_state` are automatically excluded from the LLM's parameter schema.
|
||||
The model never sees or provides them:
|
||||
|
||||
```python
|
||||
from typing import Annotated
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.tools import tool
|
||||
|
||||
|
||||
@tool(inputs_from_state={"user_name": "user_context"})
|
||||
def search_documents(
|
||||
query: Annotated[str, "The search query"],
|
||||
user_context: str, # injected from state; excluded from LLM schema
|
||||
) -> dict:
|
||||
"""Search documents using query and user context."""
|
||||
return {"results": [f"Found results for '{query}' (user: {user_context})"]}
|
||||
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
tools=[search_documents],
|
||||
system_prompt="Use the search_documents tool to find information.",
|
||||
streaming_callback=print_streaming_chunk,
|
||||
state_schema={"user_name": {"type": str}},
|
||||
)
|
||||
|
||||
result = agent.run(
|
||||
messages=[ChatMessage.from_user("Search for Python tutorials")],
|
||||
user_name="Alice", # state key "user_name" is pre-populated by passing user_name= to agent.run()
|
||||
)
|
||||
|
||||
print(result["last_message"].text)
|
||||
```
|
||||
|
||||
### Writing to State: `outputs_to_state`
|
||||
|
||||
The `outputs_to_state` parameter maps tool output keys to state keys. Each entry supports two optional fields:
|
||||
|
||||
```python
|
||||
{
|
||||
"state_key": {
|
||||
"source": "tool_output_key", # which key to read from the tool's return dict; omit to store the entire dict
|
||||
"handler": some_func, # override the schema's merge handler for this mapping only
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
```python
|
||||
from typing import Annotated
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.tools import tool
|
||||
|
||||
|
||||
@tool(
|
||||
outputs_to_state={
|
||||
"documents": {"source": "documents"},
|
||||
"result_count": {"source": "count"},
|
||||
"last_query": {"source": "query"},
|
||||
},
|
||||
)
|
||||
def retrieve_documents(
|
||||
query: Annotated[str, "The search query"],
|
||||
) -> dict:
|
||||
"""Retrieve relevant documents."""
|
||||
return {
|
||||
"documents": [
|
||||
{"title": "Doc 1", "content": "Content about Python"},
|
||||
{"title": "Doc 2", "content": "More about Python"},
|
||||
],
|
||||
"count": 2,
|
||||
"query": query,
|
||||
}
|
||||
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
tools=[retrieve_documents],
|
||||
system_prompt="Use the retrieve_documents tool to find information.",
|
||||
streaming_callback=print_streaming_chunk,
|
||||
state_schema={
|
||||
"documents": {"type": list},
|
||||
"result_count": {"type": int},
|
||||
"last_query": {"type": str},
|
||||
},
|
||||
)
|
||||
|
||||
result = agent.run(messages=[ChatMessage.from_user("Find information about Python")])
|
||||
|
||||
print(f"Documents: {result['documents']}")
|
||||
print(f"Result count: {result['result_count']}")
|
||||
print(f"Last query: {result['last_query']}")
|
||||
```
|
||||
|
||||
If you omit `source`, the entire tool result dict is stored under the state key:
|
||||
|
||||
```python
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.tools import tool
|
||||
|
||||
|
||||
@tool(outputs_to_state={"user_info": {}})
|
||||
def get_user_info() -> dict:
|
||||
"""Get user information."""
|
||||
return {"name": "Alice", "email": "alice@example.com", "role": "admin"}
|
||||
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
tools=[get_user_info],
|
||||
system_prompt="Use the get_user_info tool to look up user details.",
|
||||
streaming_callback=print_streaming_chunk,
|
||||
state_schema={"user_info": {"type": dict}},
|
||||
)
|
||||
|
||||
result = agent.run(messages=[ChatMessage.from_user("What are the user's details?")])
|
||||
|
||||
print(result["last_message"].text)
|
||||
print(f"User info: {result['user_info']}")
|
||||
```
|
||||
|
||||
### Combining Inputs and Outputs
|
||||
|
||||
Tools can both read from and write to State, enabling tool chaining across iterations.
|
||||
This example builds on `retrieve_documents` from the previous section:
|
||||
|
||||
```python
|
||||
@tool(
|
||||
inputs_from_state={"documents": "documents"},
|
||||
outputs_to_state={
|
||||
"final_docs": {"source": "processed_docs"},
|
||||
"final_count": {"source": "processed_count"},
|
||||
},
|
||||
)
|
||||
def process_documents(
|
||||
max_results: Annotated[int, "Maximum number of documents to return"],
|
||||
documents: list = None, # injected from state; LLM does not provide this
|
||||
) -> dict:
|
||||
"""Process retrieved documents and return a filtered subset."""
|
||||
processed = (documents or [])[:max_results]
|
||||
return {"processed_docs": processed, "processed_count": len(processed)}
|
||||
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
tools=[retrieve_documents, process_documents], # chained through state
|
||||
system_prompt="Use the available tools to retrieve and process documents.",
|
||||
streaming_callback=print_streaming_chunk,
|
||||
state_schema={
|
||||
"documents": {"type": list},
|
||||
"result_count": {"type": int},
|
||||
"last_query": {"type": str},
|
||||
"final_docs": {"type": list},
|
||||
"final_count": {"type": int},
|
||||
},
|
||||
)
|
||||
|
||||
result = agent.run(
|
||||
messages=[ChatMessage.from_user("Find and process 3 documents about Python")],
|
||||
)
|
||||
print(f"Processed {result['final_count']} documents")
|
||||
```
|
||||
|
||||
### Injecting State Directly into Tools
|
||||
|
||||
As an alternative to `inputs_from_state` and `outputs_to_state`, a tool can declare a `state: State` parameter to receive the live `State` object at invocation time.
|
||||
This lets the tool read from and write to any number of state keys without declaring mappings upfront.
|
||||
|
||||
The ToolInvoker detects the `State` annotation and injects the object automatically.
|
||||
It is excluded from the LLM-facing schema. The model is never asked to supply it.
|
||||
Both `State` and `State | None` annotations are supported.
|
||||
|
||||
For function-based tools, add the `state` parameter and use the `@tool` decorator:
|
||||
|
||||
```python
|
||||
from typing import Annotated
|
||||
from haystack.components.agents import Agent, State
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
from haystack.dataclasses import ChatMessage, Document
|
||||
from haystack.tools import tool
|
||||
|
||||
|
||||
@tool
|
||||
def retrieve_and_store(
|
||||
query: Annotated[str, "The search query"],
|
||||
state: State,
|
||||
) -> str:
|
||||
"""Retrieve documents and store them directly in state."""
|
||||
documents = [Document(content=f"Result for '{query}'")]
|
||||
state.set("documents", documents)
|
||||
user_name = state.get("user_name", "unknown")
|
||||
return f"Retrieved {len(documents)} document(s) for {user_name}"
|
||||
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
tools=[retrieve_and_store],
|
||||
system_prompt="Use the retrieve_and_store tool to find documents.",
|
||||
streaming_callback=print_streaming_chunk,
|
||||
state_schema={"documents": {"type": list[Document]}, "user_name": {"type": str}},
|
||||
)
|
||||
|
||||
result = agent.run(
|
||||
messages=[ChatMessage.from_user("Find documents about Python")],
|
||||
user_name="Alice",
|
||||
)
|
||||
|
||||
print(result["last_message"].text)
|
||||
print(result["documents"])
|
||||
```
|
||||
|
||||
For component-based tools, declare a `State` input socket on the `run` method and wrap it with `ComponentTool`:
|
||||
|
||||
```python
|
||||
from haystack import component
|
||||
from haystack.components.agents import Agent, State
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
from haystack.dataclasses import ChatMessage, Document
|
||||
from haystack.tools import ComponentTool
|
||||
|
||||
|
||||
@component
|
||||
class DocumentRetriever:
|
||||
"""Retrieve documents and store them in state."""
|
||||
|
||||
@component.output_types(reply=str)
|
||||
def run(self, query: str, state: State) -> dict[str, str]:
|
||||
"""
|
||||
Retrieve documents based on query and store them in state.
|
||||
|
||||
:param query: The search query
|
||||
"""
|
||||
documents = [Document(content=f"Result for '{query}'")]
|
||||
state.set("documents", documents)
|
||||
return {"reply": f"Retrieved {len(documents)} document(s)"}
|
||||
|
||||
|
||||
retriever_tool = ComponentTool(
|
||||
component=DocumentRetriever(),
|
||||
name="retrieve",
|
||||
description="Retrieve documents based on a search query",
|
||||
)
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
tools=[retriever_tool],
|
||||
system_prompt="Use the retrieve tool to find documents.",
|
||||
streaming_callback=print_streaming_chunk,
|
||||
state_schema={"documents": {"type": list[Document]}},
|
||||
)
|
||||
|
||||
result = agent.run(messages=[ChatMessage.from_user("Find documents about Python")])
|
||||
|
||||
print(result["last_message"].text)
|
||||
print(result["documents"])
|
||||
```
|
||||
@@ -0,0 +1,224 @@
|
||||
---
|
||||
title: "Tool Result Offloading"
|
||||
id: tool-result-offloading
|
||||
slug: "/tool-result-offloading"
|
||||
description: "Tool result offloading writes large tool results to a store and replaces them in the conversation with a compact pointer, keeping the Agent's context window small."
|
||||
---
|
||||
|
||||
# Tool Result Offloading
|
||||
|
||||
Tool result offloading writes selected tool results to a store and replaces them in the conversation with a compact pointer — a reference plus a short preview — so the next LLM call sees a reference instead of the full result.
|
||||
This keeps the context window small when tools return large outputs (web pages, file contents, query results), and it is a step towards letting an Agent operate on offloaded results with follow-up tools, such as a file-reading tool that opens the referenced files.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Configured on** | The [`Agent`](./agent.mdx) component, as a `ToolResultOffloadHook` registered under the `after_tool` [hook point](./hooks.mdx) |
|
||||
| **Key classes** | `ToolResultOffloadHook`, `FileSystemToolResultStore`, `AlwaysOffload`, `NeverOffload`, `OffloadOverChars` |
|
||||
| **Import path** | `haystack.hooks.tool_result_offloading` |
|
||||
| **API reference** | [Hooks](/reference/hooks-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/hooks/tool_result_offloading/ |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
Tool result offloading is one application of the Agent's general [hooks](./hooks.mdx) mechanism: a `ToolResultOffloadHook` registered under the `after_tool` hook point runs after each step's tools execute and rewrites the freshly produced tool-result messages in the Agent's [`State`](./state.mdx). It only considers the current step's results; earlier conversation history is left untouched.
|
||||
|
||||
The system is composed of these layers:
|
||||
|
||||
- **`ToolResultOffloadHook`** - the `after_tool` hook that applies your offload strategies to fresh tool results. Its `offload_strategies` mapping accepts a single tool name, a tuple of tool names, or the wildcard `"*"` that applies to any tool without a more specific entry.
|
||||
- **Policy** - decides *whether* a given result is offloaded. Built-in policies: `AlwaysOffload`, `NeverOffload`, `OffloadOverChars`.
|
||||
- **Store** - decides *where* the full result lives. The built-in `FileSystemToolResultStore` writes results to the local file system.
|
||||
|
||||
When a result is offloaded, the hook writes the full text to the store and rebuilds the message with a one-line pointer in its place:
|
||||
|
||||
```
|
||||
Tool result offloaded to '/abs/path/tool_results/2_search_call-123.txt' (18234 characters). Preview: Fusion startups reported...
|
||||
```
|
||||
|
||||
The pointer carries the store reference, the original length, and a preview of the first `preview_chars` characters (200 by default, configurable on the hook), so the model knows roughly what was offloaded and where to find it.
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic setup
|
||||
|
||||
The example below offloads any tool result longer than 4,000 characters to files under a local `tool_results` directory:
|
||||
|
||||
```python
|
||||
from typing import Annotated
|
||||
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.hooks.tool_result_offloading import (
|
||||
FileSystemToolResultStore,
|
||||
OffloadOverChars,
|
||||
ToolResultOffloadHook,
|
||||
)
|
||||
from haystack.tools import tool
|
||||
|
||||
|
||||
@tool
|
||||
def search(query: Annotated[str, "The search query"]) -> str:
|
||||
"""Search the web and return the (potentially large) results."""
|
||||
# Placeholder: would call a real search API
|
||||
return f"... large result for {query} ..."
|
||||
|
||||
|
||||
offload_hook = ToolResultOffloadHook(
|
||||
store=FileSystemToolResultStore(root="tool_results"),
|
||||
offload_strategies={"*": OffloadOverChars(4000)},
|
||||
)
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(model="gpt-5.4-nano"),
|
||||
tools=[search],
|
||||
hooks={"after_tool": [offload_hook]},
|
||||
)
|
||||
|
||||
result = agent.run(messages=[ChatMessage.from_user("Summarize today's tech news")])
|
||||
```
|
||||
|
||||
### Configuring what gets offloaded per tool
|
||||
|
||||
Each key in `offload_strategies` may be a single tool name, a tuple of tool names sharing one policy, or the wildcard `"*"`. More specific keys win over `"*"`, and a tool with no matching key (and no `"*"`) is never offloaded:
|
||||
|
||||
```python
|
||||
from haystack.hooks.tool_result_offloading import (
|
||||
AlwaysOffload,
|
||||
FileSystemToolResultStore,
|
||||
NeverOffload,
|
||||
OffloadOverChars,
|
||||
ToolResultOffloadHook,
|
||||
)
|
||||
|
||||
offload_hook = ToolResultOffloadHook(
|
||||
store=FileSystemToolResultStore(root="tool_results"),
|
||||
offload_strategies={
|
||||
"web_search": AlwaysOffload(), # force offload
|
||||
"get_time": NeverOffload(), # opt out of the wildcard default
|
||||
("read_file", "list_dir"): OffloadOverChars(4000), # tuple key: shared policy
|
||||
"*": OffloadOverChars(8000), # default for any unlisted tool
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
### What is offloaded
|
||||
|
||||
The hook only offloads **successful, text** tool results:
|
||||
|
||||
- Error results — including rejections produced by a `before_tool` [Human-in-the-Loop](./human-in-the-loop.mdx) hook — are always left in context, so the model sees what went wrong.
|
||||
- Non-text results (image or file content) are left in context; supporting only text is a deliberate choice for now. A warning is logged when a non-text result has a matching offload policy.
|
||||
- Each result is offloaded at most once, even though the hook runs on every tool step. This also means two offload hooks registered under `after_tool` won't offload each other's pointers.
|
||||
|
||||
## Policies
|
||||
|
||||
Policies control *whether* a result is offloaded.
|
||||
|
||||
| Policy | Behavior |
|
||||
| --- | --- |
|
||||
| `AlwaysOffload` | Offload every result of the tool it is assigned to |
|
||||
| `NeverOffload` | Never offload - keep the full result in context (useful to opt a tool out of a wildcard default) |
|
||||
| `OffloadOverChars(threshold)` | Offload only when the result is longer than `threshold` characters |
|
||||
|
||||
### Custom policy
|
||||
|
||||
Subclass the `OffloadPolicy` protocol from `haystack.hooks.tool_result_offloading` for custom conditions. A policy needs a `should_offload` method, which receives the tool name, the result text, and the Agent's live [`State`](./state.mdx), so it can also decide based on run context:
|
||||
|
||||
```python
|
||||
from haystack.components.agents.state import State
|
||||
from haystack.hooks.tool_result_offloading import OffloadPolicy
|
||||
|
||||
|
||||
class OffloadLateSteps(OffloadPolicy):
|
||||
"""Offload results only once the run is several steps deep and context pressure builds up."""
|
||||
|
||||
def should_offload(self, tool_name: str, result: str, state: State) -> bool:
|
||||
return state.data.get("step_count", 0) >= 3 and len(result) > 1000
|
||||
```
|
||||
|
||||
The protocol provides default `to_dict` / `from_dict` implementations, so a policy like this one, whose constructor takes no arguments, is serializable as-is. A policy with constructor arguments should implement both methods itself, following `OffloadOverChars` as an example.
|
||||
|
||||
## Stores
|
||||
|
||||
### `FileSystemToolResultStore`
|
||||
|
||||
`FileSystemToolResultStore(root=...)` writes each offloaded result to a file under its root directory and returns the absolute file path as the reference. The directory is created on first write. Store keys are derived from the step count, tool name, and tool call ID (for example `2_search_call-123.txt`), so results from different tools and steps do not collide. A key that would resolve outside the root directory is rejected.
|
||||
|
||||
### Custom store
|
||||
|
||||
Subclass the `ToolResultStore` protocol to target other backends, such as object storage or an isolated sandbox file system. A store needs two methods: `write(key=..., content=...)` persists the content and returns an opaque reference string, and `read(reference)` resolves that reference back to the content:
|
||||
|
||||
```python
|
||||
from haystack.hooks.tool_result_offloading import ToolResultStore
|
||||
|
||||
|
||||
class InMemoryToolResultStore(ToolResultStore):
|
||||
"""Keep offloaded results in a dict - useful for tests."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._data: dict[str, str] = {}
|
||||
|
||||
def write(self, *, key: str, content: str) -> str:
|
||||
self._data[key] = content
|
||||
return key
|
||||
|
||||
def read(self, reference: str) -> str:
|
||||
return self._data[reference]
|
||||
```
|
||||
|
||||
Like `OffloadPolicy`, the protocol provides default `to_dict` / `from_dict` implementations covering stores whose constructor takes no arguments; implement both methods for stores with constructor arguments.
|
||||
|
||||
### Per-run stores via `hook_context`
|
||||
|
||||
The constructor `store` is shared by every run - fine for single-user or local use. In a multi-user server, give each run its own isolated store (for example, a per-session directory) by passing it in the Agent's generic `hook_context` run argument under the key `RESULT_STORE_CONTEXT_KEY`. It overrides the constructor store for that run:
|
||||
|
||||
```python
|
||||
from haystack.hooks.tool_result_offloading import (
|
||||
RESULT_STORE_CONTEXT_KEY,
|
||||
FileSystemToolResultStore,
|
||||
)
|
||||
|
||||
per_request_store = FileSystemToolResultStore(root=f"tool_results/{session_id}")
|
||||
result = agent.run(
|
||||
messages=[ChatMessage.from_user("...")],
|
||||
hook_context={RESULT_STORE_CONTEXT_KEY: per_request_store},
|
||||
)
|
||||
```
|
||||
|
||||
Isolating the store per run keeps concurrent users from colliding on store keys or reading each other's offloaded results — especially important when a file-reading tool is scoped to the store. The hook itself keeps no mutable state, so a single instance is safe to share across concurrent runs.
|
||||
|
||||
## Letting the Agent read offloaded results back
|
||||
|
||||
The pointer left in the conversation tells the model where the full result lives, but the model can only act on it if the Agent has a tool that can read from the store. With `FileSystemToolResultStore`, that can be a simple file-reading tool:
|
||||
|
||||
```python
|
||||
from typing import Annotated
|
||||
|
||||
from haystack.tools import tool
|
||||
|
||||
|
||||
@tool
|
||||
def read_offloaded_result(
|
||||
path: Annotated[str, "Absolute path of an offloaded tool result"],
|
||||
) -> str:
|
||||
"""Read back the full content of an offloaded tool result."""
|
||||
return FileSystemToolResultStore(root="tool_results").read(path)
|
||||
```
|
||||
|
||||
With this tool available, the Agent can work with a compact conversation and selectively re-read only the offloaded results it actually needs — instead of carrying every full result in context on every LLM call.
|
||||
|
||||
## Serialization
|
||||
|
||||
`ToolResultOffloadHook` implements `to_dict` / `from_dict`, so an Agent using it can be serialized as long as the configured store and policies are serializable too. The built-in store and policies all are; for custom ones, see the notes in [Policies](#custom-policy) and [Stores](#custom-store) above.
|
||||
|
||||
## Additional References
|
||||
|
||||
📖 Related docs:
|
||||
|
||||
- [Hooks](./hooks.mdx) — the general mechanism behind this feature, including the `after_tool` hook point
|
||||
- [Human in the Loop](./human-in-the-loop.mdx) — another ready-made hook, intercepting tool calls for human review
|
||||
- [State](./state.mdx) — the live run state hooks and policies receive
|
||||
@@ -0,0 +1,16 @@
|
||||
---
|
||||
title: "Audio"
|
||||
id: audio
|
||||
slug: "/audio"
|
||||
description: "Use these components to work with audio in Haystack by transcribing files or converting text to audio."
|
||||
---
|
||||
|
||||
# Audio
|
||||
|
||||
Use these components to work with audio in Haystack by transcribing files or converting text to audio.
|
||||
|
||||
| Name | Description |
|
||||
| --- | --- |
|
||||
| [FunASRTranscriber](audio/funasrtranscriber.mdx) | Transcribe audio files using FunASR — a local, open-source speech recognition toolkit supporting 50+ languages. |
|
||||
| [LocalWhisperTranscriber](audio/localwhispertranscriber.mdx) | Transcribe audio files using OpenAI's Whisper model using your local installation of Whisper. |
|
||||
| [RemoteWhisperTranscriber](audio/remotewhispertranscriber.mdx) | Transcribe audio files using OpenAI's Whisper model. |
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
title: "External Integrations"
|
||||
id: external-integrations-audio
|
||||
slug: "/external-integrations-audio"
|
||||
description: "External integrations that enable working with audio in Haystack by transcribing files or converting text to audio."
|
||||
---
|
||||
|
||||
# External Integrations
|
||||
|
||||
External integrations that enable working with audio in Haystack by transcribing files or converting text to audio.
|
||||
|
||||
| Name | Description |
|
||||
| --- | --- |
|
||||
| [AssemblyAI](https://haystack.deepset.ai/integrations/assemblyai) | Perform speech recognition, speaker diarization and summarization. |
|
||||
| [Elevenlabs](https://haystack.deepset.ai/integrations/elevenlabs) | Convert text to speech using ElevenLabs’ API. |
|
||||
@@ -0,0 +1,66 @@
|
||||
---
|
||||
title: "FunASRTranscriber"
|
||||
id: funasrtranscriber
|
||||
slug: "/funasrtranscriber"
|
||||
description: "Transcribe audio files to Documents using FunASR — a local, open-source speech recognition toolkit supporting 50+ languages."
|
||||
---
|
||||
|
||||
# FunASRTranscriber
|
||||
|
||||
Transcribe audio files to Haystack Documents using FunASR — a local, open-source speech recognition toolkit supporting 50+ languages.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | As the first component in an indexing pipeline |
|
||||
| **Mandatory run variables** | `sources`: A list of audio file paths (`str` or `Path`) or `ByteStream` objects |
|
||||
| **Output variables** | `documents`: A list of Haystack Documents, one per source, with transcript text in `content` |
|
||||
| **API reference** | [FunASR integration](/reference/integrations-funasr) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/blob/main/integrations/funasr/src/haystack_integrations/components/audio/funasr/transcriber.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`FunASRTranscriber` uses [FunASR](https://github.com/modelscope/FunASR), an open-source speech recognition toolkit from Alibaba DAMO Academy, to transcribe audio files into Haystack `Document` objects. It runs entirely locally — no API key required.
|
||||
|
||||
The default model is `iic/SenseVoiceSmall`, a multilingual model supporting 50+ languages that is 5–10x faster than Whisper. Models are downloaded from ModelScope on first use and cached in `~/.cache/modelscope`.
|
||||
|
||||
The component accepts audio file paths (`str` or `Path`) as well as `ByteStream` objects. The model is loaded into memory automatically the first time the component runs.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.audio.funasr import FunASRTranscriber
|
||||
|
||||
transcriber = FunASRTranscriber()
|
||||
|
||||
result = transcriber.run(sources=["speech.wav"])
|
||||
print(result["documents"][0].content)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.fetchers import LinkContentFetcher
|
||||
from haystack_integrations.components.audio.funasr import FunASRTranscriber
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("fetcher", LinkContentFetcher())
|
||||
pipe.add_component("transcriber", FunASRTranscriber())
|
||||
|
||||
pipe.connect("fetcher", "transcriber")
|
||||
|
||||
result = pipe.run(
|
||||
data={
|
||||
"fetcher": {
|
||||
"urls": ["https://example.com/interview.wav"],
|
||||
},
|
||||
},
|
||||
)
|
||||
print(result["transcriber"]["documents"][0].content)
|
||||
```
|
||||
@@ -0,0 +1,90 @@
|
||||
---
|
||||
title: "LocalWhisperTranscriber"
|
||||
id: localwhispertranscriber
|
||||
slug: "/localwhispertranscriber"
|
||||
description: "Use `LocalWhisperTranscriber` to transcribe audio files using OpenAI's Whisper model using your local installation of Whisper."
|
||||
---
|
||||
|
||||
# LocalWhisperTranscriber
|
||||
|
||||
Use `LocalWhisperTranscriber` to transcribe audio files using OpenAI's Whisper model using your local installation of Whisper.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | As the first component in an indexing pipeline |
|
||||
| **Mandatory run variables** | `sources`: A list of paths or binary streams that you want to transcribe |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Whisper](/reference/integrations-whisper) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/whisper |
|
||||
| **Package name** | `whisper-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The component also needs to know which Whisper model to work with. Specify this in the `model` parameter when initializing the component. All transcription is completed on the executing machine, and the audio is never sent to a third-party provider.
|
||||
|
||||
See other optional parameters you can specify in our [API documentation](/reference/integrations-whisper).
|
||||
|
||||
See the [Whisper API documentation](https://platform.openai.com/docs/guides/speech-to-text) and the official Whisper [GitHub repo](https://github.com/openai/whisper) for the supported audio formats and languages.
|
||||
|
||||
The `LocalWhisperTranscriber` is part of the `whisper-haystack` integration package. To work with it, install the package along with [Whisper](https://github.com/openai/whisper) (which also pulls in torch) using the following commands:
|
||||
|
||||
```python
|
||||
pip install whisper-haystack
|
||||
pip install -U openai-whisper
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Here’s an example of how to use `LocalWhisperTranscriber` on its own:
|
||||
|
||||
```python
|
||||
import requests
|
||||
from haystack_integrations.components.audio.whisper import LocalWhisperTranscriber
|
||||
|
||||
response = requests.get(
|
||||
"https://ia903102.us.archive.org/19/items/100-Best--Speeches/EK_19690725_64kb.mp3",
|
||||
)
|
||||
with open("kennedy_speech.mp3", "wb") as file:
|
||||
file.write(response.content)
|
||||
|
||||
transcriber = LocalWhisperTranscriber(model="tiny")
|
||||
|
||||
transcription = transcriber.run(sources=["./kennedy_speech.mp3"])
|
||||
print(transcription["documents"][0].content)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
The pipeline below fetches an audio file from a specified URL and transcribes it. It first retrieves the audio file using `LinkContentFetcher`, then transcribes the audio into text with `LocalWhisperTranscriber`, and finally outputs the transcription text.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.audio.whisper import LocalWhisperTranscriber
|
||||
from haystack.components.fetchers import LinkContentFetcher
|
||||
from haystack import Pipeline
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("fetcher", LinkContentFetcher())
|
||||
pipe.add_component("transcriber", LocalWhisperTranscriber(model="tiny"))
|
||||
|
||||
pipe.connect("fetcher", "transcriber")
|
||||
result = pipe.run(
|
||||
data={
|
||||
"fetcher": {
|
||||
"urls": [
|
||||
"https://ia903102.us.archive.org/19/items/100-Best--Speeches/EK_19690725_64kb.mp3",
|
||||
],
|
||||
},
|
||||
},
|
||||
)
|
||||
print(result["transcriber"]["documents"][0].content)
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Multilingual RAG from a podcast with Whisper, Qdrant and Mistral](https://haystack.deepset.ai/cookbook/multilingual_rag_podcast)
|
||||
@@ -0,0 +1,104 @@
|
||||
---
|
||||
title: "RemoteWhisperTranscriber"
|
||||
id: remotewhispertranscriber
|
||||
slug: "/remotewhispertranscriber"
|
||||
description: "Use `RemoteWhisperTranscriber` to transcribe audio files using OpenAI's Whisper model."
|
||||
---
|
||||
|
||||
# RemoteWhisperTranscriber
|
||||
|
||||
Use `RemoteWhisperTranscriber` to transcribe audio files using OpenAI's Whisper model.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | As the first component in an indexing pipeline |
|
||||
| **Mandatory init variables** | `api_key`: An OpenAI API key. Can be set with an environment variable `OPENAI_API_KEY`. |
|
||||
| **Mandatory run variables** | `sources`: A list of paths or binary streams that you want to transcribe |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Whisper](/reference/integrations-whisper) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/whisper |
|
||||
| **Package name** | `whisper-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `RemoteWhisperTranscriber` is part of the `whisper-haystack` integration package. Install it with:
|
||||
|
||||
```python
|
||||
pip install whisper-haystack
|
||||
```
|
||||
|
||||
`RemoteWhisperTranscriber` works with OpenAI-compatible clients and isn't limited to just OpenAI as a provider. For example, [Groq](https://console.groq.com/docs/speech-text) offers a drop-in replacement that can be used as well. You can set the API key in one of two ways:
|
||||
|
||||
1. Through the `api_key` initialization parameter, where the key is resolved using [Secret API](../../concepts/secret-management.mdx).
|
||||
2. By setting it in the `OPENAI_API_KEY` environment variable, which the system will use to access the key.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.audio.whisper import RemoteWhisperTranscriber
|
||||
|
||||
transcriber = RemoteWhisperTranscriber()
|
||||
```
|
||||
|
||||
Additionally, the component requires the following parameters to work:
|
||||
|
||||
- `model` specifies the Whisper model.
|
||||
- `api_base_url` specifies the OpenAI base URL and defaults to `"https://api.openai.com/v1"`. If you are using Whisper provider other than OpenAI set this parameter according to provider's documentation.
|
||||
|
||||
See other optional parameters in our [API documentation](/reference/integrations-whisper).
|
||||
|
||||
See the [Whisper API documentation](https://platform.openai.com/docs/guides/speech-to-text) and the official Whisper [GitHub repo](https://github.com/openai/whisper) for the supported audio formats and languages.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Here’s an example of how to use `RemoteWhisperTranscriber` to transcribe a local file:
|
||||
|
||||
```python
|
||||
import requests
|
||||
from haystack_integrations.components.audio.whisper import RemoteWhisperTranscriber
|
||||
|
||||
response = requests.get(
|
||||
"https://ia903102.us.archive.org/19/items/100-Best--Speeches/EK_19690725_64kb.mp3",
|
||||
)
|
||||
with open("kennedy_speech.mp3", "wb") as file:
|
||||
file.write(response.content)
|
||||
|
||||
transcriber = RemoteWhisperTranscriber()
|
||||
transcription = transcriber.run(sources=["./kennedy_speech.mp3"])
|
||||
|
||||
print(transcription["documents"][0].content)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
The pipeline below fetches an audio file from a specified URL and transcribes it. It first retrieves the audio file using `LinkContentFetcher`, then transcribes the audio into text with `RemoteWhisperTranscriber`, and finally outputs the transcription text.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.audio.whisper import RemoteWhisperTranscriber
|
||||
from haystack.components.fetchers import LinkContentFetcher
|
||||
from haystack import Pipeline
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("fetcher", LinkContentFetcher())
|
||||
pipe.add_component("transcriber", RemoteWhisperTranscriber())
|
||||
|
||||
pipe.connect("fetcher", "transcriber")
|
||||
result = pipe.run(
|
||||
data={
|
||||
"fetcher": {
|
||||
"urls": [
|
||||
"https://ia903102.us.archive.org/19/items/100-Best--Speeches/EK_19690725_64kb.mp3",
|
||||
],
|
||||
},
|
||||
},
|
||||
)
|
||||
print(result["transcriber"]["documents"][0].content)
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Multilingual RAG from a podcast with Whisper, Qdrant and Mistral](https://haystack.deepset.ai/cookbook/multilingual_rag_podcast)
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
title: "Builders"
|
||||
id: builders
|
||||
slug: "/builders"
|
||||
---
|
||||
|
||||
# Builders
|
||||
|
||||
| Component | Description |
|
||||
| --- | --- |
|
||||
| [AnswerBuilder](builders/answerbuilder.mdx) | Creates `GeneratedAnswer` objects from the query and the answer. |
|
||||
| [PromptBuilder](builders/promptbuilder.mdx) | Renders prompt templates with given parameters. |
|
||||
| [ChatPromptBuilder](builders/chatpromptbuilder.mdx) | PromptBuilder for chat messages. |
|
||||
@@ -0,0 +1,114 @@
|
||||
---
|
||||
title: "AnswerBuilder"
|
||||
id: answerbuilder
|
||||
slug: "/answerbuilder"
|
||||
description: "Use this component in pipelines that contain a Generator to parse its replies."
|
||||
---
|
||||
|
||||
# AnswerBuilder
|
||||
|
||||
Use this component in pipelines that contain a Generator to parse its replies.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Use in pipelines (such as a RAG pipeline) after a [Generator](../generators.mdx) component to create [`GeneratedAnswer`](../../concepts/data-classes.mdx#generatedanswer) objects from its replies. |
|
||||
| **Mandatory run variables** | `query`: A query string <br /> <br />`replies`: A list of strings, or a list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects that are replies from a Generator |
|
||||
| **Output variables** | `answers`: A list of `GeneratedAnswer` objects |
|
||||
| **API reference** | [Builders](/reference/builders-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/builders/answer_builder.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`AnswerBuilder` takes a query and the replies a Generator returns as input and parses them into `GeneratedAnswer` objects. Optionally, it also takes documents and metadata from the Generator as inputs to enrich the `GeneratedAnswer` objects.
|
||||
|
||||
The `AnswerBuilder` works with both Chat and non-Chat Generators.
|
||||
|
||||
The optional `pattern` parameter defines how to extract answer texts from replies. It needs to be a regular expression with a maximum of one capture group. If a capture group is present, the text matched by the capture group is used as the answer. If no capture group is present, the whole match is used as the answer. If no `pattern` is set, the whole reply is used as the answer text.
|
||||
|
||||
The optional `reference_pattern` parameter can be set to a regular expression that parses referenced documents from the replies so that only those referenced documents are listed in the `GeneratedAnswer` objects. Haystack assumes that documents are referenced by their index in the list of input documents and that indices start at 1. For example, if you set the `reference_pattern` to _`\\[(\\d+)\\]`,_ it finds “1” in a string "This is an answer[1]". If `reference_pattern` is not set, all input documents are listed in the `GeneratedAnswer` objects.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Below is an example where we’re using the `AnswerBuilder` to parse a string that could be the reply received from a Generator using a custom regular expression. Any text other than the answer will not be included in the `GeneratedAnswer` object constructed by the builder.
|
||||
|
||||
```python
|
||||
from haystack.components.builders import AnswerBuilder
|
||||
|
||||
builder = AnswerBuilder(pattern="Answer: (.*)")
|
||||
builder.run(
|
||||
query="What's the answer?",
|
||||
replies=["This is an argument. Answer: This is the answer."],
|
||||
)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Below is an example of a RAG pipeline where we use an `AnswerBuilder` to create `GeneratedAnswer` objects from the replies returned by a Generator. In addition to the text of the reply, these objects also hold the query, the referenced docs, and metadata returned by the Generator.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
|
||||
from haystack.components.builders.answer_builder import AnswerBuilder
|
||||
from haystack.utils import Secret
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.dataclasses import Document
|
||||
|
||||
prompt_template = [
|
||||
ChatMessage.from_system("You are a helpful assistant."),
|
||||
ChatMessage.from_user(
|
||||
"Given these documents, answer the question.\nDocuments:\n"
|
||||
"{% for doc in documents %}{{ doc.content }}{% endfor %}\n"
|
||||
"Question: {{query}}\nAnswer:",
|
||||
),
|
||||
]
|
||||
|
||||
docs = [
|
||||
Document(content="The capital of France is Paris"),
|
||||
Document(content="The capital of England is London"),
|
||||
]
|
||||
document_store = InMemoryDocumentStore()
|
||||
document_store.write_documents(docs)
|
||||
|
||||
p = Pipeline()
|
||||
p.add_component(
|
||||
instance=InMemoryBM25Retriever(document_store=document_store),
|
||||
name="retriever",
|
||||
)
|
||||
p.add_component(
|
||||
instance=ChatPromptBuilder(
|
||||
template=prompt_template,
|
||||
required_variables={"query", "documents"},
|
||||
),
|
||||
name="prompt_builder",
|
||||
)
|
||||
p.add_component(
|
||||
instance=OpenAIChatGenerator(api_key=Secret.from_env_var("OPENAI_API_KEY")),
|
||||
name="llm",
|
||||
)
|
||||
p.add_component(instance=AnswerBuilder(), name="answer_builder")
|
||||
p.connect("retriever", "prompt_builder.documents")
|
||||
p.connect("prompt_builder", "llm.messages")
|
||||
p.connect("llm.replies", "answer_builder.replies")
|
||||
p.connect("retriever", "answer_builder.documents")
|
||||
|
||||
query = "What is the capital of France?"
|
||||
result = p.run(
|
||||
{
|
||||
"retriever": {"query": query},
|
||||
"prompt_builder": {"query": query},
|
||||
"answer_builder": {"query": query},
|
||||
},
|
||||
)
|
||||
|
||||
print(result)
|
||||
```
|
||||
@@ -0,0 +1,477 @@
|
||||
---
|
||||
title: "ChatPromptBuilder"
|
||||
id: chatpromptbuilder
|
||||
slug: "/chatpromptbuilder"
|
||||
description: "This component constructs prompts dynamically by processing chat messages."
|
||||
---
|
||||
|
||||
# ChatPromptBuilder
|
||||
|
||||
This component constructs prompts dynamically by processing chat messages.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a [Generator](../generators.mdx) |
|
||||
| **Mandatory init variables** | `template`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects or a special string template. Needs to be provided either during init or run. |
|
||||
| **Mandatory run variables** | `**kwargs`: Any strings that should be used to render the prompt template. See [Variables](#variables) section for more details. |
|
||||
| **Output variables** | `prompt`: A dynamically constructed prompt |
|
||||
| **API reference** | [Builders](/reference/builders-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/builders/chat_prompt_builder.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `ChatPromptBuilder` component creates prompts using static or dynamic templates written in [Jinja2](https://palletsprojects.com/p/jinja/) syntax, by processing a list of chat messages or a special string template. The templates contain placeholders like `{{ variable }}` that are filled with values provided during runtime. You can use it for static prompts set at initialization or change the templates and variables dynamically while running.
|
||||
|
||||
To use it, start by providing a list of `ChatMessage` objects or a special string as the template.
|
||||
|
||||
[`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) is a data class that includes message content, a role (who generated the message, such as `user`, `assistant`, `system`, `tool`), and optional metadata.
|
||||
|
||||
The builder looks for placeholders in the template and identifies the required variables. You can also list these variables manually. During runtime, the `run` method takes the template and the variables, fills in the placeholders, and returns the completed prompt. If required variables are missing. If the template is invalid, the builder raises an error.
|
||||
|
||||
For example, you can create a simple translation prompt:
|
||||
|
||||
```python
|
||||
template = [ChatMessage.from_user("Translate to {{ target_language }}: {{ text }}")]
|
||||
builder = ChatPromptBuilder(template=template)
|
||||
result = builder.run(target_language="French", text="Hello, how are you?")
|
||||
```
|
||||
|
||||
Or you can also replace the template at runtime with a new one:
|
||||
|
||||
```python
|
||||
new_template = [
|
||||
ChatMessage.from_user("Summarize in {{ target_language }}: {{ content }}"),
|
||||
]
|
||||
result = builder.run(
|
||||
template=new_template,
|
||||
target_language="English",
|
||||
content="A detailed paragraph.",
|
||||
)
|
||||
```
|
||||
|
||||
### Variables
|
||||
|
||||
The template variables found in the init template are used as input types for the component. If there are no `required_variables` set, all variables are considered optional by default. In this case, any missing variables are replaced with empty strings, which can lead to unintended behavior, especially in complex pipelines.
|
||||
|
||||
Use `required_variables` and `variables` to specify the input types and required variables:
|
||||
|
||||
- `required_variables`
|
||||
- Defines which template variables must be provided when the component runs.
|
||||
- If any required variable is missing, the component raises an error and halts execution.
|
||||
- You can:
|
||||
- Pass a list of required variable names (such as `["name"]`), or
|
||||
- Use `"*"` to mark all variables in the template as required.
|
||||
|
||||
- `variables`
|
||||
- Lists all variables that can appear in the template, whether required or optional.
|
||||
- Optional variables that aren't provided are replaced with an empty string in the rendered prompt.
|
||||
- This allows partial prompts to be constructed without errors, unless a variable is marked as required.
|
||||
|
||||
In the example below, only _name_ is required to run the component, while _topic_ is only an optional variable:
|
||||
|
||||
```python
|
||||
template = [
|
||||
ChatMessage.from_user("Hello, {{ name }}. How can I assist you with {{ topic }}?"),
|
||||
]
|
||||
|
||||
builder = ChatPromptBuilder(
|
||||
template=template,
|
||||
required_variables=["name"],
|
||||
variables=["name", "topic"],
|
||||
)
|
||||
|
||||
result = builder.run(name="Alice")
|
||||
# Output: "Hello, Alice. How can I assist you with ?"
|
||||
```
|
||||
|
||||
The component only waits for the required inputs before running.
|
||||
|
||||
### Roles
|
||||
|
||||
A [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) represents a single message in the conversation and can have one of three class methods that build the chat messages: `from_user`, `from_system`, or `from_assistant`. `from_user` messages are inputs provided by the user, such as a query or request. `from_system` messages provide context or instructions to guide the LLM’s behavior, such as setting a tone or purpose for the conversation. `from_assistant` defines the expected or actual response from the LLM.
|
||||
|
||||
Here’s how the roles work together in a `ChatPromptBuilder`:
|
||||
|
||||
```python
|
||||
system_message = ChatMessage.from_system(
|
||||
"You are an assistant helping tourists in {{ language }}.",
|
||||
)
|
||||
|
||||
user_message = ChatMessage.from_user("What are the best places to visit in {{ city }}?")
|
||||
|
||||
assistant_message = ChatMessage.from_assistant(
|
||||
"The best places to visit in {{ city }} include the Eiffel Tower, Louvre Museum, and Montmartre.",
|
||||
)
|
||||
```
|
||||
|
||||
### String Templates
|
||||
|
||||
Instead of a list of `ChatMessage` objects, you can also express the template as a special string.
|
||||
|
||||
This template format allows you to define `ChatMessage` sequences using Jinja2 syntax. Each `{% message %}` block defines a single message with a specific role, and you can insert dynamic content using `{{ variables }}`.
|
||||
|
||||
Compared to using a list of `ChatMessage`s, this format is more flexible and allows including structured parts like images in the templatized `ChatMessage`; to better understand this use case, check out the [multimodal example](#multimodal) in the Usage section below.
|
||||
|
||||
#### The `insert` Tag
|
||||
|
||||
String templates also support an `{% insert %}` tag. It is a placeholder that evaluates an expression to a `ChatMessage` or a list of `ChatMessage` objects and expands it into the prompt, so messages provided at runtime can be interleaved with literal `{% message %}` blocks. For example, you can wrap the runtime messages with a system message above and a templated user message below, then pass the messages (and any template variables) at run time:
|
||||
|
||||
```python
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
template = """
|
||||
{% message role="system" %}You are a helpful assistant.{% endmessage %}
|
||||
{% insert messages %}
|
||||
{% message role="user" %}{{ query }}{% endmessage %}
|
||||
"""
|
||||
|
||||
builder = ChatPromptBuilder(template=template)
|
||||
result = builder.run(
|
||||
messages=[ChatMessage.from_user("Hi"), ChatMessage.from_assistant("Hello!")],
|
||||
query="What's the weather?",
|
||||
)
|
||||
# result["prompt"] -> [system, user "Hi", assistant "Hello!", user "What's the weather?"]
|
||||
```
|
||||
|
||||
All content types (tool calls, tool call results, images, reasoning, `name`, and `meta`) round trip without loss. A missing or empty value expands to nothing.
|
||||
|
||||
The expression can be a plain variable (`{% insert messages %}`), a slice or index (`{% insert messages[-1:] %}`, `{% insert messages[-1] %}`), or a combination of variables (`{% insert previous + current %}`). Multiple `{% insert %}` tags can be used in a single template, so the runtime messages can be split, reordered, or repeated across different positions.
|
||||
|
||||
### Jinja2 Time Extension
|
||||
|
||||
`PromptBuilder` supports the Jinja2 TimeExtension, which allows you to work with datetime formats.
|
||||
|
||||
The Time Extension provides two main features:
|
||||
|
||||
1. A `now` tag that gives you access to the current time,
|
||||
2. Date/time formatting capabilities through Python's datetime module.
|
||||
|
||||
To use the Jinja2 TimeExtension, you need to install a dependency with:
|
||||
|
||||
```shell
|
||||
pip install arrow>=1.3.0
|
||||
```
|
||||
|
||||
#### The `now` Tag
|
||||
|
||||
The `now` tag creates a datetime object representing the current time, which you can then store in a variable:
|
||||
|
||||
```jinja2
|
||||
{% now 'utc' as current_time %}
|
||||
The current UTC time is: {{ current_time }}
|
||||
```
|
||||
|
||||
You can specify different timezones:
|
||||
|
||||
```jinja2
|
||||
{% now 'America/New_York' as ny_time %}
|
||||
The time in New York is: {{ ny_time }}
|
||||
```
|
||||
|
||||
If you don't specify a timezone, your system's local timezone will be used:
|
||||
|
||||
```jinja2
|
||||
{% now as local_time %}
|
||||
Local time: {{ local_time }}
|
||||
```
|
||||
|
||||
#### Date Formatting
|
||||
|
||||
You can format the datetime objects using Python's `strftime` syntax:
|
||||
|
||||
```jinja2
|
||||
{% now as current_time %}
|
||||
Formatted date: {{ current_time.strftime('%Y-%m-%d %H:%M:%S') }}
|
||||
```
|
||||
|
||||
The common format codes are:
|
||||
|
||||
- `%Y`: 4-digit year (for example, 2025)
|
||||
- `%m`: Month as a zero-padded number (01-12)
|
||||
- `%d`: Day as a zero-padded number (01-31)
|
||||
- `%H`: Hour (24-hour clock) as a zero-padded number (00-23)
|
||||
- `%M`: Minute as a zero-padded number (00-59)
|
||||
- `%S`: Second as a zero-padded number (00-59)
|
||||
|
||||
#### Example
|
||||
|
||||
```python
|
||||
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
template = [
|
||||
ChatMessage.from_user("Current date is: {% now 'UTC' %}"),
|
||||
ChatMessage.from_assistant("Thank you for providing the date"),
|
||||
ChatMessage.from_user("Yesterday was: {% now 'UTC' - 'days=1' %}"),
|
||||
]
|
||||
builder = ChatPromptBuilder(template=template)
|
||||
|
||||
result = builder.run()["prompt"]
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
#### With static template
|
||||
|
||||
```python
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
template = [
|
||||
ChatMessage.from_user(
|
||||
"Translate to {{ target_language }}. Context: {{ snippet }}; Translation:",
|
||||
),
|
||||
]
|
||||
builder = ChatPromptBuilder(template=template)
|
||||
builder.run(target_language="spanish", snippet="I can't speak spanish.")
|
||||
```
|
||||
|
||||
#### With special string template
|
||||
|
||||
```python
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
template = """
|
||||
{% message role="user" %}
|
||||
Hello, my name is {{name}}!
|
||||
{% endmessage %}
|
||||
"""
|
||||
|
||||
builder = ChatPromptBuilder(template=template)
|
||||
result = builder.run(name="John")
|
||||
|
||||
assert result["prompt"] == [ChatMessage.from_user("Hello, my name is John!")]
|
||||
```
|
||||
|
||||
#### Specifying name and meta in a ChatMessage
|
||||
|
||||
```python
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
template = """
|
||||
{% message role="user" name="John" meta={"key": "value"} %}
|
||||
Hello from {{country}}!
|
||||
{% endmessage %}
|
||||
"""
|
||||
|
||||
builder = ChatPromptBuilder(template=template)
|
||||
result = builder.run(country="Italy")
|
||||
assert result["prompt"] == [
|
||||
ChatMessage.from_user("Hello from Italy!", name="John", meta={"key": "value"}),
|
||||
]
|
||||
```
|
||||
|
||||
#### Multiple ChatMessages with different roles
|
||||
|
||||
```python
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
template = """
|
||||
{% message role="system" %}
|
||||
You are a {{adjective}} assistant.
|
||||
{% endmessage %}
|
||||
|
||||
{% message role="user" %}
|
||||
Hello, my name is {{name}}!
|
||||
{% endmessage %}
|
||||
|
||||
{% message role="assistant" %}
|
||||
Hello, {{name}}! How can I help you today?
|
||||
{% endmessage %}
|
||||
"""
|
||||
|
||||
builder = ChatPromptBuilder(template=template)
|
||||
result = builder.run(name="John", adjective="helpful")
|
||||
assert result["prompt"] == [
|
||||
ChatMessage.from_system("You are a helpful assistant."),
|
||||
ChatMessage.from_user("Hello, my name is John!"),
|
||||
ChatMessage.from_assistant("Hello, John! How can I help you today?"),
|
||||
]
|
||||
```
|
||||
|
||||
#### Overriding static template at runtime
|
||||
|
||||
```python
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
template = [
|
||||
ChatMessage.from_user(
|
||||
"Translate to {{ target_language }}. Context: {{ snippet }}; Translation:",
|
||||
),
|
||||
]
|
||||
builder = ChatPromptBuilder(template=template)
|
||||
builder.run(target_language="spanish", snippet="I can't speak spanish.")
|
||||
|
||||
summary_template = [
|
||||
ChatMessage.from_user(
|
||||
"Translate to {{ target_language }} and summarize. Context: {{ snippet }}; Summary:",
|
||||
),
|
||||
]
|
||||
builder.run(
|
||||
target_language="spanish",
|
||||
snippet="I can't speak spanish.",
|
||||
template=summary_template,
|
||||
)
|
||||
```
|
||||
|
||||
#### Multimodal
|
||||
|
||||
The `| templatize_part` filter in the example below tells the template engine to insert structured (non-text) objects, such as images, into the message content. These are treated differently from plain text and are rendered as special content parts in the final `ChatMessage`.
|
||||
|
||||
```python
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage, ImageContent
|
||||
|
||||
template = """
|
||||
{% message role="user" meta={"key": "value"}%}
|
||||
Hello! I am {{user_name}}. What's the difference between the following images?
|
||||
{% for image in images %}
|
||||
{{ image | templatize_part }}
|
||||
{% endfor %}
|
||||
{% endmessage %}
|
||||
"""
|
||||
builder = ChatPromptBuilder(template=template)
|
||||
images = [
|
||||
ImageContent.from_file_path("apple.jpg"),
|
||||
ImageContent.from_file_path("kiwi.jpg"),
|
||||
]
|
||||
result = builder.run(user_name="John", images=images)
|
||||
|
||||
assert result["prompt"] == [
|
||||
ChatMessage.from_user(
|
||||
content_parts=[
|
||||
"Hello! I am John. What's the difference between the following images?",
|
||||
*images,
|
||||
],
|
||||
meta={"key": "value"},
|
||||
),
|
||||
]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack import Pipeline
|
||||
from haystack.utils import Secret
|
||||
|
||||
# no parameter init, we don't use any runtime template variables
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
llm = OpenAIChatGenerator()
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
location = "Berlin"
|
||||
language = "English"
|
||||
system_message = ChatMessage.from_system(
|
||||
"You are an assistant giving information to tourists in {{language}}",
|
||||
)
|
||||
messages = [system_message, ChatMessage.from_user("Tell me about {{location}}")]
|
||||
|
||||
res = pipe.run(
|
||||
data={
|
||||
"prompt_builder": {
|
||||
"template_variables": {"location": location, "language": language},
|
||||
"template": messages,
|
||||
},
|
||||
},
|
||||
)
|
||||
print(res)
|
||||
```
|
||||
|
||||
Then, you could ask about the weather forecast for the said location. The `ChatPromptBuilder` fills in the template with the new `day_count` variable and passes it to an LLM once again:
|
||||
|
||||
```python
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack import Pipeline
|
||||
from haystack.utils import Secret
|
||||
|
||||
# no parameter init, we don't use any runtime template variables
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
llm = OpenAIChatGenerator()
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
location = "Berlin"
|
||||
|
||||
messages = [
|
||||
system_message,
|
||||
ChatMessage.from_user(
|
||||
"What's the weather forecast for {{location}} in the next {{day_count}} days?",
|
||||
),
|
||||
]
|
||||
res = pipe.run(
|
||||
data={
|
||||
"prompt_builder": {
|
||||
"template_variables": {"location": location, "day_count": "5"},
|
||||
"template": messages,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
print(res)
|
||||
```
|
||||
|
||||
### In YAML
|
||||
|
||||
This is the YAML representation of the pipeline shown above. It dynamically constructs a prompt and generates an answer using a chat model.
|
||||
|
||||
```yaml
|
||||
components:
|
||||
llm:
|
||||
init_parameters:
|
||||
api_base_url: null
|
||||
api_key:
|
||||
env_vars:
|
||||
- OPENAI_API_KEY
|
||||
strict: true
|
||||
type: env_var
|
||||
generation_kwargs: {}
|
||||
http_client_kwargs: null
|
||||
max_retries: null
|
||||
model: gpt-4o-mini
|
||||
organization: null
|
||||
streaming_callback: null
|
||||
timeout: null
|
||||
tools: null
|
||||
tools_strict: false
|
||||
type: haystack.components.generators.chat.openai.OpenAIChatGenerator
|
||||
prompt_builder:
|
||||
init_parameters:
|
||||
required_variables: null
|
||||
template: null
|
||||
variables: null
|
||||
type: haystack.components.builders.chat_prompt_builder.ChatPromptBuilder
|
||||
connection_type_validation: true
|
||||
connections:
|
||||
- receiver: llm.messages
|
||||
sender: prompt_builder.prompt
|
||||
max_runs_per_component: 100
|
||||
metadata: {}
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Advanced Prompt Customization for Anthropic](https://haystack.deepset.ai/cookbook/prompt_customization_for_anthropic)
|
||||
@@ -0,0 +1,306 @@
|
||||
---
|
||||
title: "PromptBuilder"
|
||||
id: promptbuilder
|
||||
slug: "/promptbuilder"
|
||||
description: "Use this component in pipelines before a Generator to render a prompt template and fill in variable values."
|
||||
---
|
||||
|
||||
# PromptBuilder
|
||||
|
||||
Use this component in pipelines before a Generator to render a prompt template and fill in variable values.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In a querying pipeline, before a [Generator](../generators.mdx) |
|
||||
| **Mandatory init variables** | `template`: A prompt template string that uses Jinja2 syntax |
|
||||
| **Mandatory run variables** | `**kwargs`: Any strings that should be used to render the prompt template. See [Variables](#variables) section for more details. |
|
||||
| **Output variables** | `prompt`: A string that represents the rendered prompt template |
|
||||
| **API reference** | [Builders](/reference/builders-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/builders/prompt_builder.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`PromptBuilder` is initialized with a prompt template and renders it by filling in parameters passed through keyword arguments, `kwargs`. With `kwargs`, you can pass a variable number of keyword arguments so that any variable used in the prompt template can be specified with the desired value. Values for all variables appearing in the prompt template need to be provided through the `kwargs`.
|
||||
|
||||
The template that is provided to the `PromptBuilder` during initialization needs to conform to the [Jinja2](https://palletsprojects.com/p/jinja/) template language.
|
||||
|
||||
### Variables
|
||||
|
||||
The template variables found in the init template are used as input types for the component. If there are no `required_variables` set, all variables are considered optional by default. In this case, any missing variables are replaced with empty strings, which can lead to unintended behavior, especially in complex pipelines.
|
||||
|
||||
Use `required_variables` and `variables` to specify the input types and required variables:
|
||||
|
||||
- `required_variables`
|
||||
- Defines which template variables must be provided when the component runs.
|
||||
- If any required variable is missing, the component raises an error and halts execution.
|
||||
- You can:
|
||||
- Pass a list of required variable names (such as `["query"]`), or
|
||||
- Use `"*"` to mark all variables in the template as required.
|
||||
|
||||
- `variables`
|
||||
- Lists all variables that can appear in the template, whether required or optional.
|
||||
- Optional variables that aren't provided are replaced with an empty string in the rendered prompt.
|
||||
- This allows partial prompts to be constructed without errors, unless a variable is marked as required.
|
||||
|
||||
```python
|
||||
from haystack.components.builders import PromptBuilder
|
||||
|
||||
# All variables optional (default to empty string)
|
||||
builder = PromptBuilder(
|
||||
template="Hello {{name}}! {{greeting}}",
|
||||
required_variables=[], # or omit this parameter entirely
|
||||
)
|
||||
|
||||
# Some variables required
|
||||
builder = PromptBuilder(
|
||||
template="Hello {{name}}! {{greeting}}",
|
||||
required_variables=["name"], # 'greeting' remains optional
|
||||
)
|
||||
```
|
||||
|
||||
The component only waits for the required inputs before running.
|
||||
|
||||
### Jinja2 Time Extension
|
||||
|
||||
`PromptBuilder` supports the Jinja2 TimeExtension, which allows you to work with datetime formats.
|
||||
|
||||
The Time Extension provides two main features:
|
||||
|
||||
1. A `now` tag that gives you access to the current time,
|
||||
2. Date/time formatting capabilities through Python's datetime module.
|
||||
|
||||
To use the Jinja2 TimeExtension, you need to install a dependency with:
|
||||
|
||||
```shell
|
||||
pip install arrow>=1.3.0
|
||||
```
|
||||
|
||||
#### The `now` Tag
|
||||
|
||||
The `now` tag creates a datetime object representing the current time, which you can then store in a variable:
|
||||
|
||||
```jinja2
|
||||
{% now 'utc' as current_time %}
|
||||
The current UTC time is: {{ current_time }}
|
||||
```
|
||||
|
||||
You can specify different timezones:
|
||||
|
||||
```jinja2
|
||||
{% now 'America/New_York' as ny_time %}
|
||||
The time in New York is: {{ ny_time }}
|
||||
```
|
||||
|
||||
If you don't specify a timezone, your system's local timezone will be used:
|
||||
|
||||
```jinja2
|
||||
{% now as local_time %}
|
||||
Local time: {{ local_time }}
|
||||
```
|
||||
|
||||
#### Date Formatting
|
||||
|
||||
You can format the datetime objects using Python's `strftime` syntax:
|
||||
|
||||
```jinja2
|
||||
{% now as current_time %}
|
||||
Formatted date: {{ current_time.strftime('%Y-%m-%d %H:%M:%S') }}
|
||||
```
|
||||
|
||||
The common format codes are:
|
||||
|
||||
- `%Y`: 4-digit year (for example, 2025)
|
||||
- `%m`: Month as a zero-padded number (01-12)
|
||||
- `%d`: Day as a zero-padded number (01-31)
|
||||
- `%H`: Hour (24-hour clock) as a zero-padded number (00-23)
|
||||
- `%M`: Minute as a zero-padded number (00-59)
|
||||
- `%S`: Second as a zero-padded number (00-59)
|
||||
|
||||
#### Example
|
||||
|
||||
```python
|
||||
from haystack.components.builders import PromptBuilder
|
||||
|
||||
# Define template using Jinja-style formatting
|
||||
template = """
|
||||
Current date is: {% now 'UTC' %}
|
||||
Thank you for providing the date
|
||||
Yesterday was: {% now 'UTC' - 'days=1' %}
|
||||
"""
|
||||
|
||||
builder = PromptBuilder(template=template)
|
||||
|
||||
result = builder.run()["prompt"]
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Below is an example of using the `PromptBuilder` to render a prompt template and fill it with `target_language` and `snippet`. The PromptBuilder returns a prompt with the string `Translate the following context to spanish. Context: I can't speak spanish.; Translation:`.
|
||||
|
||||
```python
|
||||
from haystack.components.builders import PromptBuilder
|
||||
|
||||
template = "Translate the following context to {{ target_language }}. Context: {{ snippet }}; Translation:"
|
||||
builder = PromptBuilder(template=template)
|
||||
builder.run(target_language="spanish", snippet="I can't speak spanish.")
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Below is an example of a RAG pipeline where we use a `PromptBuilder` to render a custom prompt template and fill it with the contents of retrieved documents and a query. The rendered prompt is then sent to a Generator.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline, Document
|
||||
from haystack.utils import Secret
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.builders.prompt_builder import PromptBuilder
|
||||
|
||||
# in a real world use case documents could come from a retriever, web, or any other source
|
||||
documents = [
|
||||
Document(content="Joe lives in Berlin"),
|
||||
Document(content="Joe is a software engineer"),
|
||||
]
|
||||
prompt_template = """
|
||||
Given these documents, answer the question.\nDocuments:
|
||||
{% for doc in documents %}
|
||||
{{ doc.content }}
|
||||
{% endfor %}
|
||||
|
||||
\nQuestion: {{query}}
|
||||
\nAnswer:
|
||||
"""
|
||||
p = Pipeline()
|
||||
p.add_component(instance=PromptBuilder(template=prompt_template), name="prompt_builder")
|
||||
p.add_component(
|
||||
instance=OpenAIChatGenerator(api_key=Secret.from_env_var("OPENAI_API_KEY")),
|
||||
name="llm",
|
||||
)
|
||||
p.connect("prompt_builder", "llm")
|
||||
|
||||
question = "Where does Joe live?"
|
||||
result = p.run({"prompt_builder": {"documents": documents, "query": question}})
|
||||
print(result)
|
||||
```
|
||||
|
||||
#### Changing the template at runtime (Prompt Engineering)
|
||||
|
||||
`PromptBuilder` allows you to switch the prompt template of an existing pipeline. The example below builds on top of the existing pipeline in the previous section. We are invoking the existing pipeline with a new prompt template:
|
||||
|
||||
```python
|
||||
documents = [
|
||||
Document(content="Joe lives in Berlin", meta={"name": "doc1"}),
|
||||
Document(content="Joe is a software engineer", meta={"name": "doc1"}),
|
||||
]
|
||||
new_template = """
|
||||
You are a helpful assistant.
|
||||
Given these documents, answer the question.
|
||||
Documents:
|
||||
{% for doc in documents %}
|
||||
Document {{ loop.index }}:
|
||||
Document name: {{ doc.meta['name'] }}
|
||||
{{ doc.content }}
|
||||
{% endfor %}
|
||||
|
||||
Question: {{ query }}
|
||||
Answer:
|
||||
"""
|
||||
p.run(
|
||||
{
|
||||
"prompt_builder": {
|
||||
"documents": documents,
|
||||
"query": question,
|
||||
"template": new_template,
|
||||
},
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
If you want to use different variables during prompt engineering than in the default template, you can do so by setting `PromptBuilder`'s variables init parameter accordingly.
|
||||
|
||||
#### Overwriting variables at runtime
|
||||
|
||||
In case you want to overwrite the values of variables, you can use `template_variables` during runtime, as shown below:
|
||||
|
||||
```python
|
||||
language_template = """
|
||||
You are a helpful assistant.
|
||||
Given these documents, answer the question.
|
||||
Documents:
|
||||
{% for doc in documents %}
|
||||
Document {{ loop.index }}:
|
||||
Document name: {{ doc.meta['name'] }}
|
||||
{{ doc.content }}
|
||||
{% endfor %}
|
||||
|
||||
Question: {{ query }}
|
||||
Please provide your answer in {{ answer_language | default('English') }}
|
||||
Answer:
|
||||
"""
|
||||
p.run(
|
||||
{
|
||||
"prompt_builder": {
|
||||
"documents": documents,
|
||||
"query": question,
|
||||
"template": language_template,
|
||||
"template_variables": {"answer_language": "German"},
|
||||
},
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
Note that `language_template` introduces `answer_language` variable which is not bound to any pipeline variable. If not set otherwise, it would use its default value, "English". In this example, we overwrite its value to "German".
|
||||
The `template_variables` allows you to overwrite pipeline variables (such as documents) as well.
|
||||
|
||||
### In YAML
|
||||
|
||||
This is the YAML representation of the RAG pipeline shown above. It renders a custom prompt template by filling it with the contents of retrieved documents and a query, then sends the rendered prompt to a generator.
|
||||
|
||||
```yaml
|
||||
components:
|
||||
llm:
|
||||
init_parameters:
|
||||
api_base_url: null
|
||||
api_key:
|
||||
env_vars:
|
||||
- OPENAI_API_KEY
|
||||
strict: true
|
||||
type: env_var
|
||||
generation_kwargs: {}
|
||||
http_client_kwargs: null
|
||||
max_retries: null
|
||||
model: gpt-5-mini
|
||||
organization: null
|
||||
streaming_callback: null
|
||||
timeout: null
|
||||
tools: null
|
||||
tools_strict: false
|
||||
type: haystack.components.generators.chat.openai.OpenAIChatGenerator
|
||||
prompt_builder:
|
||||
init_parameters:
|
||||
required_variables: '*'
|
||||
template: "\n Given these documents, answer the question.\nDocuments:\n \
|
||||
\ {% for doc in documents %}\n {{ doc.content }}\n {% endfor %}\n\
|
||||
\n \nQuestion: {{query}}\n \nAnswer:\n "
|
||||
variables: null
|
||||
type: haystack.components.builders.prompt_builder.PromptBuilder
|
||||
connection_type_validation: true
|
||||
connections:
|
||||
- receiver: llm.messages
|
||||
sender: prompt_builder.prompt
|
||||
max_runs_per_component: 100
|
||||
metadata: {}
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbooks:
|
||||
|
||||
- [Advanced Prompt Customization for Anthropic](https://haystack.deepset.ai/cookbook/prompt_customization_for_anthropic)
|
||||
- [Prompt Optimization with DSPy](https://haystack.deepset.ai/cookbook/prompt_optimization_with_dspy)
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
title: "CacheChecker"
|
||||
id: cachechecker
|
||||
slug: "/cachechecker"
|
||||
description: "This component checks for the presence of documents in a Document Store based on a specified cache field."
|
||||
---
|
||||
|
||||
# CacheChecker
|
||||
|
||||
This component checks for the presence of documents in a Document Store based on a specified cache field.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Flexible |
|
||||
| **Mandatory init variables** | `document_store`: A Document Store instance <br /> <br />`cache_field`: Name of the document's metadata field |
|
||||
| **Mandatory run variables** | `items`: A list of values associated with the `cache_field` in documents |
|
||||
| **Output variables** | `hits`: A list of documents that were found with the specified value in cache <br /> <br />`misses`: A list of values that could not be found |
|
||||
| **API reference** | [Caching](/reference/caching-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/caching/cache_checker.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`CacheChecker` checks if a Document Store contains any document with a value in the `cache_field` that matches any of the values provided in the `items` input variable. It returns a dictionary with two keys: `"hits"` and `"misses"`. The values are lists of documents that were found in the cache and items that were not, respectively.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.components.caching import CacheChecker
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
my_doc_store = InMemoryDocumentStore()
|
||||
|
||||
# For URL-based caching
|
||||
cache_checker = CacheChecker(document_store=my_doc_store, cache_field="url")
|
||||
cache_check_results = cache_checker.run(
|
||||
items=[
|
||||
"https://example.com/resource",
|
||||
"https://another_example.com/other_resources",
|
||||
],
|
||||
)
|
||||
print(
|
||||
cache_check_results["hits"],
|
||||
) # List of Documents that were found in the cache: all of these have 'url': <one of the above> in the metadata
|
||||
print(
|
||||
cache_check_results["misses"],
|
||||
) # URLs that were not found in the cache, like ["https://example.com/resource"]
|
||||
|
||||
# For caching based on a custom identifier
|
||||
cache_checker = CacheChecker(document_store=my_doc_store, cache_field="metadata_field")
|
||||
cache_check_results = cache_checker.run(items=["12345", "ABCDE"])
|
||||
print(
|
||||
cache_check_results["hits"],
|
||||
) # Documents that were found in the cache: all of these have 'metadata_field': <one of the above> in the metadata
|
||||
print(
|
||||
cache_check_results["misses"],
|
||||
) # Values that were not found in the cache, like: ["ABCDE"]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.converters import TextFileToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner, DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.components.caching import CacheChecker
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
pipeline = Pipeline()
|
||||
document_store = InMemoryDocumentStore()
|
||||
pipeline.add_component(
|
||||
instance=CacheChecker(document_store, cache_field="meta.file_path"),
|
||||
name="cache_checker",
|
||||
)
|
||||
pipeline.add_component(instance=TextFileToDocument(), name="text_file_converter")
|
||||
pipeline.add_component(instance=DocumentCleaner(), name="cleaner")
|
||||
pipeline.add_component(
|
||||
instance=DocumentSplitter(split_by="sentence", split_length=250, split_overlap=30),
|
||||
name="splitter",
|
||||
)
|
||||
pipeline.add_component(
|
||||
instance=DocumentWriter(document_store=document_store),
|
||||
name="writer",
|
||||
)
|
||||
pipeline.connect("cache_checker.misses", "text_file_converter.sources")
|
||||
pipeline.connect("text_file_converter.documents", "cleaner.documents")
|
||||
pipeline.connect("cleaner.documents", "splitter.documents")
|
||||
pipeline.connect("splitter.documents", "writer.documents")
|
||||
|
||||
pipeline.draw("pipeline.png")
|
||||
|
||||
# Take the current directory as input and run the pipeline
|
||||
result = pipeline.run({"cache_checker": {"items": ["code_of_conduct_1.txt"]}})
|
||||
print(result)
|
||||
|
||||
# The second execution skips the files that were already processed
|
||||
result = pipeline.run({"cache_checker": {"items": ["code_of_conduct_1.txt"]}})
|
||||
print(result)
|
||||
```
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
title: "Classifiers"
|
||||
id: classifiers
|
||||
slug: "/classifiers"
|
||||
description: "Use Classifiers to classify your documents by specific traits and update the metadata."
|
||||
---
|
||||
|
||||
# Classifiers
|
||||
|
||||
Use Classifiers to classify your documents by specific traits and update the metadata.
|
||||
|
||||
| Classifier | Description |
|
||||
| --- | --- |
|
||||
| [DocumentLanguageClassifier](classifiers/documentlanguageclassifier.mdx) | Classify documents by language. |
|
||||
| [TransformersZeroShotDocumentClassifier](classifiers/transformerszeroshotdocumentclassifier.mdx) | Classify the documents based on the provided labels. |
|
||||
@@ -0,0 +1,127 @@
|
||||
---
|
||||
title: "DocumentLanguageClassifier"
|
||||
id: documentlanguageclassifier
|
||||
slug: "/documentlanguageclassifier"
|
||||
description: "Use this component to classify documents by language and add language information to metadata."
|
||||
---
|
||||
|
||||
# DocumentLanguageClassifier
|
||||
|
||||
Use this component to classify documents by language and add language information to metadata.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [`MetadataRouter`](../routers/metadatarouter.mdx) |
|
||||
| **Mandatory run variables** | `documents`: A list of documents |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Langdetect](/reference/integrations-langdetect) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/langdetect |
|
||||
| **Package name** | `langdetect-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`DocumentLanguageClassifier` classifies the language of documents and adds the detected language to their metadata. If a document's text does not match any of the languages specified at initialization, it is classified as "unmatched". By default, the classifier classifies for English (”en”) documents, with the rest being classified as “unmatched”.
|
||||
|
||||
The set of supported languages can be specified in the init method with the `languages` variable, using ISO codes.
|
||||
|
||||
To route your documents to various branches of the pipeline based on the language, use `MetadataRouter` component right after `DocumentLanguageClassifier`.
|
||||
|
||||
For classifying and then routing plain text using the same logic, use the `TextLanguageRouter` component instead.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `langdetect-haystack` package to use the `DocumentLanguageClassifier` component:
|
||||
|
||||
```shell
|
||||
pip install langdetect-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
Below, we are using the `DocumentLanguageClassifier` to classify English and German documents:
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.classifiers.langdetect import (
|
||||
DocumentLanguageClassifier,
|
||||
)
|
||||
from haystack import Document
|
||||
|
||||
documents = [
|
||||
Document(content="Mein Name ist Jean und ich wohne in Paris."),
|
||||
Document(content="Mein Name ist Mark und ich wohne in Berlin."),
|
||||
Document(content="Mein Name ist Giorgio und ich wohne in Rome."),
|
||||
Document(content="My name is Pierre and I live in Paris"),
|
||||
Document(content="My name is Paul and I live in Berlin."),
|
||||
Document(content="My name is Alessia and I live in Rome."),
|
||||
]
|
||||
|
||||
document_classifier = DocumentLanguageClassifier(languages=["en", "de"])
|
||||
document_classifier.run(documents=documents)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Below, we are using the `DocumentLanguageClassifier` in an indexing pipeline that indexes English and German documents into two difference indexes in an `InMemoryDocumentStore`, using embedding models for each language.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack import Document
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.classifiers.langdetect import (
|
||||
DocumentLanguageClassifier,
|
||||
)
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
)
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.components.routers import MetadataRouter
|
||||
|
||||
document_store_en = InMemoryDocumentStore()
|
||||
document_store_de = InMemoryDocumentStore()
|
||||
|
||||
document_classifier = DocumentLanguageClassifier(languages=["en", "de"])
|
||||
metadata_router = MetadataRouter(
|
||||
rules={"en": {"language": {"$eq": "en"}}, "de": {"language": {"$eq": "de"}}},
|
||||
)
|
||||
english_embedder = SentenceTransformersDocumentEmbedder()
|
||||
german_embedder = SentenceTransformersDocumentEmbedder(
|
||||
model="PM-AI/bi-encoder_msmarco_bert-base_german",
|
||||
)
|
||||
en_writer = DocumentWriter(document_store=document_store_en)
|
||||
de_writer = DocumentWriter(document_store=document_store_de)
|
||||
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component(document_classifier, name="document_classifier")
|
||||
indexing_pipeline.add_component(metadata_router, name="metadata_router")
|
||||
indexing_pipeline.add_component(english_embedder, name="english_embedder")
|
||||
indexing_pipeline.add_component(german_embedder, name="german_embedder")
|
||||
indexing_pipeline.add_component(en_writer, name="en_writer")
|
||||
indexing_pipeline.add_component(de_writer, name="de_writer")
|
||||
|
||||
indexing_pipeline.connect("document_classifier.documents", "metadata_router.documents")
|
||||
indexing_pipeline.connect("metadata_router.en", "english_embedder.documents")
|
||||
indexing_pipeline.connect("metadata_router.de", "german_embedder.documents")
|
||||
indexing_pipeline.connect("english_embedder", "en_writer")
|
||||
indexing_pipeline.connect("german_embedder", "de_writer")
|
||||
|
||||
indexing_pipeline.run(
|
||||
{
|
||||
"document_classifier": {
|
||||
"documents": [
|
||||
Document(content="This is an English sentence."),
|
||||
Document(content="Dies ist ein deutscher Satz."),
|
||||
],
|
||||
},
|
||||
},
|
||||
)
|
||||
```
|
||||
+115
@@ -0,0 +1,115 @@
|
||||
---
|
||||
title: "TransformersZeroShotDocumentClassifier"
|
||||
id: transformerszeroshotdocumentclassifier
|
||||
slug: "/transformerszeroshotdocumentclassifier"
|
||||
description: "Classifies the documents based on the provided labels and adds them to their metadata."
|
||||
---
|
||||
|
||||
# TransformersZeroShotDocumentClassifier
|
||||
|
||||
Classifies the documents based on the provided labels and adds them to their metadata.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a [MetadataRouter](../routers/metadatarouter.mdx) |
|
||||
| **Mandatory init variables** | `model`: The name or path of a Hugging Face model for zero shot document classification <br /> <br />`labels`: The set of possible class labels to classify each document into, for example, [`positive`, `negative`]. The labels depend on the selected model. |
|
||||
| **Mandatory run variables** | `documents`: A list of documents to classify |
|
||||
| **Output variables** | `documents`: A list of processed documents with an added `classification` metadata field |
|
||||
| **API reference** | [Transformers](/reference/integrations-transformers) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/transformers |
|
||||
| **Package name** | `transformers-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `TransformersZeroShotDocumentClassifier` component performs zero-shot classification of documents based on the labels that you set and adds the predicted label to their metadata.
|
||||
|
||||
The component uses a Hugging Face pipeline for zero-shot classification.
|
||||
To initialize the component, provide the model and the set of labels to be used for categorization.
|
||||
You can additionally configure the component to allow multiple labels to be true with the `multi_label` boolean set to True.
|
||||
|
||||
Classification is run on the document's content field by default. If you want it to run on another field, set the`classification_field` to one of the document's metadata fields.
|
||||
|
||||
The classification results are stored in the `classification` dictionary within each document's metadata. If `multi_label` is set to `True`, you will find the scores for each label under the `details` key within the `classification` dictionary.
|
||||
|
||||
Available models for the task of zero-shot-classification are:
|
||||
- `valhalla/distilbart-mnli-12-3`
|
||||
- `cross-encoder/nli-distilroberta-base`
|
||||
- `cross-encoder/nli-deberta-v3-xsmall`
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `transformers-haystack` package to use the `TransformersZeroShotDocumentClassifier`:
|
||||
|
||||
```shell
|
||||
pip install transformers-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.classifiers.transformers import (
|
||||
TransformersZeroShotDocumentClassifier,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(id="0", content="Cats don't get teeth cavities."),
|
||||
Document(id="1", content="Cucumbers can be grown in water."),
|
||||
]
|
||||
|
||||
document_classifier = TransformersZeroShotDocumentClassifier(
|
||||
model="cross-encoder/nli-deberta-v3-xsmall",
|
||||
labels=["animals", "food"],
|
||||
)
|
||||
|
||||
document_classifier.run(documents=documents)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
The following is a pipeline that classifies documents based on predefined classification labels
|
||||
retrieved from a search pipeline:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.core.pipeline import Pipeline
|
||||
from haystack_integrations.components.classifiers.transformers import (
|
||||
TransformersZeroShotDocumentClassifier,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(id="0", content="Today was a nice day!"),
|
||||
Document(id="1", content="Yesterday was a bad day!"),
|
||||
]
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
retriever = InMemoryBM25Retriever(document_store=document_store)
|
||||
document_classifier = TransformersZeroShotDocumentClassifier(
|
||||
model="cross-encoder/nli-deberta-v3-xsmall",
|
||||
labels=["positive", "negative"],
|
||||
)
|
||||
|
||||
document_store.write_documents(documents)
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component(name="retriever", instance=retriever)
|
||||
pipeline.add_component(name="document_classifier", instance=document_classifier)
|
||||
pipeline.connect("retriever", "document_classifier")
|
||||
|
||||
queries = ["How was your day today?", "How was your day yesterday?"]
|
||||
expected_predictions = ["positive", "negative"]
|
||||
|
||||
for idx, query in enumerate(queries):
|
||||
result = pipeline.run({"retriever": {"query": query, "top_k": 1}})
|
||||
classified_docs = result["document_classifier"]["documents"]
|
||||
assert classified_docs[0].id == str(idx)
|
||||
assert (
|
||||
classified_docs[0].meta["classification"]["label"] == expected_predictions[idx]
|
||||
)
|
||||
```
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
title: "Connectors"
|
||||
id: connectors
|
||||
slug: "/connectors"
|
||||
description: "These are Haystack integrations that connect your pipelines to services by external providers."
|
||||
---
|
||||
|
||||
# Connectors
|
||||
|
||||
These are Haystack integrations that connect your pipelines to services by external providers.
|
||||
|
||||
| Component | Description |
|
||||
| --- | --- |
|
||||
| [DatadogConnector](connectors/datadogconnector.mdx) | Enables tracing in Haystack pipelines using Datadog. |
|
||||
| [GitHubFileEditor](connectors/githubfileeditor.mdx) | Enables editing files in GitHub repositories through the GitHub API. |
|
||||
| [GitHubIssueCommenter](connectors/githubissuecommenter.mdx) | Enables posting comments to GitHub issues using the GitHub API. |
|
||||
| [GitHubIssueViewer](connectors/githubissueviewer.mdx) | Enables fetching and parsing GitHub issues into Haystack documents. |
|
||||
| [GitHubPRCreator](connectors/githubprcreator.mdx) | Enables creating pull requests from a fork back to the original repository through the GitHub API. |
|
||||
| [GitHubRepoForker](connectors/githubrepoforker.mdx) | Enables forking a GitHub repository from an issue URL through the GitHub API. |
|
||||
| [GitHubRepoViewer](connectors/githubrepoviewer.mdx) | Enables navigating and fetching content from GitHub repositories through the GitHub API. |
|
||||
| [JinaReaderConnector](connectors/jinareaderconnector.mdx) | Use Jina AI’s Reader API with Haystack. |
|
||||
| [LangfuseConnector](connectors/langfuseconnector.mdx) | Enables tracing in Haystack pipelines using Langfuse. |
|
||||
| [OAuthTokenResolver](connectors/oauthtokenresolver.mdx) | Resolves an OAuth access token at runtime and emits it for downstream components. |
|
||||
| [OpenAPIConnector](connectors/openapiconnector.mdx) | Acts as an interface between the Haystack ecosystem and OpenAPI services, using explicit input arguments. |
|
||||
| [OpenAPIServiceConnector](connectors/openapiserviceconnector.mdx) | Acts as an interface between the Haystack ecosystem and OpenAPI services. |
|
||||
| [OpenTelemetryConnector](connectors/opentelemetryconnector.mdx) | Enables tracing in Haystack pipelines using OpenTelemetry. |
|
||||
| [WeaveConnector](connectors/weaveconnector.mdx) | Connects you to Weights & Biases Weave framework for tracing and monitoring your pipeline components. |
|
||||
@@ -0,0 +1,202 @@
|
||||
---
|
||||
title: "DatadogConnector"
|
||||
id: datadogconnector
|
||||
slug: "/datadogconnector"
|
||||
description: "Learn how to work with Datadog in Haystack."
|
||||
---
|
||||
|
||||
# DatadogConnector
|
||||
|
||||
Learn how to work with Datadog in Haystack.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Anywhere, as it’s not connected to other components |
|
||||
| **Mandatory init variables** | None. The connection to the Datadog backend is created at initialization time |
|
||||
| **Output variables** | `name`: The name of the tracing component |
|
||||
| **API reference** | [datadog](/reference/integrations-datadog) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/datadog |
|
||||
| **Package name** | `datadog-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`DatadogConnector` integrates tracing capabilities into Haystack pipelines using [Datadog](https://www.datadoghq.com/), through [Datadog's tracing library `ddtrace`](https://ddtrace.readthedocs.io/en/stable/). It captures detailed information about pipeline runs, like API calls, context data, prompts, and more, so you can see the complete trace of your pipeline execution in Datadog.
|
||||
|
||||
Datadog tracing is enabled as soon as the `DatadogConnector` is initialized, so you only need to add it to your pipeline – it does not need to be connected to other components or to run to take effect.
|
||||
|
||||
You can optionally pass a `name` to identify this tracing component (it defaults to `datadog`).
|
||||
|
||||
### Prerequisites
|
||||
|
||||
These are the things that you need before working with the `DatadogConnector`:
|
||||
|
||||
1. A way to receive traces, such as a running [Datadog Agent](https://docs.datadoghq.com/agent/). `ddtrace` sends traces to the Datadog Agent at `localhost:8126` by default.
|
||||
2. Set the `HAYSTACK_CONTENT_TRACING_ENABLED` environment variable to `true` – this will enable content tracing (inputs and outputs) in your pipelines.
|
||||
3. Configure `ddtrace` through the standard mechanisms, for example the `DD_SERVICE`, `DD_ENV`, and `DD_VERSION` environment variables, or by running your application with the `ddtrace-run` command. See the [ddtrace documentation](https://ddtrace.readthedocs.io/en/stable/) for more details.
|
||||
|
||||
### Installation
|
||||
|
||||
First, install the `datadog-haystack` package to use the `DatadogConnector`:
|
||||
|
||||
```shell
|
||||
pip install datadog-haystack
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
:::info[Usage Notice]
|
||||
|
||||
To ensure proper tracing, always set environment variables before importing any Haystack components. This is crucial because Haystack initializes its internal tracing components during import. In the example below, we first set the environment variables and then import the relevant Haystack components.
|
||||
|
||||
Alternatively, an even better practice is to set these environment variables in your shell before running the script. This approach keeps configuration separate from code and allows for easier management of different environments.
|
||||
:::
|
||||
|
||||
## Usage
|
||||
|
||||
In the example below, we are adding `DatadogConnector` to the pipeline as a _tracer_. Each pipeline run will produce a trace that includes the entire execution context, including prompts, completions, and metadata. You can then view the traces in your Datadog dashboard.
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
os.environ["HAYSTACK_CONTENT_TRACING_ENABLED"] = "true"
|
||||
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
from haystack_integrations.components.connectors.datadog import DatadogConnector
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("tracer", DatadogConnector("Chat example"))
|
||||
pipe.add_component("prompt_builder", ChatPromptBuilder())
|
||||
pipe.add_component("llm", OpenAIChatGenerator())
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
messages = [
|
||||
ChatMessage.from_system(
|
||||
"Always respond in German even if some input data is in other languages.",
|
||||
),
|
||||
ChatMessage.from_user("Tell me about {{location}}"),
|
||||
]
|
||||
|
||||
response = pipe.run(
|
||||
data={
|
||||
"prompt_builder": {
|
||||
"template_variables": {"location": "Berlin"},
|
||||
"template": messages,
|
||||
},
|
||||
},
|
||||
)
|
||||
print(response["llm"]["replies"][0])
|
||||
```
|
||||
|
||||
### With an Agent
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
os.environ["HAYSTACK_CONTENT_TRACING_ENABLED"] = "true"
|
||||
|
||||
from typing import Annotated
|
||||
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.tools import tool
|
||||
from haystack import Pipeline
|
||||
|
||||
from haystack_integrations.components.connectors.datadog import DatadogConnector
|
||||
|
||||
|
||||
@tool
|
||||
def get_weather(city: Annotated[str, "The city to get weather for"]) -> str:
|
||||
"""Get current weather information for a city."""
|
||||
weather_data = {
|
||||
"Berlin": "18°C, partly cloudy",
|
||||
"New York": "22°C, sunny",
|
||||
"Tokyo": "25°C, clear skies",
|
||||
}
|
||||
return weather_data.get(city, f"Weather information for {city} not available")
|
||||
|
||||
|
||||
@tool
|
||||
def calculate(
|
||||
operation: Annotated[
|
||||
str,
|
||||
"Mathematical operation: add, subtract, multiply, divide",
|
||||
],
|
||||
a: Annotated[float, "First number"],
|
||||
b: Annotated[float, "Second number"],
|
||||
) -> str:
|
||||
"""Perform basic mathematical calculations."""
|
||||
if operation == "add":
|
||||
result = a + b
|
||||
elif operation == "subtract":
|
||||
result = a - b
|
||||
elif operation == "multiply":
|
||||
result = a * b
|
||||
elif operation == "divide":
|
||||
if b == 0:
|
||||
return "Error: Division by zero"
|
||||
result = a / b
|
||||
else:
|
||||
return f"Error: Unknown operation '{operation}'"
|
||||
|
||||
return f"The result of {a} {operation} {b} is {result}"
|
||||
|
||||
|
||||
# Create the chat generator
|
||||
chat_generator = OpenAIChatGenerator()
|
||||
|
||||
# Create the agent with tools
|
||||
agent = Agent(
|
||||
chat_generator=chat_generator,
|
||||
tools=[get_weather, calculate],
|
||||
system_prompt="You are a helpful assistant with access to weather and calculator tools. Use them when needed.",
|
||||
exit_conditions=["text"],
|
||||
)
|
||||
|
||||
# Create the DatadogConnector for tracing
|
||||
datadog_connector = DatadogConnector("Agent Example")
|
||||
|
||||
# Build the pipeline
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("tracer", datadog_connector)
|
||||
pipe.add_component("agent", agent)
|
||||
|
||||
# Run the pipeline
|
||||
response = pipe.run(
|
||||
data={
|
||||
"agent": {
|
||||
"messages": [
|
||||
ChatMessage.from_user(
|
||||
"What's the weather in Berlin and calculate 15 + 27?",
|
||||
),
|
||||
],
|
||||
},
|
||||
"tracer": {},
|
||||
},
|
||||
)
|
||||
|
||||
# Display results
|
||||
print("Agent Response:")
|
||||
print(response["agent"]["last_message"].text)
|
||||
```
|
||||
|
||||
### Configuring the tracing backend directly
|
||||
|
||||
Instead of using the `DatadogConnector`, you can configure the Datadog tracing backend directly by enabling a `DatadogTracer`. Make sure to set the `HAYSTACK_CONTENT_TRACING_ENABLED` environment variable before importing any Haystack components.
|
||||
|
||||
```python
|
||||
import ddtrace
|
||||
|
||||
from haystack import tracing
|
||||
from haystack_integrations.tracing.datadog import DatadogTracer
|
||||
|
||||
tracing.enable_tracing(DatadogTracer(ddtrace.tracer))
|
||||
```
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
title: "External Integrations"
|
||||
id: external-integrations-connectors
|
||||
slug: "/external-integrations-connectors"
|
||||
description: "External integrations that connect your pipelines to services by external providers."
|
||||
---
|
||||
|
||||
# External Integrations
|
||||
|
||||
External integrations that connect your pipelines to services by external providers.
|
||||
|
||||
| Name | Description |
|
||||
| --- | --- |
|
||||
| [Arize AI](https://haystack.deepset.ai/integrations/arize) | Trace and evaluate your Haystack pipelines with Arize AI. |
|
||||
| [Arize Phoenix](https://haystack.deepset.ai/integrations/arize-phoenix) | Trace and evaluate your Haystack pipelines with Arize Phoenix. |
|
||||
| [Context AI](https://haystack.deepset.ai/integrations/context-ai) | Log conversations for analytics by Context.ai |
|
||||
| [Opik](https://haystack.deepset.ai/integrations/opik) | Trace and evaluate your Haystack pipelines with Opik platform. |
|
||||
| [Traceloop](https://haystack.deepset.ai/integrations/traceloop) | Evaluate and monitor the quality of your LLM apps and agents |
|
||||
@@ -0,0 +1,105 @@
|
||||
---
|
||||
title: "GitHubFileEditor"
|
||||
id: githubfileeditor
|
||||
slug: "/githubfileeditor"
|
||||
description: "This is a component for editing files in GitHub repositories through the GitHub API."
|
||||
---
|
||||
|
||||
# GitHubFileEditor
|
||||
|
||||
This is a component for editing files in GitHub repositories through the GitHub API.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a Chat Generator, or right at the beginning of a pipeline |
|
||||
| **Mandatory init variables** | `github_token`: GitHub personal access token. Can be set with `GITHUB_TOKEN` env var. |
|
||||
| **Mandatory run variables** | `command`: Operation type (edit, create, delete, undo) <br /> <br />`payload`: Command-specific parameters |
|
||||
| **Output variables** | `result`: String that indicates the operation result |
|
||||
| **API reference** | [GitHub](/reference/integrations-github) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/github |
|
||||
| **Package name** | `github-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`GitHubFileEditor` supports multiple file operations, including editing existing files, creating new files, deleting files, and undoing recent changes.
|
||||
|
||||
There are four main commands:
|
||||
|
||||
- **EDIT**: Edit an existing file by replacing specific content
|
||||
- **CREATE**: Create a new file with specified content
|
||||
- **DELETE**: Delete an existing file
|
||||
- **UNDO**: Revert the last commit if made by the same user
|
||||
|
||||
### Authorization
|
||||
|
||||
This component requires GitHub authentication with a personal access token. You can set the token using the `GITHUB_TOKEN` environment variable, or pass it directly during initialization via the `github_token` parameter.
|
||||
|
||||
To create a personal access token, visit [GitHub's token settings page](https://github.com/settings/tokens). Make sure to grant the appropriate permissions for repository access and content management.
|
||||
|
||||
### Installation
|
||||
|
||||
Install the GitHub integration with pip:
|
||||
|
||||
```shell
|
||||
pip install github-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
:::info[Repository Placeholder]
|
||||
|
||||
To run the following code snippets, you need to replace the `owner/repo` with your own GitHub repository name.
|
||||
:::
|
||||
|
||||
### On its own
|
||||
|
||||
Editing an existing file:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.connectors.github import GitHubFileEditor, Command
|
||||
|
||||
editor = GitHubFileEditor(repo="owner/repo", branch="main")
|
||||
|
||||
result = editor.run(
|
||||
command=Command.EDIT,
|
||||
payload={
|
||||
"path": "src/example.py",
|
||||
"original": "def old_function():",
|
||||
"replacement": "def new_function():",
|
||||
"message": "Renamed function for clarity",
|
||||
},
|
||||
)
|
||||
|
||||
print(result)
|
||||
```
|
||||
|
||||
```bash
|
||||
{'result': 'Edit successful'}
|
||||
```
|
||||
|
||||
Creating a new file:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.connectors.github import GitHubFileEditor, Command
|
||||
|
||||
editor = GitHubFileEditor(repo="owner/repo")
|
||||
|
||||
result = editor.run(
|
||||
command=Command.CREATE,
|
||||
payload={
|
||||
"path": "docs/new_file.md",
|
||||
"content": "# New Documentation\n\nThis is a new file.",
|
||||
"message": "Add new documentation file",
|
||||
},
|
||||
)
|
||||
|
||||
print(result)
|
||||
```
|
||||
|
||||
```bash
|
||||
{'result': 'File created successfully'}
|
||||
```
|
||||
@@ -0,0 +1,129 @@
|
||||
---
|
||||
title: "GitHubIssueCommenter"
|
||||
id: githubissuecommenter
|
||||
slug: "/githubissuecommenter"
|
||||
description: "This component posts comments to GitHub issues using the GitHub API."
|
||||
---
|
||||
|
||||
# GitHubIssueCommenter
|
||||
|
||||
This component posts comments to GitHub issues using the GitHub API.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a Chat Generator that provides the comment text to post or right at the beginning of a pipeline |
|
||||
| **Mandatory init variables** | `github_token`: GitHub personal access token. Can be set with `GITHUB_TOKEN` env var. |
|
||||
| **Mandatory run variables** | `url`: A GitHub issue URL <br /> <br />`comment`: Comment text to post |
|
||||
| **Output variables** | `success`: Boolean indicating whether the comment was posted successfully |
|
||||
| **API reference** | [GitHub](/reference/integrations-github) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/github |
|
||||
| **Package name** | `github-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`GitHubIssueCommenter` takes a GitHub issue URL and comment text, then posts the comment to the specified issue.
|
||||
|
||||
The component requires authentication with a GitHub personal access token since posting comments is an authenticated operation.
|
||||
|
||||
### Authorization
|
||||
|
||||
This component requires GitHub authentication with a personal access token. You can set the token using the `GITHUB_TOKEN` environment variable, or pass it directly during initialization via the `github_token` parameter.
|
||||
|
||||
To create a personal access token, visit [GitHub's token settings page](https://github.com/settings/tokens). Make sure to grant the appropriate permissions for repository access and issue management.
|
||||
|
||||
### Installation
|
||||
|
||||
Install the GitHub integration with pip:
|
||||
|
||||
```shell
|
||||
pip install github-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
:::info[Repository Placeholder]
|
||||
|
||||
To run the following code snippets, you need to replace the `owner/repo` with your own GitHub repository name.
|
||||
:::
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage with environment variable authentication:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.connectors.github import GitHubIssueCommenter
|
||||
|
||||
commenter = GitHubIssueCommenter()
|
||||
result = commenter.run(
|
||||
url="https://github.com/owner/repo/issues/123",
|
||||
comment="Thanks for reporting this issue! We'll look into it.",
|
||||
)
|
||||
|
||||
print(result)
|
||||
```
|
||||
|
||||
```bash
|
||||
{'success': True}
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
The following pipeline analyzes a GitHub issue and automatically posts a response:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.connectors.github import (
|
||||
GitHubIssueViewer,
|
||||
GitHubIssueCommenter,
|
||||
)
|
||||
|
||||
issue_viewer = GitHubIssueViewer()
|
||||
issue_commenter = GitHubIssueCommenter()
|
||||
|
||||
prompt_template = [
|
||||
ChatMessage.from_system(
|
||||
"You are a helpful assistant that analyzes GitHub issues and creates appropriate responses.",
|
||||
),
|
||||
ChatMessage.from_user(
|
||||
"Based on the following GitHub issue:\n"
|
||||
"{% for document in documents %}"
|
||||
"{% if document.meta.type == 'issue' %}"
|
||||
"**Issue Title:** {{ document.meta.title }}\n"
|
||||
"**Issue Description:** {{ document.content }}\n"
|
||||
"{% endif %}"
|
||||
"{% endfor %}\n"
|
||||
"Generate a helpful response comment for this issue. Keep it professional and concise.",
|
||||
),
|
||||
]
|
||||
|
||||
prompt_builder = ChatPromptBuilder(template=prompt_template, required_variables="*")
|
||||
llm = OpenAIChatGenerator(model="gpt-4o-mini")
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("issue_viewer", issue_viewer)
|
||||
pipeline.add_component("prompt_builder", prompt_builder)
|
||||
pipeline.add_component("llm", llm)
|
||||
pipeline.add_component("issue_commenter", issue_commenter)
|
||||
|
||||
pipeline.connect("issue_viewer.documents", "prompt_builder.documents")
|
||||
pipeline.connect("prompt_builder.prompt", "llm.messages")
|
||||
pipeline.connect("llm.replies", "issue_commenter.comment")
|
||||
|
||||
issue_url = "https://github.com/owner/repo/issues/123"
|
||||
result = pipeline.run(
|
||||
data={"issue_viewer": {"url": issue_url}, "issue_commenter": {"url": issue_url}},
|
||||
)
|
||||
|
||||
print(f"Comment posted successfully: {result['issue_commenter']['success']}")
|
||||
```
|
||||
|
||||
```
|
||||
Comment posted successfully: True
|
||||
```
|
||||
@@ -0,0 +1,127 @@
|
||||
---
|
||||
title: "GitHubIssueViewer"
|
||||
id: githubissueviewer
|
||||
slug: "/githubissueviewer"
|
||||
description: "This component fetches and parses GitHub issues into Haystack documents."
|
||||
---
|
||||
|
||||
# GitHubIssueViewer
|
||||
|
||||
This component fetches and parses GitHub issues into Haystack documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Right at the beginning of a pipeline and before a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) that expects the content of a GitHub issue as input |
|
||||
| **Mandatory run variables** | `url`: A GitHub issue URL |
|
||||
| **Output variables** | `documents`: A list of documents containing the main issue and its comments |
|
||||
| **API reference** | [GitHub](/reference/integrations-github) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/github |
|
||||
| **Package name** | `github-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`GitHubIssueViewer` takes a GitHub issue URL and returns a list of documents where:
|
||||
|
||||
- The first document contains the main issue content
|
||||
- Subsequent documents contain the issue comments (if any)
|
||||
|
||||
Each document includes rich metadata such as the issue title, number, state, creation date, author, and more.
|
||||
|
||||
### Authorization
|
||||
|
||||
The component can work without authentication for public repositories, but for private repositories or to avoid rate limiting, you can provide a GitHub personal access token.
|
||||
|
||||
You can set the token using the `GITHUB_API_KEY` environment variable, or pass it directly during initialization via the `github_token` parameter.
|
||||
|
||||
To create a personal access token, visit [GitHub's token settings page](https://github.com/settings/tokens).
|
||||
|
||||
### Installation
|
||||
|
||||
Install the GitHub integration with pip:
|
||||
|
||||
```shell
|
||||
pip install github-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
:::info[Repository Placeholder]
|
||||
|
||||
To run the following code snippets, you need to replace the `owner/repo` with your own GitHub repository name.
|
||||
:::
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage without authentication:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.connectors.github import GitHubIssueViewer
|
||||
|
||||
viewer = GitHubIssueViewer()
|
||||
result = viewer.run(url="https://github.com/deepset-ai/haystack/issues/123")
|
||||
|
||||
print(result)
|
||||
```
|
||||
|
||||
```bash
|
||||
{'documents': [Document(id=3989459bbd8c2a8420a9ba7f3cd3cf79bb41d78bd0738882e57d509e1293c67a, content: 'sentence-transformers = 0.2.6.1
|
||||
haystack = latest
|
||||
farm = 0.4.3 latest branch
|
||||
|
||||
In the call to Emb...', meta: {'type': 'issue', 'title': 'SentenceTransformer no longer accepts \'gpu" as argument', 'number': 123, 'state': 'closed', 'created_at': '2020-05-28T04:49:31Z', 'updated_at': '2020-05-28T07:11:43Z', 'author': 'predoctech', 'url': 'https://github.com/deepset-ai/haystack/issues/123'}), Document(id=a8a56b9ad119244678804d5873b13da0784587773d8f839e07f644c4d02c167a, content: 'Thanks for reporting!
|
||||
Fixed with #124 ', meta: {'type': 'comment', 'issue_number': 123, 'created_at': '2020-05-28T07:11:42Z', 'updated_at': '2020-05-28T07:11:42Z', 'author': 'tholor', 'url': 'https://github.com/deepset-ai/haystack/issues/123#issuecomment-635153940'})]}
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
The following pipeline fetches a GitHub issue, extracts relevant information, and generates a summary:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.connectors.github import GitHubIssueViewer
|
||||
|
||||
# Initialize components
|
||||
issue_viewer = GitHubIssueViewer()
|
||||
|
||||
prompt_template = [
|
||||
ChatMessage.from_system("You are a helpful assistant that analyzes GitHub issues."),
|
||||
ChatMessage.from_user(
|
||||
"Based on the following GitHub issue and comments:\n"
|
||||
"{% for document in documents %}"
|
||||
"{% if document.meta.type == 'issue' %}"
|
||||
"**Issue Title:** {{ document.meta.title }}\n"
|
||||
"**Issue Description:** {{ document.content }}\n"
|
||||
"{% else %}"
|
||||
"**Comment by {{ document.meta.author }}:** {{ document.content }}\n"
|
||||
"{% endif %}"
|
||||
"{% endfor %}\n"
|
||||
"Please provide a summary of the issue and suggest potential solutions.",
|
||||
),
|
||||
]
|
||||
|
||||
prompt_builder = ChatPromptBuilder(template=prompt_template, required_variables="*")
|
||||
llm = OpenAIChatGenerator(model="gpt-4o-mini")
|
||||
|
||||
# Create pipeline
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("issue_viewer", issue_viewer)
|
||||
pipeline.add_component("prompt_builder", prompt_builder)
|
||||
pipeline.add_component("llm", llm)
|
||||
|
||||
# Connect components
|
||||
pipeline.connect("issue_viewer.documents", "prompt_builder.documents")
|
||||
pipeline.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
# Run pipeline
|
||||
issue_url = "https://github.com/deepset-ai/haystack/issues/123"
|
||||
result = pipeline.run(data={"issue_viewer": {"url": issue_url}})
|
||||
|
||||
print(result["llm"]["replies"][0])
|
||||
```
|
||||
@@ -0,0 +1,79 @@
|
||||
---
|
||||
title: "GitHubPRCreator"
|
||||
id: githubprcreator
|
||||
slug: "/githubprcreator"
|
||||
description: "This component creates pull requests from a fork back to the original repository through the GitHub API."
|
||||
---
|
||||
|
||||
# GitHubPRCreator
|
||||
|
||||
This component creates pull requests from a fork back to the original repository through the GitHub API.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | At the end of a pipeline, after [GitHubRepoForker](githubrepoforker.mdx), [GitHubFileEditor](githubfileeditor.mdx) and other components that prepare changes for submission |
|
||||
| **Mandatory init variables** | `github_token`: GitHub personal access token. Can be set with `GITHUB_TOKEN` env var. |
|
||||
| **Mandatory run variables** | `issue_url`: GitHub issue URL <br /> <br />`title`: PR title <br /> <br />`branch`: Source branch <br /> <br />`base`: Target branch |
|
||||
| **Output variables** | `result`: String indicating the pull request creation result |
|
||||
| **API reference** | [GitHub](/reference/integrations-github) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/github |
|
||||
| **Package name** | `github-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`GitHubPRCreator` takes a GitHub issue URL and creates a pull request from your fork to the original repository, automatically linking it to the specified issue. It's designed to work with existing forks and assumes you have already made changes in a branch.
|
||||
|
||||
Key features:
|
||||
|
||||
- **Cross-repository PRs**: Creates pull requests from your fork to the original repository
|
||||
- **Issue linking**: Automatically links the PR to the specified GitHub issue
|
||||
- **Draft support**: Option to create draft pull requests
|
||||
- **Fork validation**: Checks that the required fork exists before creating the PR
|
||||
|
||||
As optional parameters, you can set `body` to provide a pull request description and the boolean parameter `draft` to open a draft pull request.
|
||||
|
||||
### Authorization
|
||||
|
||||
This component requires GitHub authentication with a personal access token from the fork owner. You can set the token using the `GITHUB_TOKEN` environment variable, or pass it directly during initialization via the `github_token` parameter.
|
||||
|
||||
To create a personal access token, visit [GitHub's token settings page](https://github.com/settings/tokens). Make sure to grant the appropriate permissions for repository access and pull request creation.
|
||||
|
||||
### Installation
|
||||
|
||||
Install the GitHub integration with pip:
|
||||
|
||||
```shell
|
||||
pip install github-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
:::info[Repository Placeholder]
|
||||
|
||||
To run the following code snippets, you need to replace the `owner/repo` with your own GitHub repository name.
|
||||
:::
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.connectors.github import GitHubPRCreator
|
||||
|
||||
pr_creator = GitHubPRCreator()
|
||||
result = pr_creator.run(
|
||||
issue_url="https://github.com/owner/repo/issues/123",
|
||||
title="Fix issue #123",
|
||||
body="This PR addresses issue #123 by implementing the requested changes.",
|
||||
branch="fix-123", # Branch in your fork with the changes
|
||||
base="main", # Branch in original repo to merge into
|
||||
)
|
||||
|
||||
print(result)
|
||||
```
|
||||
|
||||
```bash
|
||||
{'result': 'Pull request #456 created successfully and linked to issue #123'}
|
||||
```
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
title: "GitHubRepoForker"
|
||||
id: githubrepoforker
|
||||
slug: "/githubrepoforker"
|
||||
description: "This component forks a GitHub repository from an issue URL through the GitHub API."
|
||||
---
|
||||
|
||||
# GitHubRepoForker
|
||||
|
||||
This component forks a GitHub repository from an issue URL through the GitHub API.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Right at the beginning of a pipeline and before an [Agent](../agents-1/agent.mdx) component that expects the name of a GitHub branch as input |
|
||||
| **Mandatory init variables** | `github_token`: GitHub personal access token. Can be set with `GITHUB_TOKEN` env var. |
|
||||
| **Mandatory run variables** | `url`: The URL of a GitHub issue in the repository that should be forked |
|
||||
| **Output variables** | `repo`: Fork repository path <br /> <br />`issue_branch`: Issue-specific branch name (if created) |
|
||||
| **API reference** | [GitHub](/reference/integrations-github) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/github |
|
||||
| **Package name** | `github-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`GitHubRepoForker` takes a GitHub issue URL, extracts the repository information, creates or syncs a fork of that repository, and optionally creates an issue-specific branch. It's particularly useful for automated workflows that need to create pull requests or work with repository forks.
|
||||
|
||||
Key features:
|
||||
|
||||
- **Auto-sync**: Automatically syncs existing forks with the upstream repository
|
||||
- **Branch creation**: Creates issue-specific branches (e.g., "fix-123" for issue #123)
|
||||
- **Completion waiting**: Optionally waits for fork creation to complete
|
||||
- **Fork management**: Handles existing forks intelligently
|
||||
|
||||
### Authorization
|
||||
|
||||
This component requires GitHub authentication with a personal access token. You can set the token using the `GITHUB_TOKEN` environment variable, or pass it directly during initialization via the `github_token` parameter.
|
||||
|
||||
To create a personal access token, visit [GitHub's token settings page](https://github.com/settings/tokens). Make sure to grant the appropriate permissions for repository forking and management.
|
||||
|
||||
### Installation
|
||||
|
||||
Install the GitHub integration with pip:
|
||||
|
||||
```shell
|
||||
pip install github-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
:::info[Repository Placeholder]
|
||||
|
||||
To run the following code snippets, you need to replace the `owner/repo` with your own GitHub repository name.
|
||||
:::
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.connectors.github import GitHubRepoForker
|
||||
|
||||
forker = GitHubRepoForker()
|
||||
result = forker.run(url="https://github.com/owner/repo/issues/123")
|
||||
|
||||
print(result)
|
||||
```
|
||||
|
||||
```bash
|
||||
{'repo': 'owner/repo', 'issue_branch': 'fix-123'}
|
||||
```
|
||||
@@ -0,0 +1,92 @@
|
||||
---
|
||||
title: "GitHubRepoViewer"
|
||||
id: githubrepoviewer
|
||||
slug: "/githubrepoviewer"
|
||||
description: "This component navigates and fetches content from GitHub repositories through the GitHub API."
|
||||
---
|
||||
|
||||
# GitHubRepoViewer
|
||||
|
||||
This component navigates and fetches content from GitHub repositories through the GitHub API.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Right at the beginning of a pipeline and before a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) that expects the content of GitHub files as input |
|
||||
| **Mandatory run variables** | `path`: Repository path to view <br /> <br />`repo`: Repository in owner/repo format |
|
||||
| **Output variables** | `documents`: A list of documents containing repository contents |
|
||||
| **API reference** | [GitHub](/reference/integrations-github) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/github |
|
||||
| **Package name** | `github-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`GitHubRepoViewer` provides different behavior based on the path type:
|
||||
|
||||
- **For directories**: Returns a list of documents, one for each item (files and subdirectories),
|
||||
- **For files**: Returns a single document containing the file content.
|
||||
|
||||
Each document includes rich metadata such as the path, type, size, and URL.
|
||||
|
||||
### Authorization
|
||||
|
||||
The component can work without authentication for public repositories, but for private repositories or to avoid rate limiting, you can provide a GitHub personal access token.
|
||||
|
||||
You can set the token using the `GITHUB_TOKEN` environment variable, or pass it directly during initialization via the `github_token` parameter.
|
||||
|
||||
To create a personal access token, visit [GitHub's token settings page](https://github.com/settings/tokens).
|
||||
|
||||
### Installation
|
||||
|
||||
Install the GitHub integration with pip:
|
||||
|
||||
```shell
|
||||
pip install github-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
:::info[Repository Placeholder]
|
||||
|
||||
To run the following code snippets, you need to replace the `owner/repo` with your own GitHub repository name.
|
||||
:::
|
||||
|
||||
### On its own
|
||||
|
||||
Viewing a directory listing:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.connectors.github import GitHubRepoViewer
|
||||
|
||||
viewer = GitHubRepoViewer()
|
||||
result = viewer.run(
|
||||
repo="deepset-ai/haystack",
|
||||
path="haystack/components",
|
||||
branch="main",
|
||||
)
|
||||
|
||||
print(result)
|
||||
```
|
||||
|
||||
```bash
|
||||
{'documents': [Document(id=..., content: 'agents', meta: {'path': 'haystack/components/agents', 'type': 'dir', 'size': 0, 'url': 'https://github.com/deepset-ai/haystack/tree/main/haystack/components/agents'}), ...]}
|
||||
```
|
||||
|
||||
Viewing a specific file:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.connectors.github import GitHubRepoViewer
|
||||
|
||||
viewer = GitHubRepoViewer(repo="deepset-ai/haystack", branch="main")
|
||||
result = viewer.run(path="README.md")
|
||||
|
||||
print(result)
|
||||
```
|
||||
|
||||
```bash
|
||||
{'documents': [Document(id=..., content: '<div align="center">
|
||||
<a href="https://haystack.deepset.ai/"><img src="https://raw.githubuserconten...', meta: {'path': 'README.md', 'type': 'file_content', 'size': 11979, 'url': 'https://github.com/deepset-ai/haystack/blob/main/README.md'})]}
|
||||
```
|
||||
@@ -0,0 +1,169 @@
|
||||
---
|
||||
title: "JinaReaderConnector"
|
||||
id: jinareaderconnector
|
||||
slug: "/jinareaderconnector"
|
||||
description: "Use Jina AI’s Reader API with Haystack."
|
||||
---
|
||||
|
||||
# JinaReaderConnector
|
||||
|
||||
Use Jina AI’s Reader API with Haystack.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | As the first component in a pipeline that passes the resulting document downstream |
|
||||
| **Mandatory init variables** | `mode`: The operation mode for the reader (`read`, `search`, or `ground`) <br /> <br />`api_key`: The Jina API key. Can be set with `JINA_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `query`: A query string |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Jina](/reference/integrations-jina) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/jina |
|
||||
| **Package name** | `jina-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`JinaReaderConnector` interacts with Jina AI’s Reader API to process queries and output documents.
|
||||
|
||||
You need to select one of the following modes of operations when initializing the component:
|
||||
|
||||
- `read`: Processes a URL and extracts the textual content.
|
||||
- `search`: Searches the web and returns textual content from the most relevant pages.
|
||||
- `ground`: Performs fact-checking using a grounding engine.
|
||||
|
||||
You can find more information on these modes in the [Jina Reader documentation](https://jina.ai/reader/).
|
||||
|
||||
You can additionally control the response format from the Jina Reader API using the component’s `json_response` parameter:
|
||||
|
||||
- `True` (default) requests a JSON response for documents enriched with structured metadata.
|
||||
- `False` requests a raw response, resulting in one document with minimal metadata.
|
||||
|
||||
### Authorization
|
||||
|
||||
The component uses a `JINA_API_KEY` environment variable by default. Otherwise, you can pass a Jina API key at initialization with `api_key` like this:
|
||||
|
||||
```python
|
||||
ranker = JinaRanker(api_key=Secret.from_token("<your-api-key>"))
|
||||
```
|
||||
|
||||
To get your API key, head to Jina AI’s [website](https://jina.ai/reranker/).
|
||||
|
||||
### Installation
|
||||
|
||||
To start using this integration with Haystack, install the package with:
|
||||
|
||||
```shell
|
||||
pip install jina-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Read mode:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.connectors.jina import JinaReaderConnector
|
||||
|
||||
reader = JinaReaderConnector(mode="read")
|
||||
query = "https://example.com"
|
||||
result = reader.run(query=query)
|
||||
|
||||
print(result)
|
||||
# {'documents': [Document(id=fa3e51e4ca91828086dca4f359b6e1ea2881e358f83b41b53c84616cb0b2f7cf,
|
||||
# content: 'This domain is for use in illustrative examples in documents. You may use this domain in literature ...',
|
||||
# meta: {'title': 'Example Domain', 'description': '', 'url': 'https://example.com/', 'usage': {'tokens': 42}})]}
|
||||
```
|
||||
|
||||
Search mode:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.connectors.jina import JinaReaderConnector
|
||||
|
||||
reader = JinaReaderConnector(mode="search")
|
||||
query = "UEFA Champions League 2024"
|
||||
result = reader.run(query=query)
|
||||
|
||||
print(result)
|
||||
# {'documents': Document(id=6a71abf9955594232037321a476d39a835c0cb7bc575d886ee0087c973c95940,
|
||||
# content: '2024/25 UEFA Champions League: Matches, draw, final, key dates | UEFA Champions League | UEFA.com...',
|
||||
# meta: {'title': '2024/25 UEFA Champions League: Matches, draw, final, key dates',
|
||||
# 'description': 'What are the match dates? Where is the 2025 final? How will the competition work?',
|
||||
# 'url': 'https://www.uefa.com/uefachampionsleague/news/...',
|
||||
# 'usage': {'tokens': 5581}}), ...]}
|
||||
```
|
||||
|
||||
Ground mode:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.connectors.jina import JinaReaderConnector
|
||||
|
||||
reader = JinaReaderConnector(mode="ground")
|
||||
query = "ChatGPT was launched in 2017"
|
||||
result = reader.run(query=query)
|
||||
|
||||
print(result)
|
||||
# {'documents': [Document(id=f0c964dbc1ebb2d6584c8032b657150b9aa6e421f714cc1b9f8093a159127f0c,
|
||||
# content: 'The statement that ChatGPT was launched in 2017 is incorrect. Multiple references confirm that ChatG...',
|
||||
# meta: {'factuality': 0, 'result': False, 'references': [
|
||||
# {'url': 'https://en.wikipedia.org/wiki/ChatGPT',
|
||||
# 'keyQuote': 'ChatGPT is a generative artificial intelligence (AI) chatbot developed by OpenAI and launched in 2022.',
|
||||
# 'isSupportive': False}, ...],
|
||||
# 'usage': {'tokens': 10188}})]}
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
**Query pipeline with search mode**
|
||||
|
||||
The following pipeline example, the `JinaReaderConnector` first searches for relevant documents, then feeds them along with a user query into a prompt template, and finally generates a response based on the retrieved context.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.utils import Secret
|
||||
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack_integrations.components.connectors.jina import JinaReaderConnector
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
reader_connector = JinaReaderConnector(mode="search")
|
||||
|
||||
prompt_template = [
|
||||
ChatMessage.from_system("You are a helpful assistant."),
|
||||
ChatMessage.from_user(
|
||||
"Given the information below:\n"
|
||||
"{% for document in documents %}{{ document.content }}{% endfor %}\n"
|
||||
"Answer question: {{ query }}.\nAnswer:",
|
||||
),
|
||||
]
|
||||
|
||||
prompt_builder = ChatPromptBuilder(
|
||||
template=prompt_template,
|
||||
required_variables={"query", "documents"},
|
||||
)
|
||||
llm = OpenAIChatGenerator(
|
||||
model="gpt-4o-mini",
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
)
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("reader_connector", reader_connector)
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
|
||||
pipe.connect("reader_connector.documents", "prompt_builder.documents")
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
query = "What is the most famous landmark in Berlin?"
|
||||
|
||||
result = pipe.run(
|
||||
data={"reader_connector": {"query": query}, "prompt_builder": {"query": query}},
|
||||
)
|
||||
print(result)
|
||||
|
||||
# {'llm': {'replies': ['The most famous landmark in Berlin is the **Brandenburg Gate**. It is considered the symbol of the city and represents reunification.'], 'meta': [{'model': 'gpt-4o-mini-2024-07-18', 'index': 0, 'finish_reason': 'stop', 'usage': {'completion_tokens': 27, 'prompt_tokens': 4479, 'total_tokens': 4506, 'completion_tokens_details': CompletionTokensDetails(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0), 'prompt_tokens_details': PromptTokensDetails(audio_tokens=0, cached_tokens=0)}}]}}
|
||||
```
|
||||
|
||||
The same component in search mode could also be used in an indexing pipeline.
|
||||
@@ -0,0 +1,234 @@
|
||||
---
|
||||
title: "LangfuseConnector"
|
||||
id: langfuseconnector
|
||||
slug: "/langfuseconnector"
|
||||
description: "Learn how to work with Langfuse in Haystack."
|
||||
---
|
||||
|
||||
# LangfuseConnector
|
||||
|
||||
Learn how to work with Langfuse in Haystack.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Anywhere, as it’s not connected to other components |
|
||||
| **Mandatory init variables** | `name`: The name of the pipeline or component to identify the tracing run |
|
||||
| **Output variables** | `name`: The name of the tracing component <br /> <br />`trace_url`: A link to the tracing data |
|
||||
| **API reference** | [langfuse](/reference/integrations-langfuse) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/langfuse |
|
||||
| **Package name** | `langfuse-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`LangfuseConnector` integrates tracing capabilities into Haystack pipelines using [Langfuse](https://langfuse.com/). It captures detailed information about pipeline runs, like API calls, context data, prompts, and more. Use this component to:
|
||||
|
||||
- Monitor model performance, such as token usage and cost.
|
||||
- Find areas for pipeline improvement by identifying low-quality outputs and collecting user feedback.
|
||||
- Create datasets for fine-tuning and testing from your pipeline executions.
|
||||
|
||||
To work with the integration, add the `LangfuseConnector` to your pipeline, run the pipeline, and then view the tracing data on the Langfuse website. Don’t connect this component to any other – `LangfuseConnector` will simply run in your pipeline’s background.
|
||||
|
||||
You can optionally define two more parameters when working with this component:
|
||||
|
||||
- `httpx_client`: An optional custom `httpx.Client` instance for Langfuse API calls. Note that custom clients are discarded when deserializing a pipeline from YAML, as HTTPX clients cannot be serialized. In such cases, Langfuse creates a default client.
|
||||
- `span_handler`: An optional custom handler for processing spans. If not provided, the `DefaultSpanHandler` is used. The span handler defines how spans are created and processed, enabling customization of span types based on component types and post-processing of spans. See more details in the [Advanced Usage section](#advanced-usage) below.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
These are the things that you need before working with LangfuseConnector:
|
||||
|
||||
1. Make sure you have an active Langfuse [account](https://cloud.langfuse.com/).
|
||||
2. Set the `HAYSTACK_CONTENT_TRACING_ENABLED` environment variable to `true` – this will enable tracing in your pipelines.
|
||||
3. Set the `LANGFUSE_SECRET_KEY` and `LANGFUSE_PUBLIC_KEY` environment variables with your Langfuse secret and public keys found in your account profile.
|
||||
|
||||
### Installation
|
||||
|
||||
First, install `langfuse-haystack` package to use the `LangfuseConnector`:
|
||||
|
||||
```shell
|
||||
pip install langfuse-haystack
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
:::info[Usage Notice]
|
||||
|
||||
To ensure proper tracing, always set environment variables before importing any Haystack components. This is crucial because Haystack initializes its internal tracing components during import. In the example below, we first set the environmental variables and then import the relevant Haystack components.
|
||||
|
||||
Alternatively, an even better practice is to set these environment variables in your shell before running the script. This approach keeps configuration separate from code and allows for easier management of different environments.
|
||||
:::
|
||||
|
||||
## Usage
|
||||
|
||||
In the example below, we are adding `LangfuseConnector` to the pipeline as a _tracer_. Each pipeline run will produce one trace that includes the entire execution context, including prompts, completions, and metadata.
|
||||
|
||||
You can then view the trace by following a URL link printed in the output.
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
os.environ["LANGFUSE_HOST"] = "https://cloud.langfuse.com"
|
||||
os.environ["TOKENIZERS_PARALLELISM"] = "false"
|
||||
os.environ["HAYSTACK_CONTENT_TRACING_ENABLED"] = "true"
|
||||
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack import Pipeline
|
||||
|
||||
from haystack_integrations.components.connectors.langfuse import LangfuseConnector
|
||||
|
||||
if __name__ == "__main__":
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("tracer", LangfuseConnector("Chat example"))
|
||||
pipe.add_component("prompt_builder", ChatPromptBuilder())
|
||||
pipe.add_component("llm", OpenAIChatGenerator())
|
||||
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
messages = [
|
||||
ChatMessage.from_system(
|
||||
"Always respond in German even if some input data is in other languages.",
|
||||
),
|
||||
ChatMessage.from_user("Tell me about {{location}}"),
|
||||
]
|
||||
|
||||
response = pipe.run(
|
||||
data={
|
||||
"prompt_builder": {
|
||||
"template_variables": {"location": "Berlin"},
|
||||
"template": messages,
|
||||
},
|
||||
},
|
||||
)
|
||||
print(response["llm"]["replies"][0])
|
||||
print(response["tracer"]["trace_url"])
|
||||
```
|
||||
|
||||
### With an Agent
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
os.environ["LANGFUSE_HOST"] = "https://cloud.langfuse.com"
|
||||
os.environ["HAYSTACK_CONTENT_TRACING_ENABLED"] = "true"
|
||||
|
||||
from typing import Annotated
|
||||
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.tools import tool
|
||||
from haystack import Pipeline
|
||||
|
||||
from haystack_integrations.components.connectors.langfuse import LangfuseConnector
|
||||
|
||||
|
||||
@tool
|
||||
def get_weather(city: Annotated[str, "The city to get weather for"]) -> str:
|
||||
"""Get current weather information for a city."""
|
||||
weather_data = {
|
||||
"Berlin": "18°C, partly cloudy",
|
||||
"New York": "22°C, sunny",
|
||||
"Tokyo": "25°C, clear skies",
|
||||
}
|
||||
return weather_data.get(city, f"Weather information for {city} not available")
|
||||
|
||||
|
||||
@tool
|
||||
def calculate(
|
||||
operation: Annotated[
|
||||
str,
|
||||
"Mathematical operation: add, subtract, multiply, divide",
|
||||
],
|
||||
a: Annotated[float, "First number"],
|
||||
b: Annotated[float, "Second number"],
|
||||
) -> str:
|
||||
"""Perform basic mathematical calculations."""
|
||||
if operation == "add":
|
||||
result = a + b
|
||||
elif operation == "subtract":
|
||||
result = a - b
|
||||
elif operation == "multiply":
|
||||
result = a * b
|
||||
elif operation == "divide":
|
||||
if b == 0:
|
||||
return "Error: Division by zero"
|
||||
else:
|
||||
result = a / b
|
||||
else:
|
||||
return f"Error: Unknown operation '{operation}'"
|
||||
|
||||
return f"The result of {a} {operation} {b} is {result}"
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Create components
|
||||
chat_generator = OpenAIChatGenerator()
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=chat_generator,
|
||||
tools=[get_weather, calculate],
|
||||
system_prompt="You are a helpful assistant with access to weather and calculator tools. Use them when needed.",
|
||||
exit_conditions=["text"],
|
||||
)
|
||||
|
||||
langfuse_connector = LangfuseConnector("Agent Example")
|
||||
|
||||
# Create and run pipeline
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("tracer", langfuse_connector)
|
||||
pipe.add_component("agent", agent)
|
||||
|
||||
response = pipe.run(
|
||||
data={
|
||||
"agent": {
|
||||
"messages": [
|
||||
ChatMessage.from_user(
|
||||
"What's the weather in Berlin and calculate 15 + 27?",
|
||||
),
|
||||
],
|
||||
},
|
||||
"tracer": {"invocation_context": {"test": "agent_with_tools"}},
|
||||
},
|
||||
)
|
||||
|
||||
print(response["agent"]["last_message"].text)
|
||||
print(response["tracer"]["trace_url"])
|
||||
```
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### Customizing Langfuse Traces with SpanHandler
|
||||
|
||||
The `SpanHandler` interface in Haystack allows you to customize how spans are created and processed for Langfuse trace creation. This enables you to log custom metrics, add tags, or integrate metadata.
|
||||
|
||||
By extending `SpanHandler` or its default implementation, `DefaultSpanHandler`, you can define custom logic for span processing, providing precise control over what data is logged to Langfuse for tracking and analyzing pipeline executions.
|
||||
|
||||
Here's an example:
|
||||
|
||||
```python
|
||||
from haystack_integrations.tracing.langfuse import (
|
||||
LangfuseConnector,
|
||||
DefaultSpanHandler,
|
||||
LangfuseSpan,
|
||||
)
|
||||
from typing import Optional
|
||||
|
||||
|
||||
class CustomSpanHandler(DefaultSpanHandler):
|
||||
def handle(self, span: LangfuseSpan, component_type: Optional[str]) -> None:
|
||||
# Custom logic to add metadata or modify span
|
||||
if component_type == "OpenAIChatGenerator":
|
||||
output = span._data.get("haystack.component.output", {})
|
||||
if len(output.get("text", "")) < 10:
|
||||
span._span.update(level="WARNING", status_message="Response too short")
|
||||
|
||||
|
||||
# Add the custom handler to the LangfuseConnector
|
||||
connector = LangfuseConnector(span_handler=CustomSpanHandler())
|
||||
```
|
||||
@@ -0,0 +1,155 @@
|
||||
---
|
||||
title: "OAuthTokenResolver"
|
||||
id: oauthtokenresolver
|
||||
slug: "/oauthtokenresolver"
|
||||
description: "Resolves an OAuth access token at pipeline runtime and emits it for downstream components such as the SharePoint and Google Drive retrievers and fetchers."
|
||||
---
|
||||
|
||||
# OAuthTokenResolver
|
||||
|
||||
Resolves an OAuth access token at pipeline runtime and emits it for downstream components such as the SharePoint and Google Drive retrievers and fetchers.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | At the start of a pipeline, feeding `access_token` into downstream components such as [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx) or [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx) |
|
||||
| **Mandatory init variables** | `token_source`: The strategy that resolves the access token, for example `OAuthRefreshTokenSource` |
|
||||
| **Mandatory run variables** | None for config-only sources. `subject_token`: a controller-injected per-request credential, mandatory only when the source requires it (for example `OAuthTokenExchangeSource`) |
|
||||
| **Output variables** | `access_token`: A bearer token string |
|
||||
| **API reference** | [OAuth](/reference/integrations-oauth) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/oauth |
|
||||
| **Package name** | `oauth-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`OAuthTokenResolver` resolves an OAuth access token when the pipeline runs and emits it on the `access_token` output socket. Downstream components – such as [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx), [`MSSharePointFetcher`](../fetchers/mssharepointfetcher.mdx), [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx), and [`GoogleDriveFetcher`](../fetchers/googledrivefetcher.mdx) – consume the token through a normal connection and never need to know how it was obtained.
|
||||
|
||||
The resolver itself is a thin wrapper. The actual work of getting a token is delegated to a pluggable **token source** that decides *where* the token comes from. This separation lets you swap authentication strategies (refresh-token grant, per-request token exchange, or a static long-lived token) without changing the rest of your pipeline.
|
||||
|
||||
### Token sources
|
||||
|
||||
You pass a token source to the resolver through the `token_source` parameter. All sources are importable from `haystack_integrations.utils.oauth`.
|
||||
|
||||
| Source | Use it when | Per-request input |
|
||||
| --- | --- | --- |
|
||||
| `OAuthRefreshTokenSource` | You have a single, fixed identity backed by a stored refresh token and want the source to exchange it for short-lived access tokens and cache them. | None |
|
||||
| `OAuthTokenExchangeSource` | You serve multiple users (or run multiple replicas) and want to exchange an incoming per-request user assertion for a downstream token, with no persistent storage. Implements RFC 8693 token exchange and Microsoft's on-behalf-of flow. | `subject_token` |
|
||||
| `OAuthStaticTokenSource` | Your provider issues a non-expiring token that you manage out of band (for example Slack or Notion). | None |
|
||||
|
||||
When the configured source needs a per-request credential (`OAuthTokenExchangeSource` sets `requires_subject_token = True`), the resolver declares a **mandatory** `subject_token` run input. This is a controller-injected credential – for example an incoming user assertion – not a value chosen by an end user. For config-only sources (`OAuthRefreshTokenSource`, `OAuthStaticTokenSource`), the resolver declares no run input and acts as a source node.
|
||||
|
||||
:::info[Scopes are provider-specific]
|
||||
|
||||
The OAuth scopes you request depend on the downstream service. For Microsoft Graph, that means scopes such as `https://graph.microsoft.com/Files.Read.All`; for Google Drive, scopes such as `https://www.googleapis.com/auth/drive.readonly`. Always consult your identity provider's documentation for the exact scope values.
|
||||
|
||||
:::
|
||||
|
||||
### Installation
|
||||
|
||||
Install the OAuth integration with:
|
||||
|
||||
```shell
|
||||
pip install oauth-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Resolve a token with a stored refresh token using `OAuthRefreshTokenSource`. The refresh token is read from an environment variable through the [Secret API](../../concepts/secret-management.mdx):
|
||||
|
||||
```python
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.components.connectors.oauth import OAuthTokenResolver
|
||||
from haystack_integrations.utils.oauth import OAuthRefreshTokenSource
|
||||
|
||||
resolver = OAuthTokenResolver(
|
||||
token_source=OAuthRefreshTokenSource(
|
||||
token_url="https://login.microsoftonline.com/common/oauth2/v2.0/token",
|
||||
client_id="aaa-bbb-ccc",
|
||||
refresh_token=Secret.from_env_var("MS_REFRESH_TOKEN"),
|
||||
scopes=[
|
||||
"https://graph.microsoft.com/Files.Read.All",
|
||||
"offline_access",
|
||||
],
|
||||
),
|
||||
)
|
||||
|
||||
access_token = resolver.run()["access_token"]
|
||||
```
|
||||
|
||||
For a provider that issues long-lived, non-expiring tokens, use `OAuthStaticTokenSource` instead:
|
||||
|
||||
```python
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.components.connectors.oauth import OAuthTokenResolver
|
||||
from haystack_integrations.utils.oauth import OAuthStaticTokenSource
|
||||
|
||||
resolver = OAuthTokenResolver(
|
||||
token_source=OAuthStaticTokenSource(token=Secret.from_env_var("SERVICE_TOKEN")),
|
||||
)
|
||||
|
||||
access_token = resolver.run()["access_token"]
|
||||
```
|
||||
|
||||
For multi-user backends, use `OAuthTokenExchangeSource`. The resolver then requires a per-request `subject_token`:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.connectors.oauth import OAuthTokenResolver
|
||||
from haystack_integrations.utils.oauth import OAuthTokenExchangeSource
|
||||
|
||||
resolver = OAuthTokenResolver(
|
||||
token_source=OAuthTokenExchangeSource(
|
||||
token_url="https://login.microsoftonline.com/<tenant>/oauth2/v2.0/token",
|
||||
client_id="aaa-bbb-ccc",
|
||||
subject_token_param="assertion",
|
||||
grant_type="urn:ietf:params:oauth:grant-type:jwt-bearer",
|
||||
scopes=["https://graph.microsoft.com/Files.Read.All"],
|
||||
extra_token_params={"requested_token_use": "on_behalf_of"},
|
||||
),
|
||||
)
|
||||
|
||||
# `subject_token` is the incoming per-request user assertion, injected by your application.
|
||||
access_token = resolver.run(subject_token="<incoming-user-assertion>")["access_token"]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
In a pipeline, connect the resolver's `access_token` output to the `access_token` input of one or more downstream components. The example below wires the resolver into a [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx) so that searching SharePoint requires only a query at runtime:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.components.connectors.oauth import OAuthTokenResolver
|
||||
from haystack_integrations.utils.oauth import OAuthRefreshTokenSource
|
||||
from haystack_integrations.components.retrievers.microsoft_sharepoint import (
|
||||
MSSharePointRetriever,
|
||||
)
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component(
|
||||
"resolver",
|
||||
OAuthTokenResolver(
|
||||
token_source=OAuthRefreshTokenSource(
|
||||
token_url="https://login.microsoftonline.com/common/oauth2/v2.0/token",
|
||||
client_id="aaa-bbb-ccc",
|
||||
refresh_token=Secret.from_env_var("MS_REFRESH_TOKEN"),
|
||||
scopes=[
|
||||
"https://graph.microsoft.com/Files.Read.All",
|
||||
"https://graph.microsoft.com/Sites.Read.All",
|
||||
"offline_access",
|
||||
],
|
||||
),
|
||||
),
|
||||
)
|
||||
pipeline.add_component("retriever", MSSharePointRetriever(top_k=5))
|
||||
pipeline.connect("resolver.access_token", "retriever.access_token")
|
||||
|
||||
result = pipeline.run({"retriever": {"query": "quarterly roadmap"}})
|
||||
documents = result["retriever"]["documents"]
|
||||
```
|
||||
|
||||
A single `access_token` output can be connected to several downstream inputs. For a full retrieve-then-fetch pipeline that feeds the same token to both a retriever and a fetcher, see the [`MSSharePointFetcher`](../fetchers/mssharepointfetcher.mdx) and [`GoogleDriveFetcher`](../fetchers/googledrivefetcher.mdx) pages.
|
||||
@@ -0,0 +1,113 @@
|
||||
---
|
||||
title: "OpenAPIConnector"
|
||||
id: openapiconnector
|
||||
slug: "/openapiconnector"
|
||||
description: "`OpenAPIConnector` is a component that acts as an interface between the Haystack ecosystem and OpenAPI services."
|
||||
---
|
||||
|
||||
# OpenAPIConnector
|
||||
|
||||
`OpenAPIConnector` is a component that acts as an interface between the Haystack ecosystem and OpenAPI services.
|
||||
|
||||
:::tip[Consider using MCP instead]
|
||||
|
||||
These OpenAPI components are a legacy way to connect Haystack to external APIs. For most use cases, we recommend the [`MCPTool`](../../tools/mcptool.mdx) instead: it is the modern, standardized way to give your pipelines and agents access to external tools and services.
|
||||
|
||||
:::
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Anywhere, after components providing input for its run parameters |
|
||||
| **Mandatory init variables** | `openapi_spec`: The OpenAPI specification for the service. Can be a URL, file path, or raw string. |
|
||||
| **Mandatory run variables** | `operation_id`: The operationId from the OpenAPI spec to invoke. |
|
||||
| **Output variables** | `response`: A REST service response |
|
||||
| **API reference** | [OpenAPI](/reference/integrations-openapi) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/openapi |
|
||||
| **Package name** | `openapi-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `OpenAPIConnector` is a component within the Haystack ecosystem that allows direct invocation of REST endpoints defined in an OpenAPI (formerly Swagger) specification. It acts as a bridge between Haystack pipelines and any REST API that follows the OpenAPI standard, enabling dynamic method calls, authentication, and parameter handling.
|
||||
|
||||
To use the `OpenAPIConnector`, ensure that you have the `openapi-haystack` package installed:
|
||||
|
||||
```shell
|
||||
pip install openapi-haystack
|
||||
```
|
||||
|
||||
Unlike [OpenAPIServiceConnector](openapiserviceconnector.mdx), which works with LLMs, `OpenAPIConnector` directly calls REST endpoints using explicit input arguments.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
You can initialize and use the `OpenAPIConnector` on its own by passing an OpenAPI specification and other parameters:
|
||||
|
||||
```python
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.components.connectors.openapi import OpenAPIConnector
|
||||
|
||||
connector = OpenAPIConnector(
|
||||
openapi_spec="https://bit.ly/serperdev_openapi",
|
||||
credentials=Secret.from_env_var("SERPERDEV_API_KEY"),
|
||||
service_kwargs={"config_factory": my_custom_config_factory},
|
||||
)
|
||||
|
||||
response = connector.run(
|
||||
operation_id="search",
|
||||
arguments={"q": "Who was Nikola Tesla?"},
|
||||
)
|
||||
```
|
||||
|
||||
#### Output
|
||||
|
||||
The `OpenAPIConnector` returns a dictionary containing the service response:
|
||||
|
||||
```json
|
||||
{
|
||||
"response": { // here goes REST endpoint response JSON
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
The `OpenAPIConnector` can be integrated into a Haystack pipeline to interact with OpenAPI services. For example, here’s how you can link the `OpenAPIConnector` to a pipeline:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack_integrations.components.connectors.openapi import OpenAPIConnector
|
||||
from haystack.dataclasses.chat_message import ChatMessage
|
||||
from haystack.utils import Secret
|
||||
|
||||
# Initialize the OpenAPIConnector
|
||||
connector = OpenAPIConnector(
|
||||
openapi_spec="https://bit.ly/serperdev_openapi",
|
||||
credentials=Secret.from_env_var("SERPERDEV_API_KEY"),
|
||||
)
|
||||
|
||||
# Create a ChatMessage from the user
|
||||
user_message = ChatMessage.from_user(text="Who was Nikola Tesla?")
|
||||
|
||||
# Define the pipeline
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("openapi_connector", connector)
|
||||
|
||||
# Run the pipeline
|
||||
response = pipeline.run(
|
||||
data={
|
||||
"openapi_connector": {
|
||||
"operation_id": "search",
|
||||
"arguments": {"q": user_message.text},
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
# Extract the answer from the response
|
||||
answer = response.get("openapi_connector", {}).get("response", {})
|
||||
print(answer)
|
||||
```
|
||||
@@ -0,0 +1,150 @@
|
||||
---
|
||||
title: "OpenAPIServiceConnector"
|
||||
id: openapiserviceconnector
|
||||
slug: "/openapiserviceconnector"
|
||||
description: "`OpenAPIServiceConnector` is a component that acts as an interface between the Haystack ecosystem and OpenAPI services."
|
||||
---
|
||||
|
||||
# OpenAPIServiceConnector
|
||||
|
||||
`OpenAPIServiceConnector` is a component that acts as an interface between the Haystack ecosystem and OpenAPI services.
|
||||
|
||||
:::tip[Consider using MCP instead]
|
||||
|
||||
These OpenAPI components are a legacy way to connect Haystack to external APIs. For most use cases, we recommend the [`MCPTool`](../../tools/mcptool.mdx) instead: it is the modern, standardized way to give your pipelines and agents access to external tools and services.
|
||||
|
||||
:::
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Flexible |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects where the last message must be from the assistant and contain tool calls. <br /> <br />`service_openapi_spec`: OpenAPI specification of the service being invoked. It can be YAML/JSON, and all ref values must be resolved. <br /> <br />`service_credentials`: Authentication credentials for the service. We currently support two OpenAPI spec v3 security schemes: <br /> <br />1. http – for Basic, Bearer, and other HTTP authentication schemes; <br />2. apiKey – for API keys and cookie authentication. |
|
||||
| **Output variables** | `service_response`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects where each message corresponds to a tool call invocation. <br />If a message contains multiple tool calls, there will be multiple responses. |
|
||||
| **API reference** | [OpenAPI](/reference/integrations-openapi) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/openapi |
|
||||
| **Package name** | `openapi-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`OpenAPIServiceConnector` acts as a bridge between Haystack ecosystem and OpenAPI services. This component works by using information from a `ChatMessage` to dynamically invoke service methods. It handles parameter payload parsing from `ChatMessage`, service authentication, method invocation, and response formatting, making it easier to integrate OpenAPI services.
|
||||
|
||||
To use `OpenAPIServiceConnector`, you need to install the `openapi-haystack` package with:
|
||||
|
||||
```shell
|
||||
pip install openapi-haystack
|
||||
```
|
||||
|
||||
`OpenAPIServiceConnector` component doesn’t have any init parameters.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
This component is primarily meant to be used in pipelines, as [`OpenAPIServiceToFunctions`](../converters/openapiservicetofunctions.mdx), in tandem with an LLM with tool calling capabilities, resolves the actual tool call parameters that are injected as invocation parameters for `OpenAPIServiceConnector`.
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Let's say we're linking the Serper search engine to a pipeline. Here, `OpenAPIServiceConnector` uses the abilities of `OpenAPIServiceToFunctions`. `OpenAPIServiceToFunctions` first fetches and changes the [Serper's OpenAPI specification](https://bit.ly/serper_dev_spec) into function definitions that an LLM with tool calling capabilities can understand. Then, `OpenAPIServiceConnector` activates the Serper service using this specification.
|
||||
|
||||
More precisely, `OpenAPIServiceConnector` dynamically calls methods defined in the Serper OpenAPI specification. This involves reading chat messages to extract tool call parameters, handling authentication with the Serper service, and making the right API calls. The connector makes sure that the method call follows the Serper API requirements, such as correct formatting requests and handling responses.
|
||||
|
||||
Note that we used Serper just as an example here. This could be any OpenAPI-compliant service.
|
||||
|
||||
:::info
|
||||
To run the following code snippet, note that you have to have your own Serper and OpenAI API keys.
|
||||
:::
|
||||
|
||||
```python
|
||||
import json
|
||||
import requests
|
||||
|
||||
from typing import Any
|
||||
|
||||
from haystack import Pipeline
|
||||
from haystack.components.converters import OutputAdapter
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.dataclasses.byte_stream import ByteStream
|
||||
from haystack_integrations.components.connectors.openapi import OpenAPIServiceConnector
|
||||
from haystack_integrations.components.converters.openapi import (
|
||||
OpenAPIServiceToFunctions,
|
||||
)
|
||||
|
||||
|
||||
def prepare_fc_params(openai_functions_schema: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
"tools": [{"type": "function", "function": openai_functions_schema}],
|
||||
"tool_choice": {
|
||||
"type": "function",
|
||||
"function": {"name": openai_functions_schema["name"]},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
serperdev_spec = requests.get("https://bit.ly/serper_dev_spec").json()
|
||||
system_prompt = requests.get("https://bit.ly/serper_dev_system").text
|
||||
user_prompt = "Why was Sam Altman ousted from OpenAI?"
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("spec_to_functions", OpenAPIServiceToFunctions())
|
||||
pipe.add_component(
|
||||
"prepare_fc_adapter",
|
||||
OutputAdapter(
|
||||
"{{functions[0] | prepare_fc}}",
|
||||
dict[str, Any],
|
||||
{"prepare_fc": prepare_fc_params},
|
||||
),
|
||||
)
|
||||
pipe.add_component("functions_llm", OpenAIChatGenerator())
|
||||
pipe.add_component("openapi_connector", OpenAPIServiceConnector())
|
||||
pipe.add_component(
|
||||
"message_adapter",
|
||||
OutputAdapter(
|
||||
"{{system_message + service_response}}",
|
||||
list[ChatMessage],
|
||||
unsafe=True,
|
||||
),
|
||||
)
|
||||
pipe.add_component("llm", OpenAIChatGenerator())
|
||||
|
||||
pipe.connect("spec_to_functions.functions", "prepare_fc_adapter.functions")
|
||||
pipe.connect(
|
||||
"spec_to_functions.openapi_specs",
|
||||
"openapi_connector.service_openapi_spec",
|
||||
)
|
||||
pipe.connect("prepare_fc_adapter", "functions_llm.generation_kwargs")
|
||||
pipe.connect("functions_llm.replies", "openapi_connector.messages")
|
||||
pipe.connect("openapi_connector.service_response", "message_adapter.service_response")
|
||||
pipe.connect("message_adapter", "llm.messages")
|
||||
|
||||
result = pipe.run(
|
||||
data={
|
||||
"functions_llm": {
|
||||
"messages": [
|
||||
ChatMessage.from_system("Only do tool/function calling"),
|
||||
ChatMessage.from_user(user_prompt),
|
||||
],
|
||||
},
|
||||
"openapi_connector": {
|
||||
"service_credentials": serper_dev_key,
|
||||
},
|
||||
"spec_to_functions": {
|
||||
"sources": [ByteStream.from_string(json.dumps(serperdev_spec))],
|
||||
},
|
||||
"message_adapter": {
|
||||
"system_message": [ChatMessage.from_system(system_prompt)],
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
print(result["llm"]["replies"][0].text)
|
||||
|
||||
# Sam Altman was ousted from OpenAI on November 17, 2023, following
|
||||
# a "deliberative review process" by the board of directors. The board concluded
|
||||
# that he was not "consistently candid in his communications". However, he
|
||||
# returned as CEO just days after his ouster.
|
||||
```
|
||||
@@ -0,0 +1,126 @@
|
||||
---
|
||||
title: "OpenTelemetryConnector"
|
||||
id: opentelemetryconnector
|
||||
slug: "/opentelemetryconnector"
|
||||
description: "Learn how to work with OpenTelemetry in Haystack."
|
||||
---
|
||||
|
||||
# OpenTelemetryConnector
|
||||
|
||||
Learn how to work with OpenTelemetry in Haystack.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Anywhere, as it’s not connected to other components |
|
||||
| **Mandatory init variables** | None. The tracer is created at initialization time |
|
||||
| **Output variables** | `name`: The name of the tracing component |
|
||||
| **API reference** | [opentelemetry](/reference/integrations-opentelemetry) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opentelemetry |
|
||||
| **Package name** | `opentelemetry-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`OpenTelemetryConnector` integrates tracing capabilities into Haystack pipelines using [OpenTelemetry](https://opentelemetry.io/), through the [OpenTelemetry SDK](https://opentelemetry.io/docs/languages/python/). It captures detailed information about pipeline runs, like API calls, context data, prompts, and more, so you can see the complete trace of your pipeline execution in any OpenTelemetry-compatible backend.
|
||||
|
||||
OpenTelemetry tracing is enabled as soon as the `OpenTelemetryConnector` is initialized, so you only need to add it to your pipeline – it does not need to be connected to other components or to run to take effect.
|
||||
|
||||
You can optionally pass a `name` to identify this tracing component (it defaults to `opentelemetry`).
|
||||
|
||||
### Prerequisites
|
||||
|
||||
These are the things that you need before working with the `OpenTelemetryConnector`:
|
||||
|
||||
1. A configured OpenTelemetry `TracerProvider` with an exporter (for example, an OTLP exporter that sends traces to a collector or a backend). Set up the provider before initializing the connector.
|
||||
2. Set the `HAYSTACK_CONTENT_TRACING_ENABLED` environment variable to `true` – this will enable content tracing (inputs and outputs) in your pipelines.
|
||||
3. To add traces at even deeper levels, check out the available [OpenTelemetry instrumentations](https://opentelemetry.io/ecosystem/registry/?s=python), such as `opentelemetry-instrumentation-openai-v2` for tracing OpenAI requests.
|
||||
|
||||
### Installation
|
||||
|
||||
First, install the `opentelemetry-haystack` package to use the `OpenTelemetryConnector`:
|
||||
|
||||
```shell
|
||||
pip install opentelemetry-haystack
|
||||
```
|
||||
|
||||
<br />
|
||||
|
||||
:::info[Usage Notice]
|
||||
|
||||
To ensure proper tracing, always set environment variables before importing any Haystack components. This is crucial because Haystack initializes its internal tracing components during import. In the example below, we first set the environment variables and then import the relevant Haystack components.
|
||||
|
||||
Alternatively, an even better practice is to set these environment variables in your shell before running the script. This approach keeps configuration separate from code and allows for easier management of different environments.
|
||||
:::
|
||||
|
||||
## Usage
|
||||
|
||||
In the example below, we are adding `OpenTelemetryConnector` to the pipeline as a _tracer_. Each pipeline run will produce a trace that includes the entire execution context, including prompts, completions, and metadata. You can then view the traces in your OpenTelemetry backend.
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
os.environ["HAYSTACK_CONTENT_TRACING_ENABLED"] = "true"
|
||||
|
||||
from opentelemetry import trace
|
||||
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
|
||||
from opentelemetry.sdk.resources import Resource
|
||||
from opentelemetry.sdk.trace import TracerProvider
|
||||
from opentelemetry.sdk.trace.export import BatchSpanProcessor
|
||||
from opentelemetry.semconv.resource import ResourceAttributes
|
||||
|
||||
# Configure the OpenTelemetry SDK. A service name is required for most backends.
|
||||
resource = Resource(attributes={ResourceAttributes.SERVICE_NAME: "haystack"})
|
||||
tracer_provider = TracerProvider(resource=resource)
|
||||
tracer_provider.add_span_processor(
|
||||
BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")),
|
||||
)
|
||||
trace.set_tracer_provider(tracer_provider)
|
||||
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
from haystack_integrations.components.connectors.opentelemetry import (
|
||||
OpenTelemetryConnector,
|
||||
)
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("tracer", OpenTelemetryConnector("Chat example"))
|
||||
pipe.add_component("prompt_builder", ChatPromptBuilder())
|
||||
pipe.add_component("llm", OpenAIChatGenerator())
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
messages = [
|
||||
ChatMessage.from_system(
|
||||
"Always respond in German even if some input data is in other languages.",
|
||||
),
|
||||
ChatMessage.from_user("Tell me about {{location}}"),
|
||||
]
|
||||
|
||||
response = pipe.run(
|
||||
data={
|
||||
"prompt_builder": {
|
||||
"template_variables": {"location": "Berlin"},
|
||||
"template": messages,
|
||||
},
|
||||
},
|
||||
)
|
||||
print(response["llm"]["replies"][0])
|
||||
```
|
||||
|
||||
### Configuring the tracing backend directly
|
||||
|
||||
Instead of using the `OpenTelemetryConnector`, you can configure the OpenTelemetry tracing backend directly by enabling an `OpenTelemetryTracer`. Make sure to set the `HAYSTACK_CONTENT_TRACING_ENABLED` environment variable and configure your `TracerProvider` before importing any Haystack components.
|
||||
|
||||
```python
|
||||
from opentelemetry import trace
|
||||
|
||||
from haystack import tracing
|
||||
from haystack_integrations.tracing.opentelemetry import OpenTelemetryTracer
|
||||
|
||||
tracing.enable_tracing(OpenTelemetryTracer(trace.get_tracer("my_application")))
|
||||
```
|
||||
@@ -0,0 +1,184 @@
|
||||
---
|
||||
title: "WeaveConnector"
|
||||
id: weaveconnector
|
||||
slug: "/weaveconnector"
|
||||
description: "Learn how to use Weights & Biases Weave framework for tracing and monitoring your pipeline components."
|
||||
---
|
||||
|
||||
# WeaveConnector
|
||||
|
||||
Learn how to use Weights & Biases Weave framework for tracing and monitoring your pipeline components.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Anywhere, as it’s not connected to other components |
|
||||
| **Mandatory init variables** | `pipeline_name`: The name of your pipeline, which will also show up in Weaver dashboard. |
|
||||
| **Output variables** | `pipeline_name`: The name of the pipeline that just run |
|
||||
| **API reference** | [Weave](/reference/integrations-weave) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/weave |
|
||||
| **Package name** | `weave-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
This integration allows you to trace and visualize your pipeline execution in [Weights & Biases](https://wandb.ai/site/).
|
||||
|
||||
Information captured by the Haystack tracing tool, such as API calls, context data, and prompts, is sent to Weights & Biases, where you can see the complete trace of your pipeline execution.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
You need a Weave account to use this feature. You can sign up for free at [Weights & Biases website](https://wandb.ai/site).
|
||||
|
||||
You will then need to set the `WANDB_API_KEY` environment variable with your Weights & Biases API key. Once logged in, you can find your API key on [your home page](https://wandb.ai/home).
|
||||
|
||||
Then go to `https://wandb.ai/<user_name>/projects` and see the full trace for your pipeline under the pipeline name you specified when creating the `WeaveConnector`.
|
||||
|
||||
You will also need to set the `HAYSTACK_CONTENT_TRACING_ENABLED` environment variable set to `true`.
|
||||
|
||||
## Usage
|
||||
|
||||
First, install the `weights_biases-haystack` package to use this connector:
|
||||
|
||||
```shell
|
||||
pip install weights_biases-haystack
|
||||
```
|
||||
|
||||
Then, add it to your pipeline without any connections, and it will automatically start sending traces to Weights & Biases:
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
from haystack_integrations.components.connectors.weave import WeaveConnector
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", ChatPromptBuilder())
|
||||
pipe.add_component("llm", OpenAIChatGenerator())
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
connector = WeaveConnector(pipeline_name="test_pipeline")
|
||||
pipe.add_component("weave", connector)
|
||||
|
||||
messages = [
|
||||
ChatMessage.from_system(
|
||||
"Always respond in German even if some input data is in other languages.",
|
||||
),
|
||||
ChatMessage.from_user("Tell me about {{location}}"),
|
||||
]
|
||||
|
||||
response = pipe.run(
|
||||
data={
|
||||
"prompt_builder": {
|
||||
"template_variables": {"location": "Berlin"},
|
||||
"template": messages,
|
||||
},
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
You can then see the complete trace for your pipeline at `https://wandb.ai/<user_name>/projects` under the pipeline name you specified when creating the `WeaveConnector`.
|
||||
|
||||
### With an Agent
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
# Enable Haystack content tracing
|
||||
os.environ["HAYSTACK_CONTENT_TRACING_ENABLED"] = "true"
|
||||
|
||||
from typing import Annotated
|
||||
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.tools import tool
|
||||
from haystack import Pipeline
|
||||
|
||||
from haystack_integrations.components.connectors.weave import WeaveConnector
|
||||
|
||||
|
||||
@tool
|
||||
def get_weather(city: Annotated[str, "The city to get weather for"]) -> str:
|
||||
"""Get current weather information for a city."""
|
||||
weather_data = {
|
||||
"Berlin": "18°C, partly cloudy",
|
||||
"New York": "22°C, sunny",
|
||||
"Tokyo": "25°C, clear skies",
|
||||
}
|
||||
return weather_data.get(city, f"Weather information for {city} not available")
|
||||
|
||||
|
||||
@tool
|
||||
def calculate(
|
||||
operation: Annotated[
|
||||
str,
|
||||
"Mathematical operation: add, subtract, multiply, divide",
|
||||
],
|
||||
a: Annotated[float, "First number"],
|
||||
b: Annotated[float, "Second number"],
|
||||
) -> str:
|
||||
"""Perform basic mathematical calculations."""
|
||||
if operation == "add":
|
||||
result = a + b
|
||||
elif operation == "subtract":
|
||||
result = a - b
|
||||
elif operation == "multiply":
|
||||
result = a * b
|
||||
elif operation == "divide":
|
||||
if b == 0:
|
||||
return "Error: Division by zero"
|
||||
result = a / b
|
||||
else:
|
||||
return f"Error: Unknown operation '{operation}'"
|
||||
|
||||
return f"The result of {a} {operation} {b} is {result}"
|
||||
|
||||
|
||||
# Create the chat generator
|
||||
chat_generator = OpenAIChatGenerator()
|
||||
|
||||
# Create the agent with tools
|
||||
agent = Agent(
|
||||
chat_generator=chat_generator,
|
||||
tools=[get_weather, calculate],
|
||||
system_prompt="You are a helpful assistant with access to weather and calculator tools. Use them when needed.",
|
||||
exit_conditions=["text"],
|
||||
)
|
||||
|
||||
# Create the WeaveConnector for tracing
|
||||
weave_connector = WeaveConnector(pipeline_name="Agent Example")
|
||||
|
||||
# Build the pipeline
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("tracer", weave_connector)
|
||||
pipe.add_component("agent", agent)
|
||||
|
||||
# Run the pipeline
|
||||
response = pipe.run(
|
||||
data={
|
||||
"agent": {
|
||||
"messages": [
|
||||
ChatMessage.from_user(
|
||||
"What's the weather in Berlin and calculate 15 + 27?",
|
||||
),
|
||||
],
|
||||
},
|
||||
"tracer": {},
|
||||
},
|
||||
)
|
||||
|
||||
# Display results
|
||||
print("Agent Response:")
|
||||
print(response["agent"]["last_message"].text)
|
||||
print(f"\nPipeline Name: {response['tracer']['pipeline_name']}")
|
||||
print(
|
||||
"\nCheck your Weights & Biases dashboard at https://wandb.ai/<user_name>/projects to see the traces!",
|
||||
)
|
||||
```
|
||||
@@ -0,0 +1,43 @@
|
||||
---
|
||||
title: "Converters"
|
||||
id: converters
|
||||
slug: "/converters"
|
||||
description: "Use various Converters to extract data from files in different formats and cast it into the unified document format. There are several converters available for converting PDFs, images, DOCX files, and more."
|
||||
---
|
||||
|
||||
# Converters
|
||||
|
||||
Use various Converters to extract data from files in different formats and cast it into the unified document format. There are several converters available for converting PDFs, images, DOCX files, and more.
|
||||
|
||||
| Converter | Description |
|
||||
| --- | --- |
|
||||
| [AmazonTextractConverter](converters/amazontextractconverter.mdx) | Converts images and single-page PDFs to documents using AWS Textract, with optional structured analysis of tables, forms, signatures, and layout, plus natural-language queries. |
|
||||
| [AzureDocumentIntelligenceConverter](converters/azuredocumentintelligenceconverter.mdx) | Converts PDF, JPEG, PNG, BMP, TIFF, DOCX, XLSX, PPTX, and HTML to documents using Azure's Document Intelligence service with GitHub Flavored Markdown output. |
|
||||
| [AzureOCRDocumentConverter](converters/azureocrdocumentconverter.mdx) | Converts PDF (both searchable and image-only), JPEG, PNG, BMP, TIFF, DOCX, XLSX, PPTX, and HTML to documents. |
|
||||
| [CSVToDocument](converters/csvtodocument.mdx) | Converts CSV files to documents. |
|
||||
| [DoclingConverter](converters/doclingconverter.mdx) | Converts PDF, DOCX, HTML, and other document formats to documents with layout-aware chunking, Markdown, and JSON export. |
|
||||
| [DoclingServeConverter](converters/doclingserveconverter.mdx) | Converts PDF, DOCX, HTML, and other document formats to documents using a remote DoclingServe HTTP server, with no local ML dependencies. |
|
||||
| [DocumentToImageContent](converters/documenttoimagecontent.mdx) | Extracts visual data from image or PDF file-based documents and converts them into `ImageContent` objects. |
|
||||
| [DOCXToDocument](converters/docxtodocument.mdx) | Convert DOCX files to documents. |
|
||||
| [FileToFileContent](converters/filetofilecontent.mdx) | Reads files and converts them into `FileContent` objects. |
|
||||
| [HTMLToDocument](converters/htmltodocument.mdx) | Converts HTML files to documents. |
|
||||
| [ImageFileToDocument](converters/imagefiletodocument.mdx) | Converts image file references into empty `Document` objects with associated metadata. |
|
||||
| [ImageFileToImageContent](converters/imagefiletoimagecontent.mdx) | Reads local image files and converts them into `ImageContent` objects. |
|
||||
| [JSONConverter](converters/jsonconverter.mdx) | Converts JSON files to text documents. |
|
||||
| [KreuzbergConverter](converters/kreuzbergconverter.mdx) | Converts 91+ file formats to documents locally using Kreuzberg's Rust-core engine. |
|
||||
| [MarkdownToDocument](converters/markdowntodocument.mdx) | Converts markdown files to documents. |
|
||||
| [MistralOCRDocumentConverter](converters/mistralocrdocumentconverter.mdx) | Extracts text from documents using Mistral's OCR API, with optional structured annotations. |
|
||||
| [MSGToDocument](converters/msgtodocument.mdx) | Converts Microsoft Outlook .msg files to documents. |
|
||||
| [MultiFileConverter](converters/multifileconverter.mdx) | Converts CSV, DOCX, HTML, JSON, MD, PPTX, PDF, TXT, and XSLX files to documents. |
|
||||
| [OpenAPIServiceToFunctions](converters/openapiservicetofunctions.mdx) | Transforms OpenAPI service specifications into a format compatible with OpenAI's function calling mechanism. |
|
||||
| [OutputAdapter](converters/outputadapter.mdx) | Helps the output of one component fit into the input of another. |
|
||||
| [PaddleOCRVLDocumentConverter](converters/paddleocrvldocumentconverter.mdx) | Extracts text from documents using PaddleOCR's large model document parsing API. |
|
||||
| [PDFMinerToDocument](converters/pdfminertodocument.mdx) | Converts complex PDF files to documents using pdfminer arguments. |
|
||||
| [PDFToImageContent](converters/pdftoimagecontent.mdx) | Reads local PDF files and converts them into `ImageContent` objects. |
|
||||
| [PPTXToDocument](converters/pptxtodocument.mdx) | Converts PPTX files to documents. |
|
||||
| [PyPDFToDocument](converters/pypdftodocument.mdx) | Converts PDF files to documents. |
|
||||
| [TikaDocumentConverter](converters/tikadocumentconverter.mdx) | Converts various file types to documents using Apache Tika. |
|
||||
| [TextFileToDocument](converters/textfiletodocument.mdx) | Converts text files to documents. |
|
||||
| [TwelveLabsVideoConverter](converters/twelvelabsvideoconverter.mdx) | Converts videos to documents using the TwelveLabs Pegasus video-language model. |
|
||||
| [UnstructuredFileConverter](converters/unstructuredfileconverter.mdx) | Converts text files and directories to a document. |
|
||||
| [XLSXToDocument](converters/xlsxtodocument.mdx) | Converts Excel files into documents. |
|
||||
@@ -0,0 +1,142 @@
|
||||
---
|
||||
title: "AmazonTextractConverter"
|
||||
id: amazontextractconverter
|
||||
slug: "/amazontextractconverter"
|
||||
description: "`AmazonTextractConverter` converts images and single-page PDFs to documents using AWS Textract. It supports plain text OCR, structured analysis of tables, forms, signatures, and layout, as well as natural-language queries over the document."
|
||||
---
|
||||
|
||||
# AmazonTextractConverter
|
||||
|
||||
`AmazonTextractConverter` converts images and single-page PDFs to documents using AWS Textract. It supports plain text OCR, structured analysis of tables, forms, signatures, and layout, as well as natural-language queries over the document.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [PreProcessors](../preprocessors.mdx), or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory init variables** | AWS credentials are resolved via `Secret` parameters or the default boto3 credential chain (environment variables, AWS config files, IAM roles). |
|
||||
| **Mandatory run variables** | `sources`: A list of file paths or `ByteStream` objects |
|
||||
| **Output variables** | `documents`: A list of documents <br /> <br />`raw_textract_response`: A list of raw responses from the Textract API |
|
||||
| **API reference** | [Amazon Textract](/reference/integrations-amazon_textract) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/amazon_textract |
|
||||
| **Package name** | `amazon-textract-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`AmazonTextractConverter` takes a list of file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects as input and uses AWS Textract to extract text from images and single-page PDFs. Optionally, metadata can be attached to the documents through the `meta` input parameter. You need an active AWS account with access to the Textract service to use this integration. Refer to the [AWS Textract documentation](https://docs.aws.amazon.com/textract/latest/dg/getting-started.html) to set up your AWS credentials and ensure Textract is available in your selected region.
|
||||
|
||||
Supported input formats: JPEG, PNG, TIFF, BMP, and single-page PDF (up to 10 MB).
|
||||
|
||||
By default, the component uses the standard AWS environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`, `AWS_DEFAULT_REGION`, `AWS_PROFILE`) for authentication. You can also pass these as `Secret` objects at initialization. The component falls back to the default boto3 credential chain if no explicit credentials are provided, which makes it work with IAM roles when running on AWS infrastructure.
|
||||
|
||||
### Operation modes
|
||||
|
||||
The component switches between two Textract APIs depending on how you configure it:
|
||||
|
||||
- **Plain text OCR (`DetectDocumentText`)** – Used when `feature_types` is not set. This is the fastest and cheapest option, extracting raw text from the document.
|
||||
- **Structured analysis (`AnalyzeDocument`)** – Used when `feature_types` is set. You can pass any combination of `"TABLES"`, `"FORMS"`, `"SIGNATURES"`, and `"LAYOUT"` to extract richer structural information from the document.
|
||||
|
||||
### Natural-language queries
|
||||
|
||||
You can pass a list of natural-language questions through the `queries` parameter on `run()`. When queries are provided, the `QUERIES` feature type is added automatically and Textract returns the extracted answers in the raw response. This is useful for pulling specific fields out of forms, invoices, or receipts without writing custom parsing logic.
|
||||
|
||||
## Usage
|
||||
|
||||
You need to install the `amazon-textract-haystack` integration to use `AmazonTextractConverter`:
|
||||
|
||||
```shell
|
||||
pip install amazon-textract-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage with plain text OCR:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.converters.amazon_textract import (
|
||||
AmazonTextractConverter,
|
||||
)
|
||||
|
||||
converter = AmazonTextractConverter()
|
||||
result = converter.run(sources=["document.png"])
|
||||
documents = result["documents"]
|
||||
```
|
||||
|
||||
Extracting tables and forms with `AnalyzeDocument`:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.converters.amazon_textract import (
|
||||
AmazonTextractConverter,
|
||||
)
|
||||
|
||||
converter = AmazonTextractConverter(feature_types=["TABLES", "FORMS"])
|
||||
result = converter.run(sources=["invoice.pdf"])
|
||||
documents = result["documents"]
|
||||
raw_responses = result["raw_textract_response"]
|
||||
```
|
||||
|
||||
Using natural-language queries to extract specific fields:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.converters.amazon_textract import (
|
||||
AmazonTextractConverter,
|
||||
)
|
||||
|
||||
converter = AmazonTextractConverter()
|
||||
result = converter.run(
|
||||
sources=["receipt.png"],
|
||||
queries=["What is the patient name?", "What is the total due?"],
|
||||
)
|
||||
documents = result["documents"]
|
||||
raw_responses = result["raw_textract_response"]
|
||||
```
|
||||
|
||||
Passing AWS credentials explicitly:
|
||||
|
||||
```python
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.components.converters.amazon_textract import (
|
||||
AmazonTextractConverter,
|
||||
)
|
||||
|
||||
converter = AmazonTextractConverter(
|
||||
aws_access_key_id=Secret.from_env_var("AWS_ACCESS_KEY_ID"),
|
||||
aws_secret_access_key=Secret.from_env_var("AWS_SECRET_ACCESS_KEY"),
|
||||
aws_region_name=Secret.from_token("us-east-1"),
|
||||
)
|
||||
result = converter.run(sources=["document.png"])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Here's an example of an indexing pipeline that uses Textract to extract text from images and writes the resulting documents to a Document Store:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.preprocessors import DocumentCleaner, DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack_integrations.components.converters.amazon_textract import (
|
||||
AmazonTextractConverter,
|
||||
)
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("converter", AmazonTextractConverter())
|
||||
pipeline.add_component("cleaner", DocumentCleaner())
|
||||
pipeline.add_component(
|
||||
"splitter",
|
||||
DocumentSplitter(split_by="sentence", split_length=5),
|
||||
)
|
||||
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
|
||||
pipeline.connect("converter", "cleaner")
|
||||
pipeline.connect("cleaner", "splitter")
|
||||
pipeline.connect("splitter", "writer")
|
||||
|
||||
file_names = ["document.png", "invoice.pdf"]
|
||||
pipeline.run({"converter": {"sources": file_names}})
|
||||
```
|
||||
+109
@@ -0,0 +1,109 @@
|
||||
---
|
||||
title: "AzureDocumentIntelligenceConverter"
|
||||
id: azuredocumentintelligenceconverter
|
||||
slug: "/azuredocumentintelligenceconverter"
|
||||
description: "`AzureDocumentIntelligenceConverter` converts files to Documents using Azure's Document Intelligence service with GitHub Flavored Markdown output for better LLM/RAG integration. It supports PDF, JPEG, PNG, BMP, TIFF, DOCX, XLSX, PPTX, and HTML."
|
||||
---
|
||||
|
||||
# AzureDocumentIntelligenceConverter
|
||||
|
||||
`AzureDocumentIntelligenceConverter` converts files to Documents using Azure's Document Intelligence service with GitHub Flavored Markdown output for better LLM/RAG integration. It supports the following file formats: PDF (both searchable and image-only), JPEG, PNG, BMP, TIFF, DOCX, XLSX, PPTX, and HTML.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [PreProcessors](../preprocessors.mdx), or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory init variables** | `endpoint`: The endpoint URL of your Azure Document Intelligence resource <br /> <br />`api_key`: The API key for Azure authentication. Can be set with `AZURE_DI_API_KEY` environment variable. |
|
||||
| **Mandatory run variables** | `sources`: A list of file paths or ByteStream objects |
|
||||
| **Output variables** | `documents`: A list of documents <br /> <br />`raw_azure_response`: A list of raw responses from Azure |
|
||||
| **API reference** | [Azure Document Intelligence](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/azure_doc_intelligence) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/azure_doc_intelligence |
|
||||
| **Package name** | `azure-doc-intelligence-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`AzureDocumentIntelligenceConverter` takes a list of file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects as input and uses Azure's Document Intelligence service to convert the files to a list of documents. Optionally, metadata can be attached to the documents through the `meta` input parameter. You need an active Azure account and a Document Intelligence or Cognitive Services resource to use this integration. Follow the steps described in the Azure [documentation](https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/quickstarts/get-started-sdks-rest-api) to set up your resource.
|
||||
|
||||
The component uses an `AZURE_DI_API_KEY` environment variable by default. Otherwise, you can pass an `api_key` at initialization — see code examples below.
|
||||
|
||||
This component uses the `azure-ai-documentintelligence` package (v1.0.0+) and outputs GitHub Flavored Markdown, preserving document structure such as headings, tables, and lists. Tables are rendered as inline markdown tables rather than being extracted as separate documents.
|
||||
|
||||
When you initialize the component, you can optionally set the `model_id`, which refers to the model you want to use. Available options include:
|
||||
- `"prebuilt-document"`: General document analysis (default)
|
||||
- `"prebuilt-read"`: Fast OCR for text extraction
|
||||
- `"prebuilt-layout"`: Enhanced layout analysis with better table and structure detection
|
||||
- Custom model IDs from your Azure resource
|
||||
|
||||
Refer to the [Azure documentation](https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/choose-model-feature) for a full list of available models.
|
||||
|
||||
:::info
|
||||
This component replaces the legacy [`AzureOCRDocumentConverter`](azureocrdocumentconverter.mdx), which uses the older `azure-ai-formrecognizer` package. The `AzureDocumentIntelligenceConverter` uses the newer `azure-ai-documentintelligence` SDK and produces Markdown output instead of plain text, making it better suited for LLM and RAG applications.
|
||||
:::
|
||||
|
||||
:::note
|
||||
This component returns Markdown content. Avoid piping it through `DocumentCleaner()` with its default settings because `remove_extra_whitespaces=True` and `remove_empty_lines=True` can collapse line breaks and flatten headings, tables, and lists. Connect the converter directly to your next component, or disable those options if you need custom cleanup.
|
||||
:::
|
||||
|
||||
## Usage
|
||||
|
||||
You need to install the `azure-doc-intelligence-haystack` integration to use the `AzureDocumentIntelligenceConverter`:
|
||||
|
||||
```shell
|
||||
pip install azure-doc-intelligence-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from haystack_integrations.components.converters.azure_doc_intelligence import (
|
||||
AzureDocumentIntelligenceConverter,
|
||||
)
|
||||
from haystack.utils import Secret
|
||||
|
||||
converter = AzureDocumentIntelligenceConverter(
|
||||
endpoint="https://YOUR_RESOURCE.cognitiveservices.azure.com/",
|
||||
api_key=Secret.from_env_var("AZURE_DI_API_KEY"),
|
||||
)
|
||||
|
||||
result = converter.run(sources=[Path("my_file.pdf")])
|
||||
documents = result["documents"]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.components.converters.azure_doc_intelligence import (
|
||||
AzureDocumentIntelligenceConverter,
|
||||
)
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component(
|
||||
"converter",
|
||||
AzureDocumentIntelligenceConverter(
|
||||
endpoint="https://YOUR_RESOURCE.cognitiveservices.azure.com/",
|
||||
api_key=Secret.from_env_var("AZURE_DI_API_KEY"),
|
||||
),
|
||||
)
|
||||
pipeline.add_component(
|
||||
"splitter",
|
||||
DocumentSplitter(split_by="sentence", split_length=5),
|
||||
)
|
||||
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
pipeline.connect("converter", "splitter")
|
||||
pipeline.connect("splitter", "writer")
|
||||
|
||||
file_names = ["my_file.pdf"]
|
||||
pipeline.run({"converter": {"sources": file_names}})
|
||||
```
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
title: "AzureOCRDocumentConverter"
|
||||
id: azureocrdocumentconverter
|
||||
slug: "/azureocrdocumentconverter"
|
||||
description: "`AzureOCRDocumentConverter` converts files to documents using Azure's Document Intelligence service. It supports the following file formats: PDF (both searchable and image-only), JPEG, PNG, BMP, TIFF, DOCX, XLSX, PPTX, and HTML."
|
||||
---
|
||||
|
||||
# AzureOCRDocumentConverter
|
||||
|
||||
`AzureOCRDocumentConverter` converts files to documents using Azure's Document Intelligence service. It supports the following file formats: PDF (both searchable and image-only), JPEG, PNG, BMP, TIFF, DOCX, XLSX, PPTX, and HTML.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [PreProcessors](../preprocessors.mdx) , or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory init variables** | `endpoint`: The endpoint of your Azure resource <br /> <br />`api_key`: The API key of your Azure resource. Can be set with `AZURE_AI_API_KEY` environment variable. |
|
||||
| **Mandatory run variables** | `sources`: A list of file paths |
|
||||
| **Output variables** | `documents`: A list of documents <br /> <br />`raw_azure_response`: A list of raw responses from Azure |
|
||||
| **API reference** | [Azure Form Recognizer](/reference/integrations-azure_form_recognizer) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/azure_form_recognizer |
|
||||
| **Package name** | `azure-form-recognizer-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`AzureOCRDocumentConverter` takes a list of file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects as input and uses Azure services to convert the files to a list of documents. Optionally, metadata can be attached to the documents through the `meta` input parameter. You need an active Azure account and a Document Intelligence or Cognitive Services resource to use this integration. Follow the steps described in the Azure [documentation](https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/quickstarts/get-started-sdks-rest-api) to set up your resource.
|
||||
|
||||
The component uses an `AZURE_AI_API_KEY` environment variable by default. Otherwise, you can pass an `api_key` at initialization – see code examples below.
|
||||
|
||||
When you initialize the component, you can optionally set the `model_id`, which refers to the model you want to use. Please refer to [Azure documentation](https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/choose-model-feature) for a list of available models. The default model is `"prebuilt-read"`.
|
||||
|
||||
The `AzureOCRDocumentConverter` doesn’t extract the tables from a file as plain text but generates separate `Document` objects of type `table` that maintain the two-dimensional structure of the tables.
|
||||
|
||||
## Usage
|
||||
|
||||
The `AzureOCRDocumentConverter` is part of the `azure-form-recognizer-haystack` integration package. Install it with:
|
||||
|
||||
```shell
|
||||
pip install azure-form-recognizer-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from haystack_integrations.components.converters.azure_form_recognizer import (
|
||||
AzureOCRDocumentConverter,
|
||||
)
|
||||
from haystack.utils import Secret
|
||||
|
||||
converter = AzureOCRDocumentConverter(
|
||||
endpoint="azure_resource_url",
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
)
|
||||
|
||||
converter.run(sources=[Path("my_file.pdf")])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.converters.azure_form_recognizer import (
|
||||
AzureOCRDocumentConverter,
|
||||
)
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.utils import Secret
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component(
|
||||
"converter",
|
||||
AzureOCRDocumentConverter(
|
||||
endpoint="azure_resource_url",
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
),
|
||||
)
|
||||
pipeline.add_component("cleaner", DocumentCleaner())
|
||||
pipeline.add_component(
|
||||
"splitter",
|
||||
DocumentSplitter(split_by="sentence", split_length=5),
|
||||
)
|
||||
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
pipeline.connect("converter", "cleaner")
|
||||
pipeline.connect("cleaner", "splitter")
|
||||
pipeline.connect("splitter", "writer")
|
||||
|
||||
file_names = ["my_file.pdf"]
|
||||
pipeline.run({"converter": {"sources": file_names}})
|
||||
```
|
||||
@@ -0,0 +1,75 @@
|
||||
---
|
||||
title: "CSVToDocument"
|
||||
id: csvtodocument
|
||||
slug: "/csvtodocument"
|
||||
description: "Converts CSV files to documents."
|
||||
---
|
||||
|
||||
# CSVToDocument
|
||||
|
||||
Converts CSV files to documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [PreProcessors](../preprocessors.mdx) , or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory run variables** | `sources`: A list of file paths or [ByteStream](../../concepts/data-classes.mdx#bytestream) objects |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Converters](/reference/converters-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/converters/csv.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`CSVToDocument` converts one or more CSV files into a text document.
|
||||
|
||||
The component uses UTF-8 encoding by default, but you may specify a different encoding if needed during initialization.
|
||||
You can optionally attach metadata to each document with a `meta` parameter when running the component.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.components.converters.csv import CSVToDocument
|
||||
|
||||
converter = CSVToDocument()
|
||||
results = converter.run(
|
||||
sources=["sample.csv"],
|
||||
meta={"date_added": datetime.now().isoformat()},
|
||||
)
|
||||
documents = results["documents"]
|
||||
|
||||
print(documents[0].content)
|
||||
# 'col1,col2\now1,row1\nrow2row2\n'
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters import CSVToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("converter", CSVToDocument())
|
||||
pipeline.add_component("cleaner", DocumentCleaner())
|
||||
pipeline.add_component(
|
||||
"splitter",
|
||||
DocumentSplitter(split_by="sentence", split_length=5),
|
||||
)
|
||||
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
pipeline.connect("converter", "cleaner")
|
||||
pipeline.connect("cleaner", "splitter")
|
||||
pipeline.connect("splitter", "writer")
|
||||
|
||||
pipeline.run({"converter": {"sources": file_names}})
|
||||
```
|
||||
@@ -0,0 +1,138 @@
|
||||
---
|
||||
title: "DoclingConverter"
|
||||
id: doclingconverter
|
||||
slug: "/doclingconverter"
|
||||
description: "`DoclingConverter` converts PDF, DOCX, HTML, and other document formats to Haystack Documents using Docling, with support for layout-aware chunking, Markdown, and JSON export."
|
||||
---
|
||||
|
||||
# DoclingConverter
|
||||
|
||||
`DoclingConverter` converts PDF, DOCX, HTML, and other document formats to Haystack Documents using [Docling](https://ds4sd.github.io/docling/), a document parsing library that understands document structure including layout, tables, and headings.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [PreProcessors](../preprocessors.mdx), or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory run variables** | `sources`: A list of file paths, URLs, or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Docling](/reference/integrations-docling) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/docling |
|
||||
| **Package name** | `docling-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `DoclingConverter` takes a list of file paths, URLs, or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects and uses Docling to parse them into a rich document representation that captures layout, tables, headings, and other structural elements.
|
||||
|
||||
The component supports three export modes, controlled by the `export_type` parameter:
|
||||
|
||||
- **`ExportType.DOC_CHUNKS`** (default): Chunks each document using Docling's `HybridChunker` and returns one [`Document`](../../concepts/data-classes.mdx#document) per chunk. Chunk metadata includes structural context from Docling. Use this mode for indexing pipelines where downstream retrieval benefits from semantically coherent chunks.
|
||||
- **`ExportType.MARKDOWN`**: Exports each input document as a single Markdown string in one [`Document`](../../concepts/data-classes.mdx#document). Use this mode when you want to preserve the full document content as formatted text.
|
||||
- **`ExportType.JSON`**: Serializes the full Docling document to a JSON string in one [`Document`](../../concepts/data-classes.mdx#document). Use this mode when you need access to the complete structured representation.
|
||||
|
||||
You can customize parsing behavior by passing a pre-configured `DocumentConverter` instance via the `converter` parameter, and pass additional keyword arguments to Docling's conversion step via `convert_kwargs`. For `ExportType.MARKDOWN`, use `md_export_kwargs` to control Markdown rendering options (for example, image placeholder text). For `ExportType.DOC_CHUNKS`, provide a custom `BaseChunker` instance via the `chunker` parameter.
|
||||
|
||||
Document metadata is populated by a `MetaExtractor` instance. The default `MetaExtractor` adds Docling-specific metadata (chunk structure or document origin) under the `dl_meta` key. You can supply a custom `BaseMetaExtractor` implementation via the `meta_extractor` parameter. Additional metadata can be attached to all output Documents by passing a dictionary to the `meta` run parameter, or per source by passing a list of dictionaries.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the Docling integration:
|
||||
|
||||
```shell
|
||||
pip install docling-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.converters.docling import (
|
||||
DoclingConverter,
|
||||
ExportType,
|
||||
)
|
||||
|
||||
# Default: chunk-based output
|
||||
converter = DoclingConverter()
|
||||
result = converter.run(sources=["report.pdf", "notes.docx"])
|
||||
documents = result["documents"]
|
||||
|
||||
# Full document as Markdown
|
||||
converter = DoclingConverter(export_type=ExportType.MARKDOWN)
|
||||
result = converter.run(sources=["report.pdf"])
|
||||
documents = result["documents"]
|
||||
print(documents[0].content)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.converters.docling import DoclingConverter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("converter", DoclingConverter())
|
||||
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
pipeline.connect("converter", "writer")
|
||||
|
||||
pipeline.run({"converter": {"sources": ["report.pdf", "manual.docx"]}})
|
||||
```
|
||||
|
||||
Because `DoclingConverter` with `ExportType.DOC_CHUNKS` already chunks the documents, you typically don't need a separate `DocumentSplitter` in the pipeline.
|
||||
|
||||
## Additional Features
|
||||
|
||||
### Custom chunking
|
||||
|
||||
Provide a custom Docling chunker to control how documents are split:
|
||||
|
||||
```python
|
||||
from docling.chunking import HybridChunker
|
||||
from haystack_integrations.components.converters.docling import DoclingConverter
|
||||
|
||||
chunker = HybridChunker(tokenizer="BAAI/bge-small-en-v1.5", max_tokens=256)
|
||||
converter = DoclingConverter(chunker=chunker)
|
||||
result = converter.run(sources=["report.pdf"])
|
||||
```
|
||||
|
||||
### Attaching metadata
|
||||
|
||||
Pass a single dictionary to apply metadata to all output Documents, or a list to set metadata per source:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.converters.docling import DoclingConverter
|
||||
|
||||
converter = DoclingConverter()
|
||||
|
||||
# Same metadata for all sources
|
||||
result = converter.run(
|
||||
sources=["a.pdf", "b.pdf"],
|
||||
meta={"project": "research"},
|
||||
)
|
||||
|
||||
# Per-source metadata
|
||||
result = converter.run(
|
||||
sources=["a.pdf", "b.pdf"],
|
||||
meta=[{"title": "Report A"}, {"title": "Report B"}],
|
||||
)
|
||||
```
|
||||
|
||||
### Processing in-memory files
|
||||
|
||||
Pass [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects to convert files loaded into memory. Set `file_path` in the ByteStream metadata so Docling can detect the file format:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ByteStream
|
||||
from haystack_integrations.components.converters.docling import DoclingConverter
|
||||
|
||||
with open("report.pdf", "rb") as f:
|
||||
data = f.read()
|
||||
|
||||
source = ByteStream(data=data, meta={"file_path": "report.pdf"})
|
||||
converter = DoclingConverter()
|
||||
result = converter.run(sources=[source])
|
||||
```
|
||||
@@ -0,0 +1,161 @@
|
||||
---
|
||||
title: "DoclingServeConverter"
|
||||
id: doclingserveconverter
|
||||
slug: "/doclingserveconverter"
|
||||
description: "`DoclingServeConverter` converts PDF, DOCX, HTML, and other document formats to Haystack Documents by calling a remote DoclingServe HTTP server, with no local ML dependencies."
|
||||
---
|
||||
|
||||
# DoclingServeConverter
|
||||
|
||||
`DoclingServeConverter` converts PDF, DOCX, HTML, and other document formats to Haystack Documents by calling a [DoclingServe](https://github.com/docling-project/docling-serve) HTTP server. Unlike the local [`DoclingConverter`](doclingconverter.mdx), this component has no heavy ML dependencies — all document parsing happens on the remote server.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [PreProcessors](../preprocessors.mdx), or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory run variables** | `sources`: A list of file paths, URLs, or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Docling Serve](/reference/integrations-docling_serve) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/docling_serve |
|
||||
| **Package name** | `docling-serve-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `DoclingServeConverter` takes a list of file paths, URLs, or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects and sends them to a running DoclingServe instance for parsing. Local files and `ByteStream` objects are uploaded to the `/v1/convert/file` endpoint; URL strings are sent to `/v1/convert/source`.
|
||||
|
||||
The component supports three export modes, controlled by the `export_type` parameter:
|
||||
|
||||
- **`ExportType.MARKDOWN`** (default): Returns the document content as a Markdown string. Use this mode when you want well-structured text output with formatting preserved.
|
||||
- **`ExportType.TEXT`**: Returns plain text extracted from the document. Use this mode when you need clean, unformatted text.
|
||||
- **`ExportType.JSON`**: Returns the full Docling document representation as a JSON string. Use this mode when you need access to the complete structured representation.
|
||||
|
||||
Each source produces one [`Document`](../../concepts/data-classes.mdx#document) in the output. Sources that fail to convert are skipped with a warning logged.
|
||||
|
||||
You can pass additional conversion options to the DoclingServe API via the `convert_options` parameter (for example, `{"do_ocr": True, "ocr_engine": "tesseract"}`). If the DoclingServe instance requires authentication, pass the API key via the `api_key` parameter or set the `DOCLING_SERVE_API_KEY` environment variable.
|
||||
|
||||
The component supports both synchronous (`run`) and asynchronous (`run_async`) execution.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the Docling Serve integration:
|
||||
|
||||
```shell
|
||||
pip install docling-serve-haystack
|
||||
```
|
||||
|
||||
Start a DoclingServe instance locally (requires Docker):
|
||||
|
||||
```shell
|
||||
docker run -p 5001:5001 ghcr.io/docling-project/docling-serve-cpu:latest
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.converters.docling_serve import (
|
||||
DoclingServeConverter,
|
||||
)
|
||||
|
||||
# Default: Markdown output
|
||||
converter = DoclingServeConverter(base_url="http://localhost:5001")
|
||||
result = converter.run(sources=["report.pdf", "notes.docx"])
|
||||
documents = result["documents"]
|
||||
print(documents[0].content[:200])
|
||||
|
||||
# Plain text output
|
||||
from haystack_integrations.components.converters.docling_serve import ExportType
|
||||
|
||||
converter = DoclingServeConverter(
|
||||
base_url="http://localhost:5001",
|
||||
export_type=ExportType.TEXT,
|
||||
)
|
||||
result = converter.run(sources=["report.pdf"])
|
||||
print(result["documents"][0].content)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.converters.docling_serve import (
|
||||
DoclingServeConverter,
|
||||
)
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component(
|
||||
"converter",
|
||||
DoclingServeConverter(base_url="http://localhost:5001"),
|
||||
)
|
||||
pipeline.add_component("splitter", DocumentSplitter())
|
||||
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
pipeline.connect("converter", "splitter")
|
||||
pipeline.connect("splitter", "writer")
|
||||
|
||||
pipeline.run({"converter": {"sources": ["report.pdf", "manual.docx"]}})
|
||||
```
|
||||
|
||||
## Additional Features
|
||||
|
||||
### Converting URLs directly
|
||||
|
||||
Pass URL strings to convert remote documents without downloading them first:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.converters.docling_serve import (
|
||||
DoclingServeConverter,
|
||||
)
|
||||
|
||||
converter = DoclingServeConverter(base_url="http://localhost:5001")
|
||||
result = converter.run(sources=["https://arxiv.org/pdf/2602.17316"])
|
||||
print(result["documents"][0].content[:200])
|
||||
```
|
||||
|
||||
### Attaching metadata
|
||||
|
||||
Pass a single dictionary to apply metadata to all output Documents, or a list to set metadata per source:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.converters.docling_serve import (
|
||||
DoclingServeConverter,
|
||||
)
|
||||
|
||||
converter = DoclingServeConverter(base_url="http://localhost:5001")
|
||||
|
||||
# Same metadata for all sources
|
||||
result = converter.run(
|
||||
sources=["a.pdf", "b.pdf"],
|
||||
meta={"project": "research"},
|
||||
)
|
||||
|
||||
# Per-source metadata
|
||||
result = converter.run(
|
||||
sources=["a.pdf", "b.pdf"],
|
||||
meta=[{"title": "Report A"}, {"title": "Report B"}],
|
||||
)
|
||||
```
|
||||
|
||||
### Processing in-memory files
|
||||
|
||||
Pass [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects to convert files loaded into memory. Set `file_path` in the ByteStream metadata so DoclingServe can detect the file format:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ByteStream
|
||||
from haystack_integrations.components.converters.docling_serve import (
|
||||
DoclingServeConverter,
|
||||
)
|
||||
|
||||
with open("report.pdf", "rb") as f:
|
||||
data = f.read()
|
||||
|
||||
source = ByteStream(data=data, meta={"file_path": "report.pdf"})
|
||||
converter = DoclingServeConverter(base_url="http://localhost:5001")
|
||||
result = converter.run(sources=[source])
|
||||
```
|
||||
@@ -0,0 +1,154 @@
|
||||
---
|
||||
title: "DocumentToImageContent"
|
||||
id: documenttoimagecontent
|
||||
slug: "/documenttoimagecontent"
|
||||
description: "`DocumentToImageContent` extracts visual data from image or PDF file-based documents and converts them into `ImageContent` objects. These are ready for multimodal AI pipelines, including tasks like image question-answering and captioning."
|
||||
---
|
||||
|
||||
# DocumentToImageContent
|
||||
|
||||
`DocumentToImageContent` extracts visual data from image or PDF file-based documents and converts them into `ImageContent` objects. These are ready for multimodal AI pipelines, including tasks like image question-answering and captioning.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a `ChatPromptBuilder` in a query pipeline |
|
||||
| **Mandatory run variables** | `documents`: A list of documents to process. Each document should have metadata containing at minimum a 'file_path_meta_field' key. PDF documents additionally require a 'page_number' key to specify which page to convert. |
|
||||
| **Output variables** | `image_contents`: A list of `ImageContent` objects |
|
||||
| **API reference** | [Image Converters](/reference/image-converters-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/converters/image/document_to_image.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`DocumentToImageContent` processes a list of documents containing image or PDF file paths and converts them into `ImageContent` objects.
|
||||
|
||||
- For images, it reads and encodes the file directly.
|
||||
- For PDFs, it extracts the specified page (through `page_number` in metadata) and converts it to an image.
|
||||
|
||||
By default, it looks for the file path in the `file_path` metadata field. You can customize this with the `file_path_meta_field` parameter. The `root_path` lets you specify a common base directory for file resolution.
|
||||
|
||||
This component is typically used in query pipelines right before a `ChatPromptBuilder` when you would like to add Images to your user prompt.
|
||||
|
||||
If `size` is provided, the images will be resized while maintaining aspect ratio. This reduces file size, memory usage, and processing time, which is beneficial when working with models that have resolution constraints or when transmitting images to remote services.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.converters.image.document_to_image import (
|
||||
DocumentToImageContent,
|
||||
)
|
||||
|
||||
converter = DocumentToImageContent(
|
||||
file_path_meta_field="file_path",
|
||||
root_path="/data/documents",
|
||||
detail="high",
|
||||
size=(800, 600),
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(content="Photo of a mountain", meta={"file_path": "mountain.jpg"}),
|
||||
Document(
|
||||
content="First page of a report",
|
||||
meta={"file_path": "report.pdf", "page_number": 1},
|
||||
),
|
||||
]
|
||||
|
||||
result = converter.run(documents)
|
||||
image_contents = result["image_contents"]
|
||||
print(image_contents)
|
||||
|
||||
# [
|
||||
# ImageContent(
|
||||
# base64_image="/9j/4A...", mime_type="image/jpeg", detail="high",
|
||||
# meta={"file_path": "mountain.jpg"}
|
||||
# ),
|
||||
# ImageContent(
|
||||
# base64_image="/9j/4A...", mime_type="image/jpeg", detail="high",
|
||||
# meta={"file_path": "report.pdf", "page_number": 1}
|
||||
# )
|
||||
# ]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
You can use `DocumentToImageContent` in multimodal indexing pipelines before passing to an Embedder or captioning model.
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.converters.image.document_to_image import (
|
||||
DocumentToImageContent,
|
||||
)
|
||||
|
||||
# Query pipeline
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("image_converter", DocumentToImageContent(detail="auto"))
|
||||
pipeline.add_component(
|
||||
"chat_prompt_builder",
|
||||
ChatPromptBuilder(
|
||||
required_variables=["question"],
|
||||
template="""{% message role="system" %}
|
||||
You are a friendly assistant that answers questions based on provided images.
|
||||
{% endmessage %}
|
||||
|
||||
{%- message role="user" -%}
|
||||
Only provide an answer to the question using the images provided.
|
||||
|
||||
Question: {{ question }}
|
||||
Answer:
|
||||
|
||||
{%- for img in image_contents -%}
|
||||
{{ img | templatize_part }}
|
||||
{%- endfor -%}
|
||||
{%- endmessage -%}
|
||||
""",
|
||||
),
|
||||
)
|
||||
pipeline.add_component("llm", OpenAIChatGenerator(model="gpt-4o-mini"))
|
||||
|
||||
pipeline.connect("image_converter", "chat_prompt_builder.image_contents")
|
||||
pipeline.connect("chat_prompt_builder", "llm")
|
||||
|
||||
documents = [
|
||||
Document(content="Cat image", meta={"file_path": "cat.jpg"}),
|
||||
Document(content="Doc intro", meta={"file_path": "paper.pdf", "page_number": 1}),
|
||||
]
|
||||
|
||||
result = pipeline.run(
|
||||
data={
|
||||
"image_converter": {"documents": documents},
|
||||
"chat_prompt_builder": {"question": "What color is the cat?"},
|
||||
},
|
||||
)
|
||||
print(result)
|
||||
|
||||
# {
|
||||
# "llm": {
|
||||
# "replies": [
|
||||
# ChatMessage(
|
||||
# _role=<ChatRole.ASSISTANT: 'assistant'>,
|
||||
# _content=[TextContent(text="The cat is orange with some black.")],
|
||||
# _name=None,
|
||||
# _meta={
|
||||
# "model": "gpt-4o-mini-2024-07-18",
|
||||
# "index": 0,
|
||||
# "finish_reason": "stop",
|
||||
# "usage": {...},
|
||||
# },
|
||||
# )
|
||||
# ]
|
||||
# }
|
||||
# }
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Introduction to Multimodality](https://haystack.deepset.ai/cookbook/multimodal_intro)
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
title: "DOCXToDocument"
|
||||
id: docxtodocument
|
||||
slug: "/docxtodocument"
|
||||
description: "Convert DOCX files to documents."
|
||||
---
|
||||
|
||||
# DOCXToDocument
|
||||
|
||||
Convert DOCX files to documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [PreProcessors](../preprocessors.mdx) or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory run variables** | `sources`: DOCX file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Converters](/reference/converters-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/converters/docx.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `DOCXToDocument` component converts DOCX files into documents. It takes a list of file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects as input and outputs the converted result as a list of documents. By defining the table format (CSV or Markdown), you can use this component to extract tables in your DOCX files. Optionally, you can attach metadata to the documents through the `meta` input parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
First, install the`python-docx` package to start using this converter:
|
||||
|
||||
```shell
|
||||
pip install python-docx
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.components.converters.docx import DOCXToDocument, DOCXTableFormat
|
||||
|
||||
converter = DOCXToDocument()
|
||||
# or define the table format
|
||||
converter = DOCXToDocument(table_format=DOCXTableFormat.CSV)
|
||||
|
||||
results = converter.run(
|
||||
sources=["sample.docx"],
|
||||
meta={"date_added": datetime.now().isoformat()},
|
||||
)
|
||||
documents = results["documents"]
|
||||
|
||||
print(documents[0].content)
|
||||
|
||||
# 'This is the text from the DOCX file.'
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters import DOCXToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("converter", DOCXToDocument())
|
||||
pipeline.add_component("cleaner", DocumentCleaner())
|
||||
pipeline.add_component(
|
||||
"splitter",
|
||||
DocumentSplitter(split_by="sentence", split_length=5),
|
||||
)
|
||||
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
pipeline.connect("converter", "cleaner")
|
||||
pipeline.connect("cleaner", "splitter")
|
||||
pipeline.connect("splitter", "writer")
|
||||
|
||||
pipeline.run({"converter": {"sources": file_names}})
|
||||
```
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
title: "FileToFileContent"
|
||||
id: filetofilecontent
|
||||
slug: "/filetofilecontent"
|
||||
description: "`FileToFileContent` reads local files and converts them into `FileContent` objects"
|
||||
---
|
||||
|
||||
# FileToFileContent
|
||||
|
||||
`FileToFileContent` reads local files and converts them into `FileContent` objects. These are ready for multimodal AI pipelines that need to pass PDFs and other file types to an LLM.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a `ChatPromptBuilder` in a query pipeline |
|
||||
| **Mandatory run variables** | `sources`: A list of file paths or ByteStreams |
|
||||
| **Output variables** | `file_contents`: A list of `FileContent` objects |
|
||||
| **API reference** | [Converters](/reference/converters-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/converters/file_to_file_content.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`FileToFileContent` processes a list of file sources and converts them into `FileContent` objects that can be embedded
|
||||
into a `ChatMessage` and passed to a Language Model.
|
||||
|
||||
Each source can be:
|
||||
|
||||
- A file path (string or `Path`), or
|
||||
- A `ByteStream` object.
|
||||
|
||||
Optionally, you can provide extra provider-specific information using the `extra` parameter. This can be a single dictionary (applied to all files) or a list matching the length of `sources`.
|
||||
|
||||
Support for passing files to LLMs varies by provider. Some providers do not support file inputs, some restrict support
|
||||
to PDF files, and others accept a wider range of file types.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.components.converters import FileToFileContent
|
||||
|
||||
converter = FileToFileContent()
|
||||
|
||||
sources = ["document.pdf", "recording.mp3"]
|
||||
|
||||
result = converter.run(sources=sources)
|
||||
file_contents = result["file_contents"]
|
||||
print(file_contents)
|
||||
|
||||
# [
|
||||
# FileContent(
|
||||
# base64_data='JVBERi0x...', mime_type='application/pdf',
|
||||
# filename='document.pdf', extra={}
|
||||
# ),
|
||||
# FileContent(
|
||||
# base64_data='SUQzBA...', mime_type='audio/mpeg',
|
||||
# filename='recording.mp3', extra={}
|
||||
# )
|
||||
# ]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Use `FileToFileContent` together with a `LinkContentFetcher` and a `ChatPromptBuilder` to build a pipeline that fetches a remote file, converts it, and passes it to an LLM.
|
||||
|
||||
```python
|
||||
from haystack.components.converters import FileToFileContent
|
||||
from haystack.components.fetchers import LinkContentFetcher
|
||||
from haystack.components.generators.chat.openai import OpenAIChatGenerator
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
|
||||
from haystack import Pipeline
|
||||
|
||||
template = """
|
||||
{% message role="user"%}
|
||||
{% for file in files %}
|
||||
{{ file | templatize_part }}
|
||||
{% endfor %}
|
||||
What's the main takeaway of the following document? Just one sentence.
|
||||
{% endmessage %}
|
||||
"""
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("fetcher", LinkContentFetcher())
|
||||
pipeline.add_component("converter", FileToFileContent())
|
||||
pipeline.add_component("prompt_builder", ChatPromptBuilder(template=template))
|
||||
pipeline.add_component("llm", OpenAIChatGenerator(model="gpt-4.1-mini"))
|
||||
|
||||
pipeline.connect("fetcher", "converter")
|
||||
pipeline.connect("converter", "prompt_builder")
|
||||
pipeline.connect("prompt_builder", "llm")
|
||||
|
||||
results = pipeline.run({"fetcher": {"urls": ["https://arxiv.org/pdf/2309.08632"]}})
|
||||
|
||||
print(results["llm"]["replies"][0].text)
|
||||
|
||||
# The document is a satirical paper humorously claiming that pretraining a
|
||||
# small language model exclusively on evaluation benchmark test sets can achieve
|
||||
# perfect performance, highlighting issues of data contamination in model
|
||||
# evaluation.
|
||||
```
|
||||
@@ -0,0 +1,71 @@
|
||||
---
|
||||
title: "HTMLToDocument"
|
||||
id: htmltodocument
|
||||
slug: "/htmltodocument"
|
||||
description: "A component that converts HTML files to documents."
|
||||
---
|
||||
|
||||
# HTMLToDocument
|
||||
|
||||
A component that converts HTML files to documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [PreProcessors](../preprocessors.mdx) , or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory run variables** | `sources`: A list of HTML file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Converters](/reference/converters-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/converters/html.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `HTMLToDocument` component converts HTML files into documents. It can be used in an indexing pipeline to index the contents of an HTML file into a Document Store or even in a querying pipeline after the [`LinkContentFetcher`](../fetchers/linkcontentfetcher.mdx). The `HTMLToDocument` component takes a list of HTML file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects as input and converts the files to a list of documents. Optionally, you can attach metadata to the documents through the `meta` input parameter.
|
||||
|
||||
When you initialize the component, you can optionally set `extraction_kwargs`, a dictionary containing keyword arguments to customize the extraction process. These are passed to the underlying Trafilatura `extract` function. For the full list of available arguments, see the [Trafilatura documentation](https://trafilatura.readthedocs.io/en/latest/corefunctions.html#extract).
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
from haystack.components.converters import HTMLToDocument
|
||||
|
||||
converter = HTMLToDocument()
|
||||
|
||||
docs = converter.run(sources=[Path("saved_page.html")])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Here's an example of an indexing pipeline that writes the contents of an HTML file into an `InMemoryDocumentStore`:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters import HTMLToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("converter", HTMLToDocument())
|
||||
pipeline.add_component("cleaner", DocumentCleaner())
|
||||
pipeline.add_component(
|
||||
"splitter",
|
||||
DocumentSplitter(split_by="sentence", split_length=5),
|
||||
)
|
||||
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
pipeline.connect("converter", "cleaner")
|
||||
pipeline.connect("cleaner", "splitter")
|
||||
pipeline.connect("splitter", "writer")
|
||||
|
||||
pipeline.run({"converter": {"sources": file_names}})
|
||||
```
|
||||
@@ -0,0 +1,111 @@
|
||||
---
|
||||
title: "ImageFileToDocument"
|
||||
id: imagefiletodocument
|
||||
slug: "/imagefiletodocument"
|
||||
description: "Converts image file references into empty `Document` objects with associated metadata."
|
||||
---
|
||||
|
||||
# ImageFileToDocument
|
||||
|
||||
Converts image file references into empty `Document` objects with associated metadata.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a component that processes images, like `SentenceTransformersImageDocumentEmbedder` or `LLMDocumentContentExtractor` |
|
||||
| **Mandatory run variables** | `sources`: A list of image file paths or ByteStreams |
|
||||
| **Output variables** | `documents`: A list of empty Document objects with associated metadata |
|
||||
| **API reference** | [Image Converters](/reference/image-converters-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/converters/image/file_to_document.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`ImageFileToDocument` converts image file sources into empty `Document` objects with associated metadata.
|
||||
|
||||
This component is useful in pipelines where image file paths need to be wrapped in `Document` objects to be processed by downstream components such as `SentenceTransformersImageDocumentEmbedder` or `LLMDocumentContentExtractor`.
|
||||
|
||||
It _does not_ extract any content from the image files, but instead creates `Document` objects with `None` as their content and attaches metadata such as file path and any user-provided values.
|
||||
|
||||
Each source can be:
|
||||
|
||||
- A file path (string or `Path`), or
|
||||
- A `ByteStream` object.
|
||||
|
||||
Optionally, you can provide metadata using the `meta` parameter. This can be a single dictionary (applied to all documents) or a list matching the length of `sources`.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
This component is primarily meant to be used in pipelines.
|
||||
|
||||
```python
|
||||
|
||||
from haystack.components.converters.image import ImageFileToDocument
|
||||
|
||||
converter = ImageFileToDocument()
|
||||
|
||||
sources = ["image.jpg", "another_image.png"]
|
||||
|
||||
result = converter.run(sources=sources)
|
||||
documents = result["documents"]
|
||||
|
||||
print(documents)
|
||||
|
||||
# [Document(id=..., content=None, meta={'file_path': 'image.jpg'}),
|
||||
# Document(id=..., content=None, meta={'file_path': 'another_image.png'})]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
In the following Pipeline, image documents are created using the `ImageFileToDocument` component, then they are enriched with image embeddings and saved in the Document Store.
|
||||
|
||||
The examples on this page use Sentence Transformers embedders that have moved to the `sentence-transformers-haystack` package. Install it to run the examples:
|
||||
|
||||
```shell
|
||||
pip install sentence-transformers-haystack
|
||||
```
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.converters.image import ImageFileToDocument
|
||||
from haystack_integrations.components.embedders.sentence_transformers import (
|
||||
SentenceTransformersDocumentImageEmbedder,
|
||||
)
|
||||
from haystack.components.writers.document_writer import DocumentWriter
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
# Create our document store
|
||||
doc_store = InMemoryDocumentStore()
|
||||
|
||||
# Define pipeline with components
|
||||
indexing_pipe = Pipeline()
|
||||
indexing_pipe.add_component(
|
||||
"image_converter",
|
||||
ImageFileToDocument(store_full_path=True),
|
||||
)
|
||||
indexing_pipe.add_component(
|
||||
"image_doc_embedder",
|
||||
SentenceTransformersDocumentImageEmbedder(),
|
||||
)
|
||||
indexing_pipe.add_component("document_writer", DocumentWriter(doc_store))
|
||||
|
||||
indexing_pipe.connect("image_converter.documents", "image_doc_embedder.documents")
|
||||
indexing_pipe.connect("image_doc_embedder.documents", "document_writer.documents")
|
||||
|
||||
indexing_result = indexing_pipe.run(
|
||||
data={"image_converter": {"sources": ["apple.jpg", "kiwi.png"]}},
|
||||
)
|
||||
|
||||
indexed_documents = doc_store.filter_documents()
|
||||
print(f"Indexed {len(indexed_documents)} documents")
|
||||
# Indexed 2 documents
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Introduction to Multimodality](https://haystack.deepset.ai/cookbook/multimodal_intro)
|
||||
@@ -0,0 +1,129 @@
|
||||
---
|
||||
title: "ImageFileToImageContent"
|
||||
id: imagefiletoimagecontent
|
||||
slug: "/imagefiletoimagecontent"
|
||||
description: "`ImageFileToImageContent` reads local image files and converts them into `ImageContent` objects. These are ready for multimodal AI pipelines, including tasks like image captioning, visual QA, or prompt-based generation."
|
||||
---
|
||||
|
||||
# ImageFileToImageContent
|
||||
|
||||
`ImageFileToImageContent` reads local image files and converts them into `ImageContent` objects. These are ready for multimodal AI pipelines, including tasks like image captioning, visual QA, or prompt-based generation.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a `ChatPromptBuilder` in a query pipeline |
|
||||
| **Mandatory run variables** | `sources`: A list of image file paths or ByteStreams |
|
||||
| **Output variables** | `image_contents`: A list of ImageContent objects |
|
||||
| **API reference** | [Image Converters](/reference/image-converters-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/converters/image/file_to_image.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`ImageFileToImageContent` processes a list of image sources and converts them into `ImageContent` objects. These can be used in multimodal pipelines that require base64-encoded image input.
|
||||
|
||||
Each source can be:
|
||||
|
||||
- A file path (string or `Path`), or
|
||||
- A `ByteStream` object.
|
||||
|
||||
Optionally, you can provide metadata using the `meta` parameter. This can be a single dictionary (applied to all images) or a list matching the length of `sources`.
|
||||
|
||||
Use the `size` parameter to resize images while preserving aspect ratio. This reduces memory usage and transmission size, which is helpful when working with remote models or limited-resource environments.
|
||||
|
||||
This component is often used in query pipelines just before a `ChatPromptBuilder`.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
|
||||
from haystack.components.converters.image import ImageFileToImageContent
|
||||
|
||||
converter = ImageFileToImageContent(detail="high", size=(800, 600))
|
||||
|
||||
sources = ["cat.jpg", "scenery.png"]
|
||||
|
||||
result = converter.run(sources=sources)
|
||||
image_contents = result["image_contents"]
|
||||
print(image_contents)
|
||||
|
||||
# [
|
||||
# ImageContent(
|
||||
# base64_image="/9j/4A...", mime_type="image/jpeg", detail="high",
|
||||
# meta={"file_path": "cat.jpg"}
|
||||
# ),
|
||||
# ImageContent(
|
||||
# base64_image="/9j/4A...", mime_type="image/png", detail="high",
|
||||
# meta={"file_path": "scenery.png"}
|
||||
# )
|
||||
# ]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Use `ImageFileToImageContent` to supply image data to a `ChatPromptBuilder` for multimodal QA or captioning with an LLM.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.converters.image import ImageFileToImageContent
|
||||
|
||||
# Query pipeline
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("image_converter", ImageFileToImageContent(detail="auto"))
|
||||
pipeline.add_component(
|
||||
"chat_prompt_builder",
|
||||
ChatPromptBuilder(
|
||||
required_variables=["question"],
|
||||
template="""{% message role="system" %}
|
||||
You are a helpful assistant that answers questions using the provided images.
|
||||
{% endmessage %}
|
||||
|
||||
{% message role="user" %}
|
||||
Question: {{ question }}
|
||||
|
||||
{% for img in image_contents %}
|
||||
{{ img | templatize_part }}
|
||||
{% endfor %}
|
||||
{% endmessage %}
|
||||
""",
|
||||
),
|
||||
)
|
||||
pipeline.add_component("llm", OpenAIChatGenerator(model="gpt-4o-mini"))
|
||||
|
||||
pipeline.connect("image_converter", "chat_prompt_builder.image_contents")
|
||||
pipeline.connect("chat_prompt_builder", "llm")
|
||||
|
||||
sources = ["apple.jpg", "haystack-logo.png"]
|
||||
|
||||
result = pipeline.run(
|
||||
data={
|
||||
"image_converter": {"sources": sources},
|
||||
"chat_prompt_builder": {"question": "Describe the Haystack logo."},
|
||||
},
|
||||
)
|
||||
print(result)
|
||||
|
||||
# {
|
||||
# "llm": {
|
||||
# "replies": [
|
||||
# ChatMessage(
|
||||
# _role=<ChatRole.ASSISTANT: 'assistant'>,
|
||||
# _content=[TextContent(text="The Haystack logo features...")],
|
||||
# ...
|
||||
# )
|
||||
# ]
|
||||
# }
|
||||
# }
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Introduction to Multimodality](https://haystack.deepset.ai/cookbook/multimodal_intro)
|
||||
@@ -0,0 +1,119 @@
|
||||
---
|
||||
title: "JSONConverter"
|
||||
id: jsonconverter
|
||||
slug: "/jsonconverter"
|
||||
description: "Converts JSON files to text documents."
|
||||
---
|
||||
|
||||
# JSONConverter
|
||||
|
||||
Converts JSON files to text documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [PreProcessors](../preprocessors.mdx) , or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory init variables** | ONE OF, OR BOTH: <br /> <br />`jq_schema`: A jq filter string to extract content <br /> <br />`content_key`: A key string to extract document content |
|
||||
| **Mandatory run variables** | `sources`: A list of file paths or [ByteStream](../../concepts/data-classes.mdx#bytestream) objects |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Converters](/reference/converters-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/converters/json.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`JSONConverter` converts one or more JSON files into a text document.
|
||||
|
||||
### Parameters Overview
|
||||
|
||||
To initialize `JSONConverter`, you must provide either `jq_schema`, or `content_key` parameter, or both.
|
||||
|
||||
`jq_schema` parameter filter extracts nested data from JSON files. Refer to the [jq documentation](https://jqlang.github.io/jq/) for filter syntax. If not set, the entire JSON file is used.
|
||||
|
||||
The `content_key` parameter lets you specify which key in the extracted data will be the document's content.
|
||||
|
||||
- If both `jq_schema` and `content_key` are set, the `content_key` is searched in the data extracted by `jq_schema`. Non-object data will be skipped.
|
||||
- If only `jq_schema` is set, the extracted value must be scalar; objects or arrays will be skipped.
|
||||
- If only `content_key` is set, the source must be a JSON object, or it will be skipped.
|
||||
|
||||
Check out the [API reference](../converters.mdx) for the full list of parameters.
|
||||
|
||||
## Usage
|
||||
|
||||
You need to install the `jq` package to use this Converter:
|
||||
|
||||
```shell
|
||||
pip install jq
|
||||
```
|
||||
|
||||
### Example
|
||||
|
||||
Here is an example of simple component usage:
|
||||
|
||||
```python
|
||||
import json
|
||||
|
||||
from haystack.components.converters import JSONConverter
|
||||
from haystack.dataclasses import ByteStream
|
||||
|
||||
source = ByteStream.from_string(
|
||||
json.dumps({"text": "This is the content of my document"}),
|
||||
)
|
||||
|
||||
converter = JSONConverter(content_key="text")
|
||||
results = converter.run(sources=[source])
|
||||
documents = results["documents"]
|
||||
print(documents[0].content)
|
||||
# 'This is the content of my document'
|
||||
```
|
||||
|
||||
In the following more complex example, we provide a `jq_schema` string to filter the JSON source files and `extra_meta_fields` to extract from the filtered data:
|
||||
|
||||
```python
|
||||
import json
|
||||
|
||||
from haystack.components.converters import JSONConverter
|
||||
from haystack.dataclasses import ByteStream
|
||||
|
||||
data = {
|
||||
"laureates": [
|
||||
{
|
||||
"firstname": "Enrico",
|
||||
"surname": "Fermi",
|
||||
"motivation": "for his demonstrations of the existence of new radioactive elements produced "
|
||||
"by neutron irradiation, and for his related discovery of nuclear reactions brought about by"
|
||||
" slow neutrons",
|
||||
},
|
||||
{
|
||||
"firstname": "Rita",
|
||||
"surname": "Levi-Montalcini",
|
||||
"motivation": "for their discoveries of growth factors",
|
||||
},
|
||||
],
|
||||
}
|
||||
source = ByteStream.from_string(json.dumps(data))
|
||||
converter = JSONConverter(
|
||||
jq_schema=".laureates[]",
|
||||
content_key="motivation",
|
||||
extra_meta_fields={"firstname", "surname"},
|
||||
)
|
||||
|
||||
results = converter.run(sources=[source])
|
||||
documents = results["documents"]
|
||||
print(documents[0].content)
|
||||
# 'for his demonstrations of the existence of new radioactive elements produced by
|
||||
# neutron irradiation, and for his related discovery of nuclear reactions brought
|
||||
# about by slow neutrons'
|
||||
|
||||
print(documents[0].meta)
|
||||
# {'firstname': 'Enrico', 'surname': 'Fermi'}
|
||||
|
||||
print(documents[1].content)
|
||||
# 'for their discoveries of growth factors'
|
||||
|
||||
print(documents[1].meta)
|
||||
# {'firstname': 'Rita', 'surname': 'Levi-Montalcini'}
|
||||
```
|
||||
@@ -0,0 +1,148 @@
|
||||
---
|
||||
title: "KreuzbergConverter"
|
||||
id: kreuzbergconverter
|
||||
slug: "/kreuzbergconverter"
|
||||
description: "`KreuzbergConverter` converts files to Haystack Documents using Kreuzberg, a document intelligence framework with a Rust core that extracts text from 91+ file formats entirely locally with no external API calls."
|
||||
---
|
||||
|
||||
# KreuzbergConverter
|
||||
|
||||
`KreuzbergConverter` converts files to Haystack Documents using [Kreuzberg](https://docs.kreuzberg.dev/), a document intelligence framework with a Rust core that extracts text from 91+ file formats entirely locally with no external API calls.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [PreProcessors](../preprocessors.mdx), or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory run variables** | `sources`: A list of file paths, directory paths, or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Kreuzberg](/reference/integrations-kreuzberg) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/kreuzberg |
|
||||
| **Package name** | `kreuzberg-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `KreuzbergConverter` takes a list of file paths, directory paths, or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects and uses Kreuzberg to extract text and metadata. All processing is performed locally with no external API calls.
|
||||
|
||||
**Supported format categories:**
|
||||
- **Documents**: PDF, DOCX, DOC, PPTX, PPT, XLSX, XLS, ODT, ODS, ODP, RTF, Pages, Keynote, Numbers, and more
|
||||
- **Images (via OCR)**: PNG, JPEG, TIFF, GIF, BMP, WebP, JPEG 2000, SVG
|
||||
- **Text/Markup**: Markdown, HTML, XML, LaTeX, Typst, JSON, YAML, reStructuredText, Jupyter notebooks
|
||||
- **Email**: EML, MSG (with attachment extraction)
|
||||
- **Archives**: ZIP, TAR, GZIP, 7Z (extracts and processes contents recursively)
|
||||
- **eBooks & Academic**: EPUB, BibTeX, DocBook, JATS
|
||||
|
||||
The component returns one Haystack [`Document`](../../concepts/data-classes.mdx#document) per source by default. When per-page extraction or chunking is enabled, it returns one Document per page or chunk instead. Documents include rich metadata such as quality scores, detected languages, extracted keywords, table data, and PDF annotations.
|
||||
|
||||
By default, batch processing is enabled, leveraging Rust's rayon thread pool for parallel extraction. Set `batch=False` for sequential processing.
|
||||
|
||||
You can customize extraction behavior with Kreuzberg's `ExtractionConfig`, either passed directly or loaded from a TOML, YAML, or JSON configuration file via `config_path`. See the [Kreuzberg documentation](https://docs.kreuzberg.dev/) for the full configuration reference.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the Kreuzberg integration:
|
||||
|
||||
```shell
|
||||
pip install kreuzberg-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.converters.kreuzberg import KreuzbergConverter
|
||||
|
||||
converter = KreuzbergConverter()
|
||||
result = converter.run(sources=["report.pdf", "notes.docx"])
|
||||
documents = result["documents"]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.converters.kreuzberg import KreuzbergConverter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("converter", KreuzbergConverter())
|
||||
pipeline.add_component(
|
||||
"splitter",
|
||||
DocumentSplitter(split_by="sentence", split_length=5),
|
||||
)
|
||||
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
|
||||
pipeline.connect("converter", "splitter")
|
||||
pipeline.connect("splitter", "writer")
|
||||
|
||||
pipeline.run({"converter": {"sources": ["report.pdf", "presentation.pptx"]}})
|
||||
```
|
||||
|
||||
## Additional Features
|
||||
|
||||
### Markdown Output with OCR
|
||||
|
||||
Use `ExtractionConfig` to customize the output format and OCR settings:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.converters.kreuzberg import KreuzbergConverter
|
||||
from kreuzberg import ExtractionConfig, OcrConfig
|
||||
|
||||
converter = KreuzbergConverter(
|
||||
config=ExtractionConfig(
|
||||
output_format="markdown",
|
||||
ocr=OcrConfig(backend="tesseract", language="eng"),
|
||||
),
|
||||
)
|
||||
result = converter.run(sources=["scanned_document.pdf"])
|
||||
documents = result["documents"]
|
||||
```
|
||||
|
||||
### Per-Page Extraction
|
||||
|
||||
Create one Document per page using `PageConfig`:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.converters.kreuzberg import KreuzbergConverter
|
||||
from kreuzberg import ExtractionConfig, PageConfig
|
||||
|
||||
converter = KreuzbergConverter(
|
||||
config=ExtractionConfig(
|
||||
page=PageConfig(extract_pages=True),
|
||||
),
|
||||
)
|
||||
result = converter.run(sources=["multipage.pdf"])
|
||||
# One Document per page, each with page_number in metadata
|
||||
```
|
||||
|
||||
### Token Reduction
|
||||
|
||||
Reduce output size for LLM consumption with `TokenReductionConfig`. Token reduction uses TF-IDF-based extractive summarization to identify and preserve the most important terms and phrases, progressively removing less critical content such as extra whitespace, filler words, and redundant phrases. Five levels are available: `"off"` (no reduction), `"light"` (~15%), `"moderate"` (~30%), `"aggressive"` (~50%), and `"maximum"` (>50% reduction):
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.converters.kreuzberg import KreuzbergConverter
|
||||
from kreuzberg import ExtractionConfig, TokenReductionConfig
|
||||
|
||||
converter = KreuzbergConverter(
|
||||
config=ExtractionConfig(
|
||||
token_reduction=TokenReductionConfig(mode="moderate"),
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
### Config from File
|
||||
|
||||
Load extraction settings from a TOML, YAML, or JSON file:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.converters.kreuzberg import KreuzbergConverter
|
||||
|
||||
converter = KreuzbergConverter(config_path="extraction_config.toml")
|
||||
```
|
||||
|
||||
For the full configuration reference and format support matrix, see the [Kreuzberg documentation](https://docs.kreuzberg.dev/).
|
||||
@@ -0,0 +1,96 @@
|
||||
---
|
||||
title: "LibreOfficeFileConverter"
|
||||
id: libreofficefileconverter
|
||||
slug: "/libreofficefileconverter"
|
||||
description: "A component that converts office files (documents, spreadsheets, presentations) between formats using LibreOffice's command line interface."
|
||||
---
|
||||
|
||||
# LibreOfficeFileConverter
|
||||
|
||||
A component that converts office files between formats using LibreOffice's command line interface (`soffice`).
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a document converter (e.g. [`DOCXToDocument`](./docxtodocument.mdx)) when the source files need to be converted to a format that the converter supports |
|
||||
| **Mandatory run variables** | `sources`: File paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects; `output_file_type`: The target file format |
|
||||
| **Output variables** | `output`: A list of [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects |
|
||||
| **API reference** | [LibreOffice](/reference/integrations-libreoffice) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/libreoffice |
|
||||
| **Package name** | `libreoffice-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`LibreOfficeFileConverter` converts office files from one format to another using LibreOffice's `soffice` command line utility. It supports a wide range of document, spreadsheet, and presentation formats and is useful when your pipeline receives files in a format that downstream converters don't support.
|
||||
|
||||
Unlike most converters, `LibreOfficeFileConverter` outputs [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects rather than Haystack Documents. This means it's typically chained with a document converter (such as [`DOCXToDocument`](./docxtodocument.mdx) or [`PyPDFToDocument`](./pypdftodocument.mdx)) to produce the final Documents.
|
||||
|
||||
**Requires LibreOffice to be installed** and available in `PATH` as `soffice`. See the [LibreOffice installation guide](https://www.libreoffice.org/get-help/install-howto/) for details.
|
||||
|
||||
### Supported conversions
|
||||
|
||||
| Category | Input formats | Possible output formats |
|
||||
| --- | --- | --- |
|
||||
| Documents | `doc`, `docx`, `odt`, `rtf`, `txt`, `html` | `pdf`, `docx`, `doc`, `odt`, `rtf`, `txt`, `html`, `epub` |
|
||||
| Spreadsheets | `xlsx`, `xls`, `ods`, `csv` | `pdf`, `xlsx`, `xls`, `ods`, `csv`, `html` |
|
||||
| Presentations | `pptx`, `ppt`, `odp` | `pdf`, `pptx`, `ppt`, `odp`, `html`, `png`, `jpg` |
|
||||
|
||||
This is a non-exhaustive list. See the [LibreOffice filter documentation](https://help.libreoffice.org/latest/en-GB/text/shared/guide/convertfilters.html) for all supported conversions.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the LibreOffice integration:
|
||||
|
||||
```shell
|
||||
pip install libreoffice-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
from haystack_integrations.components.converters.libreoffice import (
|
||||
LibreOfficeFileConverter,
|
||||
)
|
||||
|
||||
converter = LibreOfficeFileConverter()
|
||||
result = converter.run(sources=[Path("sample.doc")], output_file_type="docx")
|
||||
bytestreams = result["output"]
|
||||
```
|
||||
|
||||
You can also set `output_file_type` at initialization to avoid passing it on every `run()` call:
|
||||
|
||||
```python
|
||||
converter = LibreOfficeFileConverter(output_file_type="pdf")
|
||||
result = converter.run(sources=[Path("report.pptx")])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
A common pattern is to chain `LibreOfficeFileConverter` with a document converter. The example below converts a legacy `.doc` file to `.docx` and then extracts it as a Haystack Document:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
from haystack import Pipeline
|
||||
from haystack.components.converters import DOCXToDocument
|
||||
from haystack_integrations.components.converters.libreoffice import (
|
||||
LibreOfficeFileConverter,
|
||||
)
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component(
|
||||
"libreoffice_converter",
|
||||
LibreOfficeFileConverter(output_file_type="docx"),
|
||||
)
|
||||
pipeline.add_component("docx_converter", DOCXToDocument())
|
||||
|
||||
pipeline.connect("libreoffice_converter.output", "docx_converter.sources")
|
||||
|
||||
result = pipeline.run(
|
||||
{"libreoffice_converter": {"sources": [Path("legacy_report.doc")]}},
|
||||
)
|
||||
documents = result["docx_converter"]["documents"]
|
||||
```
|
||||
@@ -0,0 +1,105 @@
|
||||
---
|
||||
title: "MarkdownToDocument"
|
||||
id: markdowntodocument
|
||||
slug: "/markdowntodocument"
|
||||
description: "A component that converts Markdown files to documents."
|
||||
---
|
||||
|
||||
# MarkdownToDocument
|
||||
|
||||
A component that converts Markdown files to documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [PreProcessors](../preprocessors.mdx) , or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory run variables** | `sources`: Markdown file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Converters](/reference/converters-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/converters/markdown.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `MarkdownToDocument` component converts Markdown files into documents. You can use it in an indexing pipeline to index the contents of a Markdown file into a Document Store. It takes a list of file paths or [ByteStream](../../concepts/data-classes.mdx#bytestream) objects as input and outputs the converted result as a list of documents. Optionally, you can attach metadata to the documents through the `meta` input parameter.
|
||||
|
||||
When you initialize the component, you can optionally turn off progress bars by setting `progress_bar` to `False`. If you want to convert the contents of tables into a single line, you can enable that through the `table_to_single_line` parameter.
|
||||
|
||||
If your Markdown files start with YAML frontmatter, set `extract_frontmatter=True` to move that data into `Document.meta` and remove it from the converted document content. Metadata passed through the `meta` input takes precedence over frontmatter keys.
|
||||
|
||||
## Usage
|
||||
|
||||
You need to install `markdown-it-py` and `mdit_plain packages` to use the `MarkdownToDocument` component:
|
||||
|
||||
```shell
|
||||
pip install markdown-it-py mdit_plain
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.components.converters import MarkdownToDocument
|
||||
|
||||
converter = MarkdownToDocument()
|
||||
|
||||
docs = converter.run(sources=Path("my_file.md"))
|
||||
```
|
||||
|
||||
### With YAML frontmatter
|
||||
|
||||
Given `equity_note.md`:
|
||||
|
||||
```markdown
|
||||
---
|
||||
ticker: AAPL
|
||||
source: earnings_call
|
||||
date: 2026-06-12
|
||||
---
|
||||
|
||||
# Thesis
|
||||
Revenue guidance improved.
|
||||
```
|
||||
|
||||
```python
|
||||
from haystack.components.converters import MarkdownToDocument
|
||||
|
||||
converter = MarkdownToDocument(extract_frontmatter=True)
|
||||
|
||||
docs = converter.run(sources=["equity_note.md"])["documents"]
|
||||
print(docs[0].meta["ticker"])
|
||||
print(docs[0].content)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters import MarkdownToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("converter", MarkdownToDocument())
|
||||
pipeline.add_component("cleaner", DocumentCleaner())
|
||||
pipeline.add_component(
|
||||
"splitter",
|
||||
DocumentSplitter(split_by="sentence", split_length=5),
|
||||
)
|
||||
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
pipeline.connect("converter", "cleaner")
|
||||
pipeline.connect("cleaner", "splitter")
|
||||
pipeline.connect("splitter", "writer")
|
||||
|
||||
pipeline.run({"converter": {"sources": file_names}})
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
:notebook: Tutorial: [Preprocessing Different File Types](https://haystack.deepset.ai/tutorials/30_file_type_preprocessing_index_pipeline)
|
||||
@@ -0,0 +1,75 @@
|
||||
---
|
||||
title: "MarkItDownConverter"
|
||||
id: markitdownconverter
|
||||
slug: "/markitdownconverter"
|
||||
description: "A component that converts files (PDF, Word, PowerPoint, Excel, HTML, images, and more) to Documents using Microsoft's MarkItDown library."
|
||||
---
|
||||
|
||||
# MarkItDownConverter
|
||||
|
||||
A component that converts files to Documents using Microsoft's MarkItDown library.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [PreProcessors](../preprocessors.mdx) or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory run variables** | `sources`: File paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [MarkItDown](/reference/integrations-markitdown) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/markitdown |
|
||||
| **Package name** | `markitdown-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`MarkItDownConverter` converts files into Haystack Documents using Microsoft's [MarkItDown](https://github.com/microsoft/markitdown) library. MarkItDown converts many file formats to Markdown, including PDF, Word (.docx), PowerPoint (.pptx), Excel (.xlsx), HTML, and more. All processing is performed locally without relying on external APIs.
|
||||
|
||||
The converter accepts file paths or [ByteStream](../../concepts/data-classes.mdx#bytestream) objects as input and outputs the converted result as a list of Documents. You can attach metadata to the Documents through the `meta` input parameter.
|
||||
|
||||
:::note
|
||||
This component returns Markdown content. Avoid piping it through `DocumentCleaner()` with its default settings because `remove_extra_whitespaces=True` and `remove_empty_lines=True` can collapse line breaks and flatten headings, tables, lists, and image tags. Connect the converter directly to your next component, or disable those options if you need custom cleanup.
|
||||
:::
|
||||
|
||||
## Usage
|
||||
|
||||
Install the MarkItDown integration:
|
||||
|
||||
```shell
|
||||
pip install markitdown-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.converters.markitdown import MarkItDownConverter
|
||||
|
||||
converter = MarkItDownConverter()
|
||||
result = converter.run(sources=["document.pdf", "report.docx"])
|
||||
documents = result["documents"]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.converters.markitdown import MarkItDownConverter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("converter", MarkItDownConverter())
|
||||
pipeline.add_component(
|
||||
"splitter",
|
||||
DocumentSplitter(split_by="sentence", split_length=5),
|
||||
)
|
||||
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
pipeline.connect("converter", "splitter")
|
||||
pipeline.connect("splitter", "writer")
|
||||
|
||||
pipeline.run({"converter": {"sources": ["document.pdf", "report.docx"]}})
|
||||
```
|
||||
@@ -0,0 +1,192 @@
|
||||
---
|
||||
title: "MistralOCRDocumentConverter"
|
||||
id: mistralocrdocumentconverter
|
||||
slug: "/mistralocrdocumentconverter"
|
||||
description: "`MistralOCRDocumentConverter` extracts text from documents using Mistral's OCR API, with optional structured annotations for both individual image regions and full documents. It supports various input formats including local files, URLs, and Mistral file IDs."
|
||||
---
|
||||
|
||||
# MistralOCRDocumentConverter
|
||||
|
||||
`MistralOCRDocumentConverter` extracts text from documents using Mistral's OCR API, with optional structured annotations for both individual image regions and full documents. It supports various input formats including local files, URLs, and Mistral file IDs.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [PreProcessors](../preprocessors.mdx), or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory init variables** | `api_key`: The Mistral API key. Can be set with `MISTRAL_API_KEY` environment variable. |
|
||||
| **Mandatory run variables** | `sources`: A list of document sources (file paths, ByteStreams, URLs, or Mistral chunks) |
|
||||
| **Output variables** | `documents`: A list of documents <br /> <br />`raw_mistral_response`: A list of raw OCR responses from Mistral API |
|
||||
| **API reference** | [Mistral](/reference/integrations-mistral) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/mistral |
|
||||
| **Package name** | `mistral-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `MistralOCRDocumentConverter` takes a list of document sources and uses Mistral's OCR API to extract text from images and PDFs. It supports multiple input formats:
|
||||
|
||||
- **Local files**: File paths (str or Path) or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects
|
||||
- **Remote resources**: Document URLs, image URLs using Mistral's `DocumentURLChunk` and `ImageURLChunk`
|
||||
- **Mistral storage**: File IDs using Mistral's `FileChunk` for files previously uploaded to Mistral
|
||||
|
||||
The component returns one Haystack [`Document`](../../concepts/data-classes.mdx#document) per source, with all pages concatenated using form feed characters (`\f`) as separators. This format ensures compatibility with Haystack's [`DocumentSplitter`](../preprocessors/documentsplitter.mdx) for accurate page-wise splitting and overlap handling. The content is returned in markdown format, with images represented as `` tags.
|
||||
|
||||
By default, the component uses the `MISTRAL_API_KEY` environment variable for authentication. You can also pass an `api_key` at initialization. Local files are automatically uploaded to Mistral's storage for processing and deleted afterward (configurable with `cleanup_uploaded_files`).
|
||||
|
||||
When you initialize the component, you can optionally specify which pages to process, set limits on image extraction, configure minimum image sizes, or include base64-encoded images in the response. The default model is `"mistral-ocr-2505"`. See the [Mistral models documentation](https://docs.mistral.ai/getting-started/models/models_overview/) for available models.
|
||||
|
||||
### Structured Annotations
|
||||
|
||||
A unique feature of `MistralOCRDocumentConverter` is its support for structured annotations using Pydantic schemas:
|
||||
|
||||
- **Bounding box annotations** (`bbox_annotation_schema`): Annotate individual image regions with structured data (for example, image type, description, summary). These annotations are inserted inline after the corresponding image tags in the markdown content.
|
||||
- **Document annotations** (`document_annotation_schema`): Annotate the full document with structured data (for example, language, chapter titles, URLs). These annotations are unpacked into the document's metadata with a `source_` prefix (for example, `source_language`, `source_chapter_titles`).
|
||||
|
||||
When annotation schemas are provided, the OCR model first extracts text and structure, then a Vision LLM analyzes the content and generates structured annotations according to your defined Pydantic schemas. Note that document annotation is limited to a maximum of 8 pages. For more details, see the [Mistral documentation on annotations](https://docs.mistral.ai/capabilities/document_ai/annotations/).
|
||||
|
||||
:::note
|
||||
This component returns Markdown content. Avoid piping it through `DocumentCleaner()` with its default settings because `remove_extra_whitespaces=True` and `remove_empty_lines=True` can collapse line breaks and flatten headings, tables, and image tags. For page-aware chunking, connect the converter directly to `DocumentSplitter`, or disable those options if you need custom cleanup.
|
||||
:::
|
||||
|
||||
## Usage
|
||||
|
||||
You need to install the `mistral-haystack` integration to use `MistralOCRDocumentConverter`:
|
||||
|
||||
```shell
|
||||
pip install mistral-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage with a local file:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.components.converters.mistral import (
|
||||
MistralOCRDocumentConverter,
|
||||
)
|
||||
|
||||
converter = MistralOCRDocumentConverter(
|
||||
api_key=Secret.from_env_var("MISTRAL_API_KEY"),
|
||||
model="mistral-ocr-2505",
|
||||
)
|
||||
|
||||
result = converter.run(sources=[Path("my_document.pdf")])
|
||||
documents = result["documents"]
|
||||
```
|
||||
|
||||
Processing multiple sources with different types:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.components.converters.mistral import (
|
||||
MistralOCRDocumentConverter,
|
||||
)
|
||||
from mistralai.models import DocumentURLChunk, ImageURLChunk
|
||||
|
||||
converter = MistralOCRDocumentConverter(
|
||||
api_key=Secret.from_env_var("MISTRAL_API_KEY"),
|
||||
model="mistral-ocr-2505",
|
||||
)
|
||||
|
||||
sources = [
|
||||
Path("local_document.pdf"),
|
||||
DocumentURLChunk(document_url="https://example.com/document.pdf"),
|
||||
ImageURLChunk(image_url="https://example.com/receipt.jpg"),
|
||||
]
|
||||
|
||||
result = converter.run(sources=sources)
|
||||
documents = result["documents"] # List of 3 Documents
|
||||
raw_responses = result["raw_mistral_response"] # List of 3 raw responses
|
||||
```
|
||||
|
||||
Using structured annotations:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
from pydantic import BaseModel, Field
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.components.converters.mistral import (
|
||||
MistralOCRDocumentConverter,
|
||||
)
|
||||
from mistralai.models import DocumentURLChunk
|
||||
|
||||
|
||||
# Define schema for image region annotations
|
||||
class ImageAnnotation(BaseModel):
|
||||
image_type: str = Field(..., description="The type of image content")
|
||||
short_description: str = Field(
|
||||
...,
|
||||
description="Short natural-language description",
|
||||
)
|
||||
summary: str = Field(..., description="Detailed summary of the image content")
|
||||
|
||||
|
||||
# Define schema for document-level annotations
|
||||
class DocumentAnnotation(BaseModel):
|
||||
language: str = Field(..., description="Primary language of the document")
|
||||
chapter_titles: List[str] = Field(
|
||||
...,
|
||||
description="Detected chapter or section titles",
|
||||
)
|
||||
urls: List[str] = Field(..., description="URLs found in the text")
|
||||
|
||||
|
||||
converter = MistralOCRDocumentConverter(
|
||||
api_key=Secret.from_env_var("MISTRAL_API_KEY"),
|
||||
model="mistral-ocr-2505",
|
||||
)
|
||||
|
||||
sources = [DocumentURLChunk(document_url="https://example.com/report.pdf")]
|
||||
result = converter.run(
|
||||
sources=sources,
|
||||
bbox_annotation_schema=ImageAnnotation,
|
||||
document_annotation_schema=DocumentAnnotation,
|
||||
)
|
||||
|
||||
documents = result["documents"]
|
||||
# Document metadata will include:
|
||||
# - source_language: extracted from DocumentAnnotation
|
||||
# - source_chapter_titles: extracted from DocumentAnnotation
|
||||
# - source_urls: extracted from DocumentAnnotation
|
||||
# Document content will include inline image annotations
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Here's an example of an indexing pipeline that processes PDFs with OCR and writes them to a Document Store:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.components.converters.mistral import (
|
||||
MistralOCRDocumentConverter,
|
||||
)
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component(
|
||||
"converter",
|
||||
MistralOCRDocumentConverter(
|
||||
api_key=Secret.from_env_var("MISTRAL_API_KEY"),
|
||||
model="mistral-ocr-2505",
|
||||
),
|
||||
)
|
||||
pipeline.add_component("splitter", DocumentSplitter(split_by="page", split_length=1))
|
||||
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
|
||||
pipeline.connect("converter", "splitter")
|
||||
pipeline.connect("splitter", "writer")
|
||||
|
||||
file_paths = ["invoice.pdf", "receipt.jpg", "contract.pdf"]
|
||||
pipeline.run({"converter": {"sources": file_paths}})
|
||||
```
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
title: "MSGToDocument"
|
||||
id: msgtodocument
|
||||
slug: "/msgtodocument"
|
||||
description: "Converts Microsoft Outlook .msg files to documents."
|
||||
---
|
||||
|
||||
# MSGToDocument
|
||||
|
||||
Converts Microsoft Outlook .msg files to documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [PreProcessors](../preprocessors.mdx) , or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory run variables** | `sources`: A list of .msg file paths or [ByteStream](../../concepts/data-classes.mdx#bytestream) objects |
|
||||
| **Output variables** | `documents`: A list of documents <br /> <br />`attachments`: A list of ByteStream objects representing file attachments |
|
||||
| **API reference** | [Converters](/reference/converters-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/converters/msg.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `MSGToDocument` component converts Microsoft Outlook `.msg` files into documents. This component extracts the email metadata (such as sender, recipients, CC, BCC, subject) and body content. Additionally, any file attachments within the `.msg` file are extracted as `ByteStream` objects.
|
||||
|
||||
## Usage
|
||||
|
||||
First, install the `python-oxmsg` package to start using this converter:
|
||||
|
||||
```
|
||||
pip install python-oxmsg
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.components.converters.msg import MSGToDocument
|
||||
from datetime import datetime
|
||||
|
||||
converter = MSGToDocument()
|
||||
results = converter.run(
|
||||
sources=["sample.msg"],
|
||||
meta={"date_added": datetime.now().isoformat()},
|
||||
)
|
||||
documents = results["documents"]
|
||||
attachments = results["attachments"]
|
||||
|
||||
print(documents[0].content)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
The following setup enables efficient extraction, preprocessing, and indexing of `.msg` email files within a Haystack pipeline:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.routers import FileTypeRouter
|
||||
from haystack.components.converters import MSGToDocument
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
router = FileTypeRouter(mime_types=["application/vnd.ms-outlook"])
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("router", router)
|
||||
pipeline.add_component("converter", MSGToDocument())
|
||||
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
|
||||
pipeline.connect("router.application/vnd.ms-outlook", "converter.sources")
|
||||
pipeline.connect("converter.documents", "writer.documents")
|
||||
|
||||
file_names = ["email1.msg", "email2.msg"]
|
||||
pipeline.run({"converter": {"sources": file_names}})
|
||||
```
|
||||
@@ -0,0 +1,80 @@
|
||||
---
|
||||
title: "MultiFileConverter"
|
||||
id: multifileconverter
|
||||
slug: "/multifileconverter"
|
||||
description: "Converts CSV, DOCX, HTML, JSON, MD, PPTX, PDF, TXT, and XSLX files to documents."
|
||||
---
|
||||
|
||||
# MultiFileConverter
|
||||
|
||||
Converts CSV, DOCX, HTML, JSON, MD, PPTX, PDF, TXT, and XSLX files to documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before PreProcessors , or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory run variables** | `sources`: A list of file paths or ByteStream objects |
|
||||
| **Output variables** | `documents`: A list of converted documents <br /> <br />`unclassified`: A list of uncategorized file paths or byte streams |
|
||||
| **API reference** | [Converters](/reference/converters-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/converters/multi_file_converter.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`MultiFileConverter` converts input files of various file types into documents.
|
||||
|
||||
It is a SuperComponent that combines a [`FileTypeRouter`](../routers/filetyperouter.mdx), nine converters and a [`DocumentJoiner`](../joiners/documentjoiner.mdx) into a single component.
|
||||
|
||||
### Parameters
|
||||
|
||||
To initialize `MultiFileConverter`, there are no mandatory parameters. Optionally, you can provide `encoding` and `json_content_key` parameters.
|
||||
|
||||
The `json_content_key` parameter lets you specify for the JSON files which key in the extracted data will be the document's content. The parameter is passed on to the underlying [`JSONConverter`](jsonconverter.mdx) component.
|
||||
|
||||
The `encoding` parameter lets you specify the default encoding of the TXT, CSV, and MD files. If you don't provide any value, the component uses `utf-8` by default. Note that if the encoding is specified in the metadata of an input ByteStream, it will override this parameter's setting. The parameter is passed on to the underlying [`TextFileToDocument`](textfiletodocument.mdx) and [`CSVToDocument`](csvtodocument.mdx) components.
|
||||
|
||||
## Usage
|
||||
|
||||
Install dependencies for all supported file types to use the `MultiFileConverter`:
|
||||
|
||||
```shell
|
||||
pip install pypdf markdown-it-py mdit_plain trafilatura python-pptx python-docx jq openpyxl tabulate pandas
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.components.converters import MultiFileConverter
|
||||
|
||||
converter = MultiFileConverter()
|
||||
converter.run(sources=["test.txt", "test.pdf"], meta={})
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
You can also use `MultiFileConverter` in your indexing pipeline.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.converters import MultiFileConverter
|
||||
from haystack.components.preprocessors import DocumentPreprocessor
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("converter", MultiFileConverter())
|
||||
pipeline.add_component("preprocessor", DocumentPreprocessor())
|
||||
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
pipeline.connect("converter", "preprocessor")
|
||||
pipeline.connect("preprocessor", "writer")
|
||||
|
||||
result = pipeline.run(data={"sources": ["test.txt", "test.pdf"]})
|
||||
|
||||
print(result)
|
||||
# {'writer': {'documents_written': 3}}
|
||||
```
|
||||
@@ -0,0 +1,148 @@
|
||||
---
|
||||
title: "OpenAPIServiceToFunctions"
|
||||
id: openapiservicetofunctions
|
||||
slug: "/openapiservicetofunctions"
|
||||
description: "`OpenAPIServiceToFunctions` is a component that transforms OpenAPI service specifications into a format compatible with LLM tool calling."
|
||||
---
|
||||
|
||||
# OpenAPIServiceToFunctions
|
||||
|
||||
`OpenAPIServiceToFunctions` is a component that transforms OpenAPI service specifications into a format compatible with LLM tool calling.
|
||||
|
||||
:::tip[Consider using MCP instead]
|
||||
|
||||
These OpenAPI components are a legacy way to connect Haystack to external APIs. For most use cases, we recommend the [`MCPTool`](../../tools/mcptool.mdx) instead: it is the modern, standardized way to give your pipelines and agents access to external tools and services.
|
||||
|
||||
:::
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Flexible |
|
||||
| **Mandatory run variables** | `sources`: A list of OpenAPI specification sources, which can be file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects |
|
||||
| **Output variables** | `functions`: A list of JSON function definitions objects. For each path definition in OpenAPI specification, a corresponding function definition is generated. <br /> <br />`openapi_specs`: A list of JSON/YAML objects with references resolved. Such OpenAPI spec (with references resolved) can, in turn, be used as input to OpenAPIServiceConnector. |
|
||||
| **API reference** | [OpenAPI](/reference/integrations-openapi) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/openapi |
|
||||
| **Package name** | `openapi-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`OpenAPIServiceToFunctions` transforms OpenAPI service specifications into a function calling format suitable for LLM tool calling. It takes an OpenAPI specification, processes it to extract function definitions, and formats these definitions to be compatible with LLM tool calling.
|
||||
|
||||
`OpenAPIServiceToFunctions` is valuable when used together with [`OpenAPIServiceConnector`](../connectors/openapiserviceconnector.mdx) component. It converts OpenAPI specifications into function definitions, allowing `OpenAPIServiceConnector` to handle input parameters for the OpenAPI specification and facilitate their use in REST API calls through `OpenAPIServiceConnector`.
|
||||
|
||||
To use `OpenAPIServiceToFunctions`, you need to install the `openapi-haystack` package with:
|
||||
|
||||
```shell
|
||||
pip install openapi-haystack
|
||||
```
|
||||
|
||||
`OpenAPIServiceToFunctions` component doesn’t have any init parameters.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
This component is primarily meant to be used in pipelines. Using this component alone is useful when you want to convert OpenAPI specification into function definitions and then perhaps save them in a file and subsequently use them for tool calling.
|
||||
|
||||
### In a pipeline
|
||||
|
||||
In a pipeline context, `OpenAPIServiceToFunctions` is most valuable when used alongside `OpenAPIServiceConnector`. For instance, let’s consider integrating [serper.dev](http://serper.dev/) search engine bridge into a pipeline. `OpenAPIServiceToFunctions` retrieves the OpenAPI specification of Serper from https://bit.ly/serper_dev_spec, converts this specification into function definitions that an LLM with tool calling capabilities can understand, and then seamlessly passes these definitions as `generation_kwargs` to the Chat Generator component.
|
||||
|
||||
:::info
|
||||
To run the following code snippet, note that you have to have your own Serper and OpenAI API keys.
|
||||
:::
|
||||
|
||||
```python
|
||||
import json
|
||||
import requests
|
||||
|
||||
from typing import Any
|
||||
|
||||
from haystack import Pipeline
|
||||
from haystack.components.converters import OutputAdapter
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.dataclasses.byte_stream import ByteStream
|
||||
from haystack_integrations.components.connectors.openapi import OpenAPIServiceConnector
|
||||
from haystack_integrations.components.converters.openapi import (
|
||||
OpenAPIServiceToFunctions,
|
||||
)
|
||||
|
||||
|
||||
def prepare_fc_params(openai_functions_schema: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
"tools": [{"type": "function", "function": openai_functions_schema}],
|
||||
"tool_choice": {
|
||||
"type": "function",
|
||||
"function": {"name": openai_functions_schema["name"]},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
serperdev_spec = requests.get("https://bit.ly/serper_dev_spec").json()
|
||||
system_prompt = requests.get("https://bit.ly/serper_dev_system").text
|
||||
user_prompt = "Why was Sam Altman ousted from OpenAI?"
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("spec_to_functions", OpenAPIServiceToFunctions())
|
||||
pipe.add_component(
|
||||
"prepare_fc_adapter",
|
||||
OutputAdapter(
|
||||
"{{functions[0] | prepare_fc}}",
|
||||
dict[str, Any],
|
||||
{"prepare_fc": prepare_fc_params},
|
||||
),
|
||||
)
|
||||
pipe.add_component("functions_llm", OpenAIChatGenerator())
|
||||
pipe.add_component("openapi_connector", OpenAPIServiceConnector())
|
||||
pipe.add_component(
|
||||
"message_adapter",
|
||||
OutputAdapter(
|
||||
"{{system_message + service_response}}",
|
||||
list[ChatMessage],
|
||||
unsafe=True,
|
||||
),
|
||||
)
|
||||
pipe.add_component("llm", OpenAIChatGenerator())
|
||||
|
||||
pipe.connect("spec_to_functions.functions", "prepare_fc_adapter.functions")
|
||||
pipe.connect(
|
||||
"spec_to_functions.openapi_specs",
|
||||
"openapi_connector.service_openapi_spec",
|
||||
)
|
||||
pipe.connect("prepare_fc_adapter", "functions_llm.generation_kwargs")
|
||||
pipe.connect("functions_llm.replies", "openapi_connector.messages")
|
||||
pipe.connect("openapi_connector.service_response", "message_adapter.service_response")
|
||||
pipe.connect("message_adapter", "llm.messages")
|
||||
|
||||
result = pipe.run(
|
||||
data={
|
||||
"functions_llm": {
|
||||
"messages": [
|
||||
ChatMessage.from_system("Only do tool/function calling"),
|
||||
ChatMessage.from_user(user_prompt),
|
||||
],
|
||||
},
|
||||
"openapi_connector": {
|
||||
"service_credentials": serper_dev_key,
|
||||
},
|
||||
"spec_to_functions": {
|
||||
"sources": [ByteStream.from_string(json.dumps(serperdev_spec))],
|
||||
},
|
||||
"message_adapter": {
|
||||
"system_message": [ChatMessage.from_system(system_prompt)],
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
print(result["llm"]["replies"][0].text)
|
||||
|
||||
# Sam Altman was ousted from OpenAI on November 17, 2023, following
|
||||
# a "deliberative review process" by the board of directors. The board concluded
|
||||
# that he was not "consistently candid in his communications". However, he
|
||||
# returned as CEO just days after his ouster.
|
||||
```
|
||||
@@ -0,0 +1,135 @@
|
||||
---
|
||||
title: "OutputAdapter"
|
||||
id: outputadapter
|
||||
slug: "/outputadapter"
|
||||
description: "This component helps the output of one component fit smoothly into the input of another. It uses Jinja expressions to define how this adaptation occurs."
|
||||
---
|
||||
|
||||
# OutputAdapter
|
||||
|
||||
This component helps the output of one component fit smoothly into the input of another. It uses Jinja expressions to define how this adaptation occurs.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Flexible |
|
||||
| **Mandatory init variables** | `template`: A Jinja template string that defines how to adapt the data <br /> <br />`output_type`: Type alias that this instance will return |
|
||||
| **Mandatory run variables** | `**kwargs`: Input variables to be used in Jinja expression. See [Variables](#variables) section for more details. |
|
||||
| **Output variables** | The output is specified under the `output` key dictionary |
|
||||
| **API reference** | [Converters](/reference/converters-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/converters/output_adapter.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
To use `OutputAdapter`, you need to specify the adaptation rule that includes:
|
||||
|
||||
- `template`: A Jinja template string that defines how to adapt the input data.
|
||||
- `output_type`: The type of the output data (such as `str`, `List[int]`..). This doesn't change the actual output type and is only needed to validate connection with other components.
|
||||
- `custom_filters`: An optional dictionary of custom Jinja filters to be used in the template.
|
||||
|
||||
### Variables
|
||||
|
||||
The `OutputAdapter` requires all template variables to be present before running and raises an error if any template variable is missing at pipeline connect time.
|
||||
|
||||
```python
|
||||
from haystack.components.converters import OutputAdapter
|
||||
|
||||
adapter = OutputAdapter(template="Hello {{name}}!", output_type=str)
|
||||
```
|
||||
|
||||
### Unsafe behavior
|
||||
|
||||
The `OutputAdapter` internally renders the `template` using Jinja, and by default, this is safe behavior. However, it limits the output types to strings, bytes, numbers, tuples, lists, dicts, sets, booleans, `None`, and `Ellipsis` (`...`), as well as any combination of these structures.
|
||||
|
||||
If you want to use other types such as `ChatMessage`, `Document`, or `Answer`, you must enable unsafe template rendering by setting the `unsafe` init argument to `True`.
|
||||
|
||||
Be cautious, as enabling this can be unsafe and may lead to remote code execution if the `template` is a string customizable by the end user.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
This component is primarily meant to be used in pipelines.
|
||||
|
||||
In this example, `OutputAdapter` simply outputs the content field of the first document in the arrays of documents:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.converters import OutputAdapter
|
||||
|
||||
adapter = OutputAdapter(template="{{ documents[0].content }}", output_type=str)
|
||||
input_data = {"documents": [Document(content="Test content")]}
|
||||
expected_output = {"output": "Test content"}
|
||||
assert adapter.run(**input_data) == expected_output
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
The example below demonstrates a straightforward pipeline that uses the `OutputAdapter` to capitalize the first document in the list. If needed, you can also utilize the predefined Jinja [filters](https://jinja.palletsprojects.com/en/3.1.x/templates/#builtin-filters).
|
||||
|
||||
```python
|
||||
from haystack import Pipeline, component, Document
|
||||
from haystack.components.converters import OutputAdapter
|
||||
|
||||
|
||||
@component
|
||||
class DocumentProducer:
|
||||
@component.output_types(documents=dict)
|
||||
def run(self):
|
||||
return {"documents": [Document(content="haystack")]}
|
||||
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component(
|
||||
name="output_adapter",
|
||||
instance=OutputAdapter(
|
||||
template="{{ documents[0].content | capitalize}}",
|
||||
output_type=str,
|
||||
),
|
||||
)
|
||||
pipe.add_component(name="document_producer", instance=DocumentProducer())
|
||||
pipe.connect("document_producer", "output_adapter")
|
||||
result = pipe.run(data={})
|
||||
|
||||
assert result["output_adapter"]["output"] == "Haystack"
|
||||
```
|
||||
|
||||
You can also define your own custom filters, which can then be added to an `OutputAdapter` instance through its init method and used in templates. Here’s an example of this approach:
|
||||
|
||||
```python
|
||||
|
||||
from haystack import Pipeline, component, Document
|
||||
from haystack.components.converters import OutputAdapter
|
||||
|
||||
|
||||
def reverse_string(s):
|
||||
return s[::-1]
|
||||
|
||||
|
||||
@component
|
||||
class DocumentProducer:
|
||||
@component.output_types(documents=dict)
|
||||
def run(self):
|
||||
return {"documents": [Document(content="haystack")]}
|
||||
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component(
|
||||
name="output_adapter",
|
||||
instance=OutputAdapter(
|
||||
template="{{ documents[0].content | reverse_string}}",
|
||||
output_type=str,
|
||||
custom_filters={"reverse_string": reverse_string},
|
||||
),
|
||||
)
|
||||
|
||||
pipe.add_component(name="document_producer", instance=DocumentProducer())
|
||||
pipe.connect("document_producer", "output_adapter")
|
||||
result = pipe.run(data={})
|
||||
|
||||
assert result["output_adapter"]["output"] == "kcatsyah"
|
||||
```
|
||||
@@ -0,0 +1,157 @@
|
||||
---
|
||||
title: "PaddleOCRVLDocumentConverter"
|
||||
id: paddleocrvldocumentconverter
|
||||
slug: "/paddleocrvldocumentconverter"
|
||||
description: "`PaddleOCRVLDocumentConverter` extracts text from documents using PaddleOCR's large model document parsing API."
|
||||
---
|
||||
|
||||
# PaddleOCRVLDocumentConverter
|
||||
|
||||
`PaddleOCRVLDocumentConverter` extracts text from documents using PaddleOCR's large model document parsing API. PaddleOCR-VL is used behind the scenes. For more information, please refer to the [PaddleOCR-VL documentation](https://www.paddleocr.ai/latest/en/version3.x/algorithm/PaddleOCR-VL/PaddleOCR-VL.html).
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [PreProcessors](../preprocessors.mdx), or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory init variables** | `api_url`: The URL of the PaddleOCR-VL API. <br /> <br /> `access_token`: The AI Studio access token. Can be set with `AISTUDIO_ACCESS_TOKEN` environment variable. |
|
||||
| **Mandatory run variables** | `sources`: A list of image or PDF file paths or ByteStream objects. |
|
||||
| **Output variables** | `documents`: A list of documents. <br /> <br />`raw_paddleocr_responses`: A list of raw OCR responses from PaddleOCR API. |
|
||||
| **API reference** | [PaddleOCR](/reference/integrations-paddleocr) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/paddleocr |
|
||||
| **Package name** | `paddleocr-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `PaddleOCRVLDocumentConverter` takes a list of document sources and uses PaddleOCR's large model document parsing API to extract text from images and PDFs. It supports both images and PDF files.
|
||||
|
||||
The component returns one Haystack [`Document`](../../concepts/data-classes.mdx#document) per source, with all pages concatenated using form feed characters (`\f`) as separators. This format ensures compatibility with Haystack's [`DocumentSplitter`](../preprocessors/documentsplitter.mdx) for accurate page-wise splitting and overlap handling. The content is returned in markdown format, with images represented as `` tags.
|
||||
|
||||
The component takes `api_url` as a required parameter. To obtain the API URL, visit the [PaddleOCR official website](https://aistudio.baidu.com/paddleocr), click the **API** button, choose the example code for PaddleOCR-VL, and copy the `API_URL`.
|
||||
|
||||
By default, the component uses the `AISTUDIO_ACCESS_TOKEN` environment variable for authentication. You can also pass an `access_token` at initialization. The AI Studio access token can be obtained from [this page](https://aistudio.baidu.com/account/accessToken).
|
||||
|
||||
`raw_paddleocr_responses` can be useful while tuning layout thresholds, prompt settings, or Markdown post-processing options because it gives you access to the original API output alongside the converted Haystack documents.
|
||||
|
||||
:::note
|
||||
This component returns Markdown content. Avoid piping it through `DocumentCleaner()` with its default settings because `remove_extra_whitespaces=True` and `remove_empty_lines=True` can collapse line breaks and flatten headings, tables, and image tags. For page-aware chunking, connect the converter directly to `DocumentSplitter`, or disable those options if you need custom cleanup.
|
||||
:::
|
||||
|
||||
## When to use it
|
||||
|
||||
`PaddleOCRVLDocumentConverter` is a strong fit when you need more than plain OCR text:
|
||||
|
||||
- **Scanned PDFs and camera-captured documents** where page orientation and warped text can reduce extraction quality.
|
||||
- **Layout-sensitive documents** such as invoices, reports, forms, and multi-column PDFs where preserving structure matters for downstream chunking and retrieval.
|
||||
- **Tables, formulas, charts, or seals** where you want more targeted extraction behavior than plain text OCR.
|
||||
- **RAG ingestion pipelines** where Markdown output is useful because headings, lists, tables, and page breaks can be preserved for later splitting.
|
||||
|
||||
## Useful configuration areas
|
||||
|
||||
The full parameter list is available in the [API reference](/reference/integrations-paddleocr). In practice, the most useful options tend to fall into these groups:
|
||||
|
||||
- **Input handling and image cleanup**: `file_type`, `use_doc_orientation_classify`, and `use_doc_unwarping` help when you mix PDFs and images or work with skewed scans and mobile photos.
|
||||
- **Layout-aware extraction**: `use_layout_detection`, `layout_threshold`, `layout_nms`, `layout_unclip_ratio`, `layout_merge_bboxes_mode`, `layout_shape_mode`, and `merge_layout_blocks` help you tune how regions are detected and merged before Markdown is generated.
|
||||
- **Content focus**: `prompt_label`, `use_ocr_for_image_block`, `use_chart_recognition`, and `use_seal_recognition` let you bias extraction toward a particular type of content, such as plain OCR, formulas, tables, charts, or seals.
|
||||
- **Markdown output shaping**: `format_block_content`, `markdown_ignore_labels`, `prettify_markdown`, `show_formula_number`, `restructure_pages`, `merge_tables`, and `relevel_titles` help you control how much cleanup and restructuring happens before the result becomes a Haystack document.
|
||||
- **VLM generation controls**: `repetition_penalty`, `temperature`, `top_p`, `min_pixels`, `max_pixels`, `max_new_tokens`, `vlm_extra_args`, and `additional_params` are useful when you need to trade off output quality, determinism, and cost.
|
||||
- **Debugging and inspection**: `visualize=True` and the returned `raw_paddleocr_responses` are helpful when you are tuning extraction quality for a new document type.
|
||||
|
||||
## Typical scenarios
|
||||
|
||||
These settings are especially useful in a few common workflows:
|
||||
|
||||
- **Scanned contracts or receipts from phones**: start with `use_doc_orientation_classify=True` and `use_doc_unwarping=True`.
|
||||
- **Table-heavy financial or operations PDFs**: consider `use_layout_detection=True`, `merge_tables=True`, and `restructure_pages=True`.
|
||||
- **Formula-heavy documents**: use `prompt_label="formula"` together with `show_formula_number=True` if formula numbering matters in the final Markdown.
|
||||
- **Mixed business documents with figures or seals**: enable `use_chart_recognition=True`, `use_seal_recognition=True`, or `use_ocr_for_image_block=True` depending on the content you want to preserve.
|
||||
|
||||
## Usage
|
||||
|
||||
You need to install the `paddleocr-haystack` integration to use `PaddleOCRVLDocumentConverter`:
|
||||
|
||||
```shell
|
||||
pip install paddleocr-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage with a local file:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.components.converters.paddleocr import (
|
||||
PaddleOCRVLDocumentConverter,
|
||||
)
|
||||
|
||||
converter = PaddleOCRVLDocumentConverter(
|
||||
api_url="<your-api-url>",
|
||||
access_token=Secret.from_env_var("AISTUDIO_ACCESS_TOKEN"),
|
||||
)
|
||||
|
||||
result = converter.run(sources=[Path("my_document.pdf")])
|
||||
documents = result["documents"]
|
||||
```
|
||||
|
||||
Advanced configuration for structure-heavy PDFs:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.components.converters.paddleocr import (
|
||||
PaddleOCRVLDocumentConverter,
|
||||
)
|
||||
|
||||
converter = PaddleOCRVLDocumentConverter(
|
||||
api_url="<your-api-url>",
|
||||
access_token=Secret.from_env_var("AISTUDIO_ACCESS_TOKEN"),
|
||||
use_doc_orientation_classify=True,
|
||||
use_doc_unwarping=True,
|
||||
use_layout_detection=True,
|
||||
use_ocr_for_image_block=True,
|
||||
merge_tables=True,
|
||||
restructure_pages=True,
|
||||
prettify_markdown=True,
|
||||
)
|
||||
|
||||
result = converter.run(sources=[Path("quarterly_report.pdf")])
|
||||
documents = result["documents"]
|
||||
raw_responses = result["raw_paddleocr_responses"]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Here's an example of an indexing pipeline that processes PDFs with OCR and writes them to a Document Store:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.components.converters.paddleocr import (
|
||||
PaddleOCRVLDocumentConverter,
|
||||
)
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component(
|
||||
"converter",
|
||||
PaddleOCRVLDocumentConverter(
|
||||
api_url="<your-api-url>",
|
||||
access_token=Secret.from_env_var("AISTUDIO_ACCESS_TOKEN"),
|
||||
),
|
||||
)
|
||||
pipeline.add_component("splitter", DocumentSplitter(split_by="page", split_length=1))
|
||||
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
|
||||
pipeline.connect("converter", "splitter")
|
||||
pipeline.connect("splitter", "writer")
|
||||
|
||||
file_paths = ["invoice.pdf", "receipt.jpg", "contract.pdf"]
|
||||
pipeline.run({"converter": {"sources": file_paths}})
|
||||
```
|
||||
@@ -0,0 +1,83 @@
|
||||
---
|
||||
title: "PDFMinerToDocument"
|
||||
id: pdfminertodocument
|
||||
slug: "/pdfminertodocument"
|
||||
description: "A component that converts complex PDF files to documents using pdfminer arguments."
|
||||
---
|
||||
|
||||
# PDFMinerToDocument
|
||||
|
||||
A component that converts complex PDF files to documents using pdfminer arguments.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [PreProcessors](../preprocessors.mdx) or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory run variables** | `sources`: PDF file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Converters](/reference/converters-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/converters/pdfminer.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `PDFMinerToDocument` component converts PDF files into documents using [PDFMiner](https://pdfminersix.readthedocs.io/en/latest/) extraction tool arguments.
|
||||
|
||||
You can use it in an indexing pipeline to index the contents of a PDF file in a Document Store. It takes a list of file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream)objects as input and outputs the converted result as a list of documents. Optionally, you can attach metadata to the documents through the `meta` input parameter.
|
||||
|
||||
When initializing the component, you can adjust several parameters to fit your PDF. See the full parameter list and descriptions in our [API reference](/reference/converters-api#pdfminertodocument).
|
||||
|
||||
## Usage
|
||||
|
||||
First, install `pdfminer` package to start using this converter:
|
||||
|
||||
```shell
|
||||
pip install pdfminer.six
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.components.converters import PDFMinerToDocument
|
||||
|
||||
converter = PDFMinerToDocument()
|
||||
results = converter.run(
|
||||
sources=["sample.pdf"],
|
||||
meta={"date_added": datetime.now().isoformat()},
|
||||
)
|
||||
documents = results["documents"]
|
||||
|
||||
print(documents[0].content)
|
||||
|
||||
# 'This is a text from the PDF file.'
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters import PDFMinerToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("converter", PDFMinerToDocument())
|
||||
pipeline.add_component("cleaner", DocumentCleaner())
|
||||
pipeline.add_component(
|
||||
"splitter",
|
||||
DocumentSplitter(split_by="sentence", split_length=5),
|
||||
)
|
||||
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
pipeline.connect("converter", "cleaner")
|
||||
pipeline.connect("cleaner", "splitter")
|
||||
pipeline.connect("splitter", "writer")
|
||||
|
||||
pipeline.run({"converter": {"sources": file_names}})
|
||||
```
|
||||
@@ -0,0 +1,117 @@
|
||||
---
|
||||
title: "PDFToImageContent"
|
||||
id: pdftoimagecontent
|
||||
slug: "/pdftoimagecontent"
|
||||
description: "`PDFToImageContent` reads local PDF files and converts them into `ImageContent` objects. These are ready for multimodal AI pipelines, including tasks like image captioning, visual QA, or prompt-based generation."
|
||||
---
|
||||
|
||||
# PDFToImageContent
|
||||
|
||||
`PDFToImageContent` reads local PDF files and converts them into `ImageContent` objects. These are ready for multimodal AI pipelines, including tasks like image captioning, visual QA, or prompt-based generation.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a `ChatPromptBuilder` in a query pipeline |
|
||||
| **Mandatory run variables** | `sources`: A list of PDF file paths or ByteStreams |
|
||||
| **Output variables** | `image_contents`: A list of ImageContent objects |
|
||||
| **API reference** | [Image Converters](/reference/image-converters-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/converters/image/pdf_to_image.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`PDFToImageContent` processes a list of PDF sources and converts them into `ImageContent` objects, one for each page of the PDF. These can be used in multimodal pipelines that require base64-encoded image input.
|
||||
|
||||
Each source can be:
|
||||
|
||||
- A file path (string or `Path`), or
|
||||
- A `ByteStream` object.
|
||||
|
||||
Optionally, you can provide metadata using the `meta` parameter. This can be a single dictionary (applied to all images) or a list matching the length of `sources`.
|
||||
|
||||
Use the `size` parameter to resize images while preserving aspect ratio. This reduces memory usage and transmission size, which is helpful when working with remote models or limited-resource environments.
|
||||
|
||||
This component is often used in query pipelines just before a `ChatPromptBuilder`.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.components.converters.image import PDFToImageContent
|
||||
|
||||
converter = PDFToImageContent()
|
||||
|
||||
sources = ["file.pdf", "another_file.pdf"]
|
||||
|
||||
image_contents = converter.run(sources=sources)["image_contents"]
|
||||
print(image_contents)
|
||||
|
||||
# [ImageContent(base64_image='...',
|
||||
# mime_type='application/pdf',
|
||||
# detail=None,
|
||||
# meta={'file_path': 'file.pdf', 'page_number': 1}),
|
||||
# ...]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Use `ImageFileToImageContent` to supply image data to a `ChatPromptBuilder` for multimodal QA or captioning with an LLM.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.components.converters.image import PDFToImageContent
|
||||
|
||||
# Query pipeline
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("image_converter", PDFToImageContent(detail="auto"))
|
||||
pipeline.add_component(
|
||||
"chat_prompt_builder",
|
||||
ChatPromptBuilder(
|
||||
required_variables=["question"],
|
||||
template="""{% message role="system" %}
|
||||
You are a helpful assistant that answers questions using the provided images.
|
||||
{% endmessage %}
|
||||
|
||||
{% message role="user" %}
|
||||
Question: {{ question }}
|
||||
|
||||
{% for img in image_contents %}
|
||||
{{ img | templatize_part }}
|
||||
{% endfor %}
|
||||
{% endmessage %}
|
||||
""",
|
||||
),
|
||||
)
|
||||
pipeline.add_component("llm", OpenAIChatGenerator(model="gpt-4o-mini"))
|
||||
|
||||
pipeline.connect("image_converter", "chat_prompt_builder.image_contents")
|
||||
pipeline.connect("chat_prompt_builder", "llm")
|
||||
|
||||
sources = ["flan_paper.pdf"]
|
||||
|
||||
result = pipeline.run(
|
||||
data={
|
||||
"image_converter": {"sources": ["flan_paper.pdf"], "page_range": "9"},
|
||||
"chat_prompt_builder": {"question": "What is the main takeaway of Figure 6?"},
|
||||
},
|
||||
)
|
||||
print(result["replies"][0].text)
|
||||
|
||||
# ('The main takeaway of Figure 6 is that Flan-PaLM demonstrates improved '
|
||||
# 'performance in zero-shot reasoning tasks when utilizing chain-of-thought '
|
||||
# '(CoT) reasoning, as indicated by higher accuracy across different model '
|
||||
# 'sizes compared to PaLM without finetuning. This highlights the importance of '
|
||||
# 'instruction finetuning combined with CoT for enhancing reasoning '
|
||||
# 'capabilities in models.')
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Introduction to Multimodality](https://haystack.deepset.ai/cookbook/multimodal_intro)
|
||||
@@ -0,0 +1,79 @@
|
||||
---
|
||||
title: "PPTXToDocument"
|
||||
id: pptxtodocument
|
||||
slug: "/pptxtodocument"
|
||||
description: "Convert PPTX files to documents."
|
||||
---
|
||||
|
||||
# PPTXToDocument
|
||||
|
||||
Convert PPTX files to documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [PreProcessors](../preprocessors.mdx) or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory run variables** | `sources`: PPTX file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Converters](/reference/converters-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/converters/pptx.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `PPTXToDocument` component converts PPTX files into documents. It takes a list of file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects as input and outputs the converted result as a list of documents. Optionally, you can attach metadata to the documents through the `meta` input parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
First, install the`python-pptx` package to start using this converter:
|
||||
|
||||
```shell
|
||||
pip install python-pptx
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.components.converters import PPTXToDocument
|
||||
|
||||
converter = PPTXToDocument()
|
||||
results = converter.run(
|
||||
sources=["sample.pptx"],
|
||||
meta={"date_added": datetime.now().isoformat()},
|
||||
)
|
||||
documents = results["documents"]
|
||||
|
||||
print(documents[0].content)
|
||||
|
||||
# 'This is the text from the PPTX file.'
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters import PPTXToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("converter", PPTXToDocument())
|
||||
pipeline.add_component("cleaner", DocumentCleaner())
|
||||
pipeline.add_component(
|
||||
"splitter",
|
||||
DocumentSplitter(split_by="sentence", split_length=5),
|
||||
)
|
||||
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
pipeline.connect("converter", "cleaner")
|
||||
pipeline.connect("cleaner", "splitter")
|
||||
pipeline.connect("splitter", "writer")
|
||||
|
||||
pipeline.run({"converter": {"sources": file_names}})
|
||||
```
|
||||
@@ -0,0 +1,79 @@
|
||||
---
|
||||
title: "PyPDFToDocument"
|
||||
id: pypdftodocument
|
||||
slug: "/pypdftodocument"
|
||||
description: "A component that converts PDF files to Documents."
|
||||
---
|
||||
|
||||
# PyPDFToDocument
|
||||
|
||||
A component that converts PDF files to Documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [PreProcessors](../preprocessors.mdx) , or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory run variables** | `sources`: PDF file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Converters](/reference/converters-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/converters/pypdf.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `PyPDFToDocument` component converts PDF files into documents. You can use it in an indexing pipeline to index the contents of a PDF file into a Document Store. It takes a list of file paths or [ByteStream](../../concepts/data-classes.mdx#bytestream) objects as input and outputs the converted result as a list of documents. Optionally, you can attach metadata to the documents through the `meta` input parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
You need to install `pypdf` package to use the `PyPDFToDocument` converter:
|
||||
|
||||
```shell
|
||||
pip install pypdf
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
from haystack.components.converters import PyPDFToDocument
|
||||
|
||||
converter = PyPDFToDocument()
|
||||
|
||||
docs = converter.run(sources=[Path("my_file.pdf")])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters import PyPDFToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("converter", PyPDFToDocument())
|
||||
pipeline.add_component("cleaner", DocumentCleaner())
|
||||
pipeline.add_component(
|
||||
"splitter",
|
||||
DocumentSplitter(split_by="sentence", split_length=5),
|
||||
)
|
||||
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
pipeline.connect("converter", "cleaner")
|
||||
pipeline.connect("cleaner", "splitter")
|
||||
pipeline.connect("splitter", "writer")
|
||||
|
||||
pipeline.run({"converter": {"sources": file_names}})
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [PDF-Based Question Answering with Amazon Bedrock and Haystack](https://haystack.deepset.ai/cookbook/amazon_bedrock_for_documentation_qa)
|
||||
|
||||
📓 Tutorial: [Preprocessing Different File Types](https://haystack.deepset.ai/tutorials/30_file_type_preprocessing_index_pipeline)
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
title: "TextFileToDocument"
|
||||
id: textfiletodocument
|
||||
slug: "/textfiletodocument"
|
||||
description: "Converts text files to documents."
|
||||
---
|
||||
|
||||
# TextFileToDocument
|
||||
|
||||
Converts text files to documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [PreProcessors](../preprocessors.mdx) or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory run variables** | `sources`: A list of paths to text files you want to convert |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Converters](/reference/converters-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/converters/txt.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `TextFileToDocument` component converts text files into documents. You can use it in an indexing pipeline to index the contents of text files into a Document Store. It takes a list of file paths or [ByteStream](../../concepts/data-classes.mdx#bytestream) objects as input and outputs the converted result as a list of documents. Optionally, you can attach metadata to the documents through the `meta` input parameter.
|
||||
|
||||
When you initialize the component, you can optionally set the default encoding of the text files through the `encoding` parameter. If you don't provide any value, the component uses `"utf-8"` by default. Note that if the encoding is specified in the metadata of an input ByteStream, it will override this parameter's setting.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
from haystack.components.converters import TextFileToDocument
|
||||
|
||||
converter = TextFileToDocument()
|
||||
|
||||
docs = converter.run(sources=[Path("my_file.txt")])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters import TextFileToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("converter", TextFileToDocument())
|
||||
pipeline.add_component("cleaner", DocumentCleaner())
|
||||
pipeline.add_component(
|
||||
"splitter",
|
||||
DocumentSplitter(split_by="sentence", split_length=5),
|
||||
)
|
||||
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
pipeline.connect("converter", "cleaner")
|
||||
pipeline.connect("cleaner", "splitter")
|
||||
pipeline.connect("splitter", "writer")
|
||||
|
||||
pipeline.run({"converter": {"sources": file_names}})
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
:notebook: Tutorial: [Preprocessing Different File Types](https://haystack.deepset.ai/tutorials/30_file_type_preprocessing_index_pipeline)
|
||||
@@ -0,0 +1,80 @@
|
||||
---
|
||||
title: "TikaDocumentConverter"
|
||||
id: tikadocumentconverter
|
||||
slug: "/tikadocumentconverter"
|
||||
description: "An integration for converting files of different types (PDF, DOCX, HTML, and more) to documents."
|
||||
---
|
||||
|
||||
# TikaDocumentConverter
|
||||
|
||||
An integration for converting files of different types (PDF, DOCX, HTML, and more) to documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [PreProcessors](../preprocessors.mdx) , or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory run variables** | `sources`: File paths |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Tika](/reference/integrations-tika) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/tika |
|
||||
| **Package name** | `tika-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `TikaDocumentConverter` component converts files of different types (pdf, docx, html, and others) into documents. You can use it in an indexing pipeline to index the contents of files into a Document Store. It takes a list of file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects as input and outputs the converted result as a list of documents. Optionally, you can attach metadata to the documents through the `meta` input parameter.
|
||||
|
||||
This integration uses [Apache Tika](https://tika.apache.org/) to parse the files and requires a running Tika server.
|
||||
|
||||
The easiest way to run Tika is by using Docker: `docker run -d -p 127.0.0.1:9998:9998 apache/tika:latest`.
|
||||
For more options on running Tika on Docker, see the [Tika documentation](https://github.com/apache/tika-docker/blob/main/README.md#usage).
|
||||
|
||||
When you initialize the `TikaDocumentConverter` component, you can specify a custom URL of the Tika server you are using through the parameter `tika_url`. The default URL is `"http://localhost:9998/tika"`.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `tika-haystack` package to use the `TikaDocumentConverter` component:
|
||||
|
||||
```shell
|
||||
pip install tika-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.converters.tika import TikaDocumentConverter
|
||||
from pathlib import Path
|
||||
|
||||
converter = TikaDocumentConverter()
|
||||
|
||||
converter.run(sources=[Path("my_file.pdf")])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.converters.tika import TikaDocumentConverter
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("converter", TikaDocumentConverter())
|
||||
pipeline.add_component("cleaner", DocumentCleaner())
|
||||
pipeline.add_component(
|
||||
"splitter",
|
||||
DocumentSplitter(split_by="sentence", split_length=5),
|
||||
)
|
||||
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
pipeline.connect("converter", "cleaner")
|
||||
pipeline.connect("cleaner", "splitter")
|
||||
pipeline.connect("splitter", "writer")
|
||||
|
||||
pipeline.run({"converter": {"sources": file_paths}})
|
||||
```
|
||||
@@ -0,0 +1,128 @@
|
||||
---
|
||||
title: "TwelveLabsVideoConverter"
|
||||
id: twelvelabsvideoconverter
|
||||
slug: "/twelvelabsvideoconverter"
|
||||
description: "`TwelveLabsVideoConverter` converts videos to Haystack Documents using the TwelveLabs Pegasus video-language model. Pegasus analyzes each video on the fly — its visuals and its own audio (via ASR) — and returns text, so each video becomes a Document whose content is the analysis."
|
||||
---
|
||||
|
||||
# TwelveLabsVideoConverter
|
||||
|
||||
`TwelveLabsVideoConverter` converts videos to Haystack Documents using the TwelveLabs Pegasus video-language model. Pegasus analyzes each video on the fly — its visuals **and** its own audio (via ASR) — and returns text, so each source video becomes one Document whose content is Pegasus's analysis (for example, a description plus a transcript). There is no frame extraction or separate transcription step.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | At the beginning of an indexing pipeline, before [PreProcessors](../preprocessors.mdx) or an embedder |
|
||||
| **Mandatory init variables** | `api_key`: The TwelveLabs API key. Can be set with `TWELVELABS_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `sources`: A list of video URLs or local file paths |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [TwelveLabs](/reference/integrations-twelvelabs) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/twelvelabs |
|
||||
| **Package name** | `twelvelabs-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `TwelveLabsVideoConverter` takes a list of video sources and produces one [`Document`](../../concepts/data-classes.mdx#document) per source, with the Document content set to Pegasus's text analysis. Sources may be publicly accessible direct video URLs or local file paths (uploaded to TwelveLabs, up to 200 MB). Sources that fail to process are skipped with a warning logged, so one bad source does not fail the whole batch.
|
||||
|
||||
Each produced Document carries metadata about the request, including `source`, `asset_id`, `analysis_id`, `model`, and `provider`. The default model is `pegasus1.5`.
|
||||
|
||||
You can steer the analysis with a custom `prompt` and tune `temperature` and `max_tokens`.
|
||||
|
||||
To start using this integration with Haystack, install the package with:
|
||||
|
||||
```shell
|
||||
pip install twelvelabs-haystack
|
||||
```
|
||||
|
||||
The component uses a `TWELVELABS_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`. To get an API key, head to [playground.twelvelabs.io](https://playground.twelvelabs.io).
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.converters.twelvelabs import (
|
||||
TwelveLabsVideoConverter,
|
||||
)
|
||||
|
||||
converter = TwelveLabsVideoConverter()
|
||||
result = converter.run(sources=["https://example.com/clip.mp4"])
|
||||
|
||||
document = result["documents"][0]
|
||||
print(document.content) # Pegasus's description + transcript of the video
|
||||
print(document.meta) # includes source, asset_id, analysis_id, model, provider
|
||||
```
|
||||
|
||||
:::info
|
||||
We recommend setting `TWELVELABS_API_KEY` as an environment variable instead of setting it as a parameter.
|
||||
:::
|
||||
|
||||
### With a custom prompt
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.converters.twelvelabs import (
|
||||
TwelveLabsVideoConverter,
|
||||
)
|
||||
|
||||
converter = TwelveLabsVideoConverter(
|
||||
prompt="Summarize this video in three bullet points and list any products shown.",
|
||||
temperature=0.2,
|
||||
max_tokens=1024,
|
||||
)
|
||||
result = converter.run(sources=["https://example.com/clip.mp4"])
|
||||
print(result["documents"][0].content)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
This indexing pipeline analyzes videos with Pegasus, embeds the resulting analysis with the [`TwelveLabsDocumentEmbedder`](../embedders/twelvelabsdocumentembedder.mdx), and writes the documents to a document store:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack_integrations.components.converters.twelvelabs import (
|
||||
TwelveLabsVideoConverter,
|
||||
)
|
||||
from haystack_integrations.components.embedders.twelvelabs import (
|
||||
TwelveLabsDocumentEmbedder,
|
||||
)
|
||||
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component("converter", TwelveLabsVideoConverter())
|
||||
indexing_pipeline.add_component("embedder", TwelveLabsDocumentEmbedder())
|
||||
indexing_pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
indexing_pipeline.connect("converter", "embedder")
|
||||
indexing_pipeline.connect("embedder", "writer")
|
||||
|
||||
indexing_pipeline.run({"converter": {"sources": ["https://example.com/clip.mp4"]}})
|
||||
```
|
||||
|
||||
### Attaching metadata
|
||||
|
||||
Pass a single dictionary to apply metadata to all output Documents, or a list to set metadata per source:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.converters.twelvelabs import (
|
||||
TwelveLabsVideoConverter,
|
||||
)
|
||||
|
||||
converter = TwelveLabsVideoConverter()
|
||||
|
||||
# Same metadata for all sources
|
||||
result = converter.run(
|
||||
sources=["https://example.com/a.mp4", "https://example.com/b.mp4"],
|
||||
meta={"campaign": "demo"},
|
||||
)
|
||||
|
||||
# Per-source metadata
|
||||
result = converter.run(
|
||||
sources=["https://example.com/a.mp4", "https://example.com/b.mp4"],
|
||||
meta=[{"title": "Clip A"}, {"title": "Clip B"}],
|
||||
)
|
||||
```
|
||||
@@ -0,0 +1,116 @@
|
||||
---
|
||||
title: "UnstructuredFileConverter"
|
||||
id: unstructuredfileconverter
|
||||
slug: "/unstructuredfileconverter"
|
||||
description: "Use this component to convert text files and directories to a document."
|
||||
---
|
||||
|
||||
# UnstructuredFileConverter
|
||||
|
||||
Use this component to convert text files and directories to a document.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [PreProcessors](../preprocessors.mdx) or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory run variables** | `paths`: A union of lists of paths |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Unstructured](/reference/integrations-unstructured) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/unstructured |
|
||||
| **Package name** | `unstructured-fileconverter-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`UnstructuredFileConverter` converts files and directories into documents using the Unstructured API.
|
||||
|
||||
[Unstructured](https://docs.unstructured.io/) provides a series of tools to do ETL for LLMs. The `UnstructuredFileConverter` calls the Unstructured API that extracts text and other information from a vast range of file [formats](https://docs.unstructured.io/api-reference/api-services/overview#supported-file-types).
|
||||
|
||||
This Converter supports different modes for creating documents from the elements returned by Unstructured:
|
||||
|
||||
- `"one-doc-per-file"`: One Haystack document per file. All elements are concatenated into one text field.
|
||||
- `"one-doc-per-page"`: One Haystack document per page. All elements on a page are concatenated into one text field.
|
||||
- `"one-doc-per-element"`: One Haystack document per element. Each element is converted to a Haystack document.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the Unstructured integration to use `UnstructuredFileConverter`component:
|
||||
|
||||
```shell
|
||||
pip install unstructured-fileconverter-haystack
|
||||
```
|
||||
|
||||
There are free and paid versions of Unstructured API: **Free Unstructured API** and **Unstructured Serverless API**.
|
||||
|
||||
1. **Free Unstructured API**:
|
||||
- API URL: `https://api.unstructured.io/general/v0/general`
|
||||
- This version is free, but comes with certain limitations.
|
||||
|
||||
2. **Unstructured Serverless API**:
|
||||
- You'll find your unique API URL in your Unstructured account after signing up for the paid version.
|
||||
- This is a full-tier paid version of Unstructured.
|
||||
|
||||
For more details about the two tiers refer to Unstructured [FAQ](https://docs.unstructured.io/faq/faq).
|
||||
|
||||
> ❗️ The API keys for the free and paid versions are different and cannot be used interchangeably.
|
||||
|
||||
Regardless of the chosen tier, we recommend to set the Unstructured API key as an environment variable `UNSTRUCTURED_API_KEY`:
|
||||
|
||||
```shell
|
||||
export UNSTRUCTURED_API_KEY=your_api_key
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
import os
|
||||
from haystack_integrations.components.converters.unstructured import (
|
||||
UnstructuredFileConverter,
|
||||
)
|
||||
|
||||
converter = UnstructuredFileConverter()
|
||||
documents = converter.run(paths=["a/file/path.pdf", "a/directory/path"])["documents"]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
import os
|
||||
from haystack import Pipeline
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.converters.unstructured import (
|
||||
UnstructuredFileConverter,
|
||||
)
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
indexing = Pipeline()
|
||||
indexing.add_component("converter", UnstructuredFileConverter())
|
||||
indexing.add_component("writer", DocumentWriter(document_store))
|
||||
indexing.connect("converter", "writer")
|
||||
|
||||
indexing.run({"converter": {"paths": ["a/file/path.pdf", "a/directory/path"]}})
|
||||
```
|
||||
|
||||
### With Docker
|
||||
|
||||
To use `UnstructuredFileConverter` through Docker, first, set up an Unstructured Docker container:
|
||||
|
||||
```
|
||||
docker run -p 8000:8000 -d --rm --name unstructured-api quay.io/unstructured-io/unstructured-api:latest --port 8000 --host 0.0.0.0
|
||||
```
|
||||
|
||||
When initializing the component, specify the localhost URL:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.converters.unstructured import (
|
||||
UnstructuredFileConverter,
|
||||
)
|
||||
|
||||
converter = UnstructuredFileConverter(
|
||||
api_url="http://localhost:8000/general/v0/general",
|
||||
)
|
||||
```
|
||||
@@ -0,0 +1,80 @@
|
||||
---
|
||||
title: "XLSXToDocument"
|
||||
id: xlsxtodocument
|
||||
slug: "/xlsxtodocument"
|
||||
description: "Converts Excel files into documents."
|
||||
---
|
||||
|
||||
# XLSXToDocument
|
||||
|
||||
Converts Excel files into documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before [PreProcessors](../preprocessors.mdx) or right at the beginning of an indexing pipeline |
|
||||
| **Mandatory run variables** | `sources`: File paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Converters](/reference/converters-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/converters/xlsx.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `XLSXToDocument` component converts XLSX files into Haystack Documents with a CSV (default) or Markdown format. It takes a list of file paths or [`ByteStream`](../../concepts/data-classes.mdx#bytestream) objects as input and outputs the converted result as a list of documents. Optionally, you can attach metadata to the documents through the `meta` input parameter.
|
||||
|
||||
To see the additional parameters that you can specify with the component initialization, check out the [API Reference](/reference/converters-api#xlsxtodocument).
|
||||
|
||||
## Usage
|
||||
|
||||
First, install the openpyxl and tabulate packages to start using this converter:
|
||||
|
||||
```shell
|
||||
pip install pandas openpyxl
|
||||
pip install tabulate
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.components.converters import XLSXToDocument
|
||||
|
||||
converter = XLSXToDocument()
|
||||
results = converter.run(
|
||||
sources=["sample.xlsx"],
|
||||
meta={"date_added": datetime.now().isoformat()},
|
||||
)
|
||||
documents = results["documents"]
|
||||
print(documents[0].content)
|
||||
# ",A,B\n1,col_a,col_b\n2,1.5,test\n"
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters import XLSXToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("converter", XLSXToDocument())
|
||||
pipeline.add_component("cleaner", DocumentCleaner())
|
||||
pipeline.add_component(
|
||||
"splitter",
|
||||
DocumentSplitter(split_by="sentence", split_length=5),
|
||||
)
|
||||
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
pipeline.connect("converter", "cleaner")
|
||||
pipeline.connect("cleaner", "splitter")
|
||||
pipeline.connect("splitter", "writer")
|
||||
|
||||
pipeline.run({"converter": {"sources": file_names}})
|
||||
```
|
||||
@@ -0,0 +1,273 @@
|
||||
---
|
||||
title: "S3Downloader"
|
||||
id: s3downloader
|
||||
slug: "/s3downloader"
|
||||
description: "`S3Downloader` downloads files from AWS S3 buckets to the local filesystem and enriches documents with the local file path."
|
||||
---
|
||||
|
||||
# S3Downloader
|
||||
|
||||
`S3Downloader` downloads files from AWS S3 buckets to the local filesystem and enriches documents with the local file path.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before File Converters or Routers that need local file paths |
|
||||
| **Mandatory init variables** | `file_root_path`: Path where files will be downloaded. Can be set with `FILE_ROOT_PATH` env var. <br /> <br />`aws_access_key_id`: AWS access key ID. Can be set with AWS_ACCESS_KEY_ID env var. <br /> <br />`aws_secret_access_key`: AWS secret access key. Can be set with AWS_SECRET_ACCESS_KEY env var. <br /> <br />`aws_region_name`: AWS region name. Can be set with AWS_DEFAULT_REGION env var. |
|
||||
| **Mandatory run variables** | `documents`: A list of documents containing name of the file to download in metadata. |
|
||||
| **Output variables** | `documents`: A list of documents enriched with the local file path in `meta['file_path']` |
|
||||
| **API reference** | [S3Downloader](/reference/integrations-amazon-bedrock) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/amazon_bedrock |
|
||||
| **Package name** | `amazon-bedrock-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`S3Downloader` downloads files from AWS S3 buckets to your local filesystem and enriches Document objects with the local file path. This component is useful for pipelines that need to process files stored in S3, such as PDFs, images, or text files.
|
||||
|
||||
The component supports AWS authentication through environment variables by default. You can set `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_DEFAULT_REGION` environment variables. Alternatively, you can pass credentials directly at initialization using the [Secret API](../../concepts/secret-management.mdx):
|
||||
|
||||
```python
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.components.downloaders.s3 import S3Downloader
|
||||
|
||||
downloader = S3Downloader(
|
||||
aws_access_key_id=Secret.from_token("<your-access-key-id>"),
|
||||
aws_secret_access_key=Secret.from_token("<your-secret-access-key>"),
|
||||
aws_region_name=Secret.from_token("<your-region>"),
|
||||
file_root_path="/path/to/download/directory",
|
||||
)
|
||||
```
|
||||
|
||||
The component downloads multiple files in parallel using the `max_workers` parameter (default is 32 workers) to speed up processing of large document sets. Downloaded files are cached locally, and when the cache exceeds `max_cache_size` (default is 100 files), least recently accessed files are automatically removed. Already downloaded files are touched to update their access time without re-downloading.
|
||||
|
||||
:::info[Required Configuration]
|
||||
|
||||
The component requires two critical configurations:
|
||||
|
||||
1. `file_root_path` parameter or `FILE_ROOT_PATH` environment variable: Specifies where files will be downloaded. This directory will be created if it doesn't exist.
|
||||
2. `S3_DOWNLOADER_BUCKET` environment variable: Specifies which S3 bucket to download files from.
|
||||
:::
|
||||
|
||||
The optional environment variable `S3_DOWNLOADER_PREFIX` can be set to add a prefix of the files to all generated S3 keys.
|
||||
|
||||
### File Extension Filtering
|
||||
|
||||
You can use the `file_extensions` parameter to download only specific file types, reducing unnecessary downloads and processing time. For example, `file_extensions=[".pdf", ".txt"]` downloads only PDF and TXT files while skipping others.
|
||||
|
||||
### Custom S3 Key Generation
|
||||
|
||||
By default, the component uses the `file_name` from Document metadata as the S3 key. If your S3 file structure doesn't match the file names in metadata, you can provide an optional `s3_key_generation_function` to customize how S3 keys are generated from Document metadata.
|
||||
|
||||
## Usage
|
||||
|
||||
You need to install the `amazon-bedrock-haystack` package to use `S3Downloader`:
|
||||
|
||||
```shell
|
||||
pip install amazon-bedrock-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
Before running the examples, ensure you have set the required environment variables:
|
||||
|
||||
```shell
|
||||
export AWS_ACCESS_KEY_ID="<your-access-key-id>"
|
||||
export AWS_SECRET_ACCESS_KEY="<your-secret-access-key>"
|
||||
export AWS_DEFAULT_REGION="<your-region>"
|
||||
export S3_DOWNLOADER_BUCKET="<your-bucket-name>"
|
||||
```
|
||||
|
||||
Here's how to use `S3Downloader` to download files from S3:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import Document
|
||||
from haystack_integrations.components.downloaders.s3 import S3Downloader
|
||||
|
||||
# Create documents with file names in metadata
|
||||
documents = [
|
||||
Document(meta={"file_name": "report.pdf"}),
|
||||
Document(meta={"file_name": "data.txt"}),
|
||||
]
|
||||
|
||||
# Initialize the downloader
|
||||
downloader = S3Downloader(file_root_path="/tmp/s3_downloads")
|
||||
|
||||
# Download the files
|
||||
result = downloader.run(documents=documents)
|
||||
|
||||
# Access the downloaded files
|
||||
for doc in result["documents"]:
|
||||
print(f"File downloaded to: {doc.meta['file_path']}")
|
||||
```
|
||||
|
||||
With file extension filtering:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import Document
|
||||
from haystack_integrations.components.downloaders.s3 import S3Downloader
|
||||
|
||||
documents = [
|
||||
Document(meta={"file_name": "report.pdf"}),
|
||||
Document(meta={"file_name": "image.png"}),
|
||||
Document(meta={"file_name": "data.txt"}),
|
||||
]
|
||||
|
||||
# Only download PDF files
|
||||
downloader = S3Downloader(file_root_path="/tmp/s3_downloads", file_extensions=[".pdf"])
|
||||
|
||||
result = downloader.run(documents=documents)
|
||||
|
||||
# Only report.pdf is downloaded
|
||||
print(f"Downloaded {len(result['documents'])} file(s)")
|
||||
# Output: Downloaded 1 file(s)
|
||||
```
|
||||
|
||||
With custom S3 key generation:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import Document
|
||||
from haystack_integrations.components.downloaders.s3 import S3Downloader
|
||||
|
||||
|
||||
def custom_s3_key_function(document: Document) -> str:
|
||||
"""Generate S3 key from custom metadata."""
|
||||
folder = document.meta.get("folder", "default")
|
||||
file_name = document.meta.get("file_name")
|
||||
if not file_name:
|
||||
raise ValueError("Document must have 'file_name' in metadata")
|
||||
return f"{folder}/{file_name}"
|
||||
|
||||
|
||||
documents = [
|
||||
Document(meta={"file_name": "report.pdf", "folder": "reports/2025"}),
|
||||
]
|
||||
|
||||
downloader = S3Downloader(
|
||||
file_root_path="/tmp/s3_downloads",
|
||||
s3_key_generation_function=custom_s3_key_function,
|
||||
)
|
||||
|
||||
result = downloader.run(documents=documents)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Here's an example of using `S3Downloader` in a document processing pipeline:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.converters import PDFMinerToDocument
|
||||
from haystack.components.routers import DocumentTypeRouter
|
||||
from haystack.dataclasses import Document
|
||||
|
||||
from haystack_integrations.components.downloaders.s3 import S3Downloader
|
||||
|
||||
# Create a pipeline
|
||||
pipe = Pipeline()
|
||||
|
||||
# Add S3Downloader to download files from S3
|
||||
pipe.add_component(
|
||||
"downloader",
|
||||
S3Downloader(file_root_path="/tmp/s3_downloads", file_extensions=[".pdf", ".txt"]),
|
||||
)
|
||||
|
||||
# Route documents by file type
|
||||
pipe.add_component(
|
||||
"router",
|
||||
DocumentTypeRouter(
|
||||
file_path_meta_field="file_path",
|
||||
mime_types=["application/pdf", "text/plain"],
|
||||
),
|
||||
)
|
||||
|
||||
# Convert PDFs to documents
|
||||
pipe.add_component("pdf_converter", PDFMinerToDocument())
|
||||
|
||||
# Connect components
|
||||
pipe.connect("downloader.documents", "router.documents")
|
||||
pipe.connect("router.application/pdf", "pdf_converter.documents")
|
||||
|
||||
# Create documents with S3 file names
|
||||
documents = [
|
||||
Document(meta={"file_name": "report.pdf"}),
|
||||
Document(meta={"file_name": "summary.txt"}),
|
||||
]
|
||||
|
||||
# Run the pipeline
|
||||
result = pipe.run({"downloader": {"documents": documents}})
|
||||
```
|
||||
|
||||
For a more complex example with image processing and LLM:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.converters.image import DocumentToImageContent
|
||||
from haystack.components.routers import DocumentTypeRouter
|
||||
from haystack.dataclasses import Document
|
||||
|
||||
from haystack_integrations.components.downloaders.s3 import S3Downloader
|
||||
from haystack_integrations.components.generators.amazon_bedrock import (
|
||||
AmazonBedrockChatGenerator,
|
||||
)
|
||||
|
||||
# Create documents with file names
|
||||
documents = [
|
||||
Document(meta={"file_name": "chart.png"}),
|
||||
Document(meta={"file_name": "report.pdf"}),
|
||||
]
|
||||
|
||||
# Create pipeline
|
||||
pipe = Pipeline()
|
||||
|
||||
# Download files from S3
|
||||
pipe.add_component("downloader", S3Downloader(file_root_path="/tmp/s3_downloads"))
|
||||
|
||||
# Route by document type
|
||||
pipe.add_component(
|
||||
"router",
|
||||
DocumentTypeRouter(
|
||||
file_path_meta_field="file_path",
|
||||
mime_types=["image/png", "application/pdf"],
|
||||
),
|
||||
)
|
||||
|
||||
# Convert images for LLM
|
||||
pipe.add_component("image_converter", DocumentToImageContent(detail="auto"))
|
||||
|
||||
# Create chat prompt with template
|
||||
template = """{% message role="user" %}
|
||||
Answer the question based on the provided images.
|
||||
|
||||
Question: {{ question }}
|
||||
|
||||
{% for image in image_contents %}
|
||||
{{ image | templatize_part }}
|
||||
{% endfor %}
|
||||
{% endmessage %}"""
|
||||
|
||||
pipe.add_component("prompt_builder", ChatPromptBuilder(template=template))
|
||||
|
||||
# Generate response
|
||||
pipe.add_component(
|
||||
"llm",
|
||||
AmazonBedrockChatGenerator(model="anthropic.claude-3-haiku-20240307-v1:0"),
|
||||
)
|
||||
|
||||
# Connect components
|
||||
pipe.connect("downloader.documents", "router.documents")
|
||||
pipe.connect("router.image/png", "image_converter.documents")
|
||||
pipe.connect("image_converter.image_contents", "prompt_builder.image_contents")
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
# Run pipeline
|
||||
result = pipe.run(
|
||||
{
|
||||
"downloader": {"documents": documents},
|
||||
"prompt_builder": {"question": "What information is shown in the chart?"},
|
||||
},
|
||||
)
|
||||
```
|
||||
@@ -0,0 +1,68 @@
|
||||
---
|
||||
title: "Embedders"
|
||||
id: embedders
|
||||
slug: "/embedders"
|
||||
description: "Embedders in Haystack transform texts or documents into vector representations using pre-trained models. You can then use the embedding for tasks like question answering, information retrieval, and more."
|
||||
---
|
||||
|
||||
# Embedders
|
||||
|
||||
Embedders in Haystack transform texts or documents into vector representations using pre-trained models. You can then use the embedding for tasks like question answering, information retrieval, and more.
|
||||
|
||||
:::info
|
||||
For general guidance on how to choose an Embedder that would be right for you, read our [Choosing the Right Embedder](embedders/choosing-the-right-embedder.mdx) page.
|
||||
:::
|
||||
|
||||
These are the Embedders available in Haystack:
|
||||
|
||||
| Embedder | Description |
|
||||
| --- | --- |
|
||||
| [AmazonBedrockTextEmbedder](embedders/amazonbedrocktextembedder.mdx) | Computes embeddings for text (such as a query) using models through Amazon Bedrock API. |
|
||||
| [AmazonBedrockDocumentEmbedder](embedders/amazonbedrockdocumentembedder.mdx) | Computes embeddings for documents using models through Amazon Bedrock API. |
|
||||
| [AmazonBedrockDocumentImageEmbedder](embedders/amazonbedrockdocumentimageembedder.mdx) | Computes image embeddings for a document. |
|
||||
| [AzureOpenAITextEmbedder](embedders/azureopenaitextembedder.mdx) | Computes embeddings for text (such as a query) using OpenAI models deployed through Azure. |
|
||||
| [AzureOpenAIDocumentEmbedder](embedders/azureopenaidocumentembedder.mdx) | Computes embeddings for documents using OpenAI models deployed through Azure. |
|
||||
| [CohereTextEmbedder](embedders/coheretextembedder.mdx) | Embeds a simple string (such as a query) with a Cohere model. Requires an API key from Cohere |
|
||||
| [CohereDocumentEmbedder](embedders/coheredocumentembedder.mdx) | Embeds a list of documents with a Cohere model. Requires an API key from Cohere. |
|
||||
| [CohereDocumentImageEmbedder](embedders/coheredocumentimageembedder.mdx) | Computes the image embeddings of a list of documents and stores the obtained vectors in the embedding field of each document. |
|
||||
| [FastembedTextEmbedder](embedders/fastembedtextembedder.mdx) | Computes the embeddings of a string using embedding models supported by Fastembed. |
|
||||
| [FastembedDocumentEmbedder](embedders/fastembeddocumentembedder.mdx) | Computes the embeddings of a list of documents using the models supported by Fastembed. |
|
||||
| [FastembedSparseTextEmbedder](embedders/fastembedsparsetextembedder.mdx) | Embeds a simple string (such as a query) into a sparse vector using the models supported by Fastembed. |
|
||||
| [FastembedSparseDocumentEmbedder](embedders/fastembedsparsedocumentembedder.mdx) | Enriches a list of documents with their sparse embeddings using the models supported by Fastembed. |
|
||||
| [GoogleGenAITextEmbedder](embedders/googlegenaitextembedder.mdx) | Embeds a simple string (such as a query) with a Google AI model. Requires an API key from Google. |
|
||||
| [GoogleGenAIDocumentEmbedder](embedders/googlegenaidocumentembedder.mdx) | Embeds a list of documents with a Google AI model. Requires an API key from Google. |
|
||||
| [GoogleGenAIMultimodalDocumentEmbedder](embedders/googlegenaimultimodaldocumentembedder.mdx) | Embeds a list of non-textual documents with a Google AI model. Requires an API key from Google. |
|
||||
| [HuggingFaceAPIDocumentEmbedder](embedders/huggingfaceapidocumentembedder.mdx) | Computes document embeddings using various Hugging Face APIs. |
|
||||
| [HuggingFaceAPITextEmbedder](embedders/huggingfaceapitextembedder.mdx) | Embeds strings using various Hugging Face APIs. |
|
||||
| [JinaTextEmbedder](embedders/jinatextembedder.mdx) | Embeds a simple string (such as a query) with a Jina AI Embeddings model. Requires an API key from Jina AI. |
|
||||
| [JinaDocumentEmbedder](embedders/jinadocumentembedder.mdx) | Embeds a list of documents with a Jina AI Embeddings model. Requires an API key from Jina AI. |
|
||||
| [JinaDocumentImageEmbedder](embedders/jinadocumentimageembedder.mdx) | Computes the image embeddings of a list of documents and stores the obtained vectors in the embedding field of each document. |
|
||||
| [MistralTextEmbedder](embedders/mistraltextembedder.mdx) | Transforms a string into a vector using the Mistral API and models. |
|
||||
| [MistralDocumentEmbedder](embedders/mistraldocumentembedder.mdx) | Computes the embeddings of a list of documents using the Mistral API and models. |
|
||||
| [MockTextEmbedder](embedders/mocktextembedder.mdx) | Returns deterministic embeddings for a string without calling any API — a zero-cost stand-in for real Text Embedders in tests and prototypes. |
|
||||
| [MockDocumentEmbedder](embedders/mockdocumentembedder.mdx) | Returns deterministic embeddings for a list of documents without calling any API — a zero-cost stand-in for real Document Embedders in tests and prototypes. |
|
||||
| [NvidiaTextEmbedder](embedders/nvidiatextembedder.mdx) | Embeds a simple string (such as a query) into a vector. |
|
||||
| [NvidiaDocumentEmbedder](embedders/nvidiadocumentembedder.mdx) | Enriches the metadata of documents with an embedding of their content. |
|
||||
| [OllamaTextEmbedder](embedders/ollamatextembedder.mdx) | Computes the embeddings of a string using embedding models compatible with the Ollama Library. |
|
||||
| [OllamaDocumentEmbedder](embedders/ollamadocumentembedder.mdx) | Computes the embeddings of a list of documents using embedding models compatible with the Ollama Library. |
|
||||
| [OpenAIDocumentEmbedder](embedders/openaidocumentembedder.mdx) | Embeds a list of documents with an OpenAI embedding model. Requires an API key from an active OpenAI account. |
|
||||
| [OpenAITextEmbedder](embedders/openaitextembedder.mdx) | Embeds a simple string (such as a query) with an OpenAI embedding model. Requires an API key from an active OpenAI account. |
|
||||
| [OptimumTextEmbedder](embedders/optimumtextembedder.mdx) | Embeds text using models loaded with the Hugging Face Optimum library. |
|
||||
| [OptimumDocumentEmbedder](embedders/optimumdocumentembedder.mdx) | Computes documents’ embeddings using models loaded with the Hugging Face Optimum library. |
|
||||
| [PerplexityDocumentEmbedder](embedders/perplexitydocumentembedder.mdx) | Computes embeddings for a list of documents using Perplexity embedding models. Requires an API key from Perplexity. |
|
||||
| [PerplexityTextEmbedder](embedders/perplexitytextembedder.mdx) | Embeds a simple string (such as a query) using a Perplexity embedding model. Requires an API key from Perplexity. |
|
||||
| [SentenceTransformersTextEmbedder](embedders/sentencetransformerstextembedder.mdx) | Embeds a simple string (such as a query) using a Sentence Transformer model. |
|
||||
| [SentenceTransformersDocumentEmbedder](embedders/sentencetransformersdocumentembedder.mdx) | Embeds a list of documents with a Sentence Transformer model. |
|
||||
| [SentenceTransformersDocumentImageEmbedder](embedders/sentencetransformersdocumentimageembedder.mdx) | Computes the image embeddings of a list of documents and stores the obtained vectors in the embedding field of each document. |
|
||||
| [SentenceTransformersSparseTextEmbedder](embedders/sentencetransformerssparsetextembedder.mdx) | Embeds a simple string (such as a query) into a sparse vector using Sentence Transformers models. |
|
||||
| [SentenceTransformersSparseDocumentEmbedder](embedders/sentencetransformerssparsedocumentembedder.mdx) | Enriches a list of documents with their sparse embeddings using Sentence Transformers models. |
|
||||
| [STACKITTextEmbedder](embedders/stackittextembedder.mdx) | Enables text embedding using the STACKIT API. |
|
||||
| [STACKITDocumentEmbedder](embedders/stackitdocumentembedder.mdx) | Enables document embedding using the STACKIT API. |
|
||||
| [TwelveLabsTextEmbedder](embedders/twelvelabstextembedder.mdx) | Embeds a simple string (such as a query) with the TwelveLabs Marengo multimodal model. Requires an API key from TwelveLabs. |
|
||||
| [TwelveLabsDocumentEmbedder](embedders/twelvelabsdocumentembedder.mdx) | Embeds a list of documents with the TwelveLabs Marengo multimodal model. Requires an API key from TwelveLabs. |
|
||||
| [VertexAITextEmbedder](embedders/vertexaitextembedder.mdx) | Computes embeddings for text (such as a query) using models through VertexAI Embeddings API. **_This integration will be deprecated soon. We recommend using [GoogleGenAITextEmbedder](embedders/googlegenaitextembedder.mdx) integration instead._** |
|
||||
| [VertexAIDocumentEmbedder](embedders/vertexaidocumentembedder.mdx) | Computes embeddings for documents using models through VertexAI Embeddings API. **_This integration will be deprecated soon. We recommend using [GoogleGenAIDocumentEmbedder](embedders/googlegenaidocumentembedder.mdx) integration instead._** |
|
||||
| [VLLMTextEmbedder](embedders/vllmtextembedder.mdx) | Computes the embeddings of a string using models served with vLLM. |
|
||||
| [VLLMDocumentEmbedder](embedders/vllmdocumentembedder.mdx) | Computes the embeddings of a list of documents using models served with vLLM. |
|
||||
| [WatsonxTextEmbedder](embedders/watsonxtextembedder.mdx) | Computes embeddings for text (such as a query) using IBM Watsonx models. |
|
||||
| [WatsonxDocumentEmbedder](embedders/watsonxdocumentembedder.mdx) | Computes embeddings for documents using IBM Watsonx models. |
|
||||
@@ -0,0 +1,172 @@
|
||||
---
|
||||
title: "AmazonBedrockDocumentEmbedder"
|
||||
id: amazonbedrockdocumentembedder
|
||||
slug: "/amazonbedrockdocumentembedder"
|
||||
description: "This component computes embeddings for documents using models through Amazon Bedrock API."
|
||||
---
|
||||
|
||||
# AmazonBedrockDocumentEmbedder
|
||||
|
||||
This component computes embeddings for documents using models through Amazon Bedrock API.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline |
|
||||
| **Mandatory init variables** | `model`: The embedding model to use <br /> <br />`aws_access_key_id`: AWS access key ID. Can be set with `AWS_ACCESS_KEY_ID` env var. <br /> <br />`aws_secret_access_key`: AWS secret access key. Can be set with `AWS_SECRET_ACCESS_KEY` env var. <br /> <br />`aws_region_name`: AWS region name. Can be set with `AWS_DEFAULT_REGION` env var. |
|
||||
| **Mandatory run variables** | `documents`: A list of documents to be embedded |
|
||||
| **Output variables** | `documents`: A list of documents (enriched with embeddings) |
|
||||
| **API reference** | [Amazon Bedrock](/reference/integrations-amazon-bedrock) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/amazon_bedrock |
|
||||
| **Package name** | `amazon-bedrock-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
[Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) is a fully managed service that makes language models from leading AI startups and Amazon available for your use through a unified API.
|
||||
|
||||
Supported models are `amazon.titan-embed-text-v1`, `cohere.embed-english-v3`, `cohere.embed-multilingual-v3`, and `amazon.titan-embed-text-v2:0`.
|
||||
|
||||
:::info[Batch Inference]
|
||||
|
||||
Note that only Cohere models support batch inference – computing embeddings for more documents with the same request.
|
||||
:::
|
||||
|
||||
This component should be used to embed a list of documents. To embed a string, you should use the [`AmazonBedrockTextEmbedder`](amazonbedrocktextembedder.mdx).
|
||||
|
||||
### Authentication
|
||||
|
||||
`AmazonBedrockDocumentEmbedder` uses AWS for authentication. You can either provide credentials as parameters directly to the component or use the AWS CLI and authenticate through your IAM. For more information on how to set up an IAM identity-based policy, see the [official documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html).
|
||||
To initialize `AmazonBedrockDocumentEmbedder` and authenticate by providing credentials, provide the `model_name`, as well as `aws_access_key_id`, `aws_secret_access_key` and `aws_region_name`. Other parameters are optional. You can check them out in our [API reference](/reference/integrations-amazon-bedrock#amazonbedrockdocumentembedder).
|
||||
|
||||
### Model-specific parameters
|
||||
|
||||
Even if Haystack provides a unified interface, each model offered by Bedrock can accept specific parameters. You can pass these parameters at initialization.
|
||||
|
||||
For example, Cohere models support `input_type` and `truncate`, as seen in [Bedrock documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters.html).
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.amazon_bedrock import (
|
||||
AmazonBedrockDocumentEmbedder,
|
||||
)
|
||||
|
||||
embedder = AmazonBedrockDocumentEmbedder(
|
||||
model="cohere.embed-english-v3",
|
||||
input_type="search_document",
|
||||
truncate="LEFT",
|
||||
)
|
||||
```
|
||||
|
||||
### Embedding Metadata
|
||||
|
||||
Text documents often come with a set of metadata. If they are distinctive and semantically meaningful, you can embed them along with the text of the document to improve retrieval.
|
||||
|
||||
You can do this easily by using the Document Embedder:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.embedders.amazon_bedrock import (
|
||||
AmazonBedrockDocumentEmbedder,
|
||||
)
|
||||
|
||||
doc = Document(content="some text", meta={"title": "relevant title", "page number": 18})
|
||||
|
||||
embedder = AmazonBedrockDocumentEmbedder(
|
||||
model="cohere.embed-english-v3",
|
||||
meta_fields_to_embed=["title"],
|
||||
)
|
||||
|
||||
docs_w_embeddings = embedder.run(documents=[doc])["documents"]
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Installation
|
||||
|
||||
You need to install `amazon-bedrock-haystack` package to use the `AmazonBedrockTextEmbedder`:
|
||||
|
||||
```shell
|
||||
pip install amazon-bedrock-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage:
|
||||
|
||||
```python
|
||||
import os
|
||||
from haystack_integrations.components.embedders.amazon_bedrock import AmazonBedrockDocumentEmbedder
|
||||
from haystack.dataclasses import DOcument
|
||||
|
||||
os.environ["AWS_ACCESS_KEY_ID"] = "..."
|
||||
os.environ["AWS_SECRET_ACCESS_KEY"] = "..."
|
||||
os.environ["AWS_DEFAULT_REGION"] = "us-east-1" # just an example
|
||||
|
||||
doc = Document(content="I love pizza!")
|
||||
|
||||
embedder = AmazonBedrockDocumentEmbedder(model="cohere.embed-english-v3",
|
||||
input_type="search_document"
|
||||
|
||||
result = document_embedder.run([doc])
|
||||
print(result['documents'][0].embedding)
|
||||
|
||||
# [0.017020374536514282, -0.023255806416273117, ...]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
In a RAG pipeline:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.embedders.amazon_bedrock import (
|
||||
AmazonBedrockDocumentEmbedder,
|
||||
AmazonBedrockTextEmbedder,
|
||||
)
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
Document(content="Germany has many big cities"),
|
||||
]
|
||||
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component(
|
||||
"embedder",
|
||||
AmazonBedrockDocumentEmbedder(model="cohere.embed-english-v3"),
|
||||
)
|
||||
indexing_pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
indexing_pipeline.connect("embedder", "writer")
|
||||
|
||||
indexing_pipeline.run({"embedder": {"documents": documents}})
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component(
|
||||
"text_embedder",
|
||||
AmazonBedrockTextEmbedder(model="cohere.embed-english-v3"),
|
||||
)
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "Who lives in Berlin?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
|
||||
# Document(id=..., content: 'My name is Wolfgang and I live in Berlin')
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [PDF-Based Question Answering with Amazon Bedrock and Haystack](https://haystack.deepset.ai/cookbook/amazon_bedrock_for_documentation_qa)
|
||||
+165
@@ -0,0 +1,165 @@
|
||||
---
|
||||
title: "AmazonBedrockDocumentImageEmbedder"
|
||||
id: amazonbedrockdocumentimageembedder
|
||||
slug: "/amazonbedrockdocumentimageembedder"
|
||||
description: "`AmazonBedrockDocumentImageEmbedder` computes image embeddings for documents using models exposed through the Amazon Bedrock API. It stores the obtained vectors in the embedding field of each document."
|
||||
---
|
||||
|
||||
# AmazonBedrockDocumentImageEmbedder
|
||||
|
||||
`AmazonBedrockDocumentImageEmbedder` computes image embeddings for documents using models exposed through the Amazon Bedrock API. It stores the obtained vectors in the embedding field of each document.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline |
|
||||
| **Mandatory init variables** | `model`: The multimodal embedding model to use. <br /> <br />`aws_access_key_id`: AWS access key ID. Can be set with `AWS_ACCESS_KEY_ID` env var. <br /> <br />`aws_secret_access_key`: AWS secret access key. Can be set with `AWS_SECRET_ACCESS_KEY` env var. <br /> <br />`aws_region_name`: AWS region name. Can be set with `AWS_DEFAULT_REGION` env var. |
|
||||
| **Mandatory run variables** | `documents`: A list of documents, with a meta field containing an image file path |
|
||||
| **Output variables** | `documents`: A list of documents (enriched with embeddings) |
|
||||
| **API reference** | [Amazon Bedrock](/reference/integrations-amazon-bedrock) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/amazon_bedrock |
|
||||
| **Package name** | `amazon-bedrock-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
Amazon Bedrock is a fully managed service that provides access to foundation models through a unified API.
|
||||
|
||||
`AmazonBedrockDocumentImageEmbedder` expects a list of documents containing an image or a PDF file path in a meta field. The meta field can be specified with the `file_path_meta_field` init parameter of this component.
|
||||
|
||||
The embedder efficiently loads the images, computes the embeddings using selected Bedrock model, and stores each of them in the `embedding` field of the document.
|
||||
|
||||
Supported models are `amazon.titan-embed-image-v1`, `cohere.embed-english-v3` , and `cohere.embed-multilingual-v3`.
|
||||
|
||||
`AmazonBedrockDocumentImageEmbedder` is commonly used in indexing pipelines. At retrieval time, you need to use the same model with `AmazonBedrockTextEmbedder` to embed the query, before using an Embedding Retriever.
|
||||
|
||||
### Installation
|
||||
|
||||
To start using this integration with Haystack, install the package with:
|
||||
|
||||
```shell
|
||||
pip install amazon-bedrock-haystack
|
||||
```
|
||||
|
||||
### Authentication
|
||||
|
||||
`AmazonBedrockDocumentImageEmbedder` uses AWS for authentication. You can either provide credentials as parameters directly to the component or use the AWS CLI and authenticate through your IAM. For more information on how to set up an IAM identity-based policy, see the [official documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html).
|
||||
|
||||
To initialize `AmazonBedrockDocumentImageEmbedder` and authenticate by providing credentials, provide the `model` name, as well as `aws_access_key_id`, `aws_secret_access_key`, and `aws_region_name`. Other parameters are optional, you can check them out in our [API reference](/reference/integrations-amazon-bedrock#amazonbedrocktextembedder).
|
||||
|
||||
### Model-specific parameters
|
||||
|
||||
Even if Haystack provides a unified interface, each model offered by Bedrock can accept specific parameters. You can pass these parameters at initialization.
|
||||
|
||||
- **Amazon Titan**: Use `embeddingConfig` to control embedding behavior.
|
||||
- **Cohere v3**: Use `embedding_types` to select a single embedding type for images.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.amazon_bedrock import (
|
||||
AmazonBedrockDocumentImageEmbedder,
|
||||
)
|
||||
|
||||
embedder = AmazonBedrockDocumentImageEmbedder(
|
||||
model="cohere.embed-english-v3",
|
||||
embedding_types=["float"], # single value only
|
||||
)
|
||||
```
|
||||
|
||||
Note that only _one_ value in `embedding_types` is supported by this component. Passing multiple values raises an error.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
import os
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.embedders.amazon_bedrock import (
|
||||
AmazonBedrockDocumentImageEmbedder,
|
||||
)
|
||||
|
||||
os.environ["AWS_ACCESS_KEY_ID"] = "..."
|
||||
os.environ["AWS_SECRET_ACCESS_KEY"] = "..."
|
||||
os.environ["AWS_DEFAULT_REGION"] = "us-east-1" # example
|
||||
|
||||
# Point Documents to image/PDF files via metadata (default key: "file_path")
|
||||
documents = [
|
||||
Document(content="A photo of a cat", meta={"file_path": "cat.jpg"}),
|
||||
Document(
|
||||
content="Invoice page",
|
||||
meta={
|
||||
"file_path": "invoice.pdf",
|
||||
"mime_type": "application/pdf",
|
||||
"page_number": 1,
|
||||
},
|
||||
),
|
||||
]
|
||||
|
||||
embedder = AmazonBedrockDocumentImageEmbedder(
|
||||
model="amazon.titan-embed-image-v1",
|
||||
image_size=(1024, 1024), # optional downscaling
|
||||
)
|
||||
|
||||
result = embedder.run(documents=documents)
|
||||
embedded_docs = result["documents"]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
In this example, we can see an indexing pipeline with 3 components:
|
||||
|
||||
- `ImageFileToDocument` Converter that creates empty documents with a reference to an image in the `meta.file_path` field;
|
||||
- `AmazonBedrockDocumentImageEmbedder` that loads the images, computes embeddings and stores them in documents;
|
||||
- `DocumentWriter` that write the documents in the `InMemoryDocumentStore`.
|
||||
|
||||
There is also a multimodal retrieval pipeline, composed of an `AmazonBedrockTextEmbedder` (using the same model as before) and an `InMemoryEmbeddingRetriever`.
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
from haystack_integrations.components.embedders.amazon_bedrock import (
|
||||
AmazonBedrockDocumentImageEmbedder,
|
||||
AmazonBedrockTextEmbedder,
|
||||
)
|
||||
|
||||
# Document store using vector similarity for retrieval
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
# Sample corpus with file paths in metadata
|
||||
documents = [
|
||||
Document(content="A sketch of a horse", meta={"file_path": "horse.png"}),
|
||||
Document(content="A city map", meta={"file_path": "map.jpg"}),
|
||||
]
|
||||
|
||||
# Indexing pipeline: image embeddings -> write to store
|
||||
indexing = Pipeline()
|
||||
indexing.add_component(
|
||||
"image_embedder",
|
||||
AmazonBedrockDocumentImageEmbedder(model="cohere.embed-english-v3"),
|
||||
)
|
||||
indexing.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
indexing.connect("image_embedder", "writer")
|
||||
indexing.run({"image_embedder": {"documents": documents}})
|
||||
|
||||
# Query pipeline: text -> embedding -> vector retriever
|
||||
query = Pipeline()
|
||||
query.add_component(
|
||||
"text_embedder",
|
||||
AmazonBedrockTextEmbedder(model="cohere.embed-english-v3"),
|
||||
)
|
||||
query.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
res = query.run({"text_embedder": {"text": "Which document shows a horse?"}})
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
:notebook: Tutorial: [Creating Vision+Text RAG Pipelines](https://haystack.deepset.ai/tutorials/46_multimodal_rag)
|
||||
@@ -0,0 +1,140 @@
|
||||
---
|
||||
title: "AmazonBedrockTextEmbedder"
|
||||
id: amazonbedrocktextembedder
|
||||
slug: "/amazonbedrocktextembedder"
|
||||
description: "This component computes embeddings for text (such as a query) using models through Amazon Bedrock API."
|
||||
---
|
||||
|
||||
# AmazonBedrockTextEmbedder
|
||||
|
||||
This component computes embeddings for text (such as a query) using models through Amazon Bedrock API.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before an embedding [Retriever](../retrievers.mdx) in a query/RAG pipeline |
|
||||
| **Mandatory init variables** | `model`: The embedding model to use <br /> <br />`aws_access_key_id`: AWS access key ID. Can be set with `AWS_ACCESS_KEY_ID` env var. <br /> <br />`aws_secret_access_key`: AWS secret access key. Can be set with `AWS_SECRET_ACCESS_KEY` env var. <br /> <br />`aws_region_name`: AWS region name. Can be set with `AWS_DEFAULT_REGION` env var. |
|
||||
| **Mandatory run variables** | `text`: A string |
|
||||
| **Output variables** | `embedding`: A list of float numbers (vector) |
|
||||
| **API reference** | [Amazon Bedrock](/reference/integrations-amazon-bedrock) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/amazon_bedrock |
|
||||
| **Package name** | `amazon-bedrock-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
[Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) is a fully managed service that makes language models from leading AI startups and Amazon available for your use through a unified API.
|
||||
|
||||
Supported models are `amazon.titan-embed-text-v1`, `cohere.embed-english-v3` and `cohere.embed-multilingual-v3`.
|
||||
|
||||
Use `AmazonBedrockTextEmbedder` to embed a simple string (such as a query) into a vector. Use the [`AmazonBedrockDocumentEmbedder`](amazonbedrockdocumentembedder.mdx) to enrich the documents with the computed embedding, also known as vector.
|
||||
|
||||
### Authentication
|
||||
|
||||
`AmazonBedrockTextEmbedder` uses AWS for authentication. You can either provide credentials as parameters directly to the component or use the AWS CLI and authenticate through your IAM. For more information on how to set up an IAM identity-based policy, see the [official documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html).
|
||||
To initialize `AmazonBedrockTextEmbedder` and authenticate by providing credentials, provide the `model` name, as well as `aws_access_key_id`, `aws_secret_access_key`, and `aws_region_name`. Other parameters are optional, you can check them out in our [API reference](/reference/integrations-amazon-bedrock#amazonbedrocktextembedder).
|
||||
|
||||
### Model-specific parameters
|
||||
|
||||
Even if Haystack provides a unified interface, each model offered by Bedrock can accept specific parameters. You can pass these parameters at initialization.
|
||||
|
||||
For example, the Cohere models support `input_type` and `truncate`, as seen in [Bedrock documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters.html).
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.amazon_bedrock import (
|
||||
AmazonBedrockTextEmbedder,
|
||||
)
|
||||
|
||||
embedder = AmazonBedrockTextEmbedder(
|
||||
model="cohere.embed-english-v3",
|
||||
input_type="search_query",
|
||||
truncate="LEFT",
|
||||
)
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Installation
|
||||
|
||||
You need to install `amazon-bedrock-haystack` package to use the `AmazonBedrockTextEmbedder`:
|
||||
|
||||
```shell
|
||||
pip install amazon-bedrock-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage:
|
||||
|
||||
```python
|
||||
import os
|
||||
from haystack_integrations.components.embedders.amazon_bedrock import (
|
||||
AmazonBedrockTextEmbedder,
|
||||
)
|
||||
|
||||
os.environ["AWS_ACCESS_KEY_ID"] = "..."
|
||||
os.environ["AWS_SECRET_ACCESS_KEY"] = "..."
|
||||
os.environ["AWS_DEFAULT_REGION"] = "us-east-1" # just an example
|
||||
|
||||
text_to_embed = "I love pizza!"
|
||||
|
||||
text_embedder = AmazonBedrockTextEmbedder(
|
||||
model="cohere.embed-english-v3",
|
||||
input_type="search_query",
|
||||
)
|
||||
|
||||
print(text_embedder.run(text_to_embed))
|
||||
# {'embedding': [-0.453125, 1.2236328, 2.0058594, 0.67871094...]}
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
In a RAG pipeline:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.embedders.amazon_bedrock import (
|
||||
AmazonBedrockDocumentEmbedder,
|
||||
AmazonBedrockTextEmbedder,
|
||||
)
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
Document(content="Germany has many big cities"),
|
||||
]
|
||||
|
||||
document_embedder = AmazonBedrockDocumentEmbedder(model="cohere.embed-english-v3")
|
||||
documents_with_embeddings = document_embedder.run(documents)["documents"]
|
||||
document_store.write_documents(documents_with_embeddings)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component(
|
||||
"text_embedder",
|
||||
AmazonBedrockTextEmbedder(model="cohere.embed-english-v3"),
|
||||
)
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "Who lives in Berlin?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
|
||||
# Document(id=..., content: 'My name is Wolfgang and I live in Berlin')
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [PDF-Based Question Answering with Amazon Bedrock and Haystack](https://haystack.deepset.ai/cookbook/amazon_bedrock_for_documentation_qa)
|
||||
@@ -0,0 +1,128 @@
|
||||
---
|
||||
title: "AzureOpenAIDocumentEmbedder"
|
||||
id: azureopenaidocumentembedder
|
||||
slug: "/azureopenaidocumentembedder"
|
||||
description: "This component computes the embeddings of a list of documents and stores the obtained vectors in the embedding field of each document. It uses Azure cognitive services for text and document embedding with models deployed on Azure."
|
||||
---
|
||||
|
||||
# AzureOpenAIDocumentEmbedder
|
||||
|
||||
This component computes the embeddings of a list of documents and stores the obtained vectors in the embedding field of each document. It uses Azure cognitive services for text and document embedding with models deployed on Azure.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: The Azure OpenAI API key. Can be set with `AZURE_OPENAI_API_KEY` env var. <br />`azure_endpoint`: The endpoint of the model deployed on Azure. |
|
||||
| **Mandatory run variables** | `documents`: A list of documents |
|
||||
| **Output variables** | `documents`: A list of documents (enriched with embeddings) <br /> <br />`meta`: A dictionary of metadata |
|
||||
| **API reference** | [Embedders](/reference/embedders-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/embedders/azure_document_embedder.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The vectors computed by this component are necessary to perform embedding retrieval on a collection of documents. At retrieval time, the vector representing the query is compared with those of the documents to find the most similar or relevant documents.
|
||||
|
||||
To see the list of compatible embedding models, head over to Azure [documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models?source=recommendations). The default model for `AzureOpenAITextEmbedder` is `text-embedding-ada-002`.
|
||||
|
||||
This component should be used to embed a list of documents. To embed a string, you should use the [`AzureOpenAITextEmbedder`](azureopenaitextembedder.mdx).
|
||||
|
||||
To work with Azure components, you will need an Azure OpenAI API key, as well as an Azure OpenAI Endpoint. You can learn more about them in Azure [documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference).
|
||||
|
||||
The component uses `AZURE_OPENAI_API_KEY` or `AZURE_OPENAI_AD_TOKEN` environment variables by default. Otherwise, you can pass `api_key` or `azure_ad_token` at initialization:
|
||||
|
||||
```python
|
||||
client = AzureOpenAIDocumentEmbedder(
|
||||
azure_endpoint="<Your Azure endpoint e.g. `https://your-company.azure.openai.com/>",
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
azure_deployment="<a model name>",
|
||||
)
|
||||
```
|
||||
|
||||
:::info
|
||||
We recommend using environment variables instead of initialization parameters.
|
||||
:::
|
||||
|
||||
### Embedding Metadata
|
||||
|
||||
Text documents often come with a set of metadata. If they are distinctive and semantically meaningful, you can embed them along with the text of the document to improve retrieval.
|
||||
|
||||
You can do this easily by using the Document Embedder:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.embedders import AzureOpenAIDocumentEmbedder
|
||||
|
||||
doc = Document(content="some text", meta={"title": "relevant title", "page number": 18})
|
||||
|
||||
embedder = AzureOpenAIDocumentEmbedder(meta_fields_to_embed=["title"])
|
||||
|
||||
docs_w_embeddings = embedder.run(documents=[doc])["documents"]
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.embedders import AzureOpenAIDocumentEmbedder
|
||||
|
||||
doc = Document(content="I love pizza!")
|
||||
|
||||
document_embedder = AzureOpenAIDocumentEmbedder()
|
||||
|
||||
result = document_embedder.run([doc])
|
||||
print(result["documents"][0].embedding)
|
||||
|
||||
# [0.017020374536514282, -0.023255806416273117, ...]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.embedders import (
|
||||
AzureOpenAITextEmbedder,
|
||||
AzureOpenAIDocumentEmbedder,
|
||||
)
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
Document(content="Germany has many big cities"),
|
||||
]
|
||||
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component("embedder", AzureOpenAIDocumentEmbedder())
|
||||
indexing_pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
indexing_pipeline.connect("embedder", "writer")
|
||||
|
||||
indexing_pipeline.run({"embedder": {"documents": documents}})
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", AzureOpenAITextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "Who lives in Berlin?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
|
||||
# Document(id=..., mimetype: 'text/plain',
|
||||
# text: 'My name is Wolfgang and I live in Berlin')
|
||||
```
|
||||
@@ -0,0 +1,110 @@
|
||||
---
|
||||
title: "AzureOpenAITextEmbedder"
|
||||
id: azureopenaitextembedder
|
||||
slug: "/azureopenaitextembedder"
|
||||
description: "When you perform embedding retrieval, you use this component to transform your query into a vector. Then, the embedding Retriever looks for similar or relevant documents."
|
||||
---
|
||||
|
||||
# AzureOpenAITextEmbedder
|
||||
|
||||
When you perform embedding retrieval, you use this component to transform your query into a vector. Then, the embedding Retriever looks for similar or relevant documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before an embedding [Retriever](../retrievers.mdx) in a query/RAG pipeline |
|
||||
| **Mandatory init variables** | `api_key`: The Azure OpenAI API key. Can be set with `AZURE_OPENAI_API_KEY` env var. <br />`azure_endpoint`: The endpoint of the model deployed on Azure. |
|
||||
| **Mandatory run variables** | `text`: A string |
|
||||
| **Output variables** | `embedding`: A list of float numbers <br /> <br />`meta`: A dictionary of metadata |
|
||||
| **API reference** | [Embedders](/reference/embedders-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/embedders/azure_text_embedder.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`AzureOpenAITextEmbedder` transforms a string into a vector that captures its semantics using an OpenAI embedding model. It uses Azure cognitive services for text and document embedding with models deployed on Azure.
|
||||
|
||||
To see the list of compatible embedding models, head over to Azure [documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models?source=recommendations). The default model for `AzureOpenAITextEmbedder` is `text-embedding-ada-002`.
|
||||
|
||||
Use `AzureOpenAITextEmbedder` to embed a simple string (such as a query) into a vector. For embedding lists of documents, use the [`AzureOpenAIDocumentEmbedder`](azureopenaidocumentembedder.mdx), which enriches the documents with the computed embedding, also known as vector.
|
||||
|
||||
To work with Azure components, you will need an Azure OpenAI API key, as well as an Azure OpenAI Endpoint. You can learn more about them in Azure [documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference).
|
||||
|
||||
The component uses `AZURE_OPENAI_API_KEY` or `AZURE_OPENAI_AD_TOKEN` environment variables by default. Otherwise, you can pass `api_key` or `azure_ad_token` at initialization:
|
||||
|
||||
```python
|
||||
client = AzureOpenAITextEmbedder(
|
||||
azure_endpoint="<Your Azure endpoint e.g. `https://your-company.azure.openai.com/>",
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
azure_deployment="<a model name>",
|
||||
)
|
||||
```
|
||||
|
||||
:::info
|
||||
We recommend using environment variables instead of initialization parameters.
|
||||
:::
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Here is how you can use the component on its own:
|
||||
|
||||
```python
|
||||
from haystack.components.embedders import AzureOpenAITextEmbedder
|
||||
|
||||
text_to_embed = "I love pizza!"
|
||||
|
||||
text_embedder = AzureOpenAITextEmbedder()
|
||||
|
||||
print(text_embedder.run(text_to_embed))
|
||||
|
||||
# {'embedding': [0.017020374536514282, -0.023255806416273117, ...],
|
||||
# 'meta': {'model': 'text-embedding-ada-002-v2',
|
||||
# 'usage': {'prompt_tokens': 4, 'total_tokens': 4}}}
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.embedders import (
|
||||
AzureOpenAITextEmbedder,
|
||||
AzureOpenAIDocumentEmbedder,
|
||||
)
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
Document(content="Germany has many big cities"),
|
||||
]
|
||||
|
||||
document_embedder = AzureOpenAIDocumentEmbedder()
|
||||
documents_with_embeddings = document_embedder.run(documents)["documents"]
|
||||
document_store.write_documents(documents_with_embeddings)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", AzureOpenAITextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "Who lives in Berlin?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
|
||||
# Document(id=..., mimetype: 'text/plain',
|
||||
# text: 'My name is Wolfgang and I live in Berlin')
|
||||
```
|
||||
@@ -0,0 +1,61 @@
|
||||
---
|
||||
title: "Choosing the Right Embedder"
|
||||
id: choosing-the-right-embedder
|
||||
slug: "/choosing-the-right-embedder"
|
||||
description: "This page provides information on choosing the right Embedder when working with Haystack. It explains the distinction between Text and Document Embedders and discusses API-based Embedders and Embedders with models running on-premise."
|
||||
---
|
||||
|
||||
# Choosing the Right Embedder
|
||||
|
||||
This page provides information on choosing the right Embedder when working with Haystack. It explains the distinction between Text and Document Embedders and discusses API-based Embedders and Embedders with models running on-premise.
|
||||
|
||||
Embedders in Haystack transform texts or documents into vector representations using pre-trained models. The embeddings produced by Haystack Embedders are fixed-length vectors. They capture contextual information and semantic relationships within the text.
|
||||
|
||||
Embeddings in isolation are only used for information retrieval purposes (to do semantic search/vector search). You can use the embeddings in your pipeline for tasks like question answering. The QA pipeline with embedding retrieval would then include the following steps:
|
||||
|
||||
1. Transform the query into a vector/embedding.
|
||||
2. Find similar documents based on the embedding similarity.
|
||||
3. Pass the query and the retrieved documents to a Language Model, which can be extractive or generative.
|
||||
|
||||
## Text and Document Embedders
|
||||
|
||||
There are two types of Embedders: text and document.
|
||||
|
||||
Text Embedders work with text strings and are most often used at the beginning of query pipelines. They convert query text into vector embeddings and send them to a Retriever.
|
||||
|
||||
Document Embedders embed Document objects and are most often used in indexing pipelines, after Converters, and before a DocumentWriter. They preserve the Document object format and add an embedding field with a list of float numbers.
|
||||
|
||||
You must use the same embedding model for text and documents. This means that if you use CohereDocumentEmbedder in your indexing pipeline, you must then use CohereTextEmbedder with the same model in your query pipeline.
|
||||
|
||||
## API-Based Embedders
|
||||
|
||||
These Embedders use external APIs to generate embeddings. They give you access to powerful models without needing to handle the computing yourself.
|
||||
|
||||
The costs associated with these solutions can vary. Depending on the solution you choose, you pay for the tokens consumed, both sent and generated, or for the hosting of the model, often billed per hour. Refer to the individual providers’ websites for detailed information.
|
||||
|
||||
Haystack supports the models offered by a variety of providers: **OpenAI**, **Cohere**, **Jina**, **Azure**, **Mistral**, and **Amazon Bedrock**, with more being added constantly.
|
||||
|
||||
Additionally, you could use Haystack’s **Hugging Face API Embedders** for prototyping with [HF Serverless Inference API](https://huggingface.co/docs/api-inference/en/index) or the [paid HF Inference Endpoints](https://huggingface.co/inference-endpoints/dedicated).
|
||||
|
||||
## On-Premise Embedders
|
||||
|
||||
On-premise Embedders allow you to host open models on your machine/infrastructure. This choice is ideal for local experimentation.
|
||||
|
||||
When you self-host an embedder, you can choose the model from plenty of open model options. The [Massive Text Embedding Benchmark (MTEB) Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) can be a good reference point for understanding retrieval performance and model size.
|
||||
|
||||
It is suitable in production scenarios where data privacy concerns drive the decision not to transmit data to external providers and you have ample computational resources (CPU or GPU).
|
||||
|
||||
Here are some options available in Haystack:
|
||||
|
||||
- **Sentence Transformers**: This library mostly uses PyTorch, so it can be a fast-running option if you’re using a GPU. On the other hand, Sentence Transformers are progressively adding support for more efficient backends, which do not require GPU.
|
||||
- **Hugging Face Text Embedding Inference**: This is a library for efficiently serving open embedding models on both CPU and GPU. In Haystack, it can be used via HuggingFace API Embedders.
|
||||
- **Hugging Face Optimum:** These Embedders are designed to run models faster on targeted hardware. They implement optimizations that are specific for a certain hardware, such as Intel IPEX.
|
||||
- **Fastembed**: Fastembed is optimized for running on standard machines even with low resources. It supports several types of embeddings, including sparse techniques (BM25, SPLADE) and classic dense embeddings.
|
||||
- **Ollama:** These Embedders run quantized models on CPU(+GPU). Embedding quality might be lower due to the quantization of regular models. However, this makes these models run efficiently on standard machines.
|
||||
- **Nvidia**: Nvidia Embedders are built on Nvidia's NIM and hosted on their optimized cloud platform. They give you both options: using models through their API or deploying models locally with Nvidia NIM.
|
||||
|
||||
***
|
||||
|
||||
:::info
|
||||
See the full list of Embedders available in Haystack on the main [Embedders](../embedders.mdx) page.
|
||||
:::
|
||||
@@ -0,0 +1,136 @@
|
||||
---
|
||||
title: "CohereDocumentEmbedder"
|
||||
id: coheredocumentembedder
|
||||
slug: "/coheredocumentembedder"
|
||||
description: "This component computes the embeddings of a list of documents and stores the obtained vectors in the embedding field of each document. It uses Cohere embedding models."
|
||||
---
|
||||
|
||||
# CohereDocumentEmbedder
|
||||
|
||||
This component computes the embeddings of a list of documents and stores the obtained vectors in the embedding field of each document. It uses Cohere embedding models.
|
||||
|
||||
The vectors computed by this component are necessary to perform embedding retrieval on a collection of documents. At retrieval time, the vector that represents the query is compared with those of the documents to find the most similar or relevant documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline |
|
||||
| **Mandatory init variables** | `api_key`: The Cohere API key. Can be set with `COHERE_API_KEY` or `CO_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `documents`: A list of documents to be embedded |
|
||||
| **Output variables** | `documents`: A list of documents (enriched with embeddings) <br /> <br />`meta`: A dictionary of metadata strings |
|
||||
| **API reference** | [Cohere](/reference/integrations-cohere) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/cohere |
|
||||
| **Package name** | `cohere-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`CohereDocumentEmbedder` enriches the metadata of documents with an embedding of their content. To embed a string, you should use the [`CohereTextEmbedder`](coheretextembedder.mdx).
|
||||
|
||||
The component supports the following Cohere models:
|
||||
`"embed-english-v3.0"`, `"embed-english-light-v3.0"`, `"embed-multilingual-v3.0"`,
|
||||
`"embed-multilingual-light-v3.0"`, `"embed-english-v2.0"`, `"embed-english-light-v2.0"`,
|
||||
`"embed-multilingual-v2.0"`. The default model is `embed-english-v2.0`. This list of all supported models can be found in Cohere’s [model documentation](https://docs.cohere.com/docs/models#representation).
|
||||
|
||||
To start using this integration with Haystack, install it with:
|
||||
|
||||
```shell
|
||||
pip install cohere-haystack
|
||||
```
|
||||
|
||||
The component uses a `COHERE_API_KEY` or `CO_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`:
|
||||
|
||||
```python
|
||||
embedder = CohereDocumentEmbedder(api_key=Secret.from_token("<your-api-key>"))
|
||||
```
|
||||
|
||||
To get a Cohere API key, head over to https://cohere.com/.
|
||||
|
||||
### Embedding Metadata
|
||||
|
||||
Text documents often come with a set of metadata. If they are distinctive and semantically meaningful, you can embed them along with the text of the document to improve retrieval.
|
||||
|
||||
You can do this by using the Document Embedder:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from cohere_haystack.embedders.document_embedder import CohereDocumentEmbedder
|
||||
|
||||
doc = Document(content="some text", meta={"title": "relevant title", "page number": 18})
|
||||
|
||||
embedder = CohereDocumentEmbedder(api_key=Secret.from_token("<your-api-key>", meta_fields_to_embed=["title"])
|
||||
|
||||
docs_w_embeddings = embedder.run(documents=[doc])["documents"]
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Remember to set `COHERE_API_KEY` as an environment variable first, or pass it in directly.
|
||||
|
||||
Here is how you can use the component on its own:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.embedders.cohere.document_embedder import (
|
||||
CohereDocumentEmbedder,
|
||||
)
|
||||
|
||||
doc = Document(content="I love pizza!")
|
||||
|
||||
embedder = CohereDocumentEmbedder()
|
||||
|
||||
result = embedder.run([doc])
|
||||
print(result["documents"][0].embedding)
|
||||
# [-0.453125, 1.2236328, 2.0058594, 0.67871094...]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
|
||||
from haystack_integrations.components.embedders.cohere.document_embedder import (
|
||||
CohereDocumentEmbedder,
|
||||
)
|
||||
from haystack_integrations.components.embedders.cohere.text_embedder import (
|
||||
CohereTextEmbedder,
|
||||
)
|
||||
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
Document(content="Germany has many big cities"),
|
||||
]
|
||||
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component("embedder", CohereDocumentEmbedder())
|
||||
indexing_pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
indexing_pipeline.connect("embedder", "writer")
|
||||
|
||||
indexing_pipeline.run({"embedder": {"documents": documents}})
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", CohereTextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "Who lives in Berlin?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
|
||||
# Document(id=..., text: 'My name is Wolfgang and I live in Berlin')
|
||||
```
|
||||
@@ -0,0 +1,166 @@
|
||||
---
|
||||
title: "CohereDocumentImageEmbedder"
|
||||
id: coheredocumentimageembedder
|
||||
slug: "/coheredocumentimageembedder"
|
||||
description: "`CohereDocumentImageEmbedder` computes the image embeddings of a list of documents and stores the obtained vectors in the embedding field of each document. It uses Cohere embedding models with the ability to embed text and images into the same vector space."
|
||||
---
|
||||
|
||||
# CohereDocumentImageEmbedder
|
||||
|
||||
`CohereDocumentImageEmbedder` computes the image embeddings of a list of documents and stores the obtained vectors in the embedding field of each document. It uses Cohere embedding models with the ability to embed text and images into the same vector space.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline |
|
||||
| **Mandatory init variables** | `api_key`: The Cohere API key. Can be set with `COHERE_API_KEY` or `CO_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `documents`: A list of documents, with a meta field containing an image file path |
|
||||
| **Output variables** | `documents`: A list of documents (enriched with embeddings) |
|
||||
| **API reference** | [Cohere](/reference/integrations-cohere) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/cohere |
|
||||
| **Package name** | `cohere-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`CohereDocumentImageEmbedder` expects a list of documents containing an image or a PDF file path in a meta field. The meta field can be specified with the `file_path_meta_field` init parameter of this component.
|
||||
|
||||
The embedder efficiently loads the images, computes the embeddings using a Cohere model, and stores each of them in the `embedding` field of the document.
|
||||
|
||||
`CohereDocumentImageEmbedder` is commonly used in indexing pipelines. At retrieval time, you need to use the same model with a `CohereTextEmbedder` to embed the query, before using an Embedding Retriever.
|
||||
|
||||
This component is compatible with Cohere Embed models v3 and later. For a complete list of supported models, see the [Cohere documentation](https://docs.cohere.com/docs/models#embed).
|
||||
|
||||
### Installation
|
||||
|
||||
To start using this integration with Haystack, install the package with:
|
||||
|
||||
```shell
|
||||
pip install cohere-haystack
|
||||
```
|
||||
|
||||
### Authentication
|
||||
|
||||
The component uses a `COHERE_API_KEY` or `CO_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with a [Secret](../../concepts/secret-management.mdx) and `Secret.from_token` method:
|
||||
|
||||
```python
|
||||
embedder = CohereTextEmbedder(api_key=Secret.from_token("<your-api-key>"))
|
||||
```
|
||||
|
||||
To get a Cohere API key, head over to https://cohere.com/.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Remember to set `COHERE_API_KEY` as an environment variable first.
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.embedders.cohere import (
|
||||
CohereDocumentImageEmbedder,
|
||||
)
|
||||
|
||||
embedder = CohereDocumentImageEmbedder(model="embed-v4.0")
|
||||
|
||||
documents = [
|
||||
Document(content="A photo of a cat", meta={"file_path": "cat.jpg"}),
|
||||
Document(content="A photo of a dog", meta={"file_path": "dog.jpg"}),
|
||||
]
|
||||
|
||||
result = embedder.run(documents=documents)
|
||||
documents_with_embeddings = result["documents"]
|
||||
print(documents_with_embeddings)
|
||||
|
||||
# [Document(id=...,
|
||||
# content='A photo of a cat',
|
||||
# meta={'file_path': 'cat.jpg',
|
||||
# 'embedding_source': {'type': 'image', 'file_path_meta_field': 'file_path'}},
|
||||
# embedding=vector of size 1536),
|
||||
# ...]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
In this example, we can see an indexing pipeline with three components:
|
||||
|
||||
- `ImageFileToDocument` converter that creates empty documents with a reference to an image in the `meta.file_path` field;
|
||||
- `CohereDocumentImageEmbedder` that loads the images, computes embeddings and store them in documents;
|
||||
- `DocumentWriter` that writes the documents in the `InMemoryDocumentStore`.
|
||||
|
||||
There is also a multimodal retrieval pipeline, composed of a `CohereTextEmbedder` (using the same model as before) and an `InMemoryEmbeddingRetriever`.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.converters.image import ImageFileToDocument
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
from haystack_integrations.components.embedders.cohere import (
|
||||
CohereDocumentImageEmbedder,
|
||||
CohereTextEmbedder,
|
||||
)
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
# Indexing pipeline
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component("image_converter", ImageFileToDocument())
|
||||
indexing_pipeline.add_component(
|
||||
"embedder",
|
||||
CohereDocumentImageEmbedder(model="embed-v4.0"),
|
||||
)
|
||||
indexing_pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
indexing_pipeline.connect("image_converter", "embedder")
|
||||
indexing_pipeline.connect("embedder", "writer")
|
||||
|
||||
indexing_pipeline.run(data={"image_converter": {"sources": ["dog.jpg", "hyena.jpeg"]}})
|
||||
|
||||
# Multimodal retrieval pipeline
|
||||
retrieval_pipeline = Pipeline()
|
||||
retrieval_pipeline.add_component("embedder", CohereTextEmbedder(model="embed-v4.0"))
|
||||
retrieval_pipeline.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store, top_k=2),
|
||||
)
|
||||
retrieval_pipeline.connect("embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
result = retrieval_pipeline.run(data={"text": "man's best friend"})
|
||||
print(result)
|
||||
|
||||
# {
|
||||
# 'retriever': {
|
||||
# 'documents': [
|
||||
# Document(
|
||||
# id=0c96...,
|
||||
# meta={
|
||||
# 'file_path': 'dog.jpg',
|
||||
# 'embedding_source': {
|
||||
# 'type': 'image',
|
||||
# 'file_path_meta_field': 'file_path'
|
||||
# }
|
||||
# },
|
||||
# score=0.288
|
||||
# ),
|
||||
# Document(
|
||||
# id=5e76...,
|
||||
# meta={
|
||||
# 'file_path': 'hyena.jpeg',
|
||||
# 'embedding_source': {
|
||||
# 'type': 'image',
|
||||
# 'file_path_meta_field': 'file_path'
|
||||
# }
|
||||
# },
|
||||
# score=0.248
|
||||
# )
|
||||
# ]
|
||||
# }
|
||||
# }
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
:notebook: Tutorial: [Creating Vision+Text RAG Pipelines](https://haystack.deepset.ai/tutorials/46_multimodal_rag)
|
||||
@@ -0,0 +1,110 @@
|
||||
---
|
||||
title: "CohereTextEmbedder"
|
||||
id: coheretextembedder
|
||||
slug: "/coheretextembedder"
|
||||
description: "This component transforms a string into a vector that captures its semantics using a Cohere embedding model. When you perform embedding retrieval, you use this component to transform your query into a vector. Then, the embedding Retriever looks for similar or relevant documents."
|
||||
---
|
||||
|
||||
# CohereTextEmbedder
|
||||
|
||||
This component transforms a string into a vector that captures its semantics using a Cohere embedding model. When you perform embedding retrieval, you use this component to transform your query into a vector. Then, the embedding Retriever looks for similar or relevant documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before an embedding [Retriever](../retrievers.mdx) in a query/RAG pipeline |
|
||||
| **Mandatory init variables** | `api_key`: The Cohere API key. Can be set with `COHERE_API_KEY` or `CO_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `text`: A string |
|
||||
| **Output variables** | `embedding`: A list of float numbers (vectors) <br /> <br />`meta`: A dictionary of metadata strings |
|
||||
| **API reference** | [Cohere](/reference/integrations-cohere) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/cohere |
|
||||
| **Package name** | `cohere-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`CohereTextEmbedder` embeds a simple string (such as a query) into a vector. For embedding lists of documents, use the use the [`CohereDocumentEmbedder`](coheredocumentembedder.mdx), which enriches the document with the computed embedding, also known as vector.
|
||||
|
||||
The component supports the following Cohere models:
|
||||
`"embed-english-v3.0"`, `"embed-english-light-v3.0"`, `"embed-multilingual-v3.0"`,
|
||||
`"embed-multilingual-light-v3.0"`, `"embed-english-v2.0"`, `"embed-english-light-v2.0"`,
|
||||
`"embed-multilingual-v2.0"`. The default model is `embed-english-v2.0`. This list of all supported models can be found in Cohere’s [model documentation](https://docs.cohere.com/docs/models#representation).
|
||||
|
||||
To start using this integration with Haystack, install it with:
|
||||
|
||||
```shell
|
||||
pip install cohere-haystack
|
||||
```
|
||||
|
||||
The component uses a `COHERE_API_KEY` or `CO_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with a [Secret](../../concepts/secret-management.mdx) and `Secret.from_token` static method:
|
||||
|
||||
```python
|
||||
embedder = CohereTextEmbedder(api_key=Secret.from_token("<your-api-key>"))
|
||||
```
|
||||
|
||||
To get a Cohere API key, head over to https://cohere.com/.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Here is how you can use the component on its own. You’ll need to pass in your Cohere API key via Secret or set it as an environment variable called `COHERE_API_KEY`. The examples below assume you've set the environment variable.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.cohere.text_embedder import (
|
||||
CohereTextEmbedder,
|
||||
)
|
||||
|
||||
text_to_embed = "I love pizza!"
|
||||
|
||||
text_embedder = CohereTextEmbedder()
|
||||
|
||||
print(text_embedder.run(text_to_embed))
|
||||
# {'embedding': [-0.453125, 1.2236328, 2.0058594, 0.67871094...],
|
||||
# 'meta': {'api_version': {'version': '1'}, 'billed_units': {'input_tokens': 4}}}
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.embedders.cohere.text_embedder import (
|
||||
CohereTextEmbedder,
|
||||
)
|
||||
from haystack_integrations.components.embedders.cohere.document_embedder import (
|
||||
CohereDocumentEmbedder,
|
||||
)
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
Document(content="Germany has many big cities"),
|
||||
]
|
||||
|
||||
document_embedder = CohereDocumentEmbedder()
|
||||
documents_with_embeddings = document_embedder.run(documents)["documents"]
|
||||
document_store.write_documents(documents_with_embeddings)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", CohereTextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "Who lives in Berlin?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
|
||||
# Document(id=..., content: 'My name is Wolfgang and I live in Berlin')
|
||||
```
|
||||
@@ -0,0 +1,16 @@
|
||||
---
|
||||
title: "External Integrations"
|
||||
id: external-integrations-embedders
|
||||
slug: "/external-integrations-embedders"
|
||||
description: "External integrations that enable transforming texts or documents into vector representations using pre-trained models."
|
||||
---
|
||||
|
||||
# External Integrations
|
||||
|
||||
External integrations that enable transforming texts or documents into vector representations using pre-trained models.
|
||||
|
||||
| Name | Description |
|
||||
| --- | --- |
|
||||
| [mixedbread ai](https://haystack.deepset.ai/integrations/mixedbread-ai) | Compute embeddings for text and documents using mixedbread's API. |
|
||||
| [Isaacus](https://haystack.deepset.ai/integrations/isaacus) | Use the latest foundational legal AI models from Isaacus in Haystack. |
|
||||
| [Voyage AI](https://haystack.deepset.ai/integrations/voyage) | Computing embeddings for text and documents using Voyage AI embedding models. |
|
||||
@@ -0,0 +1,172 @@
|
||||
---
|
||||
title: "FastembedDocumentEmbedder"
|
||||
id: fastembeddocumentembedder
|
||||
slug: "/fastembeddocumentembedder"
|
||||
description: "This component computes the embeddings of a list of documents using the models supported by FastEmbed."
|
||||
---
|
||||
|
||||
# FastembedDocumentEmbedder
|
||||
|
||||
This component computes the embeddings of a list of documents using the models supported by FastEmbed.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline |
|
||||
| **Mandatory run variables** | `documents`: A list of documents |
|
||||
| **Output variables** | `documents`: A list of documents (enriched with embeddings) |
|
||||
| **API reference** | [FastEmbed](/reference/fastembed-embedders) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/fastembed |
|
||||
| **Package name** | `fastembed-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
This component should be used to embed a list of documents. To embed a string, use the [`FastembedTextEmbedder`](fastembedtextembedder.mdx).
|
||||
|
||||
## Overview
|
||||
|
||||
`FastembedDocumentEmbedder` computes the embeddings of a list of documents and stores the obtained vectors in the embedding field of each document. It uses embedding [models supported by FastEmbed](https://qdrant.github.io/fastembed/examples/Supported_Models/).
|
||||
|
||||
The vectors computed by this component are necessary to perform embedding retrieval on a collection of documents. At retrieval time, the vector that represents the query is compared with those of the documents in order to find the most similar or relevant documents.
|
||||
|
||||
### Compatible models
|
||||
|
||||
You can find the original models in the [FastEmbed documentation](https://qdrant.github.io/fastembed/).
|
||||
|
||||
Nowadays, most of the models in the [Massive Text Embedding Benchmark (MTEB) Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) are compatible with FastEmbed. You can look for compatibility in the [supported model list](https://qdrant.github.io/fastembed/examples/Supported_Models/).
|
||||
|
||||
### Installation
|
||||
|
||||
To start using this integration with Haystack, install the package with:
|
||||
|
||||
```shell
|
||||
pip install fastembed-haystack
|
||||
```
|
||||
|
||||
### Parameters
|
||||
|
||||
You can set the path where the model will be stored in a cache directory. Also, you can set the number of threads a single `onnxruntime` session can use.
|
||||
|
||||
```python
|
||||
cache_dir= "/your_cacheDirectory"
|
||||
embedder = FastembedDocumentEmbedder(
|
||||
*model="*BAAI/bge-large-en-v1.5",
|
||||
cache_dir=cache_dir,
|
||||
threads=2
|
||||
)
|
||||
```
|
||||
|
||||
If you want to use the data parallel encoding, you can set the parameters `parallel` and `batch_size`.
|
||||
|
||||
- If parallel > 1, data-parallel encoding will be used. This is recommended for offline encoding of large datasets.
|
||||
- If parallel is 0, use all available cores.
|
||||
- If None, don't use data-parallel processing; use default `onnxruntime` threading instead.
|
||||
|
||||
:::tip
|
||||
If you create a Text Embedder and a Document Embedder based on the same model, Haystack uses the same resource behind the scenes to save resources.
|
||||
:::
|
||||
|
||||
### Embedding Metadata
|
||||
|
||||
Text documents often come with a set of metadata. If they are distinctive and semantically meaningful, you can embed them along with the text of the document to improve retrieval.
|
||||
|
||||
You can do this easily by using the Document Embedder:
|
||||
|
||||
```python
|
||||
from haystack.preview import Document
|
||||
from haystack_integrations.components.embedders.fastembed import (
|
||||
FastembedDocumentEmbedder,
|
||||
)
|
||||
|
||||
doc = Document(
|
||||
text="some text",
|
||||
metadata={"title": "relevant title", "page number": 18},
|
||||
)
|
||||
|
||||
embedder = FastembedDocumentEmbedder(
|
||||
model="BAAI/bge-small-en-v1.5",
|
||||
batch_size=256,
|
||||
metadata_fields_to_embed=["title"],
|
||||
)
|
||||
|
||||
docs_w_embeddings = embedder.run(documents=[doc])["documents"]
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import Document
|
||||
from haystack_integrations.components.embedders.fastembed import (
|
||||
FastembedDocumentEmbedder,
|
||||
)
|
||||
|
||||
document_list = [
|
||||
Document(content="I love pizza!"),
|
||||
Document(content="I like spaghetti"),
|
||||
]
|
||||
|
||||
doc_embedder = FastembedDocumentEmbedder()
|
||||
|
||||
result = doc_embedder.run(document_list)
|
||||
print(result["documents"][0].embedding)
|
||||
|
||||
# [-0.04235665127635002, 0.021791068837046623, ...]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack_integrations.components.embedders.fastembed import (
|
||||
FastembedDocumentEmbedder,
|
||||
FastembedTextEmbedder,
|
||||
)
|
||||
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
Document(content="Germany has many big cities"),
|
||||
Document(content="fastembed is supported by and maintained by Qdrant."),
|
||||
]
|
||||
|
||||
document_embedder = FastembedDocumentEmbedder()
|
||||
writer = DocumentWriter(document_store=document_store, policy=DuplicatePolicy.OVERWRITE)
|
||||
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component("document_embedder", document_embedder)
|
||||
indexing_pipeline.add_component("writer", writer)
|
||||
indexing_pipeline.connect("document_embedder", "writer")
|
||||
|
||||
indexing_pipeline.run({"document_embedder": {"documents": documents}})
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", FastembedTextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "Who supports fastembed?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0]) # noqa: T201
|
||||
|
||||
# Document(id=...,
|
||||
# content: 'fastembed is supported by and maintained by Qdrant.',
|
||||
# score: 0.758..)
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [RAG Pipeline Using FastEmbed for Embeddings Generation](https://haystack.deepset.ai/cookbook/rag_fastembed)
|
||||
@@ -0,0 +1,191 @@
|
||||
---
|
||||
title: "FastembedSparseDocumentEmbedder"
|
||||
id: fastembedsparsedocumentembedder
|
||||
slug: "/fastembedsparsedocumentembedder"
|
||||
description: "Use this component to enrich a list of documents with their sparse embeddings."
|
||||
---
|
||||
|
||||
# FastembedSparseDocumentEmbedder
|
||||
|
||||
Use this component to enrich a list of documents with their sparse embeddings.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline |
|
||||
| **Mandatory run variables** | `documents`: A list of documents |
|
||||
| **Output variables** | `documents`: A list of documents (enriched with sparse embeddings) |
|
||||
| **API reference** | [FastEmbed](/reference/fastembed-embedders) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/fastembed |
|
||||
| **Package name** | `fastembed-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
To compute a sparse embedding for a string, use the [`FastembedSparseTextEmbedder`](fastembedsparsetextembedder.mdx).
|
||||
|
||||
## Overview
|
||||
|
||||
`FastembedSparseDocumentEmbedder` computes the sparse embeddings of a list of documents and stores the obtained vectors in the `sparse_embedding` field of each document. It uses sparse embedding [models](https://qdrant.github.io/fastembed/examples/Supported_Models/#supported-sparse-text-embedding-models) supported by FastEmbed.
|
||||
|
||||
The vectors calculated by this component are necessary for performing sparse embedding retrieval on a set of documents. During retrieval, the sparse vector representing the query is compared to those of the documents to identify the most similar or relevant ones.
|
||||
|
||||
### Compatible models
|
||||
|
||||
You can find the supported models in the [FastEmbed documentation](https://qdrant.github.io/fastembed/examples/Supported_Models/#supported-sparse-text-embedding-models).
|
||||
|
||||
Currently, supported models are based on SPLADE, a technique for producing sparse representations for text, where each non-zero value in the embedding is the importance weight of a term in the BERT WordPiece vocabulary. For more information, see [our docs](../retrievers.mdx#sparse-embedding-based-retrievers) that explain sparse embedding-based Retrievers further.
|
||||
|
||||
### Installation
|
||||
|
||||
To start using this integration with Haystack, install the package with:
|
||||
|
||||
```shell
|
||||
pip install fastembed-haystack
|
||||
```
|
||||
|
||||
### Parameters
|
||||
|
||||
You can set the path where the model will be stored in a cache directory. Also, you can set the number of threads a single `onnxruntime` session can use:
|
||||
|
||||
```python
|
||||
cache_dir = "/your_cacheDirectory"
|
||||
embedder = FastembedSparseDocumentEmbedder(
|
||||
model="prithivida/Splade_PP_en_v1",
|
||||
cache_dir=cache_dir,
|
||||
threads=2,
|
||||
)
|
||||
```
|
||||
|
||||
If you want to use the data parallel encoding, you can set the parameters `parallel` and `batch_size`.
|
||||
|
||||
- If `parallel` > 1, data-parallel encoding will be used. This is recommended for offline encoding of large datasets.
|
||||
- If `parallel` is 0, use all available cores.
|
||||
- If None, don't use data-parallel processing; use default `onnxruntime` threading instead.
|
||||
|
||||
:::tip
|
||||
If you create both a Sparse Text Embedder and a Sparse Document Embedder based on the same model, Haystack utilizes a shared resource behind the scenes to conserve resources.
|
||||
:::
|
||||
|
||||
### Embedding Metadata
|
||||
|
||||
Text documents often include metadata. If the metadata is distinctive and semantically meaningful, you can embed it along with the document's text to improve retrieval.
|
||||
|
||||
You can do this easily by using the sparse Document Embedder:
|
||||
|
||||
```python
|
||||
from haystack.preview import Document
|
||||
from haystack_integrations.components.embedders.fastembed import (
|
||||
FastembedSparseDocumentEmbedder,
|
||||
)
|
||||
|
||||
doc = Document(
|
||||
text="some text",
|
||||
metadata={"title": "relevant title", "page number": 18},
|
||||
)
|
||||
|
||||
embedder = FastembedSparseDocumentEmbedder(
|
||||
model="prithivida/Splade_PP_en_v1",
|
||||
metadata_fields_to_embed=["title"],
|
||||
)
|
||||
|
||||
docs_w_sparse_embeddings = embedder.run(documents=[doc])["documents"]
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import Document
|
||||
from haystack_integrations.components.embedders.fastembed import (
|
||||
FastembedSparseDocumentEmbedder,
|
||||
)
|
||||
|
||||
document_list = [
|
||||
Document(content="I love pizza!"),
|
||||
Document(content="I like spaghetti"),
|
||||
]
|
||||
|
||||
doc_embedder = FastembedSparseDocumentEmbedder()
|
||||
|
||||
result = doc_embedder.run(document_list)
|
||||
print(result["documents"][0])
|
||||
|
||||
# Document(id=...,
|
||||
# content: 'I love pizza!',
|
||||
# sparse_embedding: vector with 24 non-zero elements)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Currently, sparse embedding retrieval is only supported by `QdrantDocumentStore`.
|
||||
First, install the package with:
|
||||
|
||||
```shell
|
||||
pip install qdrant-haystack
|
||||
```
|
||||
|
||||
Then, try out this pipeline:
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack_integrations.components.retrievers.qdrant import (
|
||||
QdrantSparseEmbeddingRetriever,
|
||||
)
|
||||
from haystack_integrations.document_stores.qdrant import QdrantDocumentStore
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack_integrations.components.embedders.fastembed import (
|
||||
FastembedDocumentEmbedder,
|
||||
FastembedTextEmbedder,
|
||||
)
|
||||
|
||||
document_store = QdrantDocumentStore(
|
||||
":memory:",
|
||||
recreate_index=True,
|
||||
use_sparse_embeddings=True,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
Document(content="Germany has many big cities"),
|
||||
Document(content="fastembed is supported by and maintained by Qdrant."),
|
||||
]
|
||||
|
||||
sparse_document_embedder = FastembedSparseDocumentEmbedder()
|
||||
writer = DocumentWriter(document_store=document_store, policy=DuplicatePolicy.OVERWRITE)
|
||||
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component("sparse_document_embedder", sparse_document_embedder)
|
||||
indexing_pipeline.add_component("writer", writer)
|
||||
indexing_pipeline.connect("sparse_document_embedder", "writer")
|
||||
|
||||
indexing_pipeline.run({"sparse_document_embedder": {"documents": documents}})
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("sparse_text_embedder", FastembedSparseTextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"sparse_retriever",
|
||||
QdrantSparseEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect(
|
||||
"sparse_text_embedder.sparse_embedding",
|
||||
"sparse_retriever.query_sparse_embedding",
|
||||
)
|
||||
|
||||
query = "Who supports fastembed?"
|
||||
|
||||
result = query_pipeline.run({"sparse_text_embedder": {"text": query}})
|
||||
|
||||
print(result["sparse_retriever"]["documents"][0]) # noqa: T201
|
||||
|
||||
# Document(id=...,
|
||||
# content: 'fastembed is supported by and maintained by Qdrant.',
|
||||
# score: 0.758..)
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Sparse Embedding Retrieval with Qdrant and FastEmbed](https://haystack.deepset.ai/cookbook/sparse_embedding_retrieval)
|
||||
@@ -0,0 +1,155 @@
|
||||
---
|
||||
title: "FastembedSparseTextEmbedder"
|
||||
id: fastembedsparsetextembedder
|
||||
slug: "/fastembedsparsetextembedder"
|
||||
description: "Use this component to embed a simple string (such as a query) into a sparse vector."
|
||||
---
|
||||
|
||||
# FastembedSparseTextEmbedder
|
||||
|
||||
Use this component to embed a simple string (such as a query) into a sparse vector.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a sparse embedding [Retriever](../retrievers.mdx) in a query/RAG pipeline |
|
||||
| **Mandatory run variables** | `text`: A string |
|
||||
| **Output variables** | `sparse_embedding`: A [`SparseEmbedding`](../../concepts/data-classes.mdx#sparseembedding) object |
|
||||
| **API reference** | [FastEmbed](/reference/fastembed-embedders) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/fastembed |
|
||||
| **Package name** | `fastembed-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
For embedding lists of documents, use the [`FastembedSparseDocumentEmbedder`](fastembedsparsedocumentembedder.mdx), which enriches the document with the computed sparse embedding.
|
||||
|
||||
## Overview
|
||||
|
||||
`FastembedSparseTextEmbedder` transforms a string into a sparse vector using sparse embedding [models](https://qdrant.github.io/fastembed/examples/Supported_Models/#supported-sparse-text-embedding-models) supported by FastEmbed.
|
||||
|
||||
When you perform sparse embedding retrieval, use this component first to transform your query into a sparse vector. Then, the sparse embedding Retriever will use the vector to search for similar or relevant documents.
|
||||
|
||||
### Compatible Models
|
||||
|
||||
You can find the supported models in the [FastEmbed documentation](https://qdrant.github.io/fastembed/examples/Supported_Models/#supported-sparse-text-embedding-models).
|
||||
|
||||
Currently, supported models are based on SPLADE, a technique for producing sparse representations for text, where each non-zero value in the embedding is the importance weight of a term in the BERT WordPiece vocabulary. For more information, see [our docs](../retrievers.mdx#sparse-embedding-based-retrievers) that explain sparse embedding-based Retrievers further.
|
||||
|
||||
### Installation
|
||||
|
||||
To start using this integration with Haystack, install the package with:
|
||||
|
||||
```shell
|
||||
pip install fastembed-haystack
|
||||
```
|
||||
|
||||
### Parameters
|
||||
|
||||
You can set the path where the model will be stored in a cache directory. Also, you can set the number of threads a single `onnxruntime` session can use:
|
||||
|
||||
```python
|
||||
cache_dir = "/your_cacheDirectory"
|
||||
embedder = FastembedSparseTextEmbedder(
|
||||
model="prithivida/Splade_PP_en_v1",
|
||||
cache_dir=cache_dir,
|
||||
threads=2,
|
||||
)
|
||||
```
|
||||
|
||||
If you want to use the data parallel encoding, you can set the `parallel` parameter.
|
||||
|
||||
- If `parallel` > 1, data-parallel encoding will be used. This is recommended for offline encoding of large datasets.
|
||||
- If `parallel` is 0, use all available cores.
|
||||
- If None, don't use data-parallel processing; use the default `onnxruntime` threading instead.
|
||||
|
||||
:::tip
|
||||
If you create both a Sparse Text Embedder and a Sparse Document Embedder based on the same model, Haystack utilizes a shared resource behind the scenes to conserve resources.
|
||||
:::
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.fastembed import (
|
||||
FastembedSparseTextEmbedder,
|
||||
)
|
||||
|
||||
text = """It clearly says online this will work on a Mac OS system.
|
||||
The disk comes and it does not, only Windows.
|
||||
Do Not order this if you have a Mac!!"""
|
||||
|
||||
text_embedder = FastembedSparseTextEmbedder(model="prithivida/Splade_PP_en_v1")
|
||||
|
||||
sparse_embedding = text_embedder.run(text)["sparse_embedding"]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Currently, sparse embedding retrieval is only supported by `QdrantDocumentStore`.
|
||||
First, install the package with:
|
||||
|
||||
```shell
|
||||
pip install qdrant-haystack
|
||||
```
|
||||
|
||||
Then, try out this pipeline:
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack_integrations.document_stores.qdrant import QdrantDocumentStore
|
||||
from haystack_integrations.components.retrievers.qdrant import (
|
||||
QdrantSparseEmbeddingRetriever,
|
||||
)
|
||||
from haystack_integrations.components.embedders.fastembed import (
|
||||
FastembedSparseTextEmbedder,
|
||||
FastembedSparseDocumentEmbedder,
|
||||
FastembedTextEmbedder,
|
||||
)
|
||||
|
||||
document_store = QdrantDocumentStore(
|
||||
":memory:",
|
||||
recreate_index=True,
|
||||
use_sparse_embeddings=True,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
Document(content="Germany has many big cities"),
|
||||
Document(content="fastembed is supported by and maintained by Qdrant."),
|
||||
]
|
||||
|
||||
sparse_document_embedder = FastembedSparseDocumentEmbedder(
|
||||
model="prithivida/Splade_PP_en_v1",
|
||||
)
|
||||
|
||||
documents_with_sparse_embeddings = sparse_document_embedder.run(documents)["documents"]
|
||||
document_store.write_documents(documents_with_sparse_embeddings)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("sparse_text_embedder", FastembedSparseTextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"sparse_retriever",
|
||||
QdrantSparseEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect(
|
||||
"sparse_text_embedder.sparse_embedding",
|
||||
"sparse_retriever.query_sparse_embedding",
|
||||
)
|
||||
|
||||
query = "Who supports fastembed?"
|
||||
|
||||
result = query_pipeline.run({"sparse_text_embedder": {"text": query}})
|
||||
|
||||
print(result["sparse_retriever"]["documents"][0]) # noqa: T201
|
||||
|
||||
# Document(id=...,
|
||||
# content: 'fastembed is supported by and maintained by Qdrant.',
|
||||
# score: 0.561..)
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Sparse Embedding Retrieval with Qdrant and FastEmbed](https://haystack.deepset.ai/cookbook/sparse_embedding_retrieval)
|
||||
@@ -0,0 +1,143 @@
|
||||
---
|
||||
title: "FastembedTextEmbedder"
|
||||
id: fastembedtextembedder
|
||||
slug: "/fastembedtextembedder"
|
||||
description: "This component computes the embeddings of a string using embedding models supported by FastEmbed."
|
||||
---
|
||||
|
||||
# FastembedTextEmbedder
|
||||
|
||||
This component computes the embeddings of a string using embedding models supported by FastEmbed.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before an embedding [Retriever](../retrievers.mdx) in a query/RAG pipeline |
|
||||
| **Mandatory run variables** | `text`: A string |
|
||||
| **Output variables** | `embedding`: A vector (list of float numbers) |
|
||||
| **API reference** | [FastEmbed](/reference/fastembed-embedders) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/fastembed |
|
||||
| **Package name** | `fastembed-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
This component should be used to embed a simple string (such as a query) into a vector. For embedding lists of documents, use the [`FastembedDocumentEmbedder`](fastembeddocumentembedder.mdx), which enriches the document with the computed embedding, known as vector.
|
||||
|
||||
## Overview
|
||||
|
||||
`FastembedTextEmbedder` transforms a string into a vector that captures its semantics using embedding [models supported by FastEmbed](https://qdrant.github.io/fastembed/examples/Supported_Models/).
|
||||
|
||||
When you perform embedding retrieval, use this component first to transform your query into a vector. Then, the embedding Retriever will use the vector to search for similar or relevant documents.
|
||||
|
||||
### Compatible models
|
||||
|
||||
You can find the original models in the [FastEmbed documentation](https://qdrant.github.io/fastembed/).
|
||||
|
||||
Currently, most of the models in the [Massive Text Embedding Benchmark (MTEB) Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) are compatible with FastEmbed. You can look for compatibility in the [supported model list](https://qdrant.github.io/fastembed/examples/Supported_Models/).
|
||||
|
||||
### Installation
|
||||
|
||||
To start using this integration with Haystack, install the package with:
|
||||
|
||||
```bash
|
||||
pip install fastembed-haystack
|
||||
```
|
||||
|
||||
### Instructions
|
||||
|
||||
Some recent models that you can find in MTEB require prepending the text with an instruction to work better for retrieval.
|
||||
For example, if you use `[BAAI/bge-large-en-v1.5](https://huggingface.co/BAAI/bge-large-en-v1.5#model-list)` model, you should prefix your query with the `instruction: “passage:”`.
|
||||
|
||||
This is how it works with `FastembedTextEmbedder`:
|
||||
|
||||
```python
|
||||
instruction = "passage:"
|
||||
embedder = FastembedTextEmbedder(
|
||||
*model="*BAAI/bge-large-en-v1.5",
|
||||
prefix=instruction)
|
||||
```
|
||||
|
||||
### Parameters
|
||||
|
||||
You can set the path where the model will be stored in a cache directory. Also, you can set the number of threads a single `onnxruntime` session can use.
|
||||
|
||||
```python
|
||||
cache_dir= "/your_cacheDirectory"
|
||||
embedder = FastembedTextEmbedder(
|
||||
*model="*BAAI/bge-large-en-v1.5",
|
||||
cache_dir=cache_dir,
|
||||
threads=2
|
||||
)
|
||||
```
|
||||
|
||||
If you want to use the data parallel encoding, you can set the parameters `parallel` and `batch_size`.
|
||||
|
||||
- If parallel > 1, data-parallel encoding will be used. This is recommended for offline encoding of large datasets.
|
||||
- If parallel is 0, use all available cores.
|
||||
- If None, don't use data-parallel processing; use default `onnxruntime` threading instead.
|
||||
|
||||
:::tip
|
||||
If you create a Text Embedder and a Document Embedder based on the same model, Haystack uses the same resource behind the scenes to save resources.
|
||||
:::
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.fastembed import FastembedTextEmbedder
|
||||
|
||||
text = """It clearly says online this will work on a Mac OS system.
|
||||
The disk comes and it does not, only Windows.
|
||||
Do Not order this if you have a Mac!!"""
|
||||
text_embedder = FastembedTextEmbedder(model="BAAI/bge-small-en-v1.5")
|
||||
embedding = text_embedder.run(text)["embedding"]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.embedders.fastembed import (
|
||||
FastembedDocumentEmbedder,
|
||||
FastembedTextEmbedder,
|
||||
)
|
||||
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
Document(content="Germany has many big cities"),
|
||||
Document(content="fastembed is supported by and maintained by Qdrant."),
|
||||
]
|
||||
|
||||
document_embedder = FastembedDocumentEmbedder()
|
||||
documents_with_embeddings = document_embedder.run(documents)["documents"]
|
||||
document_store.write_documents(documents_with_embeddings)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", FastembedTextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "Who supports FastEmbed?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0]) # noqa: T201
|
||||
|
||||
# Document(id=...,
|
||||
# content: 'FastEmbed is supported by and maintained by Qdrant.',
|
||||
# score: 0.758..)
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [RAG Pipeline Using FastEmbed for Embeddings Generation](https://haystack.deepset.ai/cookbook/rag_fastembed)
|
||||
@@ -0,0 +1,182 @@
|
||||
---
|
||||
title: "GoogleGenAIDocumentEmbedder"
|
||||
id: googlegenaidocumentembedder
|
||||
slug: "/googlegenaidocumentembedder"
|
||||
description: "The vectors computed by this component are necessary to perform embedding retrieval on a collection of documents. At retrieval time, the vector representing the query is compared with those of the documents to find the most similar or relevant documents."
|
||||
---
|
||||
|
||||
# GoogleGenAIDocumentEmbedder
|
||||
|
||||
The vectors computed by this component are necessary to perform embedding retrieval on a collection of documents. At retrieval time, the vector representing the query is compared with those of the documents to find the most similar or relevant documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a [DocumentWriter](../writers/documentwriter.mdx) in an indexing pipeline |
|
||||
| **Mandatory init variables** | `api_key`: The Google API key. Can be set with `GOOGLE_API_KEY` or `GEMINI_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `documents`: A list of documents to be embedded |
|
||||
| **Output variables** | `documents`: A list of documents (enriched with embeddings) <br /> <br />`meta`: A dictionary of metadata |
|
||||
| **API reference** | [Google GenAI](/reference/integrations-google-genai) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/google_genai |
|
||||
| **Package name** | `google-genai-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`GoogleGenAIDocumentEmbedder` enriches the metadata of documents with an embedding of their content. To embed a string, you should use the [`GoogleGenAITextEmbedder`](googlegenaitextembedder.mdx).
|
||||
|
||||
The component supports [Google AI Embedding models](https://ai.google.dev/gemini-api/docs/embeddings#model-versions).
|
||||
|
||||
`gemini-embedding-001` is the default model.
|
||||
|
||||
To start using this integration with Haystack, install it with:
|
||||
|
||||
```shell
|
||||
pip install google-genai-haystack
|
||||
```
|
||||
|
||||
### Authentication
|
||||
|
||||
Google Gen AI is compatible with both the Gemini Developer API and the Vertex AI API.
|
||||
|
||||
To use this component with the Gemini Developer API and get an API key, visit [Google AI Studio](https://aistudio.google.com/).
|
||||
To use this component with the Vertex AI API, visit [Google Cloud > Vertex AI](https://cloud.google.com/vertex-ai).
|
||||
|
||||
The component uses a `GOOGLE_API_KEY` or `GEMINI_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with a [Secret](../../concepts/secret-management.mdx) and `Secret.from_token` static method:
|
||||
|
||||
```python
|
||||
embedder = GoogleGenAIDocumentEmbedder(api_key=Secret.from_token("<your-api-key>"))
|
||||
```
|
||||
|
||||
The following examples show how to use the component with the Gemini Developer API and the Vertex AI API.
|
||||
|
||||
#### Gemini Developer API (API Key Authentication)
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.google_genai import (
|
||||
GoogleGenAIDocumentEmbedder,
|
||||
)
|
||||
|
||||
# set the environment variable (GOOGLE_API_KEY or GEMINI_API_KEY)
|
||||
chat_generator = GoogleGenAIDocumentEmbedder()
|
||||
```
|
||||
|
||||
#### Vertex AI (Application Default Credentials)
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.google_genai import (
|
||||
GoogleGenAIDocumentEmbedder,
|
||||
)
|
||||
|
||||
# Using Application Default Credentials (requires gcloud auth setup)
|
||||
chat_generator = GoogleGenAIDocumentEmbedder(
|
||||
api="vertex",
|
||||
vertex_ai_project="my-project",
|
||||
vertex_ai_location="us-central1",
|
||||
)
|
||||
```
|
||||
|
||||
#### Vertex AI (API Key Authentication)
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.google_genai import (
|
||||
GoogleGenAIDocumentEmbedder,
|
||||
)
|
||||
|
||||
# set the environment variable (GOOGLE_API_KEY or GEMINI_API_KEY)
|
||||
chat_generator = GoogleGenAIDocumentEmbedder(api="vertex")
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Embedding Metadata
|
||||
|
||||
Text documents often come with a set of metadata. If they are distinctive and semantically meaningful, you can embed them along with the text of the document to improve retrieval.
|
||||
|
||||
You can do this by using the Document Embedder:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.embedders.google_genai import (
|
||||
GoogleGenAIDocumentEmbedder,
|
||||
)
|
||||
|
||||
doc = Document(content="some text", meta={"title": "relevant title", "page number": 18})
|
||||
|
||||
embedder = GoogleGenAIDocumentEmbedder(
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
meta_fields_to_embed=["title"],
|
||||
)
|
||||
|
||||
docs_w_embeddings = embedder.run(documents=[doc])["documents"]
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Here is how you can use the component on its own. You'll need to pass in your Google API key via Secret or set it as an environment variable called `GOOGLE_API_KEY` or `GEMINI_API_KEY`. The examples below assume you've set the environment variable.
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.embedders.google_genai import (
|
||||
GoogleGenAIDocumentEmbedder,
|
||||
)
|
||||
|
||||
doc = Document(content="I love pizza!")
|
||||
|
||||
document_embedder = GoogleGenAIDocumentEmbedder()
|
||||
|
||||
result = document_embedder.run([doc])
|
||||
print(result["documents"][0].embedding)
|
||||
# [0.017020374536514282, -0.023255806416273117, ...]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.embedders.google_genai import (
|
||||
GoogleGenAITextEmbedder,
|
||||
)
|
||||
from haystack_integrations.components.embedders.google_genai import (
|
||||
GoogleGenAIDocumentEmbedder,
|
||||
)
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
Document(content="Germany has many big cities"),
|
||||
]
|
||||
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component("embedder", GoogleGenAIDocumentEmbedder())
|
||||
indexing_pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
indexing_pipeline.connect("embedder", "writer")
|
||||
|
||||
indexing_pipeline.run({"embedder": {"documents": documents}})
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", GoogleGenAITextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "Who lives in Berlin?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
|
||||
# Document(id=..., content: 'My name is Wolfgang and I live in Berlin')
|
||||
```
|
||||
+197
@@ -0,0 +1,197 @@
|
||||
---
|
||||
title: "GoogleGenAIMultimodalDocumentEmbedder"
|
||||
id: googlegenaimultimodaldocumentembedder
|
||||
slug: "/googlegenaimultimodaldocumentembedder"
|
||||
description: "`GoogleGenAIMultimodalDocumentEmbedder` computes the embeddings of a list of non-textual documents and stores the obtained vectors in the embedding field of each document."
|
||||
---
|
||||
|
||||
# GoogleGenAIMultimodalDocumentEmbedder
|
||||
|
||||
`GoogleGenAIMultimodalDocumentEmbedder` computes the embeddings of a list of non-textual documents and stores the obtained vectors in the embedding field of each document.
|
||||
It uses Google AI multimodal embedding models with the ability to embed text, images, videos, and audio into the same vector space.
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a [DocumentWriter](../writers/documentwriter.mdx) in an indexing pipeline |
|
||||
| **Mandatory init variables** | `api_key`: The Google API key. Can be set with `GOOGLE_API_KEY` or `GEMINI_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `documents`: A list of documents, with a meta field containing an image file path |
|
||||
| **Output variables** | `documents`: A list of documents (enriched with embeddings) <br /> <br />`meta`: A dictionary of metadata |
|
||||
| **API reference** | [Google GenAI](/reference/integrations-google-genai) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/google_genai |
|
||||
| **Package name** | `google-genai-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`GoogleGenAIMultimodalDocumentEmbedder` expects a list of documents containing a file path in a meta field. The meta field can be specified with the `file_path_meta_field` init parameter of this component.
|
||||
|
||||
The embedder efficiently loads the files, computes the embeddings using a Google AI model, and stores each of them in the `embedding` field of the document.
|
||||
|
||||
`GoogleGenAIMultimodalDocumentEmbedder` is commonly used in indexing pipelines. At retrieval time, you need to use the same model with a `GoogleGenAITextEmbedder` to embed the query, before using an Embedding Retriever.
|
||||
|
||||
This component is compatible with Gemini multimodal models: `gemini-embedding-2` and later. For a complete list of supported models, see the [Google AI documentation](https://ai.google.dev/gemini-api/docs/embeddings).
|
||||
|
||||
To embed a textual document, you should use the [`GoogleGenAIDocumentEmbedder`](googlegenaidocumentembedder.mdx).
|
||||
To embed a string, you should use the [`GoogleGenAITextEmbedder`](googlegenaitextembedder.mdx).
|
||||
|
||||
To start using this integration with Haystack, install it with:
|
||||
|
||||
```shell
|
||||
pip install google-genai-haystack
|
||||
```
|
||||
|
||||
### Authentication
|
||||
|
||||
Google Gen AI is compatible with both the Gemini Developer API and the Vertex AI API.
|
||||
|
||||
To use this component with the Gemini Developer API and get an API key, visit [Google AI Studio](https://aistudio.google.com/).
|
||||
To use this component with the Vertex AI API, visit [Google Cloud > Vertex AI](https://cloud.google.com/vertex-ai).
|
||||
|
||||
The component uses a `GOOGLE_API_KEY` or `GEMINI_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with a [Secret](../../concepts/secret-management.mdx) and `Secret.from_token` static method:
|
||||
|
||||
```python
|
||||
embedder = GoogleGenAIMultimodalDocumentEmbedder(
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
)
|
||||
```
|
||||
|
||||
The following examples show how to use the component with the Gemini Developer API and the Vertex AI API.
|
||||
|
||||
#### Gemini Developer API (API Key Authentication)
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.google_genai import (
|
||||
GoogleGenAIMultimodalDocumentEmbedder,
|
||||
)
|
||||
|
||||
# set the environment variable (GOOGLE_API_KEY or GEMINI_API_KEY)
|
||||
embedder = GoogleGenAIMultimodalDocumentEmbedder()
|
||||
```
|
||||
|
||||
#### Vertex AI (Application Default Credentials)
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.google_genai import (
|
||||
GoogleGenAIMultimodalDocumentEmbedder,
|
||||
)
|
||||
|
||||
# Using Application Default Credentials (requires gcloud auth setup)
|
||||
embedder = GoogleGenAIMultimodalDocumentEmbedder(
|
||||
api="vertex",
|
||||
vertex_ai_project="my-project",
|
||||
vertex_ai_location="us-central1",
|
||||
)
|
||||
```
|
||||
|
||||
#### Vertex AI (API Key Authentication)
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.google_genai import (
|
||||
GoogleGenAIMultimodalDocumentEmbedder,
|
||||
)
|
||||
|
||||
# set the environment variable (GOOGLE_API_KEY or GEMINI_API_KEY)
|
||||
embedder = GoogleGenAIMultimodalDocumentEmbedder(api="vertex")
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Here is how you can use the component on its own. You'll need to pass in your Google API key via Secret or set it as an environment variable called `GOOGLE_API_KEY` or `GEMINI_API_KEY`.
|
||||
The examples below assume you've set the environment variable.
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.embedders.google_genai import (
|
||||
GoogleGenAIMultimodalDocumentEmbedder,
|
||||
)
|
||||
|
||||
docs = [
|
||||
Document(meta={"file_path": "path/to/image.jpg"}),
|
||||
Document(meta={"file_path": "path/to/video.mp4"}),
|
||||
Document(meta={"file_path": "path/to/pdf.pdf", "page_number": 1}),
|
||||
Document(meta={"file_path": "path/to/pdf.pdf", "page_number": 3}),
|
||||
]
|
||||
|
||||
document_embedder = GoogleGenAIMultimodalDocumentEmbedder()
|
||||
|
||||
result = document_embedder.run(documents=docs)
|
||||
print(result["documents"][0].embedding)
|
||||
# [0.017020374536514282, -0.023255806416273117, ...]
|
||||
```
|
||||
|
||||
### Setting embedding dimensions
|
||||
|
||||
Models like `gemini-embedding-2` have a default embedding dimension of 3072, but, thanks to
|
||||
Matryoshka Representation Learning, it's possible to reduce embedding size while keeping similar performance.
|
||||
|
||||
Check the [Google AI documentation](https://ai.google.dev/gemini-api/docs/embeddings#control-embedding-size) for more information.
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
|
||||
from haystack_integrations.components.embedders.google_genai import (
|
||||
GoogleGenAIMultimodalDocumentEmbedder,
|
||||
)
|
||||
|
||||
docs = [Document(meta={"file_path": "path/to/image.jpg"})]
|
||||
|
||||
doc_multimodal_embedder = GoogleGenAIMultimodalDocumentEmbedder(
|
||||
config={"output_dimensionality": 768},
|
||||
)
|
||||
docs_with_embeddings = doc_multimodal_embedder.run(docs)["documents"]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
In the following example, we look for a specific plot in the "Scaling Instruction-Finetuned Language Models" paper (PDF format).
|
||||
|
||||
You first need to download the PDF file from https://arxiv.org/pdf/2210.11416.pdf.
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.embedders.google_genai import (
|
||||
GoogleGenAITextEmbedder,
|
||||
)
|
||||
from haystack_integrations.components.embedders.google_genai import (
|
||||
GoogleGenAIMultimodalDocumentEmbedder,
|
||||
)
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
paper_path = "2210.11416.pdf"
|
||||
|
||||
documents = [
|
||||
Document(meta={"file_path": paper_path, "page_number": i}) for i in range(1, 16)
|
||||
]
|
||||
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component("embedder", GoogleGenAIMultimodalDocumentEmbedder())
|
||||
indexing_pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
indexing_pipeline.connect("embedder", "writer")
|
||||
|
||||
indexing_pipeline.run({"embedder": {"documents": documents}})
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", GoogleGenAITextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "plot showing BBH accuracy"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0].meta)
|
||||
|
||||
# {'file_path': '2210.11416.pdf', 'page_number': 9}
|
||||
```
|
||||
@@ -0,0 +1,154 @@
|
||||
---
|
||||
title: "GoogleGenAITextEmbedder"
|
||||
id: googlegenaitextembedder
|
||||
slug: "/googlegenaitextembedder"
|
||||
description: "This component transforms a string into a vector that captures its semantics using a Google AI embedding models. When you perform embedding retrieval, you use this component to transform your query into a vector. Then, the embedding Retriever looks for similar or relevant documents."
|
||||
---
|
||||
|
||||
# GoogleGenAITextEmbedder
|
||||
|
||||
This component transforms a string into a vector that captures its semantics using a Google AI embedding models. When you perform embedding retrieval, you use this component to transform your query into a vector. Then, the embedding Retriever looks for similar or relevant documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before an embedding [Retriever](../retrievers.mdx) in a query/RAG pipeline |
|
||||
| **Mandatory init variables** | `api_key`: The Google API key. Can be set with `GOOGLE_API_KEY` or `GEMINI_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `text`: A string |
|
||||
| **Output variables** | `embedding`: A list of float numbers <br /> <br />`meta`: A dictionary of metadata |
|
||||
| **API reference** | [Google GenAI](/reference/integrations-google-genai) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/google_genai |
|
||||
| **Package name** | `google-genai-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`GoogleGenAITextEmbedder` embeds a simple string (such as a query) into a vector. For embedding lists of documents, use the [`GoogleGenAIDocumentEmbedder`](googlegenaidocumentembedder.mdx), which enriches the document with the computed embedding, also known as vector.
|
||||
|
||||
The component supports [Google AI Embedding models](https://ai.google.dev/gemini-api/docs/embeddings#model-versions).
|
||||
|
||||
`gemini-embedding-001` is the default model.
|
||||
|
||||
To start using this integration with Haystack, install it with:
|
||||
|
||||
```shell
|
||||
pip install google-genai-haystack
|
||||
```
|
||||
|
||||
### Authentication
|
||||
|
||||
Google Gen AI is compatible with both the Gemini Developer API and the Vertex AI API.
|
||||
|
||||
To use this component with the Gemini Developer API and get an API key, visit [Google AI Studio](https://aistudio.google.com/).
|
||||
To use this component with the Vertex AI API, visit [Google Cloud > Vertex AI](https://cloud.google.com/vertex-ai).
|
||||
|
||||
The component uses a `GOOGLE_API_KEY` or `GEMINI_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with a [Secret](../../concepts/secret-management.mdx) and `Secret.from_token` static method:
|
||||
|
||||
```python
|
||||
embedder = GoogleGenAITextEmbedder(api_key=Secret.from_token("<your-api-key>"))
|
||||
```
|
||||
|
||||
The following examples show how to use the component with the Gemini Developer API and the Vertex AI API.
|
||||
|
||||
#### Gemini Developer API (API Key Authentication)
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.google_genai import (
|
||||
GoogleGenAITextEmbedder,
|
||||
)
|
||||
|
||||
# set the environment variable (GOOGLE_API_KEY or GEMINI_API_KEY)
|
||||
chat_generator = GoogleGenAITextEmbedder()
|
||||
```
|
||||
|
||||
#### Vertex AI (Application Default Credentials)
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.google_genai import (
|
||||
GoogleGenAITextEmbedder,
|
||||
)
|
||||
|
||||
# Using Application Default Credentials (requires gcloud auth setup)
|
||||
chat_generator = GoogleGenAITextEmbedder(
|
||||
api="vertex",
|
||||
vertex_ai_project="my-project",
|
||||
vertex_ai_location="us-central1",
|
||||
)
|
||||
```
|
||||
|
||||
#### Vertex AI (API Key Authentication)
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.google_genai import (
|
||||
GoogleGenAITextEmbedder,
|
||||
)
|
||||
|
||||
# set the environment variable (GOOGLE_API_KEY or GEMINI_API_KEY)
|
||||
chat_generator = GoogleGenAITextEmbedder(api="vertex")
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Here is how you can use the component on its own. You'll need to pass in your Google API key with a Secret or set it as an environment variable called `GOOGLE_API_KEY` or `GEMINI_API_KEY`. The examples below assume you've set the environment variable.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.google_genai import (
|
||||
GoogleGenAITextEmbedder,
|
||||
)
|
||||
|
||||
text_to_embed = "I love pizza!"
|
||||
|
||||
text_embedder = GoogleGenAITextEmbedder()
|
||||
|
||||
print(text_embedder.run(text_to_embed))
|
||||
# {'embedding': [0.017020374536514282, -0.023255806416273117, ...],
|
||||
# 'meta': {'model': 'gemini-embedding-001',
|
||||
# 'usage': {'prompt_tokens': 4, 'total_tokens': 4}}}
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.embedders.google_genai import (
|
||||
GoogleGenAITextEmbedder,
|
||||
)
|
||||
from haystack_integrations.components.embedders.google_genai import (
|
||||
GoogleGenAIDocumentEmbedder,
|
||||
)
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
Document(content="Germany has many big cities"),
|
||||
]
|
||||
|
||||
document_embedder = GoogleGenAIDocumentEmbedder()
|
||||
documents_with_embeddings = document_embedder.run(documents)["documents"]
|
||||
document_store.write_documents(documents_with_embeddings)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", GoogleGenAITextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "Who lives in Berlin?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
|
||||
# Document(id=..., content: 'My name is Wolfgang and I live in Berlin')
|
||||
```
|
||||
@@ -0,0 +1,209 @@
|
||||
---
|
||||
title: "HuggingFaceAPIDocumentEmbedder"
|
||||
id: huggingfaceapidocumentembedder
|
||||
slug: "/huggingfaceapidocumentembedder"
|
||||
description: "Use this component to compute document embeddings using various Hugging Face APIs."
|
||||
---
|
||||
|
||||
# HuggingFaceAPIDocumentEmbedder
|
||||
|
||||
Use this component to compute document embeddings using various Hugging Face APIs.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline |
|
||||
| **Mandatory init variables** | `api_type`: The type of Hugging Face API to use <br /> <br />`api_params`: A dictionary with one of the following keys: <br /> <br />- `model`: Hugging Face model ID. Required when `api_type` is `SERVERLESS_INFERENCE_API`.**OR** - `url`: URL of the inference endpoint. Required when `api_type` is `INFERENCE_ENDPOINTS` or `TEXT_EMBEDDINGS_INFERENCE`. <br /> <br />`token`: The Hugging Face API token. Can be set with `HF_API_TOKEN` or `HF_TOKEN` env var. |
|
||||
| **Mandatory run variables** | `documents`: A list of documents to be embedded |
|
||||
| **Output variables** | `documents`: A list of documents to be embedded (enriched with embeddings) |
|
||||
| **API reference** | [Hugging Face API](/reference/integrations-huggingface-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/huggingface_api |
|
||||
| **Package name** | `huggingface-api-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`HuggingFaceAPIDocumentEmbedder` can be used to compute document embeddings using different Hugging Face APIs:
|
||||
|
||||
- [Free Serverless Inference API](https://huggingface.co/inference-api)
|
||||
- [Paid Inference Endpoints](https://huggingface.co/inference-endpoints)
|
||||
- [Self-hosted Text Embeddings Inference](https://github.com/huggingface/text-embeddings-inference)
|
||||
|
||||
:::info
|
||||
This component should be used to embed a list of documents. To embed a string, use [`HuggingFaceAPITextEmbedder`](huggingfaceapitextembedder.mdx).
|
||||
:::
|
||||
|
||||
The component uses a `HF_API_TOKEN` environment variable by default. Otherwise, you can pass a Hugging Face API token at initialization with `token` – see code examples below.
|
||||
The token is needed:
|
||||
|
||||
- If you use the Serverless Inference API, or
|
||||
- If you use the Inference Endpoints.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `huggingface-api-haystack` package to use the `HuggingFaceAPIDocumentEmbedder`:
|
||||
|
||||
```shell
|
||||
pip install huggingface-api-haystack
|
||||
```
|
||||
|
||||
Similarly to other Document Embedders, this component allows adding prefixes (and postfixes) to include instruction and embedding metadata.
|
||||
For more fine-grained details, refer to the component’s [API reference](/reference/integrations-huggingface-api#huggingfaceapidocumentembedder).
|
||||
|
||||
### On its own
|
||||
|
||||
#### Using Free Serverless Inference API
|
||||
|
||||
Formerly known as (free) Hugging Face Inference API, this API allows you to quickly experiment with many models hosted on the Hugging Face Hub, offloading the inference to Hugging Face servers. It’s rate-limited and not meant for production.
|
||||
|
||||
To use this API, you need a [free Hugging Face token](https://huggingface.co/settings/tokens).
|
||||
The Embedder expects the `model` in `api_params`.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.huggingface_api import (
|
||||
HuggingFaceAPIDocumentEmbedder,
|
||||
)
|
||||
from haystack.utils import Secret
|
||||
from haystack.dataclasses import Document
|
||||
|
||||
doc = Document(content="I love pizza!")
|
||||
|
||||
document_embedder = HuggingFaceAPIDocumentEmbedder(
|
||||
api_type="serverless_inference_api",
|
||||
api_params={"model": "BAAI/bge-small-en-v1.5"},
|
||||
token=Secret.from_token("<your-api-key>"),
|
||||
)
|
||||
|
||||
result = document_embedder.run([doc])
|
||||
print(result["documents"][0].embedding)
|
||||
|
||||
# [0.017020374536514282, -0.023255806416273117, ...]
|
||||
```
|
||||
|
||||
#### Using Paid Inference Endpoints
|
||||
|
||||
In this case, a private instance of the model is deployed by Hugging Face, and you typically pay per hour.
|
||||
|
||||
To understand how to spin up an Inference Endpoint, visit [Hugging Face documentation](https://huggingface.co/inference-endpoints/dedicated).
|
||||
|
||||
Additionally, in this case, you need to provide your Hugging Face token.
|
||||
The Embedder expects the `url` of your endpoint in `api_params`.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.huggingface_api import (
|
||||
HuggingFaceAPIDocumentEmbedder,
|
||||
)
|
||||
from haystack.utils import Secret
|
||||
from haystack.dataclasses import Document
|
||||
|
||||
doc = Document(content="I love pizza!")
|
||||
|
||||
document_embedder = HuggingFaceAPIDocumentEmbedder(
|
||||
api_type="inference_endpoints",
|
||||
api_params={"url": "<your-inference-endpoint-url>"},
|
||||
token=Secret.from_token("<your-api-key>"),
|
||||
)
|
||||
|
||||
result = document_embedder.run([doc])
|
||||
print(result["documents"][0].embedding)
|
||||
|
||||
# [0.017020374536514282, -0.023255806416273117, ...]
|
||||
```
|
||||
|
||||
#### Using Self-Hosted Text Embeddings Inference (TEI)
|
||||
|
||||
[Hugging Face Text Embeddings Inference](https://github.com/huggingface/text-embeddings-inference) is a toolkit for efficiently deploying and serving text embedding models.
|
||||
|
||||
While it powers the most recent versions of Serverless Inference API and Inference Endpoints, it can be used easily on-premise through Docker.
|
||||
|
||||
For example, you can run a TEI container as follows:
|
||||
|
||||
```shell
|
||||
model=BAAI/bge-large-en-v1.5
|
||||
revision=refs/pr/5
|
||||
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
|
||||
|
||||
docker run --gpus all -p 8080:80 -v $volume:/data --pull always ghcr.io/huggingface/text-embeddings-inference:1.2 --model-id $model --revision $revision
|
||||
```
|
||||
|
||||
For more information, refer to the [official TEI repository](https://github.com/huggingface/text-embeddings-inference).
|
||||
|
||||
The Embedder expects the `url` of your TEI instance in `api_params`.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.huggingface_api import (
|
||||
HuggingFaceAPIDocumentEmbedder,
|
||||
)
|
||||
from haystack.dataclasses import Document
|
||||
|
||||
doc = Document(content="I love pizza!")
|
||||
|
||||
document_embedder = HuggingFaceAPIDocumentEmbedder(
|
||||
api_type="text_embeddings_inference",
|
||||
api_params={"url": "http://localhost:8080"},
|
||||
)
|
||||
|
||||
result = document_embedder.run([doc])
|
||||
print(result["documents"][0].embedding)
|
||||
|
||||
# [0.017020374536514282, -0.023255806416273117, ...]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.embedders.huggingface_api import (
|
||||
HuggingFaceAPITextEmbedder,
|
||||
HuggingFaceAPIDocumentEmbedder,
|
||||
)
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
Document(content="Germany has many big cities"),
|
||||
]
|
||||
|
||||
document_embedder = HuggingFaceAPIDocumentEmbedder(
|
||||
api_type="serverless_inference_api",
|
||||
api_params={"model": "BAAI/bge-small-en-v1.5"},
|
||||
)
|
||||
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component("document_embedder", document_embedder)
|
||||
indexing_pipeline.add_component(
|
||||
"doc_writer",
|
||||
DocumentWriter(document_store=document_store),
|
||||
)
|
||||
indexing_pipeline.connect("document_embedder", "doc_writer")
|
||||
indexing_pipeline.run({"document_embedder": {"documents": documents}})
|
||||
|
||||
text_embedder = HuggingFaceAPITextEmbedder(
|
||||
api_type="serverless_inference_api",
|
||||
api_params={"model": "BAAI/bge-small-en-v1.5"},
|
||||
)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", text_embedder)
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "Who lives in Berlin?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
|
||||
# Document(id=..., content: 'My name is Wolfgang and I live in Berlin', ...)
|
||||
```
|
||||
@@ -0,0 +1,190 @@
|
||||
---
|
||||
title: "HuggingFaceAPITextEmbedder"
|
||||
id: huggingfaceapitextembedder
|
||||
slug: "/huggingfaceapitextembedder"
|
||||
description: "Use this component to embed strings using various Hugging Face APIs."
|
||||
---
|
||||
|
||||
# HuggingFaceAPITextEmbedder
|
||||
|
||||
Use this component to embed strings using various Hugging Face APIs.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before an embedding [Retriever](../retrievers.mdx) in a query/RAG pipeline |
|
||||
| **Mandatory init variables** | `api_type`: The type of Hugging Face API to use <br /> <br />`api_params`: A dictionary with one of the following keys: <br /> <br />- `model`: Hugging Face model ID. Required when `api_type` is `SERVERLESS_INFERENCE_API`.**OR** - `url`: URL of the inference endpoint. Required when `api_type` is `INFERENCE_ENDPOINTS` or `TEXT_EMBEDDINGS_INFERENCE`. <br /> <br />`token`: The Hugging Face API token. Can be set with `HF_API_TOKEN` or `HF_TOKEN` env var. |
|
||||
| **Mandatory run variables** | `text`: A string |
|
||||
| **Output variables** | `embedding`: A list of float numbers |
|
||||
| **API reference** | [Hugging Face API](/reference/integrations-huggingface-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/huggingface_api |
|
||||
| **Package name** | `huggingface-api-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`HuggingFaceAPITextEmbedder` can be used to embed strings using different Hugging Face APIs:
|
||||
|
||||
- [Free Serverless Inference API](https://huggingface.co/inference-api)
|
||||
- [Paid Inference Endpoints](https://huggingface.co/inference-endpoints)
|
||||
- [Self-hosted Text Embeddings Inference](https://github.com/huggingface/text-embeddings-inference)
|
||||
|
||||
:::info
|
||||
This component should be used to embed plain text. To embed a list of documents, use [`HuggingFaceAPIDocumentEmbedder`](huggingfaceapidocumentembedder.mdx).
|
||||
:::
|
||||
|
||||
The component uses a `HF_API_TOKEN` environment variable by default. Otherwise, you can pass a Hugging Face API token at initialization with `token` – see code examples below.
|
||||
The token is needed:
|
||||
|
||||
- If you use the Serverless Inference API, or
|
||||
- If you use the Inference Endpoints.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `huggingface-api-haystack` package to use the `HuggingFaceAPITextEmbedder`:
|
||||
|
||||
```shell
|
||||
pip install huggingface-api-haystack
|
||||
```
|
||||
|
||||
Similarly to other text Embedders, this component allows adding prefixes (and postfixes) to include instructions.
|
||||
For more fine-grained details, refer to the component’s [API reference](/reference/integrations-huggingface-api#huggingfaceapitextembedder).
|
||||
|
||||
### On its own
|
||||
|
||||
#### Using Free Serverless Inference API
|
||||
|
||||
Formerly known as (free) Hugging Face Inference API, this API allows you to quickly experiment with many models hosted on the Hugging Face Hub, offloading the inference to Hugging Face servers. It’s rate-limited and not meant for production.
|
||||
|
||||
To use this API, you need a [free Hugging Face token](https://huggingface.co/settings/tokens).
|
||||
The Embedder expects the `model` in `api_params`.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.huggingface_api import (
|
||||
HuggingFaceAPITextEmbedder,
|
||||
)
|
||||
from haystack.utils import Secret
|
||||
|
||||
text_embedder = HuggingFaceAPITextEmbedder(
|
||||
api_type="serverless_inference_api",
|
||||
api_params={"model": "BAAI/bge-small-en-v1.5"},
|
||||
token=Secret.from_token("<your-api-key>"),
|
||||
)
|
||||
|
||||
print(text_embedder.run("I love pizza!"))
|
||||
|
||||
# {'embedding': [0.017020374536514282, -0.023255806416273117, ...]}
|
||||
```
|
||||
|
||||
#### Using Paid Inference Endpoints
|
||||
|
||||
In this case, a private instance of the model is deployed by Hugging Face, and you typically pay per hour.
|
||||
|
||||
To understand how to spin up an Inference Endpoint, visit [Hugging Face documentation](https://huggingface.co/inference-endpoints/dedicated).
|
||||
|
||||
Additionally, in this case, you need to provide your Hugging Face token.
|
||||
The Embedder expects the `url` of your endpoint in `api_params`.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.huggingface_api import (
|
||||
HuggingFaceAPITextEmbedder,
|
||||
)
|
||||
from haystack.utils import Secret
|
||||
|
||||
text_embedder = HuggingFaceAPITextEmbedder(
|
||||
api_type="inference_endpoints",
|
||||
api_params={"model": "BAAI/bge-small-en-v1.5"},
|
||||
token=Secret.from_token("<your-api-key>"),
|
||||
)
|
||||
|
||||
print(text_embedder.run("I love pizza!"))
|
||||
|
||||
# {'embedding': [0.017020374536514282, -0.023255806416273117, ...]}
|
||||
```
|
||||
|
||||
#### Using Self-Hosted Text Embeddings Inference (TEI)
|
||||
|
||||
[Hugging Face Text Embeddings Inference](https://github.com/huggingface/text-embeddings-inference) is a toolkit for efficiently deploying and serving text embedding models.
|
||||
|
||||
While it powers the most recent versions of Serverless Inference API and Inference Endpoints, it can be used easily on-premise through Docker.
|
||||
|
||||
For example, you can run a TEI container as follows:
|
||||
|
||||
```shell
|
||||
model=BAAI/bge-large-en-v1.5
|
||||
revision=refs/pr/5
|
||||
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
|
||||
|
||||
docker run --gpus all -p 8080:80 -v $volume:/data --pull always ghcr.io/huggingface/text-embeddings-inference:1.2 --model-id $model --revision $revision
|
||||
```
|
||||
|
||||
For more information, refer to the [official TEI repository](https://github.com/huggingface/text-embeddings-inference).
|
||||
|
||||
The Embedder expects the `url` of your TEI instance in `api_params`.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.huggingface_api import (
|
||||
HuggingFaceAPITextEmbedder,
|
||||
)
|
||||
from haystack.utils import Secret
|
||||
|
||||
text_embedder = HuggingFaceAPITextEmbedder(
|
||||
api_type="text_embeddings_inference",
|
||||
api_params={"url": "http://localhost:8080"},
|
||||
)
|
||||
|
||||
print(text_embedder.run("I love pizza!"))
|
||||
|
||||
# {'embedding': [0.017020374536514282, -0.023255806416273117, ...],
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.embedders.huggingface_api import (
|
||||
HuggingFaceAPITextEmbedder,
|
||||
HuggingFaceAPIDocumentEmbedder,
|
||||
)
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
Document(content="Germany has many big cities"),
|
||||
]
|
||||
|
||||
document_embedder = HuggingFaceAPIDocumentEmbedder(
|
||||
api_type="serverless_inference_api",
|
||||
api_params={"model": "BAAI/bge-small-en-v1.5"},
|
||||
)
|
||||
documents_with_embeddings = document_embedder.run(documents)["documents"]
|
||||
document_store.write_documents(documents_with_embeddings)
|
||||
|
||||
text_embedder = HuggingFaceAPITextEmbedder(
|
||||
api_type="serverless_inference_api",
|
||||
api_params={"model": "BAAI/bge-small-en-v1.5"},
|
||||
)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", text_embedder)
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "Who lives in Berlin?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
|
||||
# Document(id=..., content: 'My name is Wolfgang and I live in Berlin', ...)
|
||||
```
|
||||
@@ -0,0 +1,138 @@
|
||||
---
|
||||
title: "JinaDocumentEmbedder"
|
||||
id: jinadocumentembedder
|
||||
slug: "/jinadocumentembedder"
|
||||
description: "This component computes the embeddings of a list of documents and stores the obtained vectors in the embedding field of each document. It uses Jina AI Embeddings models. The vectors computed by this component are necessary to perform embedding retrieval on a collection of documents. At retrieval time, the vector representing the query is compared with those of the documents to find the most similar or relevant documents."
|
||||
---
|
||||
|
||||
# JinaDocumentEmbedder
|
||||
|
||||
This component computes the embeddings of a list of documents and stores the obtained vectors in the embedding field of each document. It uses Jina AI Embeddings models. The vectors computed by this component are necessary to perform embedding retrieval on a collection of documents. At retrieval time, the vector representing the query is compared with those of the documents to find the most similar or relevant documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline |
|
||||
| **Mandatory init variables** | `api_key`: The Jina API key. Can be set with `JINA_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `documents`: A list of documents |
|
||||
| **Output variables** | `documents`: A list of documents (enriched with embeddings) <br /> <br />`meta`: A dictionary of metadata |
|
||||
| **API reference** | [Jina](/reference/integrations-jina) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/jina |
|
||||
| **Package name** | `jina-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`JinaDocumentEmbedder` enriches the metadata of documents with an embedding of their content. To embed a string, you should use the [`JinaTextEmbedder`](jinatextembedder.mdx). To see the list of compatible Jina Embeddings models, head to Jina AI’s [website](https://jina.ai/embeddings/). The default model for `JinaDocumentEmbedder` is `jina-embeddings-v2-base-en`.
|
||||
|
||||
To start using this integration with Haystack, install the package with:
|
||||
|
||||
```shell
|
||||
pip install jina-haystack
|
||||
```
|
||||
|
||||
The component uses a `JINA_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`:
|
||||
|
||||
```python
|
||||
embedder = JinaDocumentEmbedder(api_key=Secret.from_token("<your-api-key>"))
|
||||
```
|
||||
|
||||
To get a Jina Embeddings API key, head to https://jina.ai/embeddings/.
|
||||
|
||||
### Embedding Metadata
|
||||
|
||||
Text documents often come with a set of metadata. If they are distinctive and semantically meaningful, you can embed them along with the text of the document to improve retrieval.
|
||||
|
||||
You can do this easily by using the Document Embedder:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.embedders.jina import JinaDocumentEmbedder
|
||||
|
||||
doc = Document(content="some text", meta={"title": "relevant title", "page number": 18})
|
||||
|
||||
embedder = JinaDocumentEmbedder(
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
meta_fields_to_embed=["title"],
|
||||
)
|
||||
|
||||
docs_w_embeddings = embedder.run(documents=[doc])["documents"]
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Here is how you can use the component on its own:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.jina import JinaDocumentEmbedder
|
||||
|
||||
doc = Document(content="I love pizza!")
|
||||
|
||||
document_embedder = JinaDocumentEmbedder(api_key=Secret.from_token("<your-api-key>"))
|
||||
|
||||
result = document_embedder.run([doc])
|
||||
print(result["documents"][0].embedding)
|
||||
|
||||
# [0.017020374536514282, -0.023255806416273117, ...]
|
||||
```
|
||||
|
||||
:::info
|
||||
We recommend setting JINA_API_KEY as an environment variable instead of setting it as a parameter.
|
||||
:::
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.embedders.jina import JinaDocumentEmbedder
|
||||
from haystack_integrations.components.embedders.jina import JinaTextEmbedder
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
Document(content="Germany has many big cities"),
|
||||
]
|
||||
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component(
|
||||
"embedder",
|
||||
JinaDocumentEmbedder(api_key=Secret.from_token("<your-api-key>")),
|
||||
)
|
||||
indexing_pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
indexing_pipeline.connect("embedder", "writer")
|
||||
|
||||
indexing_pipeline.run({"embedder": {"documents": documents}})
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component(
|
||||
"text_embedder",
|
||||
JinaTextEmbedder(api_key=Secret.from_token("<your-api-key>")),
|
||||
)
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "Who lives in Berlin?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
|
||||
# Document(id=..., mimetype: 'text/plain',
|
||||
# text: 'My name is Wolfgang and I live in Berlin')
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Using the Jina-embeddings-v2-base-en model in a Haystack RAG pipeline for legal document analysis](https://haystack.deepset.ai/cookbook/jina-embeddings-v2-legal-analysis-rag)
|
||||
@@ -0,0 +1,168 @@
|
||||
---
|
||||
title: "JinaDocumentImageEmbedder"
|
||||
id: jinadocumentimageembedder
|
||||
slug: "/jinadocumentimageembedder"
|
||||
description: "`JinaDocumentImageEmbedder` computes the image embeddings of a list of documents and stores the obtained vectors in the embedding field of each document. It uses Jina embedding models with the ability to embed text and images into the same vector space."
|
||||
---
|
||||
|
||||
# JinaDocumentImageEmbedder
|
||||
|
||||
`JinaDocumentImageEmbedder` computes the image embeddings of a list of documents and stores the obtained vectors in the embedding field of each document. It uses Jina embedding models with the ability to embed text and images into the same vector space.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline |
|
||||
| **Mandatory init variables** | `api_key`: The Jina API key. Can be set with `JINA_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `documents`: A list of documents, with a meta field containing an image file path |
|
||||
| **Output variables** | `documents`: A list of documents (enriched with embeddings) |
|
||||
| **API reference** | [Jina](/reference/integrations-jina) |
|
||||
| **GitHub link** | [https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/jina](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/cohere) |
|
||||
| **Package name** | `jina-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`JinaDocumentImageEmbedder` expects a list of documents containing an image or a PDF file path in a meta field. The meta field can be specified with the `file_path_meta_field` init parameter of this component.
|
||||
|
||||
The embedder efficiently loads the images, computes the embeddings using a Jina model, and stores each of them in the `embedding` field of the document.
|
||||
|
||||
`JinaDocumentImageEmbedder` is commonly used in indexing pipelines. At retrieval time, you need to use the same model with a `JinaTextEmbedder` to embed the query, before using an Embedding Retriever.
|
||||
|
||||
This component is compatible with Jina multimodal embedding models:
|
||||
|
||||
- `jina-clip-v1`
|
||||
- `jina-clip-v2` (default)
|
||||
- `jina-embeddings-v4` (non-commercial research only)
|
||||
|
||||
### Installation
|
||||
|
||||
To start using this integration with Haystack, install the package with:
|
||||
|
||||
```shell
|
||||
pip install jina-haystack
|
||||
```
|
||||
|
||||
### Authentication
|
||||
|
||||
The component uses a `JINA_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with a [Secret](../../concepts/secret-management.mdx) and `Secret.from_token` method:
|
||||
|
||||
```python
|
||||
embedder = JinaDocumentImageEmbedder(api_key=Secret.from_token("<your-api-key>"))
|
||||
```
|
||||
|
||||
To get a Cohere API key, head over to https://jina.ai/embeddings/.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Remember to set `JINA_API_KEY` as an environment variable first.
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.embedders.jina import JinaDocumentImageEmbedder
|
||||
|
||||
embedder = JinaDocumentImageEmbedder(model="jina-clip-v2")
|
||||
|
||||
documents = [
|
||||
Document(content="A photo of a cat", meta={"file_path": "cat.jpg"}),
|
||||
Document(content="A photo of a dog", meta={"file_path": "dog.jpg"}),
|
||||
]
|
||||
|
||||
result = embedder.run(documents=documents)
|
||||
documents_with_embeddings = result["documents"]
|
||||
print(documents_with_embeddings)
|
||||
|
||||
# [Document(id=...,
|
||||
# content='A photo of a cat',
|
||||
# meta={'file_path': 'cat.jpg',
|
||||
# 'embedding_source': {'type': 'image', 'file_path_meta_field': 'file_path'}},
|
||||
# embedding=vector of size 1024),
|
||||
# ...]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
In this example, we can see an indexing pipeline with 3 components:
|
||||
|
||||
- `ImageFileToDocument` Converter that creates empty documents with a reference to an image in the `meta.file_path` field.
|
||||
- `JinaDocumentImageEmbedder` that loads the images, computes embeddings and store them in documents. Here, we set the `image_size` parameter to resize the image to fit within the specified dimensions while maintaining aspect ratio. This reduces API usage.
|
||||
- `DocumentWriter` that writes the documents in the `InMemoryDocumentStore`.
|
||||
|
||||
There is also a multimodal retrieval pipeline, composed of a `JinaTextEmbedder` (using the same model as before) and an `InMemoryEmbeddingRetriever`.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.converters.image import ImageFileToDocument
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
from haystack_integrations.components.embedders.jina import (
|
||||
JinaDocumentImageEmbedder,
|
||||
JinaTextEmbedder,
|
||||
)
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
# Indexing pipeline
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component("image_converter", ImageFileToDocument())
|
||||
indexing_pipeline.add_component(
|
||||
"embedder",
|
||||
JinaDocumentImageEmbedder(model="jina-clip-v2", image_size=(200, 200)),
|
||||
)
|
||||
indexing_pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
indexing_pipeline.connect("image_converter", "embedder")
|
||||
indexing_pipeline.connect("embedder", "writer")
|
||||
|
||||
indexing_pipeline.run(data={"image_converter": {"sources": ["dog.jpg", "cat.jpg"]}})
|
||||
|
||||
# Multimodal retrieval pipeline
|
||||
retrieval_pipeline = Pipeline()
|
||||
retrieval_pipeline.add_component("embedder", JinaTextEmbedder(model="jina-clip-v2"))
|
||||
retrieval_pipeline.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store, top_k=2),
|
||||
)
|
||||
retrieval_pipeline.connect("embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
result = retrieval_pipeline.run(data={"text": "man's best friend"})
|
||||
print(result)
|
||||
|
||||
# {
|
||||
# 'retriever': {
|
||||
# 'documents': [
|
||||
# Document(
|
||||
# id=0c96...,
|
||||
# meta={
|
||||
# 'file_path': 'dog.jpg',
|
||||
# 'embedding_source': {
|
||||
# 'type': 'image',
|
||||
# 'file_path_meta_field': 'file_path'
|
||||
# }
|
||||
# },
|
||||
# score=0.246
|
||||
# ),
|
||||
# Document(
|
||||
# id=5e76...,
|
||||
# meta={
|
||||
# 'file_path': 'cat.jpg',
|
||||
# 'embedding_source': {
|
||||
# 'type': 'image',
|
||||
# 'file_path_meta_field': 'file_path'
|
||||
# }
|
||||
# },
|
||||
# score=0.199
|
||||
# )
|
||||
# ]
|
||||
# }
|
||||
# }
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
:notebook: Tutorial: [Creating Vision+Text RAG Pipelines](https://haystack.deepset.ai/tutorials/46_multimodal_rag)
|
||||
@@ -0,0 +1,113 @@
|
||||
---
|
||||
title: "JinaTextEmbedder"
|
||||
id: jinatextembedder
|
||||
slug: "/jinatextembedder"
|
||||
description: "This component transforms a string into a vector that captures its semantics using a Jina Embeddings model. When you perform embedding retrieval, you use this component to transform your query into a vector. Then, the embedding Retriever looks for similar or relevant documents."
|
||||
---
|
||||
|
||||
# JinaTextEmbedder
|
||||
|
||||
This component transforms a string into a vector that captures its semantics using a Jina Embeddings model. When you perform embedding retrieval, you use this component to transform your query into a vector. Then, the embedding Retriever looks for similar or relevant documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before an embedding [Retriever](../retrievers.mdx) in a query/RAG pipeline |
|
||||
| **Mandatory init variables** | `api_key`: The Jina API key. Can be set with `JINA_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `text`: A string |
|
||||
| **Output variables** | `embedding`: A list of float numbers <br /> <br />`meta`: A dictionary of metadata |
|
||||
| **API reference** | [Jina](/reference/integrations-jina) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/jina |
|
||||
| **Package name** | `jina-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`JinaTextEmbedder` embeds a simple string (such as a query) into a vector. For embedding lists of documents, use the use the [`JinaDocumentEmbedder`](jinadocumentembedder.mdx), which enriches the document with the computed embedding, also known as vector. To see the list of compatible Jina Embeddings models, head to Jina AI’s [website](https://jina.ai/embeddings/). The default model for `JinaTextEmbedder` is `jina-embeddings-v2-base-en`.
|
||||
|
||||
To start using this integration with Haystack, install the package with:
|
||||
|
||||
```shell
|
||||
pip install jina-haystack
|
||||
```
|
||||
|
||||
The component uses a `JINA_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`:
|
||||
|
||||
```python
|
||||
embedder = JinaTextEmbedder(api_key=Secret.from_token("<your-api-key>"))
|
||||
```
|
||||
|
||||
To get a Jina Embeddings API key, head to https://jina.ai/embeddings/.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Here is how you can use the component on its own:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.jina import JinaTextEmbedder
|
||||
|
||||
text_to_embed = "I love pizza!"
|
||||
|
||||
text_embedder = JinaTextEmbedder(api_key=Secret.from_token("<your-api-key>"))
|
||||
|
||||
print(text_embedder.run(text_to_embed))
|
||||
|
||||
# {'embedding': [0.017020374536514282, -0.023255806416273117, ...],
|
||||
# 'meta': {'model': 'text-embedding-ada-002-v2',
|
||||
# 'usage': {'prompt_tokens': 4, 'total_tokens': 4}}}
|
||||
```
|
||||
|
||||
:::info
|
||||
We recommend setting JINA_API_KEY as an environment variable instead of setting it as a parameter.
|
||||
:::
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.embedders.jina import JinaDocumentEmbedder
|
||||
from haystack_integrations.components.embedders.jina import JinaTextEmbedder
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
Document(content="Germany has many big cities"),
|
||||
]
|
||||
|
||||
document_embedder = JinaDocumentEmbedder(api_key=Secret.from_token("<your-api-key>"))
|
||||
documents_with_embeddings = document_embedder.run(documents)["documents"]
|
||||
document_store.write_documents(documents_with_embeddings)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component(
|
||||
"text_embedder",
|
||||
JinaTextEmbedder(api_key=Secret.from_token("<your-api-key>")),
|
||||
)
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "Who lives in Berlin?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
|
||||
# Document(id=..., mimetype: 'text/plain',
|
||||
# text: 'My name is Wolfgang and I live in Berlin')
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Using the Jina-embeddings-v2-base-en model in a Haystack RAG pipeline for legal document analysis](https://haystack.deepset.ai/cookbook/jina-embeddings-v2-legal-analysis-rag)
|
||||
@@ -0,0 +1,111 @@
|
||||
---
|
||||
title: "MistralDocumentEmbedder"
|
||||
id: mistraldocumentembedder
|
||||
slug: "/mistraldocumentembedder"
|
||||
description: "This component computes the embeddings of a list of documents using the Mistral API and models."
|
||||
---
|
||||
|
||||
# MistralDocumentEmbedder
|
||||
|
||||
This component computes the embeddings of a list of documents using the Mistral API and models.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline |
|
||||
| **Mandatory init variables** | `api_key`: The Mistral API key. Can be set with `MISTRAL_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `documents`: A list of documents to be embedded |
|
||||
| **Output variables** | `documents`: A list of documents (enriched with embeddings) <br /> <br />`meta`: A dictionary of metadata strings |
|
||||
| **API reference** | [Mistral](/reference/integrations-mistral) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/mistral |
|
||||
| **Package name** | `mistral-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
This component should be used to embed a list of Documents. To embed a string, use the [`MistralTextEmbedder`](mistraltextembedder.mdx).
|
||||
|
||||
## Overview
|
||||
|
||||
`MistralDocumentEmbedder` computes the embeddings of a list of documents and stores the obtained vectors in the embedding field of each document. It uses the Mistral API and its embedding models.
|
||||
|
||||
The component currently supports the `mistral-embed` embedding model. The list of all supported models can be found in Mistral’s [embedding models documentation](https://docs.mistral.ai/platform/endpoints/#embedding-models).
|
||||
|
||||
To start using this integration with Haystack, install it with:
|
||||
|
||||
```shell
|
||||
pip install mistral-haystack
|
||||
```
|
||||
|
||||
`MistralDocumentEmbedder` needs a Mistral API key to work. It uses an `MISTRAL_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`:
|
||||
|
||||
```python
|
||||
embedder = MistralDocumentEmbedder(
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
model="mistral-embed",
|
||||
)
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Remember first to set the`MISTRAL_API_KEY` as an environment variable or pass it in directly.
|
||||
|
||||
Here is how you can use the component on its own:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.embedders.mistral.document_embedder import (
|
||||
MistralDocumentEmbedder,
|
||||
)
|
||||
|
||||
doc = Document(content="I love pizza!")
|
||||
|
||||
embedder = MistralDocumentEmbedder(
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
model="mistral-embed",
|
||||
)
|
||||
|
||||
result = embedder.run([doc])
|
||||
print(result["documents"][0].embedding)
|
||||
# [-0.453125, 1.2236328, 2.0058594, 0.67871094...]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Below is an example of the `MistralDocumentEmbedder` in an indexing pipeline. We are indexing the contents of a webpage into an `InMemoryDocumentStore`.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.converters import HTMLToDocument
|
||||
from haystack.components.fetchers import LinkContentFetcher
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.embedders.mistral.document_embedder import (
|
||||
MistralDocumentEmbedder,
|
||||
)
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
fetcher = LinkContentFetcher()
|
||||
converter = HTMLToDocument()
|
||||
chunker = DocumentSplitter()
|
||||
embedder = MistralDocumentEmbedder()
|
||||
writer = DocumentWriter(document_store=document_store)
|
||||
|
||||
indexing = Pipeline()
|
||||
|
||||
indexing.add_component(name="fetcher", instance=fetcher)
|
||||
indexing.add_component(name="converter", instance=converter)
|
||||
indexing.add_component(name="chunker", instance=chunker)
|
||||
indexing.add_component(name="embedder", instance=embedder)
|
||||
indexing.add_component(name="writer", instance=writer)
|
||||
|
||||
indexing.connect("fetcher", "converter")
|
||||
indexing.connect("converter", "chunker")
|
||||
indexing.connect("chunker", "embedder")
|
||||
indexing.connect("embedder", "writer")
|
||||
|
||||
indexing.run(data={"fetcher": {"urls": ["https://mistral.ai/news/la-plateforme/"]}})
|
||||
```
|
||||
@@ -0,0 +1,170 @@
|
||||
---
|
||||
title: "MistralTextEmbedder"
|
||||
id: mistraltextembedder
|
||||
slug: "/mistraltextembedder"
|
||||
description: "This component transforms a string into a vector using the Mistral API and models. Use it for embedding retrieval to transform your query into an embedding."
|
||||
---
|
||||
|
||||
# MistralTextEmbedder
|
||||
|
||||
This component transforms a string into a vector using the Mistral API and models. Use it for embedding retrieval to transform your query into an embedding.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before an embedding [Retriever](../retrievers.mdx) in a query/RAG pipeline |
|
||||
| **Mandatory init variables** | `api_key`: The Mistral API key. Can be set with `MISTRAL_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `text`: A string |
|
||||
| **Output variables** | `embedding`: A list of float numbers (vectors) <br /> <br />`meta`: A dictionary of metadata strings |
|
||||
| **API reference** | [Mistral](/reference/integrations-mistral) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/mistral |
|
||||
| **Package name** | `mistral-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
Use `MistalTextEmbedder` to embed a simple string (such as a query) into a vector. For embedding lists of documents, use the [`MistralDocumentEmbedder`](mistraldocumentembedder.mdx), which enriches the document with the computed embedding, also known as vector.
|
||||
|
||||
## Overview
|
||||
|
||||
`MistralTextEmbedder` transforms a string into a vector that captures its semantics using a Mistral embedding model.
|
||||
|
||||
The component currently supports the `mistral-embed` embedding model. The list of all supported models can be found in Mistral’s [embedding models documentation](https://docs.mistral.ai/platform/endpoints/#embedding-models).
|
||||
|
||||
To start using this integration with Haystack, install it with:
|
||||
|
||||
```shell
|
||||
pip install mistral-haystack
|
||||
```
|
||||
|
||||
`MistralTextEmbedder` needs a Mistral API key to work. It uses a `MISTRAL_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`:
|
||||
|
||||
```python
|
||||
embedder = MistralTextEmbedder(
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
model="mistral-embed",
|
||||
)
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Remember to set the`MISTRAL_API_KEY` as an environment variable first or pass it in directly.
|
||||
|
||||
Here is how you can use the component on its own:
|
||||
|
||||
```python
|
||||
|
||||
from haystack_integrations.components.embedders.mistral.text_embedder import (
|
||||
MistralTextEmbedder,
|
||||
)
|
||||
|
||||
embedder = MistralTextEmbedder(
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
model="mistral-embed",
|
||||
)
|
||||
|
||||
result = embedder.run(text="How can I ise the Mistral embedding models with Haystack?")
|
||||
|
||||
print(result["embedding"])
|
||||
# [-0.0015687942504882812, 0.052154541015625, 0.037109375...]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Below is an example of the `MistralTextEmbedder` in a document search pipeline. We are building this pipeline on top of an `InMemoryDocumentStore` where we index the contents of two URLs.
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.utils import Secret
|
||||
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
|
||||
from haystack.components.fetchers import LinkContentFetcher
|
||||
from haystack.components.converters import HTMLToDocument
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.embedders.mistral.document_embedder import (
|
||||
MistralDocumentEmbedder,
|
||||
)
|
||||
from haystack_integrations.components.embedders.mistral.text_embedder import (
|
||||
MistralTextEmbedder,
|
||||
)
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
# Initialize document store
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
# Indexing components
|
||||
fetcher = LinkContentFetcher()
|
||||
converter = HTMLToDocument()
|
||||
embedder = MistralDocumentEmbedder()
|
||||
writer = DocumentWriter(document_store=document_store)
|
||||
|
||||
indexing = Pipeline()
|
||||
indexing.add_component(name="fetcher", instance=fetcher)
|
||||
indexing.add_component(name="converter", instance=converter)
|
||||
indexing.add_component(name="embedder", instance=embedder)
|
||||
indexing.add_component(name="writer", instance=writer)
|
||||
|
||||
indexing.connect("fetcher", "converter")
|
||||
indexing.connect("converter", "embedder")
|
||||
indexing.connect("embedder", "writer")
|
||||
|
||||
indexing.run(
|
||||
data={
|
||||
"fetcher": {
|
||||
"urls": [
|
||||
"https://docs.mistral.ai/self-deployment/cloudflare/",
|
||||
"https://docs.mistral.ai/platform/endpoints/",
|
||||
],
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
# Retrieval components
|
||||
text_embedder = MistralTextEmbedder()
|
||||
retriever = InMemoryEmbeddingRetriever(document_store=document_store)
|
||||
|
||||
# Define prompt template
|
||||
prompt_template = [
|
||||
ChatMessage.from_system("You are a helpful assistant."),
|
||||
ChatMessage.from_user(
|
||||
"Given the retrieved documents, answer the question.\nDocuments:\n"
|
||||
"{% for document in documents %}{{ document.content }}{% endfor %}\n"
|
||||
"Question: {{ query }}\nAnswer:",
|
||||
),
|
||||
]
|
||||
|
||||
prompt_builder = ChatPromptBuilder(
|
||||
template=prompt_template,
|
||||
required_variables={"query", "documents"},
|
||||
)
|
||||
llm = OpenAIChatGenerator(
|
||||
model="gpt-4o-mini",
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
)
|
||||
|
||||
doc_search = Pipeline()
|
||||
doc_search.add_component("text_embedder", text_embedder)
|
||||
doc_search.add_component("retriever", retriever)
|
||||
doc_search.add_component("prompt_builder", prompt_builder)
|
||||
doc_search.add_component("llm", llm)
|
||||
|
||||
doc_search.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
doc_search.connect("retriever.documents", "prompt_builder.documents")
|
||||
doc_search.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
query = "How can I deploy Mistral models with Cloudflare?"
|
||||
|
||||
result = doc_search.run(
|
||||
{
|
||||
"text_embedder": {"text": query},
|
||||
"retriever": {"top_k": 1},
|
||||
"prompt_builder": {"query": query},
|
||||
},
|
||||
)
|
||||
|
||||
print(result["llm"]["replies"])
|
||||
```
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
title: "MockDocumentEmbedder"
|
||||
id: mockdocumentembedder
|
||||
slug: "/mockdocumentembedder"
|
||||
description: "A Document Embedder that returns deterministic embeddings without calling any API, for tests and quick prototypes."
|
||||
---
|
||||
|
||||
# MockDocumentEmbedder
|
||||
|
||||
A Document Embedder that returns deterministic embeddings without calling any API, for tests and quick prototypes.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In place of a real Document Embedder, in tests and prototypes |
|
||||
| **Mandatory init variables** | None |
|
||||
| **Mandatory run variables** | `documents`: A list of documents |
|
||||
| **Output variables** | `documents`: A list of documents enriched with embeddings <br /> <br />`meta`: A dictionary of metadata |
|
||||
| **API reference** | [Embedders](/reference/embedders-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/embedders/mock_document_embedder.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`MockDocumentEmbedder` is a deterministic, zero-cost drop-in replacement for real Document Embedders such as `OpenAIDocumentEmbedder`. It implements `run`, `run_async`, and serialization like any other embedder but never contacts an external service, which makes it ideal for unit tests, smoke tests, and quick prototypes.
|
||||
|
||||
The embedding is selected based on how the component is configured:
|
||||
|
||||
- **Deterministic (default)**: With no configuration, each document's embedding is derived from a stable hash of its prepared text. The same text always yields the same unit-length embedding, and different texts yield different embeddings, so the mock works in retrieval pipelines and is reproducible across runs and processes.
|
||||
- **Fixed embedding**: Pass an `embedding` vector. The same vector is assigned to every document.
|
||||
- **Dynamic embedding**: Pass an `embedding_fn` callable that receives the prepared text of a document and returns the embedding. To support serialization, pass a named function.
|
||||
|
||||
`embedding` and `embedding_fn` are mutually exclusive.
|
||||
|
||||
Further optional parameters:
|
||||
|
||||
- `dimension`: The number of dimensions of the deterministic embedding. Defaults to `768`. Ignored when `embedding` or `embedding_fn` is provided, since their length is determined by the value or callable.
|
||||
- `model`: The model name reported in the metadata. Defaults to `"mock-model"`.
|
||||
- `meta`: Additional metadata merged into the output `meta`.
|
||||
- `prefix` / `suffix`: Strings added to the beginning and end of each text before embedding, mirroring real embedders.
|
||||
- `meta_fields_to_embed` / `embedding_separator`: Like real Document Embedders, the metadata fields listed in `meta_fields_to_embed` are concatenated with the document content before embedding, so the deterministic embedding reflects the embedded metadata.
|
||||
- `progress_bar`: Accepted for interface compatibility with real Document Embedders and ignored.
|
||||
|
||||
:::info
|
||||
The deterministic embeddings are derived from a hash: identical texts get identical vectors and the similarity between different texts is stable but arbitrary. For exact-match retrieval in tests this is exactly what you want. Do not expect semantically similar texts to end up close together.
|
||||
:::
|
||||
|
||||
Use `MockDocumentEmbedder` for documents and its counterpart [`MockTextEmbedder`](mocktextembedder.mdx) for queries. With the default deterministic mode, a query whose text matches a document's content produces the same vector, so the document is retrieved as the top hit.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.embedders import MockDocumentEmbedder
|
||||
|
||||
embedder = MockDocumentEmbedder(dimension=8)
|
||||
result = embedder.run([Document(content="I love pizza!")])
|
||||
print(result["documents"][0].embedding) # a deterministic list of 8 floats
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Use it in an indexing pipeline exactly like a real Document Embedder — no API key needed:
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.embedders import MockDocumentEmbedder
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component("embedder", MockDocumentEmbedder(dimension=8))
|
||||
indexing_pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
indexing_pipeline.connect("embedder.documents", "writer.documents")
|
||||
|
||||
indexing_pipeline.run(
|
||||
{
|
||||
"embedder": {
|
||||
"documents": [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
],
|
||||
},
|
||||
},
|
||||
)
|
||||
print(document_store.count_documents()) # 2
|
||||
```
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
title: "MockTextEmbedder"
|
||||
id: mocktextembedder
|
||||
slug: "/mocktextembedder"
|
||||
description: "A Text Embedder that returns deterministic embeddings without calling any API, for tests and quick prototypes."
|
||||
---
|
||||
|
||||
# MockTextEmbedder
|
||||
|
||||
A Text Embedder that returns deterministic embeddings without calling any API, for tests and quick prototypes.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In place of a real Text Embedder, in tests and prototypes |
|
||||
| **Mandatory init variables** | None |
|
||||
| **Mandatory run variables** | `text`: A string |
|
||||
| **Output variables** | `embedding`: A list of float numbers <br /> <br />`meta`: A dictionary of metadata |
|
||||
| **API reference** | [Embedders](/reference/embedders-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/embedders/mock_text_embedder.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`MockTextEmbedder` is a deterministic, zero-cost drop-in replacement for real Text Embedders such as `OpenAITextEmbedder`. It implements `run`, `run_async`, and serialization like any other embedder but never contacts an external service, which makes it ideal for unit tests, smoke tests, and quick prototypes.
|
||||
|
||||
The embedding is selected based on how the component is configured:
|
||||
|
||||
- **Deterministic (default)**: With no configuration, the embedding is derived from a stable hash of the input text. The same text always yields the same unit-length embedding, and different texts yield different embeddings, so the mock works in retrieval pipelines and is reproducible across runs and processes.
|
||||
- **Fixed embedding**: Pass an `embedding` vector. The same vector is returned for every input.
|
||||
- **Dynamic embedding**: Pass an `embedding_fn` callable that receives the prepared text (after `prefix`/`suffix` are applied) and returns the embedding. To support serialization, pass a named function.
|
||||
|
||||
`embedding` and `embedding_fn` are mutually exclusive.
|
||||
|
||||
Further optional parameters:
|
||||
|
||||
- `dimension`: The number of dimensions of the deterministic embedding. Defaults to `768`. Ignored when `embedding` or `embedding_fn` is provided, since their length is determined by the value or callable.
|
||||
- `model`: The model name reported in the metadata. Defaults to `"mock-model"`.
|
||||
- `meta`: Additional metadata merged into the output `meta`.
|
||||
- `prefix` / `suffix`: Strings added to the beginning and end of the text before embedding, mirroring real embedders.
|
||||
|
||||
:::info
|
||||
The deterministic embeddings are derived from a hash: identical texts get identical vectors and the similarity between different texts is stable but arbitrary. For exact-match retrieval in tests this is exactly what you want. Do not expect semantically similar texts to end up close together.
|
||||
:::
|
||||
|
||||
Use `MockTextEmbedder` for queries and its counterpart [`MockDocumentEmbedder`](mockdocumentembedder.mdx) for documents. With the default deterministic mode, a query whose text matches a document's content produces the same vector, so the document is retrieved as the top hit.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.components.embedders import MockTextEmbedder
|
||||
|
||||
embedder = MockTextEmbedder(dimension=8)
|
||||
result = embedder.run("I love pizza!")
|
||||
print(result["embedding"]) # a deterministic list of 8 floats
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
A retrieval pipeline built with mock embedders runs without any API key and always returns the same result for the same input:
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.embedders import MockDocumentEmbedder, MockTextEmbedder
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
documents = [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
]
|
||||
|
||||
indexed = MockDocumentEmbedder(dimension=8).run(documents=documents)
|
||||
document_store.write_documents(indexed["documents"])
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", MockTextEmbedder(dimension=8))
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
result = query_pipeline.run(
|
||||
{"text_embedder": {"text": "I saw a black horse running"}},
|
||||
)
|
||||
print(result["retriever"]["documents"][0].content) # "I saw a black horse running"
|
||||
```
|
||||
@@ -0,0 +1,154 @@
|
||||
---
|
||||
title: "NvidiaDocumentEmbedder"
|
||||
id: nvidiadocumentembedder
|
||||
slug: "/nvidiadocumentembedder"
|
||||
description: "This component computes the embeddings of a list of documents and stores the obtained vectors in the embedding field of each document."
|
||||
---
|
||||
|
||||
# NvidiaDocumentEmbedder
|
||||
|
||||
This component computes the embeddings of a list of documents and stores the obtained vectors in the embedding field of each document.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline |
|
||||
| **Mandatory init variables** | `api_key`: API key for the NVIDIA NIM. Can be set with `NVIDIA_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `documents`: A list of documents |
|
||||
| **Output variables** | `documents`: A list of documents (enriched with embeddings) <br /> <br />`meta`: A dictionary of metadata |
|
||||
| **API reference** | [NVIDIA](/reference/integrations-nvidia) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/nvidia |
|
||||
| **Package name** | `nvidia-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`NvidiaDocumentEmbedder` enriches documents with an embedding of their content.
|
||||
|
||||
You can use this component with self-hosted models using NVIDIA NIM or models hosted on the [NVIDIA API Catalog](https://build.nvidia.com/explore/discover).
|
||||
|
||||
To embed a string, use [`NvidiaTextEmbedder`](nvidiatextembedder.mdx).
|
||||
|
||||
## Usage
|
||||
|
||||
To start using `NvidiaDocumentEmbedder`, install the `nvidia-haystack` package:
|
||||
|
||||
```shell
|
||||
pip install nvidia-haystack
|
||||
```
|
||||
|
||||
You can use `NvidiaDocumentEmbedder` with all the embedding models available on the [NVIDIA API Catalog](https://docs.api.nvidia.com/nim/reference) or with a model deployed using NVIDIA NIM. For more information, refer to [Deploying Text Embedding Models](https://developer.nvidia.com/docs/nemo-microservices/embedding/source/deploy.html).
|
||||
|
||||
### On its own
|
||||
|
||||
To use models from the NVIDIA API Catalog, you need to specify the `api_url` and your API key. You can get your API key from the [NVIDIA API Catalog](https://build.nvidia.com/explore/discover).
|
||||
|
||||
`NvidiaDocumentEmbedder` uses the `NVIDIA_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with the `api_key` parameter:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.utils.auth import Secret
|
||||
from haystack_integrations.components.embedders.nvidia import NvidiaDocumentEmbedder
|
||||
|
||||
documents = [
|
||||
Document(content="A transformer is a deep learning architecture"),
|
||||
Document(content="Large language models use transformer architectures"),
|
||||
]
|
||||
|
||||
embedder = NvidiaDocumentEmbedder(
|
||||
model="nvidia/nv-embedqa-e5-v5",
|
||||
api_url="https://integrate.api.nvidia.com/v1",
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
)
|
||||
|
||||
result = embedder.run(documents=documents)
|
||||
print(result["documents"])
|
||||
print(result["meta"])
|
||||
```
|
||||
|
||||
To use a locally deployed model, set the `api_url` to your localhost and set `api_key` to `None`:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.embedders.nvidia import NvidiaDocumentEmbedder
|
||||
|
||||
documents = [
|
||||
Document(content="A transformer is a deep learning architecture"),
|
||||
Document(content="Large language models use transformer architectures"),
|
||||
]
|
||||
|
||||
embedder = NvidiaDocumentEmbedder(
|
||||
model="nvidia/nv-embedqa-e5-v5",
|
||||
api_url="http://localhost:9999/v1",
|
||||
api_key=None,
|
||||
)
|
||||
|
||||
result = embedder.run(documents=documents)
|
||||
print(result["documents"])
|
||||
print(result["meta"])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
The following example shows how to use `NvidiaDocumentEmbedder` in a RAG pipeline:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline, Document
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
from haystack.utils.auth import Secret
|
||||
from haystack_integrations.components.embedders.nvidia import (
|
||||
NvidiaTextEmbedder,
|
||||
NvidiaDocumentEmbedder,
|
||||
)
|
||||
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
Document(content="Germany has many big cities"),
|
||||
]
|
||||
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component(
|
||||
"embedder",
|
||||
NvidiaDocumentEmbedder(
|
||||
model="nvidia/nv-embedqa-e5-v5",
|
||||
api_url="https://integrate.api.nvidia.com/v1",
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
),
|
||||
)
|
||||
indexing_pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
indexing_pipeline.connect("embedder", "writer")
|
||||
|
||||
indexing_pipeline.run({"embedder": {"documents": documents}})
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component(
|
||||
"text_embedder",
|
||||
NvidiaTextEmbedder(
|
||||
model="nvidia/nv-embedqa-e5-v5",
|
||||
api_url="https://integrate.api.nvidia.com/v1",
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
),
|
||||
)
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "Who lives in Berlin?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
```
|
||||
|
||||
## Related
|
||||
|
||||
- Cookbook: [Haystack RAG Pipeline with Self-Deployed AI models using NVIDIA NIMs](https://haystack.deepset.ai/cookbook/rag-with-nims)
|
||||
@@ -0,0 +1,142 @@
|
||||
---
|
||||
title: "NvidiaTextEmbedder"
|
||||
id: nvidiatextembedder
|
||||
slug: "/nvidiatextembedder"
|
||||
description: "This component transforms a string into a vector that captures its semantics using NVIDIA-hosted models."
|
||||
---
|
||||
|
||||
# NvidiaTextEmbedder
|
||||
|
||||
This component transforms a string into a vector that captures its semantics using NVIDIA-hosted models.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before an embedding [Retriever](../retrievers.mdx) in a query/RAG pipeline |
|
||||
| **Mandatory init variables** | `api_key`: API key for the NVIDIA NIM. Can be set with `NVIDIA_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `text`: A string |
|
||||
| **Output variables** | `embedding`: A list of float numbers (vectors) <br /> <br />`meta`: A dictionary of metadata strings |
|
||||
| **API reference** | [NVIDIA](/reference/integrations-nvidia) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/nvidia |
|
||||
| **Package name** | `nvidia-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`NvidiaTextEmbedder` embeds a simple string (such as a query) into a vector.
|
||||
|
||||
You can use this component with self-hosted models using NVIDIA NIM or models hosted on the [NVIDIA API Catalog](https://build.nvidia.com/explore/discover).
|
||||
|
||||
To embed a list of documents, use [`NvidiaDocumentEmbedder`](nvidiadocumentembedder.mdx), which enriches each document with the computed embedding.
|
||||
|
||||
## Usage
|
||||
|
||||
To start using `NvidiaTextEmbedder`, install the `nvidia-haystack` package:
|
||||
|
||||
```shell
|
||||
pip install nvidia-haystack
|
||||
```
|
||||
|
||||
You can use `NvidiaTextEmbedder` with all the embedding models available on the [NVIDIA API Catalog](https://docs.api.nvidia.com/nim/reference) or with a model deployed using NVIDIA NIM. For more information, refer to [Deploying Text Embedding Models](https://developer.nvidia.com/docs/nemo-microservices/embedding/source/deploy.html).
|
||||
|
||||
### On its own
|
||||
|
||||
To use models from the NVIDIA API Catalog, you need to specify the `api_url` and your API key. You can get your API key from the [NVIDIA API Catalog](https://build.nvidia.com/explore/discover).
|
||||
|
||||
`NvidiaTextEmbedder` uses the `NVIDIA_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with the `api_key` parameter:
|
||||
|
||||
```python
|
||||
from haystack.utils.auth import Secret
|
||||
from haystack_integrations.components.embedders.nvidia import NvidiaTextEmbedder
|
||||
|
||||
embedder = NvidiaTextEmbedder(
|
||||
model="nvidia/nv-embedqa-e5-v5",
|
||||
api_url="https://integrate.api.nvidia.com/v1",
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
)
|
||||
|
||||
result = embedder.run("A transformer is a deep learning architecture")
|
||||
print(result["embedding"])
|
||||
print(result["meta"])
|
||||
```
|
||||
|
||||
To use a locally deployed model, set the `api_url` to your localhost and set `api_key` to `None`:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.nvidia import NvidiaTextEmbedder
|
||||
|
||||
embedder = NvidiaTextEmbedder(
|
||||
model="nvidia/nv-embedqa-e5-v5",
|
||||
api_url="http://localhost:9999/v1",
|
||||
api_key=None,
|
||||
)
|
||||
|
||||
result = embedder.run("A transformer is a deep learning architecture")
|
||||
print(result["embedding"])
|
||||
print(result["meta"])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
The following example shows how to use `NvidiaTextEmbedder` in a RAG pipeline:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline, Document
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
from haystack.utils.auth import Secret
|
||||
from haystack_integrations.components.embedders.nvidia import (
|
||||
NvidiaTextEmbedder,
|
||||
NvidiaDocumentEmbedder,
|
||||
)
|
||||
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
Document(content="Germany has many big cities"),
|
||||
]
|
||||
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component(
|
||||
"embedder",
|
||||
NvidiaDocumentEmbedder(
|
||||
model="nvidia/nv-embedqa-e5-v5",
|
||||
api_url="https://integrate.api.nvidia.com/v1",
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
),
|
||||
)
|
||||
indexing_pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
indexing_pipeline.connect("embedder", "writer")
|
||||
|
||||
indexing_pipeline.run({"embedder": {"documents": documents}})
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component(
|
||||
"text_embedder",
|
||||
NvidiaTextEmbedder(
|
||||
model="nvidia/nv-embedqa-e5-v5",
|
||||
api_url="https://integrate.api.nvidia.com/v1",
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
),
|
||||
)
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "Who lives in Berlin?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
```
|
||||
|
||||
## Related
|
||||
|
||||
- Cookbook: [Haystack RAG Pipeline with Self-Deployed AI models using NVIDIA NIMs](https://haystack.deepset.ai/cookbook/rag-with-nims)
|
||||
@@ -0,0 +1,124 @@
|
||||
---
|
||||
title: "OllamaDocumentEmbedder"
|
||||
id: ollamadocumentembedder
|
||||
slug: "/ollamadocumentembedder"
|
||||
description: "This component computes the embeddings of a list of documents using embedding models compatible with the Ollama Library."
|
||||
---
|
||||
|
||||
# OllamaDocumentEmbedder
|
||||
|
||||
This component computes the embeddings of a list of documents using embedding models compatible with the Ollama Library.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline |
|
||||
| **Mandatory run variables** | `documents`: A list of documents to be embedded |
|
||||
| **Output variables** | `documents`: A list of documents (enriched with embeddings) <br /> <br />`meta`: A dictionary of metadata strings |
|
||||
| **API reference** | [Ollama](/reference/integrations-ollama) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/ollama |
|
||||
| **Package name** | `ollama-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
`OllamaDocumentEmbedder` computes the embeddings of a list of documents and stores the obtained vectors in the embedding field of each document. It uses embedding models compatible with the Ollama Library.
|
||||
|
||||
The vectors computed by this component are necessary to perform embedding retrieval on a collection of documents. At retrieval time, the vector that represents the query is compared with those of the documents to find the most similar or relevant documents.
|
||||
|
||||
## Overview
|
||||
|
||||
`OllamaDocumentEmbedder` should be used to embed a list of documents. For embedding a string only, use the [`OllamaTextEmbedder`](ollamatextembedder.mdx).
|
||||
|
||||
The component uses `http://localhost:11434` as the default URL as most available setups (Mac, Linux, Docker) default to port 11434.
|
||||
|
||||
### Compatible Models
|
||||
|
||||
Unless specified otherwise while initializing this component, the default embedding model is "nomic-embed-text". See other possible pre-built models in Ollama's [library](https://ollama.ai/library). To load your own custom model, follow the [instructions](https://github.com/ollama/ollama/blob/main/docs/modelfile.md) from Ollama.
|
||||
|
||||
### Installation
|
||||
|
||||
To start using this integration with Haystack, install the package with:
|
||||
|
||||
```shell
|
||||
pip install ollama-haystack
|
||||
```
|
||||
|
||||
Make sure that you have a running Ollama model (either through a docker container, or locally hosted). No other configuration is necessary as Ollama has the embedding API built in.
|
||||
|
||||
### Embedding Metadata
|
||||
|
||||
Most embedded metadata contains information about the model name and type. You can pass [optional arguments](https://github.com/jmorganca/ollama/blob/main/docs/modelfile.md#valid-parameters-and-values), such as temperature, top_p, and others, to the Ollama generation endpoint.
|
||||
|
||||
The name of the model used will be automatically appended as part of the document metadata. An example payload using the nomic-embed-text model will look like this:
|
||||
|
||||
```python
|
||||
{"meta": {"model": "nomic-embed-text"}}
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.embedders.ollama import OllamaDocumentEmbedder
|
||||
|
||||
doc = Document(content="What do llamas say once you have thanked them? No probllama!")
|
||||
document_embedder = OllamaDocumentEmbedder()
|
||||
|
||||
result = document_embedder.run([doc])
|
||||
print(result["documents"][0].embedding)
|
||||
|
||||
# Calculating embeddings: 100%|██████████| 1/1 [00:02<00:00, 2.82s/it]
|
||||
|
||||
# [-0.16412407159805298, -3.8359334468841553, ... ]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
|
||||
from haystack_integrations.components.embedders.ollama import OllamaDocumentEmbedder
|
||||
|
||||
from haystack.components.preprocessors import DocumentCleaner, DocumentSplitter
|
||||
|
||||
from haystack.components.converters import PyPDFToDocument
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.document_stores.types import DuplicatePolicy
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
embedder = OllamaDocumentEmbedder(
|
||||
model="nomic-embed-text",
|
||||
url="http://localhost:11434",
|
||||
) # This is the default model and URL
|
||||
|
||||
cleaner = DocumentCleaner()
|
||||
splitter = DocumentSplitter()
|
||||
file_converter = PyPDFToDocument()
|
||||
writer = DocumentWriter(document_store=document_store, policy=DuplicatePolicy.OVERWRITE)
|
||||
|
||||
indexing_pipeline = Pipeline()
|
||||
|
||||
# Add components to pipeline
|
||||
indexing_pipeline.add_component("embedder", embedder)
|
||||
indexing_pipeline.add_component("converter", file_converter)
|
||||
indexing_pipeline.add_component("cleaner", cleaner)
|
||||
indexing_pipeline.add_component("splitter", splitter)
|
||||
indexing_pipeline.add_component("writer", writer)
|
||||
|
||||
# Connect components in pipeline
|
||||
indexing_pipeline.connect("converter", "cleaner")
|
||||
indexing_pipeline.connect("cleaner", "splitter")
|
||||
indexing_pipeline.connect("splitter", "embedder")
|
||||
indexing_pipeline.connect("embedder", "writer")
|
||||
|
||||
# Run Pipeline
|
||||
indexing_pipeline.run({"converter": {"sources": ["files/test_pdf_data.pdf"]}})
|
||||
|
||||
# Calculating embeddings: 100%|██████████| 115/115
|
||||
# {'embedder': {'meta': {'model': 'nomic-embed-text'}}, 'writer': {'documents_written': 115}}
|
||||
```
|
||||
@@ -0,0 +1,110 @@
|
||||
---
|
||||
title: "OllamaTextEmbedder"
|
||||
id: ollamatextembedder
|
||||
slug: "/ollamatextembedder"
|
||||
description: "This component computes the embeddings of a string using embedding models compatible with the Ollama Library."
|
||||
---
|
||||
|
||||
# OllamaTextEmbedder
|
||||
|
||||
This component computes the embeddings of a string using embedding models compatible with the Ollama Library.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before an embedding [Retriever](../retrievers.mdx) in a query/RAG pipeline |
|
||||
| **Mandatory run variables** | `text`: A string |
|
||||
| **Output variables** | `embedding`: A list of float numbers (vectors) <br /> <br />`meta`: A dictionary of metadata strings |
|
||||
| **API reference** | [Ollama](/reference/integrations-ollama) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/ollama |
|
||||
| **Package name** | `ollama-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
`OllamaDocumentEmbedder` computes the embeddings of a list of documents and stores the obtained vectors in the embedding field of each document. It uses embedding models compatible with the Ollama Library.
|
||||
|
||||
The vectors computed by this component are necessary to perform embedding retrieval on a collection of documents. At retrieval time, the vector that represents the query is compared with those of the documents to find the most similar or relevant documents.
|
||||
|
||||
## Overview
|
||||
|
||||
`OllamaTextEmbedder` should be used to embed a string. For embedding a list of documents, use the [`OllamaDocumentEmbedder`](ollamadocumentembedder.mdx).
|
||||
|
||||
The component uses `http://localhost:11434` as the default URL as most available setups (Mac, Linux, Docker) default to port 11434.
|
||||
|
||||
### Compatible Models
|
||||
|
||||
Unless specified otherwise while initializing this component, the default embedding model is "nomic-embed-text". See other possible pre-built models in Ollama's [library](https://ollama.ai/library). To load your own custom model, follow the [instructions](https://github.com/ollama/ollama/blob/main/docs/modelfile.md) from Ollama.
|
||||
|
||||
### Installation
|
||||
|
||||
To start using this integration with Haystack, install the package with:
|
||||
|
||||
```shell
|
||||
pip install ollama-haystack
|
||||
```
|
||||
|
||||
Make sure that you have a running Ollama model (either through a docker container, or locally hosted). No other configuration is necessary as Ollama has the embedding API built in.
|
||||
|
||||
### Embedding Metadata
|
||||
|
||||
Most embedded metadata contains information about the model name and type. You can pass [optional arguments](https://github.com/jmorganca/ollama/blob/main/docs/modelfile.md#valid-parameters-and-values), such as temperature, top_p, and others, to the Ollama generation endpoint.
|
||||
|
||||
The name of the model used will be automatically appended as part of the metadata. An example payload using the nomic-embed-text model will look like this:
|
||||
|
||||
```python
|
||||
{"meta": {"model": "nomic-embed-text"}}
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.embedders.ollama import OllamaTextEmbedder
|
||||
|
||||
embedder = OllamaTextEmbedder()
|
||||
|
||||
result = embedder.run(
|
||||
text="What do llamas say once you have thanked them? No probllama!",
|
||||
)
|
||||
|
||||
print(result["embedding"])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from cohere_haystack.embedders.text_embedder import OllamaTextEmbedder
|
||||
from cohere_haystack.embedders.document_embedder import OllamaDocumentEmbedder
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
Document(content="Germany has many big cities"),
|
||||
]
|
||||
|
||||
document_embedder = OllamaDocumentEmbedder()
|
||||
documents_with_embeddings = document_embedder.run(documents)["documents"]
|
||||
document_store.write_documents(documents_with_embeddings)
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", OllamaTextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "Who lives in Berlin?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
```
|
||||
@@ -0,0 +1,120 @@
|
||||
---
|
||||
title: "OpenAIDocumentEmbedder"
|
||||
id: openaidocumentembedder
|
||||
slug: "/openaidocumentembedder"
|
||||
description: "OpenAIDocumentEmbedder computes the embeddings of a list of documents and stores the obtained vectors in the embedding field of each document. It uses OpenAI embedding models."
|
||||
---
|
||||
|
||||
# OpenAIDocumentEmbedder
|
||||
|
||||
OpenAIDocumentEmbedder computes the embeddings of a list of documents and stores the obtained vectors in the embedding field of each document. It uses OpenAI embedding models.
|
||||
|
||||
The vectors computed by this component are necessary to perform embedding retrieval on a collection of documents. At retrieval time, the vector representing the query is compared with those of the documents to find the most similar or relevant documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Before a [`DocumentWriter`](../writers/documentwriter.mdx) in an indexing pipeline |
|
||||
| **Mandatory init variables** | `api_key`: An OpenAI API key. Can be set with `OPENAI_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `documents`: A list of documents |
|
||||
| **Output variables** | `documents`: A list of documents (enriched with embeddings) <br /> <br />`meta`: A dictionary of metadata |
|
||||
| **API reference** | [Embedders](/reference/embedders-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/embedders/openai_document_embedder.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
To see the list of compatible OpenAI embedding models, head over to OpenAI [documentation](https://platform.openai.com/docs/guides/embeddings/embedding-models). The default model for `OpenAIDocumentEmbedder` is `text-embedding-ada-002`. You can specify another model with the `model` parameter when initializing this component.
|
||||
|
||||
This component should be used to embed a list of documents. To embed a string, use the [OpenAITextEmbedder](openaitextembedder.mdx).
|
||||
|
||||
The component uses an `OPENAI_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`:
|
||||
|
||||
```
|
||||
embedder = OpenAIDocumentEmbedder(api_key=Secret.from_token("<your-api-key>"))
|
||||
```
|
||||
|
||||
### Embedding Metadata
|
||||
|
||||
Text documents often come with a set of metadata. If they are distinctive and semantically meaningful, you can embed them along with the text of the document to improve retrieval.
|
||||
|
||||
You can do this easily by using the Document Embedder:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.embedders import OpenAIDocumentEmbedder
|
||||
|
||||
doc = Document(content="some text", meta={"title": "relevant title", "page number": 18})
|
||||
|
||||
embedder = OpenAIDocumentEmbedder(meta_fields_to_embed=["title"])
|
||||
|
||||
docs_w_embeddings = embedder.run(documents=[doc])["documents"]
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Here is how you can use the component on its own:
|
||||
|
||||
```python
|
||||
from haystack.components.embedders import OpenAIDocumentEmbedder
|
||||
|
||||
doc = Document(content="I love pizza!")
|
||||
|
||||
document_embedder = OpenAIDocumentEmbedder(api_key=Secret.from_token("<your-api-key>"))
|
||||
|
||||
result = document_embedder.run([doc])
|
||||
print(result["documents"][0].embedding)
|
||||
|
||||
# [0.017020374536514282, -0.023255806416273117, ...]
|
||||
```
|
||||
|
||||
:::info
|
||||
We recommend setting OPENAI_API_KEY as an environment variable instead of setting it as a parameter.
|
||||
:::
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.embedders import OpenAITextEmbedder, OpenAIDocumentEmbedder
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
|
||||
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Wolfgang and I live in Berlin"),
|
||||
Document(content="I saw a black horse running"),
|
||||
Document(content="Germany has many big cities"),
|
||||
]
|
||||
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component("embedder", OpenAIDocumentEmbedder())
|
||||
indexing_pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
indexing_pipeline.connect("embedder", "writer")
|
||||
|
||||
indexing_pipeline.run({"embedder": {"documents": documents}})
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("text_embedder", OpenAITextEmbedder())
|
||||
query_pipeline.add_component(
|
||||
"retriever",
|
||||
InMemoryEmbeddingRetriever(document_store=document_store),
|
||||
)
|
||||
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
|
||||
|
||||
query = "Who lives in Berlin?"
|
||||
|
||||
result = query_pipeline.run({"text_embedder": {"text": query}})
|
||||
|
||||
print(result["retriever"]["documents"][0])
|
||||
|
||||
# Document(id=..., mimetype: 'text/plain',
|
||||
# text: 'My name is Wolfgang and I live in Berlin')
|
||||
```
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user