chore: import upstream snapshot with attribution
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled

This commit is contained in:
wehub-resource-sync
2026-07-13 13:22:28 +08:00
commit c56bef871b
9296 changed files with 1854228 additions and 0 deletions
@@ -0,0 +1,202 @@
---
title: "DatadogConnector"
id: datadogconnector
slug: "/datadogconnector"
description: "Learn how to work with Datadog in Haystack."
---
# DatadogConnector
Learn how to work with Datadog in Haystack.
<div className="key-value-table">
| | |
| --- | --- |
| **Most common position in a pipeline** | Anywhere, as its not connected to other components |
| **Mandatory init variables** | None. The connection to the Datadog backend is created at initialization time |
| **Output variables** | `name`: The name of the tracing component |
| **API reference** | [datadog](/reference/integrations-datadog) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/datadog |
| **Package name** | `datadog-haystack` |
</div>
## Overview
`DatadogConnector` integrates tracing capabilities into Haystack pipelines using [Datadog](https://www.datadoghq.com/), through [Datadog's tracing library `ddtrace`](https://ddtrace.readthedocs.io/en/stable/). It captures detailed information about pipeline runs, like API calls, context data, prompts, and more, so you can see the complete trace of your pipeline execution in Datadog.
Datadog tracing is enabled as soon as the `DatadogConnector` is initialized, so you only need to add it to your pipeline it does not need to be connected to other components or to run to take effect.
You can optionally pass a `name` to identify this tracing component (it defaults to `datadog`).
### Prerequisites
These are the things that you need before working with the `DatadogConnector`:
1. A way to receive traces, such as a running [Datadog Agent](https://docs.datadoghq.com/agent/). `ddtrace` sends traces to the Datadog Agent at `localhost:8126` by default.
2. Set the `HAYSTACK_CONTENT_TRACING_ENABLED` environment variable to `true` this will enable content tracing (inputs and outputs) in your pipelines.
3. Configure `ddtrace` through the standard mechanisms, for example the `DD_SERVICE`, `DD_ENV`, and `DD_VERSION` environment variables, or by running your application with the `ddtrace-run` command. See the [ddtrace documentation](https://ddtrace.readthedocs.io/en/stable/) for more details.
### Installation
First, install the `datadog-haystack` package to use the `DatadogConnector`:
```shell
pip install datadog-haystack
```
<br />
:::info[Usage Notice]
To ensure proper tracing, always set environment variables before importing any Haystack components. This is crucial because Haystack initializes its internal tracing components during import. In the example below, we first set the environment variables and then import the relevant Haystack components.
Alternatively, an even better practice is to set these environment variables in your shell before running the script. This approach keeps configuration separate from code and allows for easier management of different environments.
:::
## Usage
In the example below, we are adding `DatadogConnector` to the pipeline as a _tracer_. Each pipeline run will produce a trace that includes the entire execution context, including prompts, completions, and metadata. You can then view the traces in your Datadog dashboard.
```python
import os
os.environ["HAYSTACK_CONTENT_TRACING_ENABLED"] = "true"
from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.connectors.datadog import DatadogConnector
pipe = Pipeline()
pipe.add_component("tracer", DatadogConnector("Chat example"))
pipe.add_component("prompt_builder", ChatPromptBuilder())
pipe.add_component("llm", OpenAIChatGenerator())
pipe.connect("prompt_builder.prompt", "llm.messages")
messages = [
ChatMessage.from_system(
"Always respond in German even if some input data is in other languages.",
),
ChatMessage.from_user("Tell me about {{location}}"),
]
response = pipe.run(
data={
"prompt_builder": {
"template_variables": {"location": "Berlin"},
"template": messages,
},
},
)
print(response["llm"]["replies"][0])
```
### With an Agent
```python
import os
os.environ["HAYSTACK_CONTENT_TRACING_ENABLED"] = "true"
from typing import Annotated
from haystack.components.agents import Agent
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.tools import tool
from haystack import Pipeline
from haystack_integrations.components.connectors.datadog import DatadogConnector
@tool
def get_weather(city: Annotated[str, "The city to get weather for"]) -> str:
"""Get current weather information for a city."""
weather_data = {
"Berlin": "18°C, partly cloudy",
"New York": "22°C, sunny",
"Tokyo": "25°C, clear skies",
}
return weather_data.get(city, f"Weather information for {city} not available")
@tool
def calculate(
operation: Annotated[
str,
"Mathematical operation: add, subtract, multiply, divide",
],
a: Annotated[float, "First number"],
b: Annotated[float, "Second number"],
) -> str:
"""Perform basic mathematical calculations."""
if operation == "add":
result = a + b
elif operation == "subtract":
result = a - b
elif operation == "multiply":
result = a * b
elif operation == "divide":
if b == 0:
return "Error: Division by zero"
result = a / b
else:
return f"Error: Unknown operation '{operation}'"
return f"The result of {a} {operation} {b} is {result}"
# Create the chat generator
chat_generator = OpenAIChatGenerator()
# Create the agent with tools
agent = Agent(
chat_generator=chat_generator,
tools=[get_weather, calculate],
system_prompt="You are a helpful assistant with access to weather and calculator tools. Use them when needed.",
exit_conditions=["text"],
)
# Create the DatadogConnector for tracing
datadog_connector = DatadogConnector("Agent Example")
# Build the pipeline
pipe = Pipeline()
pipe.add_component("tracer", datadog_connector)
pipe.add_component("agent", agent)
# Run the pipeline
response = pipe.run(
data={
"agent": {
"messages": [
ChatMessage.from_user(
"What's the weather in Berlin and calculate 15 + 27?",
),
],
},
"tracer": {},
},
)
# Display results
print("Agent Response:")
print(response["agent"]["last_message"].text)
```
### Configuring the tracing backend directly
Instead of using the `DatadogConnector`, you can configure the Datadog tracing backend directly by enabling a `DatadogTracer`. Make sure to set the `HAYSTACK_CONTENT_TRACING_ENABLED` environment variable before importing any Haystack components.
```python
import ddtrace
from haystack import tracing
from haystack_integrations.tracing.datadog import DatadogTracer
tracing.enable_tracing(DatadogTracer(ddtrace.tracer))
```
@@ -0,0 +1,18 @@
---
title: "External Integrations"
id: external-integrations-connectors
slug: "/external-integrations-connectors"
description: "External integrations that connect your pipelines to services by external providers."
---
# External Integrations
External integrations that connect your pipelines to services by external providers.
| Name | Description |
| --- | --- |
| [Arize AI](https://haystack.deepset.ai/integrations/arize) | Trace and evaluate your Haystack pipelines with Arize AI. |
| [Arize Phoenix](https://haystack.deepset.ai/integrations/arize-phoenix) | Trace and evaluate your Haystack pipelines with Arize Phoenix. |
| [Context AI](https://haystack.deepset.ai/integrations/context-ai) | Log conversations for analytics by Context.ai |
| [Opik](https://haystack.deepset.ai/integrations/opik) | Trace and evaluate your Haystack pipelines with Opik platform. |
| [Traceloop](https://haystack.deepset.ai/integrations/traceloop) | Evaluate and monitor the quality of your LLM apps and agents |
@@ -0,0 +1,105 @@
---
title: "GitHubFileEditor"
id: githubfileeditor
slug: "/githubfileeditor"
description: "This is a component for editing files in GitHub repositories through the GitHub API."
---
# GitHubFileEditor
This is a component for editing files in GitHub repositories through the GitHub API.
<div className="key-value-table">
| | |
| --- | --- |
| **Most common position in a pipeline** | After a Chat Generator, or right at the beginning of a pipeline |
| **Mandatory init variables** | `github_token`: GitHub personal access token. Can be set with `GITHUB_TOKEN` env var. |
| **Mandatory run variables** | `command`: Operation type (edit, create, delete, undo) <br /> <br />`payload`: Command-specific parameters |
| **Output variables** | `result`: String that indicates the operation result |
| **API reference** | [GitHub](/reference/integrations-github) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/github |
| **Package name** | `github-haystack` |
</div>
## Overview
`GitHubFileEditor` supports multiple file operations, including editing existing files, creating new files, deleting files, and undoing recent changes.
There are four main commands:
- **EDIT**: Edit an existing file by replacing specific content
- **CREATE**: Create a new file with specified content
- **DELETE**: Delete an existing file
- **UNDO**: Revert the last commit if made by the same user
### Authorization
This component requires GitHub authentication with a personal access token. You can set the token using the `GITHUB_TOKEN` environment variable, or pass it directly during initialization via the `github_token` parameter.
To create a personal access token, visit [GitHub's token settings page](https://github.com/settings/tokens). Make sure to grant the appropriate permissions for repository access and content management.
### Installation
Install the GitHub integration with pip:
```shell
pip install github-haystack
```
## Usage
:::info[Repository Placeholder]
To run the following code snippets, you need to replace the `owner/repo` with your own GitHub repository name.
:::
### On its own
Editing an existing file:
```python
from haystack_integrations.components.connectors.github import GitHubFileEditor, Command
editor = GitHubFileEditor(repo="owner/repo", branch="main")
result = editor.run(
command=Command.EDIT,
payload={
"path": "src/example.py",
"original": "def old_function():",
"replacement": "def new_function():",
"message": "Renamed function for clarity",
},
)
print(result)
```
```bash
{'result': 'Edit successful'}
```
Creating a new file:
```python
from haystack_integrations.components.connectors.github import GitHubFileEditor, Command
editor = GitHubFileEditor(repo="owner/repo")
result = editor.run(
command=Command.CREATE,
payload={
"path": "docs/new_file.md",
"content": "# New Documentation\n\nThis is a new file.",
"message": "Add new documentation file",
},
)
print(result)
```
```bash
{'result': 'File created successfully'}
```
@@ -0,0 +1,129 @@
---
title: "GitHubIssueCommenter"
id: githubissuecommenter
slug: "/githubissuecommenter"
description: "This component posts comments to GitHub issues using the GitHub API."
---
# GitHubIssueCommenter
This component posts comments to GitHub issues using the GitHub API.
<div className="key-value-table">
| | |
| --- | --- |
| **Most common position in a pipeline** | After a Chat Generator that provides the comment text to post or right at the beginning of a pipeline |
| **Mandatory init variables** | `github_token`: GitHub personal access token. Can be set with `GITHUB_TOKEN` env var. |
| **Mandatory run variables** | `url`: A GitHub issue URL <br /> <br />`comment`: Comment text to post |
| **Output variables** | `success`: Boolean indicating whether the comment was posted successfully |
| **API reference** | [GitHub](/reference/integrations-github) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/github |
| **Package name** | `github-haystack` |
</div>
## Overview
`GitHubIssueCommenter` takes a GitHub issue URL and comment text, then posts the comment to the specified issue.
The component requires authentication with a GitHub personal access token since posting comments is an authenticated operation.
### Authorization
This component requires GitHub authentication with a personal access token. You can set the token using the `GITHUB_TOKEN` environment variable, or pass it directly during initialization via the `github_token` parameter.
To create a personal access token, visit [GitHub's token settings page](https://github.com/settings/tokens). Make sure to grant the appropriate permissions for repository access and issue management.
### Installation
Install the GitHub integration with pip:
```shell
pip install github-haystack
```
## Usage
:::info[Repository Placeholder]
To run the following code snippets, you need to replace the `owner/repo` with your own GitHub repository name.
:::
### On its own
Basic usage with environment variable authentication:
```python
from haystack_integrations.components.connectors.github import GitHubIssueCommenter
commenter = GitHubIssueCommenter()
result = commenter.run(
url="https://github.com/owner/repo/issues/123",
comment="Thanks for reporting this issue! We'll look into it.",
)
print(result)
```
```bash
{'success': True}
```
### In a pipeline
The following pipeline analyzes a GitHub issue and automatically posts a response:
```python
from haystack import Pipeline
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.connectors.github import (
GitHubIssueViewer,
GitHubIssueCommenter,
)
issue_viewer = GitHubIssueViewer()
issue_commenter = GitHubIssueCommenter()
prompt_template = [
ChatMessage.from_system(
"You are a helpful assistant that analyzes GitHub issues and creates appropriate responses.",
),
ChatMessage.from_user(
"Based on the following GitHub issue:\n"
"{% for document in documents %}"
"{% if document.meta.type == 'issue' %}"
"**Issue Title:** {{ document.meta.title }}\n"
"**Issue Description:** {{ document.content }}\n"
"{% endif %}"
"{% endfor %}\n"
"Generate a helpful response comment for this issue. Keep it professional and concise.",
),
]
prompt_builder = ChatPromptBuilder(template=prompt_template, required_variables="*")
llm = OpenAIChatGenerator(model="gpt-4o-mini")
pipeline = Pipeline()
pipeline.add_component("issue_viewer", issue_viewer)
pipeline.add_component("prompt_builder", prompt_builder)
pipeline.add_component("llm", llm)
pipeline.add_component("issue_commenter", issue_commenter)
pipeline.connect("issue_viewer.documents", "prompt_builder.documents")
pipeline.connect("prompt_builder.prompt", "llm.messages")
pipeline.connect("llm.replies", "issue_commenter.comment")
issue_url = "https://github.com/owner/repo/issues/123"
result = pipeline.run(
data={"issue_viewer": {"url": issue_url}, "issue_commenter": {"url": issue_url}},
)
print(f"Comment posted successfully: {result['issue_commenter']['success']}")
```
```
Comment posted successfully: True
```
@@ -0,0 +1,127 @@
---
title: "GitHubIssueViewer"
id: githubissueviewer
slug: "/githubissueviewer"
description: "This component fetches and parses GitHub issues into Haystack documents."
---
# GitHubIssueViewer
This component fetches and parses GitHub issues into Haystack documents.
<div className="key-value-table">
| | |
| --- | --- |
| **Most common position in a pipeline** | Right at the beginning of a pipeline and before a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) that expects the content of a GitHub issue as input |
| **Mandatory run variables** | `url`: A GitHub issue URL |
| **Output variables** | `documents`: A list of documents containing the main issue and its comments |
| **API reference** | [GitHub](/reference/integrations-github) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/github |
| **Package name** | `github-haystack` |
</div>
## Overview
`GitHubIssueViewer` takes a GitHub issue URL and returns a list of documents where:
- The first document contains the main issue content
- Subsequent documents contain the issue comments (if any)
Each document includes rich metadata such as the issue title, number, state, creation date, author, and more.
### Authorization
The component can work without authentication for public repositories, but for private repositories or to avoid rate limiting, you can provide a GitHub personal access token.
You can set the token using the `GITHUB_API_KEY` environment variable, or pass it directly during initialization via the `github_token` parameter.
To create a personal access token, visit [GitHub's token settings page](https://github.com/settings/tokens).
### Installation
Install the GitHub integration with pip:
```shell
pip install github-haystack
```
## Usage
:::info[Repository Placeholder]
To run the following code snippets, you need to replace the `owner/repo` with your own GitHub repository name.
:::
### On its own
Basic usage without authentication:
```python
from haystack_integrations.components.connectors.github import GitHubIssueViewer
viewer = GitHubIssueViewer()
result = viewer.run(url="https://github.com/deepset-ai/haystack/issues/123")
print(result)
```
```bash
{'documents': [Document(id=3989459bbd8c2a8420a9ba7f3cd3cf79bb41d78bd0738882e57d509e1293c67a, content: 'sentence-transformers = 0.2.6.1
haystack = latest
farm = 0.4.3 latest branch
In the call to Emb...', meta: {'type': 'issue', 'title': 'SentenceTransformer no longer accepts \'gpu" as argument', 'number': 123, 'state': 'closed', 'created_at': '2020-05-28T04:49:31Z', 'updated_at': '2020-05-28T07:11:43Z', 'author': 'predoctech', 'url': 'https://github.com/deepset-ai/haystack/issues/123'}), Document(id=a8a56b9ad119244678804d5873b13da0784587773d8f839e07f644c4d02c167a, content: 'Thanks for reporting!
Fixed with #124 ', meta: {'type': 'comment', 'issue_number': 123, 'created_at': '2020-05-28T07:11:42Z', 'updated_at': '2020-05-28T07:11:42Z', 'author': 'tholor', 'url': 'https://github.com/deepset-ai/haystack/issues/123#issuecomment-635153940'})]}
```
### In a pipeline
The following pipeline fetches a GitHub issue, extracts relevant information, and generates a summary:
```python
from haystack import Pipeline
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.connectors.github import GitHubIssueViewer
# Initialize components
issue_viewer = GitHubIssueViewer()
prompt_template = [
ChatMessage.from_system("You are a helpful assistant that analyzes GitHub issues."),
ChatMessage.from_user(
"Based on the following GitHub issue and comments:\n"
"{% for document in documents %}"
"{% if document.meta.type == 'issue' %}"
"**Issue Title:** {{ document.meta.title }}\n"
"**Issue Description:** {{ document.content }}\n"
"{% else %}"
"**Comment by {{ document.meta.author }}:** {{ document.content }}\n"
"{% endif %}"
"{% endfor %}\n"
"Please provide a summary of the issue and suggest potential solutions.",
),
]
prompt_builder = ChatPromptBuilder(template=prompt_template, required_variables="*")
llm = OpenAIChatGenerator(model="gpt-4o-mini")
# Create pipeline
pipeline = Pipeline()
pipeline.add_component("issue_viewer", issue_viewer)
pipeline.add_component("prompt_builder", prompt_builder)
pipeline.add_component("llm", llm)
# Connect components
pipeline.connect("issue_viewer.documents", "prompt_builder.documents")
pipeline.connect("prompt_builder.prompt", "llm.messages")
# Run pipeline
issue_url = "https://github.com/deepset-ai/haystack/issues/123"
result = pipeline.run(data={"issue_viewer": {"url": issue_url}})
print(result["llm"]["replies"][0])
```
@@ -0,0 +1,79 @@
---
title: "GitHubPRCreator"
id: githubprcreator
slug: "/githubprcreator"
description: "This component creates pull requests from a fork back to the original repository through the GitHub API."
---
# GitHubPRCreator
This component creates pull requests from a fork back to the original repository through the GitHub API.
<div className="key-value-table">
| | |
| --- | --- |
| **Most common position in a pipeline** | At the end of a pipeline, after [GitHubRepoForker](githubrepoforker.mdx), [GitHubFileEditor](githubfileeditor.mdx) and other components that prepare changes for submission |
| **Mandatory init variables** | `github_token`: GitHub personal access token. Can be set with `GITHUB_TOKEN` env var. |
| **Mandatory run variables** | `issue_url`: GitHub issue URL <br /> <br />`title`: PR title <br /> <br />`branch`: Source branch <br /> <br />`base`: Target branch |
| **Output variables** | `result`: String indicating the pull request creation result |
| **API reference** | [GitHub](/reference/integrations-github) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/github |
| **Package name** | `github-haystack` |
</div>
## Overview
`GitHubPRCreator` takes a GitHub issue URL and creates a pull request from your fork to the original repository, automatically linking it to the specified issue. It's designed to work with existing forks and assumes you have already made changes in a branch.
Key features:
- **Cross-repository PRs**: Creates pull requests from your fork to the original repository
- **Issue linking**: Automatically links the PR to the specified GitHub issue
- **Draft support**: Option to create draft pull requests
- **Fork validation**: Checks that the required fork exists before creating the PR
As optional parameters, you can set `body` to provide a pull request description and the boolean parameter `draft` to open a draft pull request.
### Authorization
This component requires GitHub authentication with a personal access token from the fork owner. You can set the token using the `GITHUB_TOKEN` environment variable, or pass it directly during initialization via the `github_token` parameter.
To create a personal access token, visit [GitHub's token settings page](https://github.com/settings/tokens). Make sure to grant the appropriate permissions for repository access and pull request creation.
### Installation
Install the GitHub integration with pip:
```shell
pip install github-haystack
```
## Usage
:::info[Repository Placeholder]
To run the following code snippets, you need to replace the `owner/repo` with your own GitHub repository name.
:::
### On its own
```python
from haystack_integrations.components.connectors.github import GitHubPRCreator
pr_creator = GitHubPRCreator()
result = pr_creator.run(
issue_url="https://github.com/owner/repo/issues/123",
title="Fix issue #123",
body="This PR addresses issue #123 by implementing the requested changes.",
branch="fix-123", # Branch in your fork with the changes
base="main", # Branch in original repo to merge into
)
print(result)
```
```bash
{'result': 'Pull request #456 created successfully and linked to issue #123'}
```
@@ -0,0 +1,71 @@
---
title: "GitHubRepoForker"
id: githubrepoforker
slug: "/githubrepoforker"
description: "This component forks a GitHub repository from an issue URL through the GitHub API."
---
# GitHubRepoForker
This component forks a GitHub repository from an issue URL through the GitHub API.
<div className="key-value-table">
| | |
| --- | --- |
| **Most common position in a pipeline** | Right at the beginning of a pipeline and before an [Agent](../agents-1/agent.mdx) component that expects the name of a GitHub branch as input |
| **Mandatory init variables** | `github_token`: GitHub personal access token. Can be set with `GITHUB_TOKEN` env var. |
| **Mandatory run variables** | `url`: The URL of a GitHub issue in the repository that should be forked |
| **Output variables** | `repo`: Fork repository path <br /> <br />`issue_branch`: Issue-specific branch name (if created) |
| **API reference** | [GitHub](/reference/integrations-github) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/github |
| **Package name** | `github-haystack` |
</div>
## Overview
`GitHubRepoForker` takes a GitHub issue URL, extracts the repository information, creates or syncs a fork of that repository, and optionally creates an issue-specific branch. It's particularly useful for automated workflows that need to create pull requests or work with repository forks.
Key features:
- **Auto-sync**: Automatically syncs existing forks with the upstream repository
- **Branch creation**: Creates issue-specific branches (e.g., "fix-123" for issue #123)
- **Completion waiting**: Optionally waits for fork creation to complete
- **Fork management**: Handles existing forks intelligently
### Authorization
This component requires GitHub authentication with a personal access token. You can set the token using the `GITHUB_TOKEN` environment variable, or pass it directly during initialization via the `github_token` parameter.
To create a personal access token, visit [GitHub's token settings page](https://github.com/settings/tokens). Make sure to grant the appropriate permissions for repository forking and management.
### Installation
Install the GitHub integration with pip:
```shell
pip install github-haystack
```
## Usage
:::info[Repository Placeholder]
To run the following code snippets, you need to replace the `owner/repo` with your own GitHub repository name.
:::
### On its own
```python
from haystack_integrations.components.connectors.github import GitHubRepoForker
forker = GitHubRepoForker()
result = forker.run(url="https://github.com/owner/repo/issues/123")
print(result)
```
```bash
{'repo': 'owner/repo', 'issue_branch': 'fix-123'}
```
@@ -0,0 +1,92 @@
---
title: "GitHubRepoViewer"
id: githubrepoviewer
slug: "/githubrepoviewer"
description: "This component navigates and fetches content from GitHub repositories through the GitHub API."
---
# GitHubRepoViewer
This component navigates and fetches content from GitHub repositories through the GitHub API.
<div className="key-value-table">
| | |
| --- | --- |
| **Most common position in a pipeline** | Right at the beginning of a pipeline and before a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) that expects the content of GitHub files as input |
| **Mandatory run variables** | `path`: Repository path to view <br /> <br />`repo`: Repository in owner/repo format |
| **Output variables** | `documents`: A list of documents containing repository contents |
| **API reference** | [GitHub](/reference/integrations-github) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/github |
| **Package name** | `github-haystack` |
</div>
## Overview
`GitHubRepoViewer` provides different behavior based on the path type:
- **For directories**: Returns a list of documents, one for each item (files and subdirectories),
- **For files**: Returns a single document containing the file content.
Each document includes rich metadata such as the path, type, size, and URL.
### Authorization
The component can work without authentication for public repositories, but for private repositories or to avoid rate limiting, you can provide a GitHub personal access token.
You can set the token using the `GITHUB_TOKEN` environment variable, or pass it directly during initialization via the `github_token` parameter.
To create a personal access token, visit [GitHub's token settings page](https://github.com/settings/tokens).
### Installation
Install the GitHub integration with pip:
```shell
pip install github-haystack
```
## Usage
:::info[Repository Placeholder]
To run the following code snippets, you need to replace the `owner/repo` with your own GitHub repository name.
:::
### On its own
Viewing a directory listing:
```python
from haystack_integrations.components.connectors.github import GitHubRepoViewer
viewer = GitHubRepoViewer()
result = viewer.run(
repo="deepset-ai/haystack",
path="haystack/components",
branch="main",
)
print(result)
```
```bash
{'documents': [Document(id=..., content: 'agents', meta: {'path': 'haystack/components/agents', 'type': 'dir', 'size': 0, 'url': 'https://github.com/deepset-ai/haystack/tree/main/haystack/components/agents'}), ...]}
```
Viewing a specific file:
```python
from haystack_integrations.components.connectors.github import GitHubRepoViewer
viewer = GitHubRepoViewer(repo="deepset-ai/haystack", branch="main")
result = viewer.run(path="README.md")
print(result)
```
```bash
{'documents': [Document(id=..., content: '<div align="center">
<a href="https://haystack.deepset.ai/"><img src="https://raw.githubuserconten...', meta: {'path': 'README.md', 'type': 'file_content', 'size': 11979, 'url': 'https://github.com/deepset-ai/haystack/blob/main/README.md'})]}
```
@@ -0,0 +1,169 @@
---
title: "JinaReaderConnector"
id: jinareaderconnector
slug: "/jinareaderconnector"
description: "Use Jina AIs Reader API with Haystack."
---
# JinaReaderConnector
Use Jina AIs Reader API with Haystack.
<div className="key-value-table">
| | |
| --- | --- |
| **Most common position in a pipeline** | As the first component in a pipeline that passes the resulting document downstream |
| **Mandatory init variables** | `mode`: The operation mode for the reader (`read`, `search`, or `ground`) <br /> <br />`api_key`: The Jina API key. Can be set with `JINA_API_KEY` env var. |
| **Mandatory run variables** | `query`: A query string |
| **Output variables** | `documents`: A list of documents |
| **API reference** | [Jina](/reference/integrations-jina) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/jina |
| **Package name** | `jina-haystack` |
</div>
## Overview
`JinaReaderConnector` interacts with Jina AIs Reader API to process queries and output documents.
You need to select one of the following modes of operations when initializing the component:
- `read`: Processes a URL and extracts the textual content.
- `search`: Searches the web and returns textual content from the most relevant pages.
- `ground`: Performs fact-checking using a grounding engine.
You can find more information on these modes in the [Jina Reader documentation](https://jina.ai/reader/).
You can additionally control the response format from the Jina Reader API using the components `json_response` parameter:
- `True` (default) requests a JSON response for documents enriched with structured metadata.
- `False` requests a raw response, resulting in one document with minimal metadata.
### Authorization
The component uses a `JINA_API_KEY` environment variable by default. Otherwise, you can pass a Jina API key at initialization with `api_key` like this:
```python
ranker = JinaRanker(api_key=Secret.from_token("<your-api-key>"))
```
To get your API key, head to Jina AIs [website](https://jina.ai/reranker/).
### Installation
To start using this integration with Haystack, install the package with:
```shell
pip install jina-haystack
```
## Usage
### On its own
Read mode:
```python
from haystack_integrations.components.connectors.jina import JinaReaderConnector
reader = JinaReaderConnector(mode="read")
query = "https://example.com"
result = reader.run(query=query)
print(result)
# {'documents': [Document(id=fa3e51e4ca91828086dca4f359b6e1ea2881e358f83b41b53c84616cb0b2f7cf,
# content: 'This domain is for use in illustrative examples in documents. You may use this domain in literature ...',
# meta: {'title': 'Example Domain', 'description': '', 'url': 'https://example.com/', 'usage': {'tokens': 42}})]}
```
Search mode:
```python
from haystack_integrations.components.connectors.jina import JinaReaderConnector
reader = JinaReaderConnector(mode="search")
query = "UEFA Champions League 2024"
result = reader.run(query=query)
print(result)
# {'documents': Document(id=6a71abf9955594232037321a476d39a835c0cb7bc575d886ee0087c973c95940,
# content: '2024/25 UEFA Champions League: Matches, draw, final, key dates | UEFA Champions League | UEFA.com...',
# meta: {'title': '2024/25 UEFA Champions League: Matches, draw, final, key dates',
# 'description': 'What are the match dates? Where is the 2025 final? How will the competition work?',
# 'url': 'https://www.uefa.com/uefachampionsleague/news/...',
# 'usage': {'tokens': 5581}}), ...]}
```
Ground mode:
```python
from haystack_integrations.components.connectors.jina import JinaReaderConnector
reader = JinaReaderConnector(mode="ground")
query = "ChatGPT was launched in 2017"
result = reader.run(query=query)
print(result)
# {'documents': [Document(id=f0c964dbc1ebb2d6584c8032b657150b9aa6e421f714cc1b9f8093a159127f0c,
# content: 'The statement that ChatGPT was launched in 2017 is incorrect. Multiple references confirm that ChatG...',
# meta: {'factuality': 0, 'result': False, 'references': [
# {'url': 'https://en.wikipedia.org/wiki/ChatGPT',
# 'keyQuote': 'ChatGPT is a generative artificial intelligence (AI) chatbot developed by OpenAI and launched in 2022.',
# 'isSupportive': False}, ...],
# 'usage': {'tokens': 10188}})]}
```
### In a pipeline
**Query pipeline with search mode**
The following pipeline example, the `JinaReaderConnector` first searches for relevant documents, then feeds them along with a user query into a prompt template, and finally generates a response based on the retrieved context.
```python
from haystack import Pipeline
from haystack.utils import Secret
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack_integrations.components.connectors.jina import JinaReaderConnector
from haystack.dataclasses import ChatMessage
reader_connector = JinaReaderConnector(mode="search")
prompt_template = [
ChatMessage.from_system("You are a helpful assistant."),
ChatMessage.from_user(
"Given the information below:\n"
"{% for document in documents %}{{ document.content }}{% endfor %}\n"
"Answer question: {{ query }}.\nAnswer:",
),
]
prompt_builder = ChatPromptBuilder(
template=prompt_template,
required_variables={"query", "documents"},
)
llm = OpenAIChatGenerator(
model="gpt-4o-mini",
api_key=Secret.from_token("<your-api-key>"),
)
pipe = Pipeline()
pipe.add_component("reader_connector", reader_connector)
pipe.add_component("prompt_builder", prompt_builder)
pipe.add_component("llm", llm)
pipe.connect("reader_connector.documents", "prompt_builder.documents")
pipe.connect("prompt_builder.prompt", "llm.messages")
query = "What is the most famous landmark in Berlin?"
result = pipe.run(
data={"reader_connector": {"query": query}, "prompt_builder": {"query": query}},
)
print(result)
# {'llm': {'replies': ['The most famous landmark in Berlin is the **Brandenburg Gate**. It is considered the symbol of the city and represents reunification.'], 'meta': [{'model': 'gpt-4o-mini-2024-07-18', 'index': 0, 'finish_reason': 'stop', 'usage': {'completion_tokens': 27, 'prompt_tokens': 4479, 'total_tokens': 4506, 'completion_tokens_details': CompletionTokensDetails(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0), 'prompt_tokens_details': PromptTokensDetails(audio_tokens=0, cached_tokens=0)}}]}}
```
The same component in search mode could also be used in an indexing pipeline.
@@ -0,0 +1,234 @@
---
title: "LangfuseConnector"
id: langfuseconnector
slug: "/langfuseconnector"
description: "Learn how to work with Langfuse in Haystack."
---
# LangfuseConnector
Learn how to work with Langfuse in Haystack.
<div className="key-value-table">
| | |
| --- | --- |
| **Most common position in a pipeline** | Anywhere, as its not connected to other components |
| **Mandatory init variables** | `name`: The name of the pipeline or component to identify the tracing run |
| **Output variables** | `name`: The name of the tracing component <br /> <br />`trace_url`: A link to the tracing data |
| **API reference** | [langfuse](/reference/integrations-langfuse) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/langfuse |
| **Package name** | `langfuse-haystack` |
</div>
## Overview
`LangfuseConnector` integrates tracing capabilities into Haystack pipelines using [Langfuse](https://langfuse.com/). It captures detailed information about pipeline runs, like API calls, context data, prompts, and more. Use this component to:
- Monitor model performance, such as token usage and cost.
- Find areas for pipeline improvement by identifying low-quality outputs and collecting user feedback.
- Create datasets for fine-tuning and testing from your pipeline executions.
To work with the integration, add the `LangfuseConnector` to your pipeline, run the pipeline, and then view the tracing data on the Langfuse website. Dont connect this component to any other `LangfuseConnector` will simply run in your pipelines background.
You can optionally define two more parameters when working with this component:
- `httpx_client`: An optional custom `httpx.Client` instance for Langfuse API calls. Note that custom clients are discarded when deserializing a pipeline from YAML, as HTTPX clients cannot be serialized. In such cases, Langfuse creates a default client.
- `span_handler`: An optional custom handler for processing spans. If not provided, the `DefaultSpanHandler` is used. The span handler defines how spans are created and processed, enabling customization of span types based on component types and post-processing of spans. See more details in the [Advanced Usage section](#advanced-usage) below.
### Prerequisites
These are the things that you need before working with LangfuseConnector:
1. Make sure you have an active Langfuse [account](https://cloud.langfuse.com/).
2. Set the `HAYSTACK_CONTENT_TRACING_ENABLED` environment variable to `true` this will enable tracing in your pipelines.
3. Set the `LANGFUSE_SECRET_KEY` and `LANGFUSE_PUBLIC_KEY` environment variables with your Langfuse secret and public keys found in your account profile.
### Installation
First, install `langfuse-haystack` package to use the `LangfuseConnector`:
```shell
pip install langfuse-haystack
```
<br />
:::info[Usage Notice]
To ensure proper tracing, always set environment variables before importing any Haystack components. This is crucial because Haystack initializes its internal tracing components during import. In the example below, we first set the environmental variables and then import the relevant Haystack components.
Alternatively, an even better practice is to set these environment variables in your shell before running the script. This approach keeps configuration separate from code and allows for easier management of different environments.
:::
## Usage
In the example below, we are adding `LangfuseConnector` to the pipeline as a _tracer_. Each pipeline run will produce one trace that includes the entire execution context, including prompts, completions, and metadata.
You can then view the trace by following a URL link printed in the output.
```python
import os
os.environ["LANGFUSE_HOST"] = "https://cloud.langfuse.com"
os.environ["TOKENIZERS_PARALLELISM"] = "false"
os.environ["HAYSTACK_CONTENT_TRACING_ENABLED"] = "true"
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack import Pipeline
from haystack_integrations.components.connectors.langfuse import LangfuseConnector
if __name__ == "__main__":
pipe = Pipeline()
pipe.add_component("tracer", LangfuseConnector("Chat example"))
pipe.add_component("prompt_builder", ChatPromptBuilder())
pipe.add_component("llm", OpenAIChatGenerator())
pipe.connect("prompt_builder.prompt", "llm.messages")
messages = [
ChatMessage.from_system(
"Always respond in German even if some input data is in other languages.",
),
ChatMessage.from_user("Tell me about {{location}}"),
]
response = pipe.run(
data={
"prompt_builder": {
"template_variables": {"location": "Berlin"},
"template": messages,
},
},
)
print(response["llm"]["replies"][0])
print(response["tracer"]["trace_url"])
```
### With an Agent
```python
import os
os.environ["LANGFUSE_HOST"] = "https://cloud.langfuse.com"
os.environ["HAYSTACK_CONTENT_TRACING_ENABLED"] = "true"
from typing import Annotated
from haystack.components.agents import Agent
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.tools import tool
from haystack import Pipeline
from haystack_integrations.components.connectors.langfuse import LangfuseConnector
@tool
def get_weather(city: Annotated[str, "The city to get weather for"]) -> str:
"""Get current weather information for a city."""
weather_data = {
"Berlin": "18°C, partly cloudy",
"New York": "22°C, sunny",
"Tokyo": "25°C, clear skies",
}
return weather_data.get(city, f"Weather information for {city} not available")
@tool
def calculate(
operation: Annotated[
str,
"Mathematical operation: add, subtract, multiply, divide",
],
a: Annotated[float, "First number"],
b: Annotated[float, "Second number"],
) -> str:
"""Perform basic mathematical calculations."""
if operation == "add":
result = a + b
elif operation == "subtract":
result = a - b
elif operation == "multiply":
result = a * b
elif operation == "divide":
if b == 0:
return "Error: Division by zero"
else:
result = a / b
else:
return f"Error: Unknown operation '{operation}'"
return f"The result of {a} {operation} {b} is {result}"
if __name__ == "__main__":
# Create components
chat_generator = OpenAIChatGenerator()
agent = Agent(
chat_generator=chat_generator,
tools=[get_weather, calculate],
system_prompt="You are a helpful assistant with access to weather and calculator tools. Use them when needed.",
exit_conditions=["text"],
)
langfuse_connector = LangfuseConnector("Agent Example")
# Create and run pipeline
pipe = Pipeline()
pipe.add_component("tracer", langfuse_connector)
pipe.add_component("agent", agent)
response = pipe.run(
data={
"agent": {
"messages": [
ChatMessage.from_user(
"What's the weather in Berlin and calculate 15 + 27?",
),
],
},
"tracer": {"invocation_context": {"test": "agent_with_tools"}},
},
)
print(response["agent"]["last_message"].text)
print(response["tracer"]["trace_url"])
```
## Advanced Usage
### Customizing Langfuse Traces with SpanHandler
The `SpanHandler` interface in Haystack allows you to customize how spans are created and processed for Langfuse trace creation. This enables you to log custom metrics, add tags, or integrate metadata.
By extending `SpanHandler` or its default implementation, `DefaultSpanHandler`, you can define custom logic for span processing, providing precise control over what data is logged to Langfuse for tracking and analyzing pipeline executions.
Here's an example:
```python
from haystack_integrations.tracing.langfuse import (
LangfuseConnector,
DefaultSpanHandler,
LangfuseSpan,
)
from typing import Optional
class CustomSpanHandler(DefaultSpanHandler):
def handle(self, span: LangfuseSpan, component_type: Optional[str]) -> None:
# Custom logic to add metadata or modify span
if component_type == "OpenAIChatGenerator":
output = span._data.get("haystack.component.output", {})
if len(output.get("text", "")) < 10:
span._span.update(level="WARNING", status_message="Response too short")
# Add the custom handler to the LangfuseConnector
connector = LangfuseConnector(span_handler=CustomSpanHandler())
```
@@ -0,0 +1,155 @@
---
title: "OAuthTokenResolver"
id: oauthtokenresolver
slug: "/oauthtokenresolver"
description: "Resolves an OAuth access token at pipeline runtime and emits it for downstream components such as the SharePoint and Google Drive retrievers and fetchers."
---
# OAuthTokenResolver
Resolves an OAuth access token at pipeline runtime and emits it for downstream components such as the SharePoint and Google Drive retrievers and fetchers.
<div className="key-value-table">
| | |
| --- | --- |
| **Most common position in a pipeline** | At the start of a pipeline, feeding `access_token` into downstream components such as [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx) or [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx) |
| **Mandatory init variables** | `token_source`: The strategy that resolves the access token, for example `OAuthRefreshTokenSource` |
| **Mandatory run variables** | None for config-only sources. `subject_token`: a controller-injected per-request credential, mandatory only when the source requires it (for example `OAuthTokenExchangeSource`) |
| **Output variables** | `access_token`: A bearer token string |
| **API reference** | [OAuth](/reference/integrations-oauth) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/oauth |
| **Package name** | `oauth-haystack` |
</div>
## Overview
`OAuthTokenResolver` resolves an OAuth access token when the pipeline runs and emits it on the `access_token` output socket. Downstream components such as [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx), [`MSSharePointFetcher`](../fetchers/mssharepointfetcher.mdx), [`GoogleDriveRetriever`](../retrievers/googledriveretriever.mdx), and [`GoogleDriveFetcher`](../fetchers/googledrivefetcher.mdx) consume the token through a normal connection and never need to know how it was obtained.
The resolver itself is a thin wrapper. The actual work of getting a token is delegated to a pluggable **token source** that decides *where* the token comes from. This separation lets you swap authentication strategies (refresh-token grant, per-request token exchange, or a static long-lived token) without changing the rest of your pipeline.
### Token sources
You pass a token source to the resolver through the `token_source` parameter. All sources are importable from `haystack_integrations.utils.oauth`.
| Source | Use it when | Per-request input |
| --- | --- | --- |
| `OAuthRefreshTokenSource` | You have a single, fixed identity backed by a stored refresh token and want the source to exchange it for short-lived access tokens and cache them. | None |
| `OAuthTokenExchangeSource` | You serve multiple users (or run multiple replicas) and want to exchange an incoming per-request user assertion for a downstream token, with no persistent storage. Implements RFC 8693 token exchange and Microsoft's on-behalf-of flow. | `subject_token` |
| `OAuthStaticTokenSource` | Your provider issues a non-expiring token that you manage out of band (for example Slack or Notion). | None |
When the configured source needs a per-request credential (`OAuthTokenExchangeSource` sets `requires_subject_token = True`), the resolver declares a **mandatory** `subject_token` run input. This is a controller-injected credential for example an incoming user assertion not a value chosen by an end user. For config-only sources (`OAuthRefreshTokenSource`, `OAuthStaticTokenSource`), the resolver declares no run input and acts as a source node.
:::info[Scopes are provider-specific]
The OAuth scopes you request depend on the downstream service. For Microsoft Graph, that means scopes such as `https://graph.microsoft.com/Files.Read.All`; for Google Drive, scopes such as `https://www.googleapis.com/auth/drive.readonly`. Always consult your identity provider's documentation for the exact scope values.
:::
### Installation
Install the OAuth integration with:
```shell
pip install oauth-haystack
```
## Usage
### On its own
Resolve a token with a stored refresh token using `OAuthRefreshTokenSource`. The refresh token is read from an environment variable through the [Secret API](../../concepts/secret-management.mdx):
```python
from haystack.utils import Secret
from haystack_integrations.components.connectors.oauth import OAuthTokenResolver
from haystack_integrations.utils.oauth import OAuthRefreshTokenSource
resolver = OAuthTokenResolver(
token_source=OAuthRefreshTokenSource(
token_url="https://login.microsoftonline.com/common/oauth2/v2.0/token",
client_id="aaa-bbb-ccc",
refresh_token=Secret.from_env_var("MS_REFRESH_TOKEN"),
scopes=[
"https://graph.microsoft.com/Files.Read.All",
"offline_access",
],
),
)
access_token = resolver.run()["access_token"]
```
For a provider that issues long-lived, non-expiring tokens, use `OAuthStaticTokenSource` instead:
```python
from haystack.utils import Secret
from haystack_integrations.components.connectors.oauth import OAuthTokenResolver
from haystack_integrations.utils.oauth import OAuthStaticTokenSource
resolver = OAuthTokenResolver(
token_source=OAuthStaticTokenSource(token=Secret.from_env_var("SERVICE_TOKEN")),
)
access_token = resolver.run()["access_token"]
```
For multi-user backends, use `OAuthTokenExchangeSource`. The resolver then requires a per-request `subject_token`:
```python
from haystack_integrations.components.connectors.oauth import OAuthTokenResolver
from haystack_integrations.utils.oauth import OAuthTokenExchangeSource
resolver = OAuthTokenResolver(
token_source=OAuthTokenExchangeSource(
token_url="https://login.microsoftonline.com/<tenant>/oauth2/v2.0/token",
client_id="aaa-bbb-ccc",
subject_token_param="assertion",
grant_type="urn:ietf:params:oauth:grant-type:jwt-bearer",
scopes=["https://graph.microsoft.com/Files.Read.All"],
extra_token_params={"requested_token_use": "on_behalf_of"},
),
)
# `subject_token` is the incoming per-request user assertion, injected by your application.
access_token = resolver.run(subject_token="<incoming-user-assertion>")["access_token"]
```
### In a pipeline
In a pipeline, connect the resolver's `access_token` output to the `access_token` input of one or more downstream components. The example below wires the resolver into a [`MSSharePointRetriever`](../retrievers/mssharepointretriever.mdx) so that searching SharePoint requires only a query at runtime:
```python
from haystack import Pipeline
from haystack.utils import Secret
from haystack_integrations.components.connectors.oauth import OAuthTokenResolver
from haystack_integrations.utils.oauth import OAuthRefreshTokenSource
from haystack_integrations.components.retrievers.microsoft_sharepoint import (
MSSharePointRetriever,
)
pipeline = Pipeline()
pipeline.add_component(
"resolver",
OAuthTokenResolver(
token_source=OAuthRefreshTokenSource(
token_url="https://login.microsoftonline.com/common/oauth2/v2.0/token",
client_id="aaa-bbb-ccc",
refresh_token=Secret.from_env_var("MS_REFRESH_TOKEN"),
scopes=[
"https://graph.microsoft.com/Files.Read.All",
"https://graph.microsoft.com/Sites.Read.All",
"offline_access",
],
),
),
)
pipeline.add_component("retriever", MSSharePointRetriever(top_k=5))
pipeline.connect("resolver.access_token", "retriever.access_token")
result = pipeline.run({"retriever": {"query": "quarterly roadmap"}})
documents = result["retriever"]["documents"]
```
A single `access_token` output can be connected to several downstream inputs. For a full retrieve-then-fetch pipeline that feeds the same token to both a retriever and a fetcher, see the [`MSSharePointFetcher`](../fetchers/mssharepointfetcher.mdx) and [`GoogleDriveFetcher`](../fetchers/googledrivefetcher.mdx) pages.
@@ -0,0 +1,113 @@
---
title: "OpenAPIConnector"
id: openapiconnector
slug: "/openapiconnector"
description: "`OpenAPIConnector` is a component that acts as an interface between the Haystack ecosystem and OpenAPI services."
---
# OpenAPIConnector
`OpenAPIConnector` is a component that acts as an interface between the Haystack ecosystem and OpenAPI services.
:::tip[Consider using MCP instead]
These OpenAPI components are a legacy way to connect Haystack to external APIs. For most use cases, we recommend the [`MCPTool`](../../tools/mcptool.mdx) instead: it is the modern, standardized way to give your pipelines and agents access to external tools and services.
:::
<div className="key-value-table">
| | |
| --- | --- |
| **Most common position in a pipeline** | Anywhere, after components providing input for its run parameters |
| **Mandatory init variables** | `openapi_spec`: The OpenAPI specification for the service. Can be a URL, file path, or raw string. |
| **Mandatory run variables** | `operation_id`: The operationId from the OpenAPI spec to invoke. |
| **Output variables** | `response`: A REST service response |
| **API reference** | [OpenAPI](/reference/integrations-openapi) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/openapi |
| **Package name** | `openapi-haystack` |
</div>
## Overview
The `OpenAPIConnector` is a component within the Haystack ecosystem that allows direct invocation of REST endpoints defined in an OpenAPI (formerly Swagger) specification. It acts as a bridge between Haystack pipelines and any REST API that follows the OpenAPI standard, enabling dynamic method calls, authentication, and parameter handling.
To use the `OpenAPIConnector`, ensure that you have the `openapi-haystack` package installed:
```shell
pip install openapi-haystack
```
Unlike [OpenAPIServiceConnector](openapiserviceconnector.mdx), which works with LLMs, `OpenAPIConnector` directly calls REST endpoints using explicit input arguments.
## Usage
### On its own
You can initialize and use the `OpenAPIConnector` on its own by passing an OpenAPI specification and other parameters:
```python
from haystack.utils import Secret
from haystack_integrations.components.connectors.openapi import OpenAPIConnector
connector = OpenAPIConnector(
openapi_spec="https://bit.ly/serperdev_openapi",
credentials=Secret.from_env_var("SERPERDEV_API_KEY"),
service_kwargs={"config_factory": my_custom_config_factory},
)
response = connector.run(
operation_id="search",
arguments={"q": "Who was Nikola Tesla?"},
)
```
#### Output
The `OpenAPIConnector` returns a dictionary containing the service response:
```json
{
"response": { // here goes REST endpoint response JSON
}
}
```
### In a pipeline
The `OpenAPIConnector` can be integrated into a Haystack pipeline to interact with OpenAPI services. For example, heres how you can link the `OpenAPIConnector` to a pipeline:
```python
from haystack import Pipeline
from haystack_integrations.components.connectors.openapi import OpenAPIConnector
from haystack.dataclasses.chat_message import ChatMessage
from haystack.utils import Secret
# Initialize the OpenAPIConnector
connector = OpenAPIConnector(
openapi_spec="https://bit.ly/serperdev_openapi",
credentials=Secret.from_env_var("SERPERDEV_API_KEY"),
)
# Create a ChatMessage from the user
user_message = ChatMessage.from_user(text="Who was Nikola Tesla?")
# Define the pipeline
pipeline = Pipeline()
pipeline.add_component("openapi_connector", connector)
# Run the pipeline
response = pipeline.run(
data={
"openapi_connector": {
"operation_id": "search",
"arguments": {"q": user_message.text},
},
},
)
# Extract the answer from the response
answer = response.get("openapi_connector", {}).get("response", {})
print(answer)
```
@@ -0,0 +1,150 @@
---
title: "OpenAPIServiceConnector"
id: openapiserviceconnector
slug: "/openapiserviceconnector"
description: "`OpenAPIServiceConnector` is a component that acts as an interface between the Haystack ecosystem and OpenAPI services."
---
# OpenAPIServiceConnector
`OpenAPIServiceConnector` is a component that acts as an interface between the Haystack ecosystem and OpenAPI services.
:::tip[Consider using MCP instead]
These OpenAPI components are a legacy way to connect Haystack to external APIs. For most use cases, we recommend the [`MCPTool`](../../tools/mcptool.mdx) instead: it is the modern, standardized way to give your pipelines and agents access to external tools and services.
:::
<div className="key-value-table">
| | |
| --- | --- |
| **Most common position in a pipeline** | Flexible |
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects where the last message must be from the assistant and contain tool calls. <br /> <br />`service_openapi_spec`: OpenAPI specification of the service being invoked. It can be YAML/JSON, and all ref values must be resolved. <br /> <br />`service_credentials`: Authentication credentials for the service. We currently support two OpenAPI spec v3 security schemes: <br /> <br />1. http for Basic, Bearer, and other HTTP authentication schemes; <br />2. apiKey for API keys and cookie authentication. |
| **Output variables** | `service_response`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects where each message corresponds to a tool call invocation. <br />If a message contains multiple tool calls, there will be multiple responses. |
| **API reference** | [OpenAPI](/reference/integrations-openapi) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/openapi |
| **Package name** | `openapi-haystack` |
</div>
## Overview
`OpenAPIServiceConnector` acts as a bridge between Haystack ecosystem and OpenAPI services. This component works by using information from a `ChatMessage` to dynamically invoke service methods. It handles parameter payload parsing from `ChatMessage`, service authentication, method invocation, and response formatting, making it easier to integrate OpenAPI services.
To use `OpenAPIServiceConnector`, you need to install the `openapi-haystack` package with:
```shell
pip install openapi-haystack
```
`OpenAPIServiceConnector` component doesnt have any init parameters.
## Usage
### On its own
This component is primarily meant to be used in pipelines, as [`OpenAPIServiceToFunctions`](../converters/openapiservicetofunctions.mdx), in tandem with an LLM with tool calling capabilities, resolves the actual tool call parameters that are injected as invocation parameters for `OpenAPIServiceConnector`.
### In a pipeline
Let's say we're linking the Serper search engine to a pipeline. Here, `OpenAPIServiceConnector` uses the abilities of `OpenAPIServiceToFunctions`. `OpenAPIServiceToFunctions` first fetches and changes the [Serper's OpenAPI specification](https://bit.ly/serper_dev_spec) into function definitions that an LLM with tool calling capabilities can understand. Then, `OpenAPIServiceConnector` activates the Serper service using this specification.
More precisely, `OpenAPIServiceConnector` dynamically calls methods defined in the Serper OpenAPI specification. This involves reading chat messages to extract tool call parameters, handling authentication with the Serper service, and making the right API calls. The connector makes sure that the method call follows the Serper API requirements, such as correct formatting requests and handling responses.
Note that we used Serper just as an example here. This could be any OpenAPI-compliant service.
:::info
To run the following code snippet, note that you have to have your own Serper and OpenAI API keys.
:::
```python
import json
import requests
from typing import Any
from haystack import Pipeline
from haystack.components.converters import OutputAdapter
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.dataclasses.byte_stream import ByteStream
from haystack_integrations.components.connectors.openapi import OpenAPIServiceConnector
from haystack_integrations.components.converters.openapi import (
OpenAPIServiceToFunctions,
)
def prepare_fc_params(openai_functions_schema: dict[str, Any]) -> dict[str, Any]:
return {
"tools": [{"type": "function", "function": openai_functions_schema}],
"tool_choice": {
"type": "function",
"function": {"name": openai_functions_schema["name"]},
},
}
serperdev_spec = requests.get("https://bit.ly/serper_dev_spec").json()
system_prompt = requests.get("https://bit.ly/serper_dev_system").text
user_prompt = "Why was Sam Altman ousted from OpenAI?"
pipe = Pipeline()
pipe.add_component("spec_to_functions", OpenAPIServiceToFunctions())
pipe.add_component(
"prepare_fc_adapter",
OutputAdapter(
"{{functions[0] | prepare_fc}}",
dict[str, Any],
{"prepare_fc": prepare_fc_params},
),
)
pipe.add_component("functions_llm", OpenAIChatGenerator())
pipe.add_component("openapi_connector", OpenAPIServiceConnector())
pipe.add_component(
"message_adapter",
OutputAdapter(
"{{system_message + service_response}}",
list[ChatMessage],
unsafe=True,
),
)
pipe.add_component("llm", OpenAIChatGenerator())
pipe.connect("spec_to_functions.functions", "prepare_fc_adapter.functions")
pipe.connect(
"spec_to_functions.openapi_specs",
"openapi_connector.service_openapi_spec",
)
pipe.connect("prepare_fc_adapter", "functions_llm.generation_kwargs")
pipe.connect("functions_llm.replies", "openapi_connector.messages")
pipe.connect("openapi_connector.service_response", "message_adapter.service_response")
pipe.connect("message_adapter", "llm.messages")
result = pipe.run(
data={
"functions_llm": {
"messages": [
ChatMessage.from_system("Only do tool/function calling"),
ChatMessage.from_user(user_prompt),
],
},
"openapi_connector": {
"service_credentials": serper_dev_key,
},
"spec_to_functions": {
"sources": [ByteStream.from_string(json.dumps(serperdev_spec))],
},
"message_adapter": {
"system_message": [ChatMessage.from_system(system_prompt)],
},
},
)
print(result["llm"]["replies"][0].text)
# Sam Altman was ousted from OpenAI on November 17, 2023, following
# a "deliberative review process" by the board of directors. The board concluded
# that he was not "consistently candid in his communications". However, he
# returned as CEO just days after his ouster.
```
@@ -0,0 +1,126 @@
---
title: "OpenTelemetryConnector"
id: opentelemetryconnector
slug: "/opentelemetryconnector"
description: "Learn how to work with OpenTelemetry in Haystack."
---
# OpenTelemetryConnector
Learn how to work with OpenTelemetry in Haystack.
<div className="key-value-table">
| | |
| --- | --- |
| **Most common position in a pipeline** | Anywhere, as its not connected to other components |
| **Mandatory init variables** | None. The tracer is created at initialization time |
| **Output variables** | `name`: The name of the tracing component |
| **API reference** | [opentelemetry](/reference/integrations-opentelemetry) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/opentelemetry |
| **Package name** | `opentelemetry-haystack` |
</div>
## Overview
`OpenTelemetryConnector` integrates tracing capabilities into Haystack pipelines using [OpenTelemetry](https://opentelemetry.io/), through the [OpenTelemetry SDK](https://opentelemetry.io/docs/languages/python/). It captures detailed information about pipeline runs, like API calls, context data, prompts, and more, so you can see the complete trace of your pipeline execution in any OpenTelemetry-compatible backend.
OpenTelemetry tracing is enabled as soon as the `OpenTelemetryConnector` is initialized, so you only need to add it to your pipeline it does not need to be connected to other components or to run to take effect.
You can optionally pass a `name` to identify this tracing component (it defaults to `opentelemetry`).
### Prerequisites
These are the things that you need before working with the `OpenTelemetryConnector`:
1. A configured OpenTelemetry `TracerProvider` with an exporter (for example, an OTLP exporter that sends traces to a collector or a backend). Set up the provider before initializing the connector.
2. Set the `HAYSTACK_CONTENT_TRACING_ENABLED` environment variable to `true` this will enable content tracing (inputs and outputs) in your pipelines.
3. To add traces at even deeper levels, check out the available [OpenTelemetry instrumentations](https://opentelemetry.io/ecosystem/registry/?s=python), such as `opentelemetry-instrumentation-openai-v2` for tracing OpenAI requests.
### Installation
First, install the `opentelemetry-haystack` package to use the `OpenTelemetryConnector`:
```shell
pip install opentelemetry-haystack
```
<br />
:::info[Usage Notice]
To ensure proper tracing, always set environment variables before importing any Haystack components. This is crucial because Haystack initializes its internal tracing components during import. In the example below, we first set the environment variables and then import the relevant Haystack components.
Alternatively, an even better practice is to set these environment variables in your shell before running the script. This approach keeps configuration separate from code and allows for easier management of different environments.
:::
## Usage
In the example below, we are adding `OpenTelemetryConnector` to the pipeline as a _tracer_. Each pipeline run will produce a trace that includes the entire execution context, including prompts, completions, and metadata. You can then view the traces in your OpenTelemetry backend.
```python
import os
os.environ["HAYSTACK_CONTENT_TRACING_ENABLED"] = "true"
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.semconv.resource import ResourceAttributes
# Configure the OpenTelemetry SDK. A service name is required for most backends.
resource = Resource(attributes={ResourceAttributes.SERVICE_NAME: "haystack"})
tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(
BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")),
)
trace.set_tracer_provider(tracer_provider)
from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.connectors.opentelemetry import (
OpenTelemetryConnector,
)
pipe = Pipeline()
pipe.add_component("tracer", OpenTelemetryConnector("Chat example"))
pipe.add_component("prompt_builder", ChatPromptBuilder())
pipe.add_component("llm", OpenAIChatGenerator())
pipe.connect("prompt_builder.prompt", "llm.messages")
messages = [
ChatMessage.from_system(
"Always respond in German even if some input data is in other languages.",
),
ChatMessage.from_user("Tell me about {{location}}"),
]
response = pipe.run(
data={
"prompt_builder": {
"template_variables": {"location": "Berlin"},
"template": messages,
},
},
)
print(response["llm"]["replies"][0])
```
### Configuring the tracing backend directly
Instead of using the `OpenTelemetryConnector`, you can configure the OpenTelemetry tracing backend directly by enabling an `OpenTelemetryTracer`. Make sure to set the `HAYSTACK_CONTENT_TRACING_ENABLED` environment variable and configure your `TracerProvider` before importing any Haystack components.
```python
from opentelemetry import trace
from haystack import tracing
from haystack_integrations.tracing.opentelemetry import OpenTelemetryTracer
tracing.enable_tracing(OpenTelemetryTracer(trace.get_tracer("my_application")))
```
@@ -0,0 +1,184 @@
---
title: "WeaveConnector"
id: weaveconnector
slug: "/weaveconnector"
description: "Learn how to use Weights & Biases Weave framework for tracing and monitoring your pipeline components."
---
# WeaveConnector
Learn how to use Weights & Biases Weave framework for tracing and monitoring your pipeline components.
<div className="key-value-table">
| | |
| --- | --- |
| **Most common position in a pipeline** | Anywhere, as its not connected to other components |
| **Mandatory init variables** | `pipeline_name`: The name of your pipeline, which will also show up in Weaver dashboard. |
| **Output variables** | `pipeline_name`: The name of the pipeline that just run |
| **API reference** | [Weave](/reference/integrations-weave) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/weave |
| **Package name** | `weave-haystack` |
</div>
## Overview
This integration allows you to trace and visualize your pipeline execution in [Weights & Biases](https://wandb.ai/site/).
Information captured by the Haystack tracing tool, such as API calls, context data, and prompts, is sent to Weights & Biases, where you can see the complete trace of your pipeline execution.
### Prerequisites
You need a Weave account to use this feature. You can sign up for free at [Weights & Biases website](https://wandb.ai/site).
You will then need to set the `WANDB_API_KEY` environment variable with your Weights & Biases API key. Once logged in, you can find your API key on [your home page](https://wandb.ai/home).
Then go to `https://wandb.ai/<user_name>/projects` and see the full trace for your pipeline under the pipeline name you specified when creating the `WeaveConnector`.
You will also need to set the `HAYSTACK_CONTENT_TRACING_ENABLED` environment variable set to `true`.
## Usage
First, install the `weights_biases-haystack` package to use this connector:
```shell
pip install weights_biases-haystack
```
Then, add it to your pipeline without any connections, and it will automatically start sending traces to Weights & Biases:
```python
import os
from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.connectors.weave import WeaveConnector
pipe = Pipeline()
pipe.add_component("prompt_builder", ChatPromptBuilder())
pipe.add_component("llm", OpenAIChatGenerator())
pipe.connect("prompt_builder.prompt", "llm.messages")
connector = WeaveConnector(pipeline_name="test_pipeline")
pipe.add_component("weave", connector)
messages = [
ChatMessage.from_system(
"Always respond in German even if some input data is in other languages.",
),
ChatMessage.from_user("Tell me about {{location}}"),
]
response = pipe.run(
data={
"prompt_builder": {
"template_variables": {"location": "Berlin"},
"template": messages,
},
},
)
```
You can then see the complete trace for your pipeline at `https://wandb.ai/<user_name>/projects` under the pipeline name you specified when creating the `WeaveConnector`.
### With an Agent
```python
import os
# Enable Haystack content tracing
os.environ["HAYSTACK_CONTENT_TRACING_ENABLED"] = "true"
from typing import Annotated
from haystack.components.agents import Agent
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.tools import tool
from haystack import Pipeline
from haystack_integrations.components.connectors.weave import WeaveConnector
@tool
def get_weather(city: Annotated[str, "The city to get weather for"]) -> str:
"""Get current weather information for a city."""
weather_data = {
"Berlin": "18°C, partly cloudy",
"New York": "22°C, sunny",
"Tokyo": "25°C, clear skies",
}
return weather_data.get(city, f"Weather information for {city} not available")
@tool
def calculate(
operation: Annotated[
str,
"Mathematical operation: add, subtract, multiply, divide",
],
a: Annotated[float, "First number"],
b: Annotated[float, "Second number"],
) -> str:
"""Perform basic mathematical calculations."""
if operation == "add":
result = a + b
elif operation == "subtract":
result = a - b
elif operation == "multiply":
result = a * b
elif operation == "divide":
if b == 0:
return "Error: Division by zero"
result = a / b
else:
return f"Error: Unknown operation '{operation}'"
return f"The result of {a} {operation} {b} is {result}"
# Create the chat generator
chat_generator = OpenAIChatGenerator()
# Create the agent with tools
agent = Agent(
chat_generator=chat_generator,
tools=[get_weather, calculate],
system_prompt="You are a helpful assistant with access to weather and calculator tools. Use them when needed.",
exit_conditions=["text"],
)
# Create the WeaveConnector for tracing
weave_connector = WeaveConnector(pipeline_name="Agent Example")
# Build the pipeline
pipe = Pipeline()
pipe.add_component("tracer", weave_connector)
pipe.add_component("agent", agent)
# Run the pipeline
response = pipe.run(
data={
"agent": {
"messages": [
ChatMessage.from_user(
"What's the weather in Berlin and calculate 15 + 27?",
),
],
},
"tracer": {},
},
)
# Display results
print("Agent Response:")
print(response["agent"]["last_message"].text)
print(f"\nPipeline Name: {response['tracer']['pipeline_name']}")
print(
"\nCheck your Weights & Biases dashboard at https://wandb.ai/<user_name>/projects to see the traces!",
)
```