chore: import upstream snapshot with attribution
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,167 @@
|
||||
---
|
||||
title: "OrcaRouterChatGenerator"
|
||||
id: orcarouterchatgenerator
|
||||
slug: "/orcarouterchatgenerator"
|
||||
description: "This component enables chat completion through [OrcaRouter](https://www.orcarouter.ai/), an OpenAI-compatible model routing gateway."
|
||||
---
|
||||
|
||||
# OrcaRouterChatGenerator
|
||||
|
||||
This component enables chat completion through [OrcaRouter](https://www.orcarouter.ai/), an OpenAI-compatible model routing gateway.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: An OrcaRouter API key. Can be set with `ORCAROUTER_API_KEY` env variable or passed to `init()` method. |
|
||||
| **Mandatory run variables** | `messages`: A list of [ChatMessage](../../concepts/data-classes/chatmessage.mdx) objects |
|
||||
| **Output variables** | `replies`: A list of [ChatMessage](../../concepts/data-classes/chatmessage.mdx) objects |
|
||||
| **API reference** | [OrcaRouter](/reference/integrations-orcarouter) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/orcarouter |
|
||||
| **Package name** | `orcarouter-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `OrcaRouterChatGenerator` enables you to use models from multiple providers (such as `openai/gpt-4o-mini`, `anthropic/claude-opus-4.8`, and `google/gemini-2.5-flash`) by making chat completion calls to the [OrcaRouter API](https://docs.orcarouter.ai). Models are addressed with a `provider/model` namespace, and you can browse the available models in the [OrcaRouter model catalog](https://www.orcarouter.ai/models).
|
||||
|
||||
This generator also supports OrcaRouter-specific features such as:
|
||||
|
||||
- Automatic routing with the `orcarouter/auto` model, which lets OrcaRouter pick a live upstream model per request based on the policy configured in your OrcaRouter console.
|
||||
- Provider routing and model fallback that are configurable with the `generation_kwargs` parameter during initialization or runtime. OrcaRouter-specific routing options are forwarded to the gateway through `extra_body`.
|
||||
|
||||
This component uses the same `ChatMessage` format as other Haystack Chat Generators for structured input and output. For more information, see the [ChatMessage documentation](../../concepts/data-classes/chatmessage.mdx).
|
||||
|
||||
### Tool Support
|
||||
|
||||
`OrcaRouterChatGenerator` supports function calling through the `tools` parameter, which accepts flexible tool configurations:
|
||||
|
||||
- **A list of Tool objects**: Pass individual tools as a list
|
||||
- **A single Toolset**: Pass an entire Toolset directly
|
||||
- **Mixed Tools and Toolsets**: Combine multiple Toolsets with standalone tools in a single list
|
||||
|
||||
This allows you to organize related tools into logical groups while also including standalone tools as needed.
|
||||
|
||||
```python
|
||||
from haystack.tools import Tool, Toolset
|
||||
from haystack_integrations.components.generators.orcarouter import OrcaRouterChatGenerator
|
||||
|
||||
# Create individual tools
|
||||
weather_tool = Tool(name="weather", description="Get weather info", ...)
|
||||
news_tool = Tool(name="news", description="Get latest news", ...)
|
||||
|
||||
# Group related tools into a toolset
|
||||
math_toolset = Toolset([add_tool, subtract_tool, multiply_tool])
|
||||
|
||||
# Pass mixed tools and toolsets to the generator
|
||||
generator = OrcaRouterChatGenerator(
|
||||
tools=[math_toolset, weather_tool, news_tool] # Mix of Toolset and Tool objects
|
||||
)
|
||||
```
|
||||
|
||||
For more details on working with tools, see the [Tool](../../tools/tool.mdx) and [Toolset](../../tools/toolset.mdx) documentation.
|
||||
|
||||
### Initialization
|
||||
|
||||
To use this integration, you need an OrcaRouter API key. You can provide it with the `ORCAROUTER_API_KEY` environment variable or by using a [Secret](../../concepts/secret-management.mdx).
|
||||
|
||||
Then, install the `orcarouter-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install orcarouter-haystack
|
||||
```
|
||||
|
||||
### Streaming
|
||||
|
||||
`OrcaRouterChatGenerator` supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) responses from the LLM, allowing tokens to be emitted as they are generated. To enable streaming, pass a callable to the `streaming_callback` parameter during initialization.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.orcarouter import (
|
||||
OrcaRouterChatGenerator,
|
||||
)
|
||||
|
||||
client = OrcaRouterChatGenerator(model="openai/gpt-4o-mini")
|
||||
response = client.run([ChatMessage.from_user("What are Agentic Pipelines? Be brief.")])
|
||||
print(response["replies"][0].text)
|
||||
```
|
||||
|
||||
With automatic routing and streaming:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.orcarouter import (
|
||||
OrcaRouterChatGenerator,
|
||||
)
|
||||
|
||||
client = OrcaRouterChatGenerator(
|
||||
model="orcarouter/auto",
|
||||
streaming_callback=lambda chunk: print(chunk.content, end="", flush=True),
|
||||
)
|
||||
|
||||
response = client.run([ChatMessage.from_user("What are Agentic Pipelines? Be brief.")])
|
||||
|
||||
# check the model used for the response
|
||||
print("\n\n Model used: ", response["replies"][0].meta["model"])
|
||||
```
|
||||
|
||||
With a fallback chain:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.orcarouter import (
|
||||
OrcaRouterChatGenerator,
|
||||
)
|
||||
|
||||
client = OrcaRouterChatGenerator(
|
||||
model="openai/gpt-4o-mini",
|
||||
generation_kwargs={
|
||||
"extra_body": {
|
||||
"route": "fallback",
|
||||
"models": [
|
||||
"openai/gpt-4o-mini",
|
||||
"anthropic/claude-haiku-4.5",
|
||||
"google/gemini-2.5-flash",
|
||||
],
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
response = client.run([ChatMessage.from_user("What is Haystack?")])
|
||||
print(response["replies"][0].text)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.orcarouter import (
|
||||
OrcaRouterChatGenerator,
|
||||
)
|
||||
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
llm = OrcaRouterChatGenerator(model="openai/gpt-4o-mini")
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
pipe.connect("builder.prompt", "llm.messages")
|
||||
|
||||
messages = [
|
||||
ChatMessage.from_system("Give brief answers."),
|
||||
ChatMessage.from_user("Tell me about {{city}}"),
|
||||
]
|
||||
|
||||
response = pipe.run(
|
||||
data={"builder": {"template": messages, "template_variables": {"city": "Berlin"}}},
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
Reference in New Issue
Block a user