chore: import upstream snapshot with attribution
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
+295
@@ -0,0 +1,295 @@
|
||||
---
|
||||
title: "AIMLAPIChatGenerator"
|
||||
id: aimllapichatgenerator
|
||||
slug: "/aimllapichatgenerator"
|
||||
description: "AIMLAPIChatGenerator enables chat completion using AI models through the AIMLAPI."
|
||||
---
|
||||
|
||||
# AIMLAPIChatGenerator
|
||||
|
||||
AIMLAPIChatGenerator enables chat completion using AI models through the AIMLAPI.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: The AIMLAPI API key. Can be set with `AIMLAPI_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `messages` A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects |
|
||||
| **Output variables** | `replies`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects <br /> <br />`meta`: A list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and so on |
|
||||
| **API reference** | [AIMLAPI](/reference/integrations-aimlapi) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/aimlapi |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`AIMLAPIChatGenerator` provides access to AI models through the AIMLAPI, a unified API gateway for models from various providers. You can use different models within a single pipeline with a consistent interface. The default model is `openai/gpt-5-chat-latest`.
|
||||
|
||||
AIMLAPI uses a single API key for all providers, which allows you to switch between or combine different models without managing multiple credentials.
|
||||
|
||||
For a complete list of available models, check the [AIMLAPI documentation](https://docs.aimlapi.com/).
|
||||
|
||||
The component needs a list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects to operate. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata.
|
||||
|
||||
You can pass any chat completion parameters valid for the underlying model directly to `AIMLAPIChatGenerator` using the `generation_kwargs` parameter, both at initialization and to the `run()` method.
|
||||
|
||||
### Authentication
|
||||
|
||||
`AIMLAPIChatGenerator` needs an AIMLAPI API key to work. You can set this key in:
|
||||
|
||||
- The `api_key` init parameter using [Secret API](../../concepts/secret-management.mdx)
|
||||
- The `AIMLAPI_API_KEY` environment variable (recommended)
|
||||
|
||||
### Structured Output
|
||||
|
||||
`AIMLAPIChatGenerator` supports structured output generation for compatible models, allowing you to receive responses in a predictable format. You can use Pydantic models or JSON schemas to define the structure of the output through the `response_format` parameter in `generation_kwargs`.
|
||||
|
||||
This is useful when you need to extract structured data from text or generate responses that match a specific format.
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.aimlapi import AIMLAPIChatGenerator
|
||||
|
||||
class CityInfo(BaseModel):
|
||||
city_name: str
|
||||
country: str
|
||||
population: int
|
||||
famous_for: str
|
||||
|
||||
client = AIMLAPIChatGenerator(
|
||||
model="openai/gpt-4o-2024-08-06",
|
||||
generation_kwargs={"response_format": CityInfo}
|
||||
)
|
||||
|
||||
response = client.run(messages=[
|
||||
ChatMessage.from_user(
|
||||
"Berlin is the capital and largest city of Germany with a population of "
|
||||
"approximately 3.7 million. It's famous for its history, culture, and nightlife."
|
||||
)
|
||||
])
|
||||
print(response["replies"][0].text)
|
||||
|
||||
>> {"city_name":"Berlin","country":"Germany","population":3700000,
|
||||
>> "famous_for":"history, culture, and nightlife"}
|
||||
```
|
||||
|
||||
:::info[Model Compatibility]
|
||||
Structured output support depends on the underlying model. OpenAI models starting from `gpt-4o-2024-08-06` support Pydantic models and JSON schemas. For details on which models support this feature, refer to the respective model provider's documentation.
|
||||
:::
|
||||
|
||||
### Tool Support
|
||||
|
||||
`AIMLAPIChatGenerator` supports function calling through the `tools` parameter, which accepts flexible tool configurations:
|
||||
|
||||
- **A list of Tool objects**: Pass individual tools as a list
|
||||
- **A single Toolset**: Pass an entire Toolset directly
|
||||
- **Mixed Tools and Toolsets**: Combine multiple Toolsets with standalone tools in a single list
|
||||
|
||||
This allows you to organize related tools into logical groups while also including standalone tools as needed.
|
||||
|
||||
```python
|
||||
from haystack.tools import Tool, Toolset
|
||||
from haystack_integrations.components.generators.aimlapi import AIMLAPIChatGenerator
|
||||
|
||||
# Create individual tools
|
||||
weather_tool = Tool(name="weather", description="Get weather info", ...)
|
||||
news_tool = Tool(name="news", description="Get latest news", ...)
|
||||
|
||||
# Group related tools into a toolset
|
||||
math_toolset = Toolset([add_tool, subtract_tool, multiply_tool])
|
||||
|
||||
# Pass mixed tools and toolsets to the generator
|
||||
generator = AIMLAPIChatGenerator(
|
||||
tools=[math_toolset, weather_tool, news_tool] # Mix of Toolset and Tool objects
|
||||
)
|
||||
```
|
||||
|
||||
For more details on working with tools, see the [Tool](../../tools/tool.mdx) and [Toolset](../../tools/toolset.mdx) documentation.
|
||||
|
||||
### Streaming
|
||||
|
||||
`AIMLAPIChatGenerator` supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
You can stream output as it's generated. Pass a callback to `streaming_callback`. Use the built-in `print_streaming_chunk` to print text tokens and tool events (tool calls and tool results).
|
||||
|
||||
```python
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
|
||||
# Configure the generator with a streaming callback
|
||||
component = AIMLAPIChatGenerator(streaming_callback=print_streaming_chunk)
|
||||
|
||||
# Pass a list of messages
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
component.run([ChatMessage.from_user("Your question here")])
|
||||
```
|
||||
|
||||
:::info
|
||||
Streaming works only with a single response. If a provider supports multiple candidates, set `n=1`.
|
||||
:::
|
||||
|
||||
See our [Streaming Support](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) docs to learn more how `StreamingChunk` works and how to write a custom callback.
|
||||
|
||||
We recommend to give preference to `print_streaming_chunk` by default. Write a custom callback only if you need a specific transport (for example, SSE/WebSocket) or custom UI formatting.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `aimlapi-haystack` package to use the `AIMLAPIChatGenerator`:
|
||||
|
||||
```shell
|
||||
pip install aimlapi-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.aimlapi import AIMLAPIChatGenerator
|
||||
|
||||
client = AIMLAPIChatGenerator(model="openai/gpt-5-chat-latest", streaming_callback=print_streaming_chunk)
|
||||
|
||||
response = client.run([ChatMessage.from_user("What's Natural Language Processing? Be brief.")])
|
||||
|
||||
>> Natural Language Processing (NLP) is a field of artificial intelligence that
|
||||
>> focuses on the interaction between computers and humans through natural language.
|
||||
>> It involves enabling machines to understand, interpret, and generate human
|
||||
>> language in a meaningful way, facilitating tasks such as language translation,
|
||||
>> sentiment analysis, and text summarization.
|
||||
|
||||
print(response)
|
||||
|
||||
>> {'replies': [ChatMessage(_role=<ChatRole.ASSISTANT: 'assistant'>, _content=
|
||||
>> [TextContent(text='Natural Language Processing (NLP) is a field of artificial
|
||||
>> intelligence that focuses on enabling computers to understand, interpret, and
|
||||
>> generate human language in a meaningful and useful way.')], _name=None,
|
||||
>> _meta={'model': 'openai/gpt-5-chat-latest', 'index': 0,
|
||||
>> 'finish_reason': 'stop', 'usage': {'completion_tokens': 36,
|
||||
>> 'prompt_tokens': 15, 'total_tokens': 51}})]}
|
||||
```
|
||||
|
||||
With multimodal inputs:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage, ImageContent
|
||||
from haystack_integrations.components.generators.aimlapi import AIMLAPIChatGenerator
|
||||
|
||||
# Use a multimodal model
|
||||
llm = AIMLAPIChatGenerator(model="openai/gpt-4o")
|
||||
|
||||
image = ImageContent.from_file_path("apple.jpg", detail="low")
|
||||
user_message = ChatMessage.from_user(content_parts=[
|
||||
"What does the image show? Max 5 words.",
|
||||
image
|
||||
])
|
||||
|
||||
response = llm.run([user_message])["replies"][0].text
|
||||
print(response)
|
||||
|
||||
>>> Red apple on straw.
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
```python
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack_integrations.components.generators.aimlapi import AIMLAPIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack import Pipeline
|
||||
|
||||
# No parameter init, we don't use any runtime template variables
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
llm = AIMLAPIChatGenerator()
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
location = "Berlin"
|
||||
messages = [
|
||||
ChatMessage.from_system("Always respond in German even if some input data is in other languages."),
|
||||
ChatMessage.from_user("Tell me about {{location}}")
|
||||
]
|
||||
pipe.run(data={"prompt_builder": {"template_variables": {"location": location}, "template": messages}})
|
||||
|
||||
>> {'llm': {'replies': [ChatMessage(_role=<ChatRole.ASSISTANT: 'assistant'>,
|
||||
>> _content=[TextContent(text='Berlin ist die Hauptstadt Deutschlands und eine der
|
||||
>> bedeutendsten Städte Europas. Es ist bekannt für ihre reiche Geschichte,
|
||||
>> kulturelle Vielfalt und kreative Scene.')],
|
||||
>> _name=None, _meta={'model': 'openai/gpt-5-chat-latest', 'index': 0,
|
||||
>> 'finish_reason': 'stop', 'usage': {'completion_tokens': 120,
|
||||
>> 'prompt_tokens': 29, 'total_tokens': 149}})]}
|
||||
```
|
||||
|
||||
Using multiple models in one pipeline:
|
||||
|
||||
```python
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack_integrations.components.generators.aimlapi import AIMLAPIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack import Pipeline
|
||||
|
||||
# Create a pipeline that uses different models for different tasks
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
# Use one model for complex reasoning
|
||||
reasoning_llm = AIMLAPIChatGenerator(model="anthropic/claude-3-5-sonnet")
|
||||
# Use another model for simple tasks
|
||||
simple_llm = AIMLAPIChatGenerator(model="openai/gpt-5-chat-latest")
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("reasoning", reasoning_llm)
|
||||
pipe.add_component("simple", simple_llm)
|
||||
|
||||
# Feed the same prompt to both models
|
||||
pipe.connect("prompt_builder.prompt", "reasoning.messages")
|
||||
pipe.connect("prompt_builder.prompt", "simple.messages")
|
||||
|
||||
messages = [ChatMessage.from_user("Explain quantum computing in simple terms.")]
|
||||
result = pipe.run(data={"prompt_builder": {"template": messages}})
|
||||
|
||||
print("Reasoning model:", result["reasoning"]["replies"][0].text)
|
||||
print("Simple model:", result["simple"]["replies"][0].text)
|
||||
```
|
||||
|
||||
With tool calling:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.tools import ToolInvoker
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.tools import Tool
|
||||
from haystack_integrations.components.generators.aimlapi import AIMLAPIChatGenerator
|
||||
|
||||
def weather(city: str) -> str:
|
||||
"""Get weather for a given city."""
|
||||
return f"The weather in {city} is sunny and 32°C"
|
||||
|
||||
tool = Tool(
|
||||
name="weather",
|
||||
description="Get weather for a given city",
|
||||
parameters={"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]},
|
||||
function=weather,
|
||||
)
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("generator", AIMLAPIChatGenerator(tools=[tool]))
|
||||
pipeline.add_component("tool_invoker", ToolInvoker(tools=[tool]))
|
||||
|
||||
pipeline.connect("generator", "tool_invoker")
|
||||
|
||||
results = pipeline.run(
|
||||
data={
|
||||
"generator": {
|
||||
"messages": [ChatMessage.from_user("What's the weather like in Paris?")],
|
||||
"generation_kwargs": {"tool_choice": "auto"},
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
print(results["tool_invoker"]["tool_messages"][0].tool_call_result.result)
|
||||
>> The weather in Paris is sunny and 32°C
|
||||
```
|
||||
+221
@@ -0,0 +1,221 @@
|
||||
---
|
||||
title: "AmazonBedrockChatGenerator"
|
||||
id: amazonbedrockchatgenerator
|
||||
slug: "/amazonbedrockchatgenerator"
|
||||
description: "This component enables chat completion using models through Amazon Bedrock service."
|
||||
---
|
||||
|
||||
# AmazonBedrockChatGenerator
|
||||
|
||||
This component enables chat completion using models through Amazon Bedrock service.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `model`: The model to use <br /> <br />`aws_access_key_id`: AWS access key ID. Can be set with `AWS_ACCESS_KEY_ID` env var. <br /> <br />`aws_secret_access_key`: AWS secret access key. Can be set with `AWS_SECRET_ACCESS_KEY` env var. <br /> <br />`aws_region_name`: AWS region name. Can be set with `AWS_DEFAULT_REGION` env var. |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) instances |
|
||||
| **Output variables** | `replies`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects <br /> <br />`meta`: A list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and so on |
|
||||
| **API reference** | [Amazon Bedrock](/reference/integrations-amazon-bedrock) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/amazon_bedrock |
|
||||
|
||||
</div>
|
||||
|
||||
[Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) is a fully managed service that makes high-performing foundation models from leading AI startups and Amazon available through a unified API. You can choose from various foundation models to find the one best suited for your use case.
|
||||
|
||||
`AmazonBedrockChatGenerator` enables chat completion using chat models from Amazon, Anthropic, Cohere, Meta, Mistral, and more with a single component.
|
||||
|
||||
## Overview
|
||||
|
||||
This component uses AWS for authentication. You can use the AWS CLI to authenticate through your IAM. For more information on setting up an IAM identity-based policy, see the [official documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html).
|
||||
|
||||
:::info[Using AWS CLI]
|
||||
|
||||
Consider using AWS CLI as a more straightforward tool to manage your AWS services. With AWS CLI, you can quickly configure your [boto3 credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html). This way, you won't need to provide detailed authentication parameters when initializing Amazon Bedrock Generator in Haystack.
|
||||
:::
|
||||
|
||||
To use this component for text generation, initialize an AmazonBedrockGenerator with the model name, the AWS credentials (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_DEFAULT_REGION`) should be set as environment variables, be configured as described above or passed as [Secret](../../concepts/secret-management.mdx) arguments. Note, make sure the region you set supports Amazon Bedrock.
|
||||
|
||||
### Tool Support
|
||||
|
||||
`AmazonBedrockChatGenerator` supports function calling through the `tools` parameter, which accepts flexible tool configurations:
|
||||
|
||||
- **A list of Tool objects**: Pass individual tools as a list
|
||||
- **A single Toolset**: Pass an entire Toolset directly
|
||||
- **Mixed Tools and Toolsets**: Combine multiple Toolsets with standalone tools in a single list
|
||||
|
||||
This allows you to organize related tools into logical groups while also including standalone tools as needed.
|
||||
|
||||
```python
|
||||
from haystack.tools import Tool, Toolset
|
||||
from haystack_integrations.components.generators.amazon_bedrock import AmazonBedrockChatGenerator
|
||||
|
||||
# Create individual tools
|
||||
weather_tool = Tool(name="weather", description="Get weather info", ...)
|
||||
news_tool = Tool(name="news", description="Get latest news", ...)
|
||||
|
||||
# Group related tools into a toolset
|
||||
math_toolset = Toolset([add_tool, subtract_tool, multiply_tool])
|
||||
|
||||
# Pass mixed tools and toolsets to the generator
|
||||
generator = AmazonBedrockChatGenerator(
|
||||
model="anthropic.claude-3-5-sonnet-20240620-v1:0",
|
||||
tools=[math_toolset, weather_tool, news_tool] # Mix of Toolset and Tool objects
|
||||
)
|
||||
```
|
||||
|
||||
For more details on working with tools, see the [Tool](../../tools/tool.mdx) and [Toolset](../../tools/toolset.mdx) documentation.
|
||||
|
||||
### Streaming
|
||||
|
||||
This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
### Prompt Caching
|
||||
|
||||
`AmazonBedrockChatGenerator` supports prompt caching, to reduce inference response latency and input token costs.
|
||||
|
||||
Prompt caching on Bedrock is available for [selected models](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html).
|
||||
It allows you to define cache points within a request, as long as the input meets a model-specific minimum token threshold.
|
||||
|
||||
Each request can contain up to four cache points.
|
||||
|
||||
#### Caching messages
|
||||
|
||||
This generator allows you to control cache points at the `ChatMessage` level via the `meta` field.
|
||||
|
||||
For example, to cache a long user message to be reused across multiple requests:
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.amazon_bedrock import (
|
||||
AmazonBedrockChatGenerator,
|
||||
)
|
||||
|
||||
msg = ChatMessage.from_user(
|
||||
"long message...",
|
||||
meta={"cachePoint": {"type": "default", "ttl": "5m"}},
|
||||
)
|
||||
|
||||
generator = AmazonBedrockChatGenerator(
|
||||
model="anthropic.claude-sonnet-4-5-20250929-v1:0",
|
||||
)
|
||||
|
||||
result = generator.run(messages=[msg])
|
||||
```
|
||||
|
||||
If the cache point is successfully written, the number of cached input tokens is available at:
|
||||
```python
|
||||
result["replies"][0].meta["usage"]["cache_write_input_tokens"]
|
||||
```
|
||||
|
||||
#### Caching tools
|
||||
|
||||
You can also cache tool definitions using the `tools_cachepoint_config` initialization parameter.
|
||||
When provided, all tools sent to the model are cached, if they exceed the minimum token threshold and the selected
|
||||
model supports prompt caching.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.amazon_bedrock import (
|
||||
AmazonBedrockChatGenerator,
|
||||
)
|
||||
|
||||
# define or load your tools
|
||||
|
||||
generator = AmazonBedrockChatGenerator(
|
||||
model="anthropic.claude-sonnet-4-5-20250929-v1:0",
|
||||
tools=my_tools,
|
||||
tools_cachepoint_config={"type": "default", "ttl": "5m"},
|
||||
)
|
||||
|
||||
# send a request to the Language Model
|
||||
```
|
||||
|
||||
For more details on how prompt caching works in Amazon Bedrock, see the [official documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html).
|
||||
|
||||
## Usage
|
||||
|
||||
To start using Amazon Bedrock with Haystack, install the `amazon-bedrock-haystack` package:
|
||||
|
||||
```shell
|
||||
pip install amazon-bedrock-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.amazon_bedrock import (
|
||||
AmazonBedrockChatGenerator,
|
||||
)
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
generator = AmazonBedrockChatGenerator(model="meta.llama2-70b-chat-v1")
|
||||
messages = [
|
||||
ChatMessage.from_system(
|
||||
"You are a helpful assistant that answers question in Spanish only",
|
||||
),
|
||||
ChatMessage.from_user("What's Natural Language Processing? Be brief."),
|
||||
]
|
||||
|
||||
response = generator.run(messages)
|
||||
print(response)
|
||||
```
|
||||
|
||||
With multimodal inputs:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage, ImageContent
|
||||
from haystack_integrations.components.generators.amazon_bedrock import (
|
||||
AmazonBedrockChatGenerator,
|
||||
)
|
||||
|
||||
llm = AmazonBedrockChatGenerator(model="anthropic.claude-3-5-sonnet-20240620-v1:0")
|
||||
|
||||
image = ImageContent.from_file_path("apple.jpg")
|
||||
user_message = ChatMessage.from_user(
|
||||
content_parts=["What does the image show? Max 5 words.", image],
|
||||
)
|
||||
|
||||
response = llm.run([user_message])["replies"][0].text
|
||||
print(response)
|
||||
|
||||
# Red apple on straw mat.
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
In a RAG pipeline:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.amazon_bedrock import (
|
||||
AmazonBedrockChatGenerator,
|
||||
)
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", ChatPromptBuilder())
|
||||
pipe.add_component("llm", AmazonBedrockChatGenerator(model="meta.llama2-70b-chat-v1"))
|
||||
pipe.connect("prompt_builder", "llm")
|
||||
|
||||
country = "Germany"
|
||||
system_message = ChatMessage.from_system(
|
||||
"You are an assistant giving out valuable information to language learners.",
|
||||
)
|
||||
messages = [
|
||||
system_message,
|
||||
ChatMessage.from_user("What's the official language of {{ country }}?"),
|
||||
]
|
||||
|
||||
res = pipe.run(
|
||||
data={
|
||||
"prompt_builder": {
|
||||
"template_variables": {"country": country},
|
||||
"template": messages,
|
||||
},
|
||||
},
|
||||
)
|
||||
print(res)
|
||||
```
|
||||
+120
@@ -0,0 +1,120 @@
|
||||
---
|
||||
title: "AmazonBedrockGenerator"
|
||||
id: amazonbedrockgenerator
|
||||
slug: "/amazonbedrockgenerator"
|
||||
description: "This component enables text generation using models through Amazon Bedrock service."
|
||||
---
|
||||
|
||||
# AmazonBedrockGenerator
|
||||
|
||||
This component enables text generation using models through Amazon Bedrock service.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`PromptBuilder`](../builders/promptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `model`: The model to use <br /> <br />`aws_access_key_id`: AWS access key ID. Can be set with `AWS_ACCESS_KEY_ID` env var. <br /> <br />`aws_secret_access_key`: AWS secret access key. Can be set with `AWS_SECRET_ACCESS_KEY` env var. <br /> <br />`aws_region_name`: AWS region name. Can be set with `AWS_DEFAULT_REGION` env var. |
|
||||
| **Mandatory run variables** | `prompt`: The instructions for the Generator |
|
||||
| **Output variables** | `replies`: A list of strings with all the replies generated by the model |
|
||||
| **API reference** | [Amazon Bedrock](/reference/integrations-amazon-bedrock) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/amazon_bedrock |
|
||||
|
||||
</div>
|
||||
|
||||
[Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html) is a fully managed service that makes high-performing foundation models from leading AI startups and Amazon available through a unified API. You can choose from various foundation models to find the one best suited for your use case.
|
||||
|
||||
`AmazonBedrockGenerator` enables text generation using models from AI21 Labs, Anthropic, Cohere, Meta, Stability AI, and Amazon with a single component.
|
||||
|
||||
The models that we currently support are Anthropic's Claude, AI21 Labs' Jurassic-2, Stability AI's Stable Diffusion, Cohere's Command and Embed, Meta's Llama 2, and the Amazon Titan language and embeddings models.
|
||||
|
||||
## Overview
|
||||
|
||||
This component uses AWS for authentication. You can use the AWS CLI to authenticate through your IAM. For more information on setting up an IAM identity-based policy, see the [official documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html).
|
||||
|
||||
:::info[Using AWS CLI]
|
||||
|
||||
Consider using AWS CLI as a more straightforward tool to manage your AWS services. With AWS CLI, you can quickly configure your [boto3 credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html). This way, you won't need to provide detailed authentication parameters when initializing Amazon Bedrock Generator in Haystack.
|
||||
:::
|
||||
|
||||
To use this component for text generation, initialize an AmazonBedrockGenerator with the model name, the AWS credentials (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_DEFAULT_REGION`) should be set as environment variables, be configured as described above or passed as [Secret](../../concepts/secret-management.mdx) arguments. Note, make sure the region you set supports Amazon Bedrock.
|
||||
|
||||
To start using Amazon Bedrock with Haystack, install the `amazon-bedrock-haystack` package:
|
||||
|
||||
```shell
|
||||
pip install amazon-bedrock-haystack
|
||||
```
|
||||
|
||||
### Streaming
|
||||
|
||||
This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.amazon_bedrock import (
|
||||
AmazonBedrockGenerator,
|
||||
)
|
||||
|
||||
aws_access_key_id = "..."
|
||||
aws_secret_access_key = "..."
|
||||
aws_region_name = "eu-central-1"
|
||||
|
||||
generator = AmazonBedrockGenerator(model="anthropic.claude-v2")
|
||||
result = generator.run("Who is the best American actor?")
|
||||
for reply in result["replies"]:
|
||||
print(reply)
|
||||
|
||||
## >>> 'There is no definitive "best" American actor, as acting skill and talent a# re subjective. However, some of the most acclaimed and influential American act# ors include Tom Hanks, Daniel Day-Lewis, Denzel Washington, Meryl Streep, Rober# t De Niro, Al Pacino, Marlon Brando, Jack Nicholson, Leonardo DiCaprio and John# ny Depp. Choosing a single "best" actor comes down to personal preference.'
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
In a RAG pipeline:
|
||||
|
||||
```python
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.components.builders import PromptBuilder
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack import Pipeline
|
||||
|
||||
from haystack_integrations.components.generators.amazon_bedrock import (
|
||||
AmazonBedrockGenerator,
|
||||
)
|
||||
|
||||
template = """
|
||||
Given the following information, answer the question.
|
||||
|
||||
Context:
|
||||
{% for document in documents %}
|
||||
{{ document.content }}
|
||||
{% endfor %}
|
||||
|
||||
Question: What's the official language of {{ country }}?
|
||||
"""
|
||||
|
||||
aws_access_key_id = "..."
|
||||
aws_secret_access_key = "..."
|
||||
aws_region_name = "eu-central-1"
|
||||
generator = AmazonBedrockGenerator(model="anthropic.claude-v2")
|
||||
docstore = InMemoryDocumentStore()
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("retriever", InMemoryBM25Retriever(document_store=docstore))
|
||||
pipe.add_component("prompt_builder", PromptBuilder(template=template))
|
||||
pipe.add_component("generator", generator)
|
||||
pipe.connect("retriever", "prompt_builder.documents")
|
||||
pipe.connect("prompt_builder", "generator")
|
||||
|
||||
pipe.run({"retriever": {"query": "France"}, "prompt_builder": {"country": "France"}})
|
||||
|
||||
## {'generator': {'replies': ['Based on the context provided, the official language of France is French.']}}
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [PDF-Based Question Answering with Amazon Bedrock and Haystack](https://haystack.deepset.ai/cookbook/amazon_bedrock_for_documentation_qa)
|
||||
+211
@@ -0,0 +1,211 @@
|
||||
---
|
||||
title: "AnthropicChatGenerator"
|
||||
id: anthropicchatgenerator
|
||||
slug: "/anthropicchatgenerator"
|
||||
description: "This component enables chat completions using Anthropic large language models (LLMs)."
|
||||
---
|
||||
|
||||
# AnthropicChatGenerator
|
||||
|
||||
This component enables chat completions using Anthropic large language models (LLMs).
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: An Anthropic API key. Can be set with `ANTHROPIC_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `messages` A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects |
|
||||
| **Output variables** | `replies`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects <br /> <br />`meta`: A list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and so on |
|
||||
| **API reference** | [Anthropic](/reference/integrations-anthropic) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/anthropic |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
This integration supports Anthropic `chat` models such as `claude-3-5-sonnet-20240620`,`claude-3-opus-20240229`, `claude-3-haiku-20240307`, and similar. Check out the most recent full list in [Anthropic documentation](https://docs.anthropic.com/en/docs/about-claude/models).
|
||||
|
||||
### Parameters
|
||||
|
||||
`AnthropicChatGenerator` needs an Anthropic API key to work. You can provide this key in:
|
||||
|
||||
- The `ANTHROPIC_API_KEY` environment variable (recommended)
|
||||
- The `api_key` init parameter and Haystack [Secret](../../concepts/secret-management.mdx) API: `Secret.from_token("your-api-key-here")`
|
||||
|
||||
Set your preferred Anthropic model with the `model` parameter when initializing the component.
|
||||
|
||||
`AnthropicChatGenerator` requires a prompt to generate text, but you can pass any text generation parameters available in the Anthropic [Messaging API](https://docs.anthropic.com/en/api/messages) method directly to this component using the `generation_kwargs` parameter, both at initialization and when running the component. For more details on the parameters supported by the Anthropic API, see the [Anthropic documentation](https://docs.anthropic.com).
|
||||
|
||||
Finally, the component needs a list of `ChatMessage` objects to operate. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata.
|
||||
|
||||
Only text input modality is supported at this time.
|
||||
|
||||
### Tool Support
|
||||
|
||||
`AnthropicChatGenerator` supports function calling through the `tools` parameter, which accepts flexible tool configurations:
|
||||
|
||||
- **A list of Tool objects**: Pass individual tools as a list
|
||||
- **A single Toolset**: Pass an entire Toolset directly
|
||||
- **Mixed Tools and Toolsets**: Combine multiple Toolsets with standalone tools in a single list
|
||||
|
||||
This allows you to organize related tools into logical groups while also including standalone tools as needed.
|
||||
|
||||
```python
|
||||
from haystack.tools import Tool, Toolset
|
||||
from haystack_integrations.components.generators.anthropic import AnthropicChatGenerator
|
||||
|
||||
# Create individual tools
|
||||
weather_tool = Tool(name="weather", description="Get weather info", ...)
|
||||
news_tool = Tool(name="news", description="Get latest news", ...)
|
||||
|
||||
# Group related tools into a toolset
|
||||
math_toolset = Toolset([add_tool, subtract_tool, multiply_tool])
|
||||
|
||||
# Pass mixed tools and toolsets to the generator
|
||||
generator = AnthropicChatGenerator(
|
||||
tools=[math_toolset, weather_tool, news_tool] # Mix of Toolset and Tool objects
|
||||
)
|
||||
```
|
||||
|
||||
For more details on working with tools, see the [Tool](../../tools/tool.mdx) and [Toolset](../../tools/toolset.mdx) documentation.
|
||||
|
||||
### Streaming
|
||||
|
||||
You can stream output as it’s generated. Pass a callback to `streaming_callback`. Use the built-in `print_streaming_chunk` to print text tokens and tool events (tool calls and tool results).
|
||||
|
||||
```python
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
|
||||
## Configure any `Generator` or `ChatGenerator` with a streaming callback
|
||||
component = SomeGeneratorOrChatGenerator(streaming_callback=print_streaming_chunk)
|
||||
|
||||
## If this is a `ChatGenerator`, pass a list of messages:
|
||||
## from haystack.dataclasses import ChatMessage
|
||||
## component.run([ChatMessage.from_user("Your question here")])
|
||||
|
||||
## If this is a (non-chat) `Generator`, pass a prompt:
|
||||
## component.run({"prompt": "Your prompt here"})
|
||||
```
|
||||
|
||||
:::info
|
||||
Streaming works only with a single response. If a provider supports multiple candidates, set `n=1`.
|
||||
:::
|
||||
|
||||
See our [Streaming Support](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) docs to learn more how `StreamingChunk` works and how to write a custom callback.
|
||||
|
||||
Give preference to `print_streaming_chunk` by default. Write a custom callback only if you need a specific transport (for example, SSE/WebSocket) or custom UI formatting.
|
||||
|
||||
### Prompt caching
|
||||
|
||||
Prompt caching is a feature for Anthropic LLMs that stores large text inputs for reuse. It allows you to send a large text block once and then refer to it in later requests without resending the entire text.
|
||||
This feature is particularly useful for coding assistants that need full codebase context and for processing large documents. It can help reduce costs and improve response times.
|
||||
|
||||
Here's an example of an instance of `AnthropicChatGenerator` being initialized with prompt caching and tagging a message to be cached:
|
||||
|
||||
```python python
|
||||
from haystack_integrations.components.generators.anthropic import AnthropicChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
generation_kwargs = {"extra_headers": {"anthropic-beta": "prompt-caching-2024-07-31"}}
|
||||
|
||||
claude_llm = AnthropicChatGenerator(
|
||||
api_key=Secret.from_env_var("ANTHROPIC_API_KEY"), generation_kwargs=generation_kwargs
|
||||
)
|
||||
|
||||
system_message = ChatMessage.from_system("Replace with some long text documents, code or instructions")
|
||||
system_message.meta["cache_control"] = {"type": "ephemeral"}
|
||||
|
||||
messages = [system_message, ChatMessage.from_user("A query about the long text for example")]
|
||||
result = claude_llm.run(messages)
|
||||
|
||||
## and now invoke again with
|
||||
|
||||
messages = [system_message, ChatMessage.from_user("Another query about the long text etc")]
|
||||
result = claude_llm.run(messages)
|
||||
|
||||
## and so on, either invoking component directly or in the pipeline
|
||||
```
|
||||
|
||||
For more details, refer to Anthropic's [documentation](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching) and integration [examples](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/anthropic/example).
|
||||
|
||||
## Usage
|
||||
|
||||
Install the`anthropic-haystack` package to use the `AnthropicChatGenerator`:
|
||||
|
||||
```shell
|
||||
pip install anthropic-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.anthropic import AnthropicChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
generator = AnthropicChatGenerator()
|
||||
message = ChatMessage.from_user("What's Natural Language Processing? Be brief.")
|
||||
print(generator.run([message]))
|
||||
```
|
||||
|
||||
With multimodal inputs:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage, ImageContent
|
||||
from haystack_integrations.components.generators.anthropic import AnthropicChatGenerator
|
||||
|
||||
llm = AnthropicChatGenerator()
|
||||
|
||||
image = ImageContent.from_file_path("apple.jpg")
|
||||
user_message = ChatMessage.from_user(
|
||||
content_parts=["What does the image show? Max 5 words.", image],
|
||||
)
|
||||
|
||||
response = llm.run([user_message])["replies"][0].text
|
||||
print(response)
|
||||
|
||||
# Red apple on straw.
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
You can also use `AnthropicChatGenerator`with the Anthropic chat models in your pipeline.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.anthropic import AnthropicChatGenerator
|
||||
from haystack.utils import Secret
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", ChatPromptBuilder())
|
||||
pipe.add_component(
|
||||
"llm",
|
||||
AnthropicChatGenerator(Secret.from_env_var("ANTHROPIC_API_KEY")),
|
||||
)
|
||||
pipe.connect("prompt_builder", "llm")
|
||||
|
||||
country = "Germany"
|
||||
system_message = ChatMessage.from_system(
|
||||
"You are an assistant giving out valuable information to language learners.",
|
||||
)
|
||||
messages = [
|
||||
system_message,
|
||||
ChatMessage.from_user("What's the official language of {{ country }}?"),
|
||||
]
|
||||
|
||||
res = pipe.run(
|
||||
data={
|
||||
"prompt_builder": {
|
||||
"template_variables": {"country": country},
|
||||
"template": messages,
|
||||
},
|
||||
},
|
||||
)
|
||||
print(res)
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Advanced Prompt Customization for Anthropic](https://haystack.deepset.ai/cookbook/prompt_customization_for_anthropic)
|
||||
+88
@@ -0,0 +1,88 @@
|
||||
---
|
||||
title: "AnthropicGenerator"
|
||||
id: anthropicgenerator
|
||||
slug: "/anthropicgenerator"
|
||||
description: "This component enables text completions using Anthropic large language models (LLMs)."
|
||||
---
|
||||
|
||||
# AnthropicGenerator
|
||||
|
||||
This component enables text completions using Anthropic large language models (LLMs).
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [PromptBuilder](../builders/promptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: An Anthropic API key. Can be set with `ANTHROPIC_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `prompt`: A string containing the prompt for the LLM |
|
||||
| **Output variables** | `replies`: A list of strings with all the replies generated by the LLM <br /> <br />`meta`: A list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and so on |
|
||||
| **API reference** | [Anthropic](/reference/integrations-anthropic) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/anthropic |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
This integration supports Anthropic models such as `claude-3-5-sonnet-20240620`,`claude-3-opus-20240229`, `claude-3-haiku-20240307`, and similar. Although these LLMs are called chat models, the main prompt interface works with the string prompts. Check out the most recent full list in the [Anthropic documentation](https://docs.anthropic.com/en/docs/about-claude/models).
|
||||
|
||||
### Parameters
|
||||
|
||||
`AnthropicGenerator` needs an Anthropic API key to work. You can provide this key in:
|
||||
|
||||
- The `ANTHROPIC_API_KEY` environment variable (recommended)
|
||||
- The `api_key` init parameter and Haystack [Secret](../../concepts/secret-management.mdx) API: `Secret.from_token("your-api-key-here")`
|
||||
|
||||
Set your preferred Anthropic model in the `model` parameter when initializing the component.
|
||||
|
||||
`AnthropicGenerator` requires a prompt to generate text, but you can pass any text generation parameters available in the Anthropic [Messaging API](https://docs.anthropic.com/en/api/messages) method directly to this component using the `generation_kwargs` parameter, both at initialization and to `run()` method. For more details on the parameters supported by the Anthropic API, see [Anthropic documentation](https://docs.anthropic.com).
|
||||
|
||||
Finally, the component run method requires a single string prompt to generate text.
|
||||
|
||||
### Streaming
|
||||
|
||||
This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `anthropic-haystack` package to use the `AnthropicGenerator`:
|
||||
|
||||
```shell
|
||||
pip install anthropic-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.anthropic import AnthropicGenerator
|
||||
|
||||
generator = AnthropicGenerator()
|
||||
print(generator.run("What's Natural Language Processing? Be brief."))
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
You can also use `AnthropicGenerator` with the Anthropic models in your pipeline.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders import PromptBuilder
|
||||
from haystack_integrations.components.generators.anthropic import AnthropicGenerator
|
||||
from haystack.utils import Secret
|
||||
|
||||
template = """
|
||||
You are an assistant giving out valuable information to language learners.
|
||||
Answer this question, be brief.
|
||||
|
||||
Question: {{ query }}?
|
||||
"""
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", PromptBuilder(template))
|
||||
pipe.add_component("llm", AnthropicGenerator(Secret.from_env_var("ANTHROPIC_API_KEY")))
|
||||
pipe.connect("prompt_builder", "llm")
|
||||
|
||||
query = "What language is spoke in Germany?"
|
||||
res = pipe.run(data={"prompt_builder": {"query": {query}}})
|
||||
print(res)
|
||||
```
|
||||
+187
@@ -0,0 +1,187 @@
|
||||
---
|
||||
title: "AnthropicVertexChatGenerator"
|
||||
id: anthropicvertexchatgenerator
|
||||
slug: "/anthropicvertexchatgenerator"
|
||||
description: "This component enables chat completions using AnthropicVertex API."
|
||||
---
|
||||
|
||||
# AnthropicVertexChatGenerator
|
||||
|
||||
This component enables chat completions using AnthropicVertex API.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`ChatPromptBuilder`](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `region`: The region where the Anthropic model is deployed <br /> <br />`project_id`: GCP project ID where the Anthropic model is deployed |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects |
|
||||
| **Output variables** | `replies`: A list of strings with all the replies generated by the LLM <br /> <br />`meta`: A list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and others |
|
||||
| **API reference** | [Anthropic](/reference/integrations-anthropic) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/anthropic |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`AnthropicVertexChatGenerator` enables text generation using state-of-the-art Claude 3 LLMs using the Anthropic Vertex AI API.
|
||||
It supports `Claude 3.5 Sonnet`, `Claude 3 Opus`, `Claude 3 Sonnet`, and `Claude 3 Haiku` models, that are accessible through the Vertex AI API endpoint. For more details about the models, refer to [Anthropic Vertex AI documentation](https://docs.anthropic.com/en/api/claude-on-vertex-ai).
|
||||
|
||||
### Parameters
|
||||
|
||||
To use the `AnthropicVertexChatGenerator`, ensure you have a GCP project with Vertex AI enabled. You need to specify your GCP `project_id` and `region`.
|
||||
|
||||
You can provide these keys in the following ways:
|
||||
|
||||
- The `REGION` and `PROJECT_ID` environment variables (recommended)
|
||||
- The `region` and `project_id` init parameters
|
||||
|
||||
Before making requests, you may need to authenticate with GCP using `gcloud auth login`.
|
||||
|
||||
Set your preferred supported Anthropic model with the `model` parameter when initializing the component. Additionally, ensure that the desired Anthropic model is activated in the Vertex AI Model Garden.
|
||||
|
||||
`AnthropicVertexChatGenerator` requires a prompt to generate text, but you can pass any text generation parameters available in the Anthropic [Messaging API](https://docs.anthropic.com/en/api/messages) method directly to this component using the `generation_kwargs` parameter, both at initialization and when running the component. For more details on the parameters supported by the Anthropic API, see the [Anthropic documentation](https://docs.anthropic.com/).
|
||||
|
||||
Finally, the component needs a list of `ChatMessage` objects to operate. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata.
|
||||
|
||||
Only text input modality is supported at this time.
|
||||
|
||||
### Streaming
|
||||
|
||||
You can stream output as it’s generated. Pass a callback to `streaming_callback`. Use the built-in `print_streaming_chunk` to print text tokens and tool events (tool calls and tool results).
|
||||
|
||||
```python
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
|
||||
## Configure any `Generator` or `ChatGenerator` with a streaming callback
|
||||
component = SomeGeneratorOrChatGenerator(streaming_callback=print_streaming_chunk)
|
||||
|
||||
## If this is a `ChatGenerator`, pass a list of messages:
|
||||
## from haystack.dataclasses import ChatMessage
|
||||
## component.run([ChatMessage.from_user("Your question here")])
|
||||
|
||||
## If this is a (non-chat) `Generator`, pass a prompt:
|
||||
## component.run({"prompt": "Your prompt here"})
|
||||
```
|
||||
|
||||
:::info
|
||||
Streaming works only with a single response. If a provider supports multiple candidates, set `n=1`.
|
||||
:::
|
||||
|
||||
See our [Streaming Support](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) docs to learn more how `StreamingChunk` works and how to write a custom callback.
|
||||
|
||||
Give preference to `print_streaming_chunk` by default. Write a custom callback only if you need a specific transport (for example, SSE/WebSocket) or custom UI formatting.
|
||||
|
||||
### Prompt Caching
|
||||
|
||||
Prompt caching is a feature for Anthropic LLMs that stores large text inputs for reuse. It allows you to send a large text block once and then refer to it in later requests without resending the entire text.
|
||||
|
||||
This feature is particularly useful for coding assistants that need full codebase context and for processing large documents. It can help reduce costs and improve response times.
|
||||
|
||||
Here's an example of an instance of `AnthropicVertexChatGenerator` being initialized with prompt caching and tagging a message to be cached:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.anthropic import (
|
||||
AnthropicVertexChatGenerator,
|
||||
)
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
generation_kwargs = {"extra_headers": {"anthropic-beta": "prompt-caching-2024-07-31"}}
|
||||
|
||||
claude_llm = AnthropicVertexChatGenerator(
|
||||
region="your_region",
|
||||
project_id="test_id",
|
||||
generation_kwargs=generation_kwargs,
|
||||
)
|
||||
|
||||
system_message = ChatMessage.from_system(
|
||||
"Replace with some long text documents, code or instructions",
|
||||
)
|
||||
system_message.meta["cache_control"] = {"type": "ephemeral"}
|
||||
|
||||
messages = [
|
||||
system_message,
|
||||
ChatMessage.from_user("A query about the long text for example"),
|
||||
]
|
||||
result = claude_llm.run(messages)
|
||||
|
||||
## and now invoke again with
|
||||
|
||||
messages = [
|
||||
system_message,
|
||||
ChatMessage.from_user("Another query about the long text etc"),
|
||||
]
|
||||
result = claude_llm.run(messages)
|
||||
|
||||
## and so on, either invoking component directly or in the pipeline
|
||||
```
|
||||
|
||||
For more details, refer to Anthropic's [documentation](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching) and integration [examples](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/anthropic/example).
|
||||
|
||||
## Usage
|
||||
|
||||
Install the`anthropic-haystack` package to use the `AnthropicVertexChatGenerator`:
|
||||
|
||||
```shell
|
||||
pip install anthropic-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.anthropic import (
|
||||
AnthropicVertexChatGenerator,
|
||||
)
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
messages = [ChatMessage.from_user("What's Natural Language Processing?")]
|
||||
client = AnthropicVertexChatGenerator(
|
||||
model="claude-3-sonnet@20240229",
|
||||
project_id="your-project-id",
|
||||
region="us-central1",
|
||||
)
|
||||
|
||||
response = client.run(messages)
|
||||
print(response)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
You can also use `AnthropicVertexChatGenerator`with the Anthropic chat models in your pipeline.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.anthropic import (
|
||||
AnthropicVertexChatGenerator,
|
||||
)
|
||||
from haystack.utils import Secret
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", ChatPromptBuilder())
|
||||
pipe.add_component(
|
||||
"llm",
|
||||
AnthropicVertexChatGenerator(project_id="test_id", region="us-central1"),
|
||||
)
|
||||
pipe.connect("prompt_builder", "llm")
|
||||
|
||||
country = "Germany"
|
||||
system_message = ChatMessage.from_system(
|
||||
"You are an assistant giving out valuable information to language learners.",
|
||||
)
|
||||
messages = [
|
||||
system_message,
|
||||
ChatMessage.from_user("What's the official language of {{ country }}?"),
|
||||
]
|
||||
|
||||
res = pipe.run(
|
||||
data={
|
||||
"prompt_builder": {
|
||||
"template_variables": {"country": country},
|
||||
"template": messages,
|
||||
},
|
||||
},
|
||||
)
|
||||
print(res)
|
||||
```
|
||||
+211
@@ -0,0 +1,211 @@
|
||||
---
|
||||
title: "AzureOpenAIChatGenerator"
|
||||
id: azureopenaichatgenerator
|
||||
slug: "/azureopenaichatgenerator"
|
||||
description: "This component enables chat completion using OpenAI’s large language models (LLMs) through Azure services."
|
||||
---
|
||||
|
||||
# AzureOpenAIChatGenerator
|
||||
|
||||
This component enables chat completion using OpenAI’s large language models (LLMs) through Azure services.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: The Azure OpenAI API key. Can be set with `AZURE_OPENAI_API_KEY` env var. <br /> <br />`azure_ad_token`: Microsoft Entra ID token. Can be set with `AZURE_OPENAI_AD_TOKEN` env var. |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects representing the chat |
|
||||
| **Output variables** | `replies`: A list of alternative replies of the LLM to the input chat |
|
||||
| **API reference** | [Generators](/reference/generators-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/generators/chat/azure.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`AzureOpenAIChatGenerator` supports OpenAI models deployed through Azure services. To see the list of supported models, head over to Azure [documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models?source=recommendations). The default model used with the component is `gpt-4o-mini`.
|
||||
|
||||
To work with Azure components, you will need an Azure OpenAI API key, as well as an Azure OpenAI Endpoint. You can learn more about them in Azure [documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference).
|
||||
|
||||
The component uses `AZURE_OPENAI_API_KEY` and `AZURE_OPENAI_AD_TOKEN` environment variables by default. Otherwise, you can pass `api_key` and `azure_ad_token` at initialization:
|
||||
|
||||
```python
|
||||
client = AzureOpenAIChatGenerator(
|
||||
azure_endpoint="<Your Azure endpoint e.g. `https://your-company.azure.openai.com/>",
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
azure_deployment="<a model name>",
|
||||
)
|
||||
```
|
||||
|
||||
:::info
|
||||
We recommend using environment variables instead of initialization parameters.
|
||||
:::
|
||||
|
||||
Then, the component needs a list of `ChatMessage` objects to operate. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata. See the [usage](https://www.notion.so/AzureOpenAIChatGenerator-c20636ac8b914ab798439a5f7a273ff0?pvs=21) section for an example.
|
||||
|
||||
You can pass any chat completion parameters that are valid for the `openai.ChatCompletion.create` method directly to `AzureOpenAIChatGenerator` using the `generation_kwargs` parameter, both at initialization and to `run()` method. For more details on the supported parameters, refer to the [Azure documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference).
|
||||
|
||||
You can also specify a model for this component through the `azure_deployment` init parameter.
|
||||
|
||||
### Structured Output
|
||||
|
||||
`AzureOpenAIChatGenerator` supports structured output generation, allowing you to receive responses in a predictable format. You can use Pydantic models or JSON schemas to define the structure of the output through the `response_format` parameter in `generation_kwargs`.
|
||||
|
||||
This is useful when you need to extract structured data from text or generate responses that match a specific format.
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel
|
||||
from haystack.components.generators.chat import AzureOpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
class NobelPrizeInfo(BaseModel):
|
||||
recipient_name: str
|
||||
award_year: int
|
||||
category: str
|
||||
achievement_description: str
|
||||
nationality: str
|
||||
|
||||
client = AzureOpenAIChatGenerator(
|
||||
azure_endpoint="<Your Azure endpoint>",
|
||||
azure_deployment="gpt-4o",
|
||||
generation_kwargs={"response_format": NobelPrizeInfo}
|
||||
)
|
||||
|
||||
response = client.run(messages=[
|
||||
ChatMessage.from_user(
|
||||
"In 2021, American scientist David Julius received the Nobel Prize in"
|
||||
" Physiology or Medicine for his groundbreaking discoveries on how the human body"
|
||||
" senses temperature and touch."
|
||||
)
|
||||
])
|
||||
print(response["replies"][0].text)
|
||||
|
||||
>> {"recipient_name":"David Julius","award_year":2021,"category":"Physiology or Medicine",
|
||||
>> "achievement_description":"David Julius was awarded for his transformative findings
|
||||
>> regarding the molecular mechanisms underlying the human body's sense of temperature
|
||||
>> and touch. Through innovative experiments, he identified specific receptors responsible
|
||||
>> for detecting heat and mechanical stimuli, ranging from gentle touch to pain-inducing
|
||||
>> pressure.","nationality":"American"}
|
||||
```
|
||||
|
||||
:::info[Model Compatibility and Limitations]
|
||||
|
||||
- Pydantic models and JSON schemas are supported for latest models starting from GPT-4o.
|
||||
- Older models only support basic JSON mode through `{"type": "json_object"}`. For details, see [OpenAI JSON mode documentation](https://platform.openai.com/docs/guides/structured-outputs#json-mode).
|
||||
- Streaming limitation: When using streaming with structured outputs, you must provide a JSON schema instead of a Pydantic model for `response_format`.
|
||||
- For complete information, check the [Azure OpenAI Structured Outputs documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/structured-outputs).
|
||||
:::
|
||||
|
||||
### Streaming
|
||||
|
||||
You can stream output as it’s generated. Pass a callback to `streaming_callback`. Use the built-in `print_streaming_chunk` to print text tokens and tool events (tool calls and tool results).
|
||||
|
||||
```python
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
|
||||
## Configure any `Generator` or `ChatGenerator` with a streaming callback
|
||||
component = SomeGeneratorOrChatGenerator(streaming_callback=print_streaming_chunk)
|
||||
|
||||
## If this is a `ChatGenerator`, pass a list of messages:
|
||||
## from haystack.dataclasses import ChatMessage
|
||||
## component.run([ChatMessage.from_user("Your question here")])
|
||||
|
||||
## If this is a (non-chat) `Generator`, pass a prompt:
|
||||
## component.run({"prompt": "Your prompt here"})
|
||||
```
|
||||
|
||||
:::info
|
||||
Streaming works only with a single response. If a provider supports multiple candidates, set `n=1`.
|
||||
:::
|
||||
|
||||
See our [Streaming Support](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) docs to learn more how `StreamingChunk` works and how to write a custom callback.
|
||||
|
||||
Give preference to `print_streaming_chunk` by default. Write a custom callback only if you need a specific transport (for example, SSE/WebSocket) or custom UI formatting.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.components.generators.chat import AzureOpenAIChatGenerator
|
||||
|
||||
client = AzureOpenAIChatGenerator()
|
||||
response = client.run(
|
||||
[ChatMessage.from_user("What's Natural Language Processing? Be brief.")],
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
||||
With streaming:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.components.generators.chat import AzureOpenAIChatGenerator
|
||||
|
||||
client = AzureOpenAIChatGenerator(
|
||||
streaming_callback=lambda chunk: print(chunk.content, end="", flush=True),
|
||||
)
|
||||
response = client.run(
|
||||
[ChatMessage.from_user("What's Natural Language Processing? Be brief.")],
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
|
||||
With multimodal inputs:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage, ImageContent
|
||||
from haystack.components.generators.chat import AzureOpenAIChatGenerator
|
||||
|
||||
llm = AzureOpenAIChatGenerator(
|
||||
azure_endpoint="<Your Azure endpoint>",
|
||||
azure_deployment="gpt-4o-mini",
|
||||
)
|
||||
|
||||
image = ImageContent.from_file_path("apple.jpg", detail="low")
|
||||
user_message = ChatMessage.from_user(
|
||||
content_parts=["What does the image show? Max 5 words.", image],
|
||||
)
|
||||
|
||||
response = llm.run([user_message])["replies"][0].text
|
||||
print(response)
|
||||
|
||||
# Fresh red apple on straw.
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import AzureOpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack import Pipeline
|
||||
|
||||
## no parameter init, we don't use any runtime template variables
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
llm = AzureOpenAIChatGenerator()
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
location = "Berlin"
|
||||
messages = [
|
||||
ChatMessage.from_system(
|
||||
"Always respond in German even if some input data is in other languages.",
|
||||
),
|
||||
ChatMessage.from_user("Tell me about {{location}}"),
|
||||
]
|
||||
pipe.run(
|
||||
data={
|
||||
"prompt_builder": {
|
||||
"template_variables": {"location": location},
|
||||
"template": messages,
|
||||
},
|
||||
},
|
||||
)
|
||||
```
|
||||
+141
@@ -0,0 +1,141 @@
|
||||
---
|
||||
title: "AzureOpenAIGenerator"
|
||||
id: azureopenaigenerator
|
||||
slug: "/azureopenaigenerator"
|
||||
description: "This component enables text generation using OpenAI's large language models (LLMs) through Azure services."
|
||||
---
|
||||
|
||||
# AzureOpenAIGenerator
|
||||
|
||||
This component enables text generation using OpenAI's large language models (LLMs) through Azure services.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`PromptBuilder`](../builders/promptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: The Azure OpenAI API key. Can be set with `AZURE_OPENAI_API_KEY` env var. <br /> <br />`azure_ad_token`: Microsoft Entra ID token. Can be set with `AZURE_OPENAI_AD_TOKEN` env var. |
|
||||
| **Mandatory run variables** | `prompt`: A string containing the prompt for the LLM |
|
||||
| **Output variables** | `replies`: A list of strings with all the replies generated by the LLM <br /> <br />`meta`: A list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and so on |
|
||||
| **API reference** | [Generators](/reference/generators-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/generators/azure.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`AzureOpenAIGenerator` supports OpenAI models deployed through Azure services. To see the list of supported models, head over to Azure [documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models?source=recommendations). The default model used with the component is `gpt-4o-mini`.
|
||||
|
||||
To work with Azure components, you will need an Azure OpenAI API key, as well as an Azure OpenAI Endpoint. You can learn more about them in Azure [documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference).
|
||||
|
||||
The component uses `AZURE_OPENAI_API_KEY` and `AZURE_OPENAI_AD_TOKEN` environment variables by default. Otherwise, you can pass `api_key` and `azure_ad_token` at initialization:
|
||||
|
||||
```python
|
||||
client = AzureOpenAIGenerator(
|
||||
azure_endpoint="<Your Azure endpoint e.g. `https://your-company.azure.openai.com/>",
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
azure_deployment="<a model name>",
|
||||
)
|
||||
```
|
||||
|
||||
:::info
|
||||
We recommend using environment variables instead of initialization parameters.
|
||||
:::
|
||||
|
||||
Then, the component needs a prompt to operate, but you can pass any text generation parameters valid for the `openai.ChatCompletion.create` method directly to this component using the `generation_kwargs` parameter, both at initialization and to `run()` method. For more details on the supported parameters, refer to the [Azure documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference).
|
||||
|
||||
You can also specify a model for this component through the `azure_deployment` init parameter.
|
||||
|
||||
### Streaming
|
||||
|
||||
`AzureOpenAIGenerator` supports streaming the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter. Note that streaming the tokens is only compatible with generating a single response, so `n` must be set to 1 for streaming to work.
|
||||
|
||||
:::info
|
||||
This component is designed for text generation, not for chat. If you want to use LLMs for chat, use [`AzureOpenAIChatGenerator`](azureopenaichatgenerator.mdx) instead.
|
||||
:::
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage:
|
||||
|
||||
```python
|
||||
from haystack.components.generators import AzureOpenAIGenerator
|
||||
client = AzureOpenAIGenerator()
|
||||
response = client.run("What's Natural Language Processing? Be brief.")
|
||||
print(response)
|
||||
|
||||
>> {'replies': ['Natural Language Processing (NLP) is a branch of artificial intelligence that focuses on
|
||||
>> the interaction between computers and human language. It involves enabling computers to understand, interpret,
|
||||
>> and respond to natural human language in a way that is both meaningful and useful.'], 'meta': [{'model':
|
||||
>> 'gpt-4o-mini', 'index': 0, 'finish_reason': 'stop', 'usage': {'prompt_tokens': 16,
|
||||
>> 'completion_tokens': 49, 'total_tokens': 65}}]}
|
||||
|
||||
```
|
||||
|
||||
With streaming:
|
||||
|
||||
```python
|
||||
from haystack.components.generators import AzureOpenAIGenerator
|
||||
|
||||
client = AzureOpenAIGenerator(streaming_callback=lambda chunk: print(chunk.content, end="", flush=True))
|
||||
response = client.run("What's Natural Language Processing? Be brief.")
|
||||
print(response)
|
||||
|
||||
>>> Natural Language Processing (NLP) is a branch of artificial
|
||||
intelligence that focuses on the interaction between computers and human
|
||||
language. It involves enabling computers to understand, interpret,and respond
|
||||
to natural human language in a way that is both meaningful and useful.
|
||||
>>> {'replies': ['Natural Language Processing (NLP) is a branch of artificial
|
||||
intelligence that focuses on the interaction between computers and human
|
||||
language. It involves enabling computers to understand, interpret,and respond
|
||||
to natural human language in a way that is both meaningful and useful.'],
|
||||
'meta': [{'model': 'gpt-4o-mini', 'index': 0, 'finish_reason':
|
||||
'stop', 'usage': {'prompt_tokens': 16, 'completion_tokens': 49,
|
||||
'total_tokens': 65}}]}
|
||||
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.components.builders.prompt_builder import PromptBuilder
|
||||
from haystack.components.generators import AzureOpenAIGenerator
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack import Document
|
||||
|
||||
docstore = InMemoryDocumentStore()
|
||||
docstore.write_documents(
|
||||
[
|
||||
Document(content="Rome is the capital of Italy"),
|
||||
Document(content="Paris is the capital of France"),
|
||||
],
|
||||
)
|
||||
|
||||
query = "What is the capital of France?"
|
||||
|
||||
template = """
|
||||
Given the following information, answer the question.
|
||||
|
||||
Context:
|
||||
{% for document in documents %}
|
||||
{{ document.content }}
|
||||
{% endfor %}
|
||||
|
||||
Question: {{ query }}?
|
||||
"""
|
||||
pipe = Pipeline()
|
||||
|
||||
pipe.add_component("retriever", InMemoryBM25Retriever(document_store=docstore))
|
||||
pipe.add_component("prompt_builder", PromptBuilder(template=template))
|
||||
pipe.add_component("llm", AzureOpenAIGenerator())
|
||||
pipe.connect("retriever", "prompt_builder.documents")
|
||||
pipe.connect("prompt_builder", "llm")
|
||||
|
||||
res = pipe.run({"prompt_builder": {"query": query}, "retriever": {"query": query}})
|
||||
|
||||
print(res)
|
||||
```
|
||||
+339
@@ -0,0 +1,339 @@
|
||||
---
|
||||
title: "AzureOpenAIResponsesChatGenerator"
|
||||
id: azureopenairesponseschatgenerator
|
||||
slug: "/azureopenairesponseschatgenerator"
|
||||
description: "This component enables chat completion using OpenAI's Responses API through Azure services with support for reasoning models."
|
||||
---
|
||||
|
||||
# AzureOpenAIResponsesChatGenerator
|
||||
|
||||
This component enables chat completion using OpenAI's Responses API through Azure services with support for reasoning models.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`ChatPromptBuilder`](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: The Azure OpenAI API key. Can be set with `AZURE_OPENAI_API_KEY` env var or a callable for Azure AD token. <br /> <br />`azure_endpoint`: The endpoint of the deployed model. Can be set with `AZURE_OPENAI_ENDPOINT` env var. |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects representing the chat |
|
||||
| **Output variables** | `replies`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects containing the generated responses |
|
||||
| **API reference** | [Generators](/reference/generators-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/generators/chat/azure_responses.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`AzureOpenAIResponsesChatGenerator` uses OpenAI's Responses API through Azure OpenAI services. It supports gpt-5 and o-series models (reasoning models like o1, o3-mini) deployed on Azure. The default model is `gpt-5-mini`.
|
||||
|
||||
The Responses API is designed for reasoning-capable models and supports features like reasoning summaries, multi-turn conversations with previous response IDs, and structured outputs. This component provides access to these capabilities through Azure's infrastructure.
|
||||
|
||||
The component requires a list of `ChatMessage` objects to operate. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`), and optional metadata. See the [usage](#usage) section for examples.
|
||||
|
||||
You can pass any parameters valid for the OpenAI Responses API directly to `AzureOpenAIResponsesChatGenerator` using the `generation_kwargs` parameter, both at initialization and to the `run()` method. For more details on the supported parameters, refer to the [Azure OpenAI documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference).
|
||||
|
||||
You can specify a model for this component through the `azure_deployment` init parameter, which should match your Azure deployment name.
|
||||
|
||||
### Authentication
|
||||
|
||||
To work with Azure components, you need an Azure OpenAI API key and an Azure OpenAI endpoint. You can learn more about them in the [Azure documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference).
|
||||
|
||||
The component uses `AZURE_OPENAI_API_KEY` and `AZURE_OPENAI_ENDPOINT` environment variables by default. Otherwise, you can pass these at initialization using a [`Secret`](../../concepts/secret-management.mdx):
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import AzureOpenAIResponsesChatGenerator
|
||||
from haystack.utils import Secret
|
||||
|
||||
client = AzureOpenAIResponsesChatGenerator(
|
||||
azure_endpoint="https://your-resource.azure.openai.com/",
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
azure_deployment="gpt-5-mini",
|
||||
)
|
||||
```
|
||||
|
||||
For Azure Active Directory authentication, you can pass a callable that returns a token:
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import AzureOpenAIResponsesChatGenerator
|
||||
|
||||
|
||||
def get_azure_ad_token():
|
||||
# Your Azure AD token retrieval logic
|
||||
return "your-azure-ad-token"
|
||||
|
||||
|
||||
client = AzureOpenAIResponsesChatGenerator(
|
||||
azure_endpoint="https://your-resource.azure.openai.com/",
|
||||
api_key=get_azure_ad_token,
|
||||
azure_deployment="gpt-5-mini",
|
||||
)
|
||||
```
|
||||
|
||||
### Reasoning Support
|
||||
|
||||
One of the key features of the Responses API is support for reasoning models. You can configure reasoning behavior using the `reasoning` parameter in `generation_kwargs`:
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import AzureOpenAIResponsesChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
client = AzureOpenAIResponsesChatGenerator(
|
||||
azure_endpoint="https://your-resource.azure.openai.com/",
|
||||
generation_kwargs={"reasoning": {"effort": "medium", "summary": "auto"}},
|
||||
)
|
||||
|
||||
messages = [
|
||||
ChatMessage.from_user(
|
||||
"What's the most efficient sorting algorithm for nearly sorted data?",
|
||||
),
|
||||
]
|
||||
response = client.run(messages)
|
||||
print(response)
|
||||
```
|
||||
|
||||
The `reasoning` parameter accepts:
|
||||
- `effort`: Level of reasoning effort - `"low"`, `"medium"`, or `"high"`
|
||||
- `summary`: How to generate reasoning summaries - `"auto"` or `"generate_summary": True/False`
|
||||
|
||||
:::note
|
||||
OpenAI does not return the actual reasoning tokens, but you can view the summary if enabled. For more details, see the [OpenAI Reasoning documentation](https://platform.openai.com/docs/guides/reasoning).
|
||||
:::
|
||||
|
||||
### Multi-turn Conversations
|
||||
|
||||
The Responses API supports multi-turn conversations using `previous_response_id`. You can pass the response ID from a previous turn to maintain conversation context:
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import AzureOpenAIResponsesChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
client = AzureOpenAIResponsesChatGenerator(
|
||||
azure_endpoint="https://your-resource.azure.openai.com/",
|
||||
)
|
||||
|
||||
# First turn
|
||||
messages = [ChatMessage.from_user("What's quantum computing?")]
|
||||
response = client.run(messages)
|
||||
response_id = response["replies"][0].meta.get("id")
|
||||
|
||||
# Second turn - reference previous response
|
||||
messages = [ChatMessage.from_user("Can you explain that in simpler terms?")]
|
||||
response = client.run(messages, generation_kwargs={"previous_response_id": response_id})
|
||||
```
|
||||
|
||||
### Structured Output
|
||||
|
||||
`AzureOpenAIResponsesChatGenerator` supports structured output generation through the `text_format` and `text` parameters in `generation_kwargs`:
|
||||
|
||||
- **`text_format`**: Pass a Pydantic model to define the structure
|
||||
- **`text`**: Pass a JSON schema directly
|
||||
|
||||
**Using a Pydantic model**:
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel
|
||||
from haystack.components.generators.chat import AzureOpenAIResponsesChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
|
||||
class ProductInfo(BaseModel):
|
||||
name: str
|
||||
price: float
|
||||
category: str
|
||||
in_stock: bool
|
||||
|
||||
|
||||
client = AzureOpenAIResponsesChatGenerator(
|
||||
azure_endpoint="https://your-resource.azure.openai.com/",
|
||||
azure_deployment="gpt-4o",
|
||||
generation_kwargs={"text_format": ProductInfo},
|
||||
)
|
||||
|
||||
response = client.run(
|
||||
messages=[
|
||||
ChatMessage.from_user(
|
||||
"Extract product info: 'Wireless Mouse, $29.99, Electronics, Available in stock'",
|
||||
),
|
||||
],
|
||||
)
|
||||
print(response["replies"][0].text)
|
||||
```
|
||||
|
||||
**Using a JSON schema**:
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import AzureOpenAIResponsesChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
json_schema = {
|
||||
"format": {
|
||||
"type": "json_schema",
|
||||
"name": "ProductInfo",
|
||||
"strict": True,
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": {"type": "string"},
|
||||
"price": {"type": "number"},
|
||||
"category": {"type": "string"},
|
||||
"in_stock": {"type": "boolean"},
|
||||
},
|
||||
"required": ["name", "price", "category", "in_stock"],
|
||||
"additionalProperties": False,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
client = AzureOpenAIResponsesChatGenerator(
|
||||
azure_endpoint="https://your-resource.azure.openai.com/",
|
||||
azure_deployment="gpt-4o",
|
||||
generation_kwargs={"text": json_schema},
|
||||
)
|
||||
|
||||
response = client.run(
|
||||
messages=[
|
||||
ChatMessage.from_user(
|
||||
"Extract product info: 'Wireless Mouse, $29.99, Electronics, Available in stock'",
|
||||
),
|
||||
],
|
||||
)
|
||||
print(response["replies"][0].text)
|
||||
```
|
||||
|
||||
:::info[Model Compatibility and Limitations]
|
||||
- Both Pydantic models and JSON schemas are supported for latest models starting from GPT-4o.
|
||||
- If both `text_format` and `text` are provided, `text_format` takes precedence and the JSON schema passed to `text` is ignored.
|
||||
- Streaming is not supported when using structured outputs.
|
||||
- Older models only support basic JSON mode through `{"type": "json_object"}`. For details, see [OpenAI JSON mode documentation](https://platform.openai.com/docs/guides/structured-outputs#json-mode).
|
||||
- For complete information, check the [Azure OpenAI Structured Outputs documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/structured-outputs).
|
||||
:::
|
||||
|
||||
### Tool Support
|
||||
|
||||
`AzureOpenAIResponsesChatGenerator` supports function calling through the `tools` parameter. It accepts flexible tool configurations:
|
||||
|
||||
- **Haystack Tool objects and Toolsets**: Pass Haystack `Tool` objects or `Toolset` objects, including mixed lists of both
|
||||
- **OpenAI/MCP tool definitions**: Pass pre-defined OpenAI or MCP tool definitions as dictionaries
|
||||
|
||||
Note that you cannot mix Haystack tools and OpenAI/MCP tools in the same call - choose one format or the other.
|
||||
|
||||
```python
|
||||
from haystack.tools import Tool
|
||||
from haystack.components.generators.chat import AzureOpenAIResponsesChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
|
||||
def get_weather(city: str) -> str:
|
||||
"""Get weather information for a city."""
|
||||
return f"Weather in {city}: Sunny, 22°C"
|
||||
|
||||
|
||||
weather_tool = Tool(
|
||||
name="get_weather",
|
||||
description="Get current weather for a city",
|
||||
function=get_weather,
|
||||
parameters={"type": "object", "properties": {"city": {"type": "string"}}},
|
||||
)
|
||||
|
||||
generator = AzureOpenAIResponsesChatGenerator(
|
||||
azure_endpoint="https://your-resource.azure.openai.com/",
|
||||
tools=[weather_tool],
|
||||
)
|
||||
messages = [ChatMessage.from_user("What's the weather in Paris?")]
|
||||
response = generator.run(messages)
|
||||
```
|
||||
|
||||
You can control strict schema adherence with the `tools_strict` parameter. When set to `True` (default is `False`), the model will follow the tool schema exactly. Note that the Responses API has its own strictness enforcement mechanisms independent of this parameter.
|
||||
|
||||
For more details on working with tools, see the [Tool](../../tools/tool.mdx) and [Toolset](../../tools/toolset.mdx) documentation.
|
||||
|
||||
### Streaming
|
||||
|
||||
You can stream output as it's generated. Pass a callback to `streaming_callback`. Use the built-in `print_streaming_chunk` to print text tokens and tool events (tool calls and tool results).
|
||||
|
||||
```python
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
|
||||
## Configure any `Generator` or `ChatGenerator` with a streaming callback
|
||||
component = SomeGeneratorOrChatGenerator(streaming_callback=print_streaming_chunk)
|
||||
|
||||
## If this is a `ChatGenerator`, pass a list of messages:
|
||||
## from haystack.dataclasses import ChatMessage
|
||||
## component.run([ChatMessage.from_user("Your question here")])
|
||||
|
||||
## If this is a (non-chat) `Generator`, pass a prompt:
|
||||
## component.run({"prompt": "Your prompt here"})
|
||||
```
|
||||
|
||||
:::info
|
||||
Streaming works only with a single response. If a provider supports multiple candidates, set `n=1`.
|
||||
:::
|
||||
|
||||
See our [Streaming Support](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) docs to learn more how `StreamingChunk` works and how to write a custom callback.
|
||||
|
||||
Give preference to `print_streaming_chunk` by default. Write a custom callback only if you need a specific transport (for example, SSE/WebSocket) or custom UI formatting.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Here is an example of using `AzureOpenAIResponsesChatGenerator` independently with reasoning and streaming:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.components.generators.chat import AzureOpenAIResponsesChatGenerator
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
|
||||
client = AzureOpenAIResponsesChatGenerator(
|
||||
azure_endpoint="https://your-resource.azure.openai.com/",
|
||||
streaming_callback=print_streaming_chunk,
|
||||
generation_kwargs={"reasoning": {"effort": "high", "summary": "auto"}},
|
||||
)
|
||||
response = client.run(
|
||||
[
|
||||
ChatMessage.from_user(
|
||||
"Solve this logic puzzle: If all roses are flowers and some flowers fade quickly, can we conclude that some roses fade quickly?",
|
||||
),
|
||||
],
|
||||
)
|
||||
print(response["replies"][0].reasoning) # Access reasoning summary if available
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
This example shows a pipeline that uses `ChatPromptBuilder` to create dynamic prompts and `AzureOpenAIResponsesChatGenerator` with reasoning enabled to generate explanations of complex topics:
|
||||
|
||||
```python
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import AzureOpenAIResponsesChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack import Pipeline
|
||||
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
llm = AzureOpenAIResponsesChatGenerator(
|
||||
azure_endpoint="https://your-resource.azure.openai.com/",
|
||||
generation_kwargs={"reasoning": {"effort": "low", "summary": "auto"}},
|
||||
)
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
topic = "quantum computing"
|
||||
messages = [
|
||||
ChatMessage.from_system(
|
||||
"You are a helpful assistant that explains complex topics clearly.",
|
||||
),
|
||||
ChatMessage.from_user("Explain {{topic}} in simple terms"),
|
||||
]
|
||||
result = pipe.run(
|
||||
data={
|
||||
"prompt_builder": {
|
||||
"template_variables": {"topic": topic},
|
||||
"template": messages,
|
||||
},
|
||||
},
|
||||
)
|
||||
print(result)
|
||||
```
|
||||
+144
@@ -0,0 +1,144 @@
|
||||
---
|
||||
title: "CohereChatGenerator"
|
||||
id: coherechatgenerator
|
||||
slug: "/coherechatgenerator"
|
||||
description: "CohereChatGenerator enables chat completions using Cohere's large language models (LLMs)."
|
||||
---
|
||||
|
||||
# CohereChatGenerator
|
||||
|
||||
CohereChatGenerator enables chat completions using Cohere's large language models (LLMs).
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: The Cohere API key. Can be set with `COHERE_API_KEY` or `CO_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `messages` A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects |
|
||||
| **Output variables** | `replies`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects <br /> <br />`meta`: A list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and so on |
|
||||
| **API reference** | [Cohere](/reference/integrations-cohere) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/cohere |
|
||||
|
||||
</div>
|
||||
|
||||
This integration supports Cohere `chat` models such as `command`,`command-r` and `comman-r-plus`. Check out the most recent full list in [Cohere documentation](https://docs.cohere.com/reference/chat).
|
||||
|
||||
## Overview
|
||||
|
||||
`CohereChatGenerator` needs a Cohere API key to work. You can set this key in:
|
||||
|
||||
- The `api_key` init parameter using [Secret API](../../concepts/secret-management.mdx)
|
||||
- The `COHERE_API_KEY` environment variable (recommended)
|
||||
|
||||
Then, the component needs a prompt to operate, but you can pass any text generation parameters valid for the `Co.chat` method directly to this component using the `generation_kwargs` parameter, both at initialization and to `run()` method. For more details on the parameters supported by the Cohere API, refer to the [Cohere documentation](https://docs.cohere.com/reference/chat).
|
||||
|
||||
Finally, the component needs a list of `ChatMessage` objects to operate. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata.
|
||||
|
||||
### Tool Support
|
||||
|
||||
`CohereChatGenerator` supports function calling through the `tools` parameter, which accepts flexible tool configurations:
|
||||
|
||||
- **A list of Tool objects**: Pass individual tools as a list
|
||||
- **A single Toolset**: Pass an entire Toolset directly
|
||||
- **Mixed Tools and Toolsets**: Combine multiple Toolsets with standalone tools in a single list
|
||||
|
||||
This allows you to organize related tools into logical groups while also including standalone tools as needed.
|
||||
|
||||
```python
|
||||
from haystack.tools import Tool, Toolset
|
||||
from haystack_integrations.components.generators.cohere import CohereChatGenerator
|
||||
|
||||
# Create individual tools
|
||||
weather_tool = Tool(name="weather", description="Get weather info", ...)
|
||||
news_tool = Tool(name="news", description="Get latest news", ...)
|
||||
|
||||
# Group related tools into a toolset
|
||||
math_toolset = Toolset([add_tool, subtract_tool, multiply_tool])
|
||||
|
||||
# Pass mixed tools and toolsets to the generator
|
||||
generator = CohereChatGenerator(
|
||||
tools=[math_toolset, weather_tool, news_tool] # Mix of Toolset and Tool objects
|
||||
)
|
||||
```
|
||||
|
||||
For more details on working with tools, see the [Tool](../../tools/tool.mdx) and [Toolset](../../tools/toolset.mdx) documentation.
|
||||
|
||||
### Streaming
|
||||
|
||||
This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
You need to install `cohere-haystack` package to use the `CohereChatGenerator`:
|
||||
|
||||
```shell
|
||||
pip install cohere-haystack
|
||||
```
|
||||
|
||||
#### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.cohere import CohereChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
generator = CohereChatGenerator()
|
||||
message = ChatMessage.from_user("What's Natural Language Processing? Be brief.")
|
||||
print(generator.run([message]))
|
||||
```
|
||||
|
||||
With multimodal inputs:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage, ImageContent
|
||||
from haystack_integrations.components.generators.cohere import CohereChatGenerator
|
||||
|
||||
# Use a multimodal model like Command A Vision
|
||||
llm = CohereChatGenerator(model="command-a-vision-07-2025")
|
||||
|
||||
image = ImageContent.from_file_path("apple.jpg")
|
||||
user_message = ChatMessage.from_user(
|
||||
content_parts=["What does the image show? Max 5 words.", image],
|
||||
)
|
||||
|
||||
response = llm.run([user_message])["replies"][0].text
|
||||
print(response)
|
||||
|
||||
# Red apple on straw.
|
||||
```
|
||||
|
||||
#### In a Pipeline
|
||||
|
||||
You can also use `CohereChatGenerator` to use cohere chat models in your pipeline.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.cohere import CohereChatGenerator
|
||||
from haystack.utils import Secret
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", ChatPromptBuilder())
|
||||
pipe.add_component("llm", CohereChatGenerator())
|
||||
pipe.connect("prompt_builder", "llm")
|
||||
|
||||
country = "Germany"
|
||||
system_message = ChatMessage.from_system(
|
||||
"You are an assistant giving out valuable information to language learners.",
|
||||
)
|
||||
messages = [
|
||||
system_message,
|
||||
ChatMessage.from_user("What's the official language of {{ country }}?"),
|
||||
]
|
||||
|
||||
res = pipe.run(
|
||||
data={
|
||||
"prompt_builder": {
|
||||
"template_variables": {"country": country},
|
||||
"template": messages,
|
||||
},
|
||||
},
|
||||
)
|
||||
print(res)
|
||||
```
|
||||
+121
@@ -0,0 +1,121 @@
|
||||
---
|
||||
title: "CohereGenerator"
|
||||
id: coheregenerator
|
||||
slug: "/coheregenerator"
|
||||
description: "`CohereGenerator` enables text generation using Cohere's large language models (LLMs)."
|
||||
---
|
||||
|
||||
# CohereGenerator
|
||||
|
||||
`CohereGenerator` enables text generation using Cohere's large language models (LLMs).
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`PromptBuilder`](../builders/promptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: The Cohere API key. Can be set with `COHERE_API_KEY` or `CO_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `prompt`: A string containing the prompt for the LLM |
|
||||
| **Output variables** | `replies`: A list of strings with all the replies generated by the LLM <br /> <br />`meta`: A list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and so on |
|
||||
| **API reference** | [Cohere](/reference/integrations-cohere) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/cohere |
|
||||
|
||||
</div>
|
||||
|
||||
This integration supports Cohere models such as `command`, `command-r` and `comman-r-plus`. Check out the most recent full list in [Cohere documentation](https://docs.cohere.com/reference/chat).
|
||||
|
||||
## Overview
|
||||
|
||||
`CohereGenerator` needs a Cohere API key to work. You can write this key in:
|
||||
|
||||
- The `api_key` init parameter using [Secret API](../../concepts/secret-management.mdx)
|
||||
- The `COHERE_API_KEY` environment variable (recommended)
|
||||
|
||||
Then, the component needs a prompt to operate, but you can pass any text generation parameters directly to this component using the `generation_kwargs` parameter at initialization. For more details on the parameters supported by the Cohere API, refer to the [Cohere documentation](https://docs.cohere.com/reference/chat).
|
||||
|
||||
### Streaming
|
||||
|
||||
This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
You need to install `cohere-haystack` package to use the `CohereGenerator`:
|
||||
|
||||
```shell
|
||||
pip install cohere-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.cohere import CohereGenerator
|
||||
|
||||
client = CohereGenerator()
|
||||
response = client.run("Briefly explain what NLP is in one sentence.")
|
||||
print(response)
|
||||
|
||||
>>> {'replies': ["Natural Language Processing (NLP) is a subfield of artificial intelligence and computational linguistics that focuses on the interaction between computers and human languages..."],
|
||||
'meta': [{'finish_reason': 'COMPLETE'}]}
|
||||
```
|
||||
|
||||
With streaming:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.cohere import CohereGenerator
|
||||
|
||||
client = CohereGenerator(streaming_callback=lambda chunk: print(chunk.content, end="", flush=True))
|
||||
response = client.run("Briefly explain what NLP is in one sentence.")
|
||||
print(response)
|
||||
|
||||
>>> Natural Language Processing (NLP) is the study of natural language and how it can be used to solve problems through computational methods, enabling machines to understand, interpret, and generate human language.
|
||||
|
||||
>>>{'replies': [' Natural Language Processing (NLP) is the study of natural language and how it can be used to solve problems through computational methods, enabling machines to understand, interpret, and generate human language.'], 'meta': [{'index': 0, 'finish_reason': 'COMPLETE'}]}
|
||||
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
In a RAG pipeline:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.components.builders.prompt_builder import PromptBuilder
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.generators.cohere import CohereGenerator
|
||||
from haystack import Document
|
||||
|
||||
docstore = InMemoryDocumentStore()
|
||||
docstore.write_documents(
|
||||
[
|
||||
Document(content="Rome is the capital of Italy"),
|
||||
Document(content="Paris is the capital of France"),
|
||||
],
|
||||
)
|
||||
|
||||
query = "What is the capital of France?"
|
||||
|
||||
template = """
|
||||
Given the following information, answer the question.
|
||||
|
||||
Context:
|
||||
{% for document in documents %}
|
||||
{{ document.content }}
|
||||
{% endfor %}
|
||||
|
||||
Question: {{ query }}?
|
||||
"""
|
||||
pipe = Pipeline()
|
||||
|
||||
pipe.add_component("retriever", InMemoryBM25Retriever(document_store=docstore))
|
||||
pipe.add_component("prompt_builder", PromptBuilder(template=template))
|
||||
pipe.add_component("llm", CohereGenerator())
|
||||
pipe.connect("retriever", "prompt_builder.documents")
|
||||
pipe.connect("prompt_builder", "llm")
|
||||
|
||||
res = pipe.run({"prompt_builder": {"query": query}, "retriever": {"query": query}})
|
||||
|
||||
print(res)
|
||||
```
|
||||
+306
@@ -0,0 +1,306 @@
|
||||
---
|
||||
title: "CometAPIChatGenerator"
|
||||
id: cometapichatgenerator
|
||||
slug: "/cometapichatgenerator"
|
||||
description: "CometAPIChatGenerator enables chat completion using AI models through the Comet API."
|
||||
---
|
||||
|
||||
# CometAPIChatGenerator
|
||||
|
||||
CometAPIChatGenerator enables chat completion using AI models through the Comet API.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: The Comet API key. Can be set with `COMET_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `messages` A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects |
|
||||
| **Output variables** | `replies`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects <br /> <br />`meta`: A list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and so on |
|
||||
| **API reference** | [Comet API](/reference/integrations-cometapi) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/cometapi |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`CometAPIChatGenerator` provides access to over 500 AI models through the Comet API, a unified API gateway for models from providers like OpenAI, Anthropic, Google, Meta, Mistral, and many more. You can use different models from different providers within a single pipeline with a consistent interface.
|
||||
|
||||
Comet API uses a single API key for all providers, which allows you to switch between or combine different models without managing multiple credentials.
|
||||
|
||||
The range of models supported by Comet API include:
|
||||
|
||||
- OpenAI models: `gpt-4o`, `gpt-4o-mini` (default), `gpt-4-turbo`, and more
|
||||
- Anthropic models: `claude-3-5-sonnet`, `claude-3-opus`, and more
|
||||
- Google models: `gemini-1.5-pro`, `gemini-1.5-flash`, and more
|
||||
- Meta models: `llama-3.3-70b`, `llama-3.1-405b`, and more
|
||||
- Mistral models: `mistral-large-latest`, `mistral-small`, and more
|
||||
|
||||
For a complete list of available models, check the [Comet API documentation](https://apidoc.cometapi.com/).
|
||||
|
||||
The component needs a list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects to operate. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata.
|
||||
|
||||
You can pass any chat completion parameters valid for the underlying model directly to `CometAPIChatGenerator` using the `generation_kwargs` parameter, both at initialization and to the `run()` method.
|
||||
|
||||
### Authentication
|
||||
|
||||
`CometAPIChatGenerator` needs a Comet API key to work. You can set this key in:
|
||||
|
||||
- The `api_key` init parameter using [Secret API](../../concepts/secret-management.mdx)
|
||||
- The `COMET_API_KEY` environment variable (recommended)
|
||||
|
||||
### Structured Output
|
||||
|
||||
`CometAPIChatGenerator` supports structured output generation for compatible models, allowing you to receive responses in a predictable format. You can use Pydantic models or JSON schemas to define the structure of the output through the `response_format` parameter in `generation_kwargs`.
|
||||
|
||||
This is useful when you need to extract structured data from text or generate responses that match a specific format.
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.cometapi import CometAPIChatGenerator
|
||||
|
||||
class CityInfo(BaseModel):
|
||||
city_name: str
|
||||
country: str
|
||||
population: int
|
||||
famous_for: str
|
||||
|
||||
client = CometAPIChatGenerator(
|
||||
model="gpt-4o-2024-08-06",
|
||||
generation_kwargs={"response_format": CityInfo}
|
||||
)
|
||||
|
||||
response = client.run(messages=[
|
||||
ChatMessage.from_user(
|
||||
"Berlin is the capital and largest city of Germany with a population of "
|
||||
"approximately 3.7 million. It's famous for its history, culture, and nightlife."
|
||||
)
|
||||
])
|
||||
print(response["replies"][0].text)
|
||||
|
||||
>> {"city_name":"Berlin","country":"Germany","population":3700000,
|
||||
>> "famous_for":"history, culture, and nightlife"}
|
||||
```
|
||||
|
||||
:::info[Model Compatibility]
|
||||
Structured output support depends on the underlying model. OpenAI models starting from `gpt-4o-2024-08-06` support Pydantic models and JSON schemas. For details on which models support this feature, refer to the respective model provider's documentation.
|
||||
:::
|
||||
|
||||
### Tool Support
|
||||
|
||||
`CometAPIChatGenerator` supports function calling through the `tools` parameter, which accepts flexible tool configurations:
|
||||
|
||||
- **A list of Tool objects**: Pass individual tools as a list
|
||||
- **A single Toolset**: Pass an entire Toolset directly
|
||||
- **Mixed Tools and Toolsets**: Combine multiple Toolsets with standalone tools in a single list
|
||||
|
||||
This allows you to organize related tools into logical groups while also including standalone tools as needed.
|
||||
|
||||
```python
|
||||
from haystack.tools import Tool, Toolset
|
||||
from haystack_integrations.components.generators.cometapi import CometAPIChatGenerator
|
||||
|
||||
# Create individual tools
|
||||
weather_tool = Tool(name="weather", description="Get weather info", ...)
|
||||
news_tool = Tool(name="news", description="Get latest news", ...)
|
||||
|
||||
# Group related tools into a toolset
|
||||
math_toolset = Toolset([add_tool, subtract_tool, multiply_tool])
|
||||
|
||||
# Pass mixed tools and toolsets to the generator
|
||||
generator = CometAPIChatGenerator(
|
||||
tools=[math_toolset, weather_tool, news_tool] # Mix of Toolset and Tool objects
|
||||
)
|
||||
```
|
||||
|
||||
For more details on working with tools, see the [Tool](../../tools/tool.mdx) and [Toolset](../../tools/toolset.mdx) documentation.
|
||||
|
||||
### Streaming
|
||||
|
||||
`CometAPIChatGenerator` supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
You can stream output as it's generated. Pass a callback to `streaming_callback`. Use the built-in `print_streaming_chunk` to print text tokens and tool events (tool calls and tool results).
|
||||
|
||||
```python
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
|
||||
# Configure the generator with a streaming callback
|
||||
component = CometAPIChatGenerator(streaming_callback=print_streaming_chunk)
|
||||
|
||||
# Pass a list of messages
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
component.run([ChatMessage.from_user("Your question here")])
|
||||
```
|
||||
|
||||
:::info
|
||||
Streaming works only with a single response. If a provider supports multiple candidates, set `n=1`.
|
||||
:::
|
||||
|
||||
See our [Streaming Support](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) docs to learn more how `StreamingChunk` works and how to write a custom callback.
|
||||
|
||||
We recommend to give preference to `print_streaming_chunk` by default. Write a custom callback only if you need a specific transport (for example, SSE/WebSocket) or custom UI formatting.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `cometapi-haystack` package to use the `CometAPIChatGenerator`:
|
||||
|
||||
```shell
|
||||
pip install cometapi-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.cometapi import CometAPIChatGenerator
|
||||
|
||||
client = CometAPIChatGenerator(model="gpt-4o-mini", streaming_callback=print_streaming_chunk)
|
||||
|
||||
response = client.run([ChatMessage.from_user("What's Natural Language Processing? Be brief.")])
|
||||
|
||||
>> Natural Language Processing (NLP) is a field of artificial intelligence that
|
||||
>> focuses on the interaction between computers and humans through natural language.
|
||||
>> It involves enabling machines to understand, interpret, and generate human
|
||||
>> language in a meaningful way, facilitating tasks such as language translation,
|
||||
>> sentiment analysis, and text summarization.
|
||||
|
||||
print(response)
|
||||
|
||||
>> {'replies': [ChatMessage(_role=<ChatRole.ASSISTANT: 'assistant'>, _content=
|
||||
>> [TextContent(text='Natural Language Processing (NLP) is a field of artificial
|
||||
>> intelligence that focuses on the interaction between computers and humans through
|
||||
>> natural language...')], _name=None, _meta={'model': 'gpt-4o-mini-2024-07-18',
|
||||
>> 'index': 0, 'finish_reason': 'stop', 'usage': {'completion_tokens': 59,
|
||||
>> 'prompt_tokens': 15, 'total_tokens': 74}})]}
|
||||
```
|
||||
|
||||
With multimodal inputs:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage, ImageContent
|
||||
from haystack_integrations.components.generators.cometapi import CometAPIChatGenerator
|
||||
|
||||
# Use a multimodal model like GPT-4o
|
||||
llm = CometAPIChatGenerator(model="gpt-4o")
|
||||
|
||||
image = ImageContent.from_file_path("apple.jpg", detail="low")
|
||||
user_message = ChatMessage.from_user(content_parts=[
|
||||
"What does the image show? Max 5 words.",
|
||||
image
|
||||
])
|
||||
|
||||
response = llm.run([user_message])["replies"][0].text
|
||||
print(response)
|
||||
|
||||
>>> Red apple on straw.
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack_integrations.components.generators.cometapi import CometAPIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack import Pipeline
|
||||
from haystack.utils import Secret
|
||||
|
||||
# No parameter init, we don't use any runtime template variables
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
llm = CometAPIChatGenerator()
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
location = "Berlin"
|
||||
messages = [
|
||||
ChatMessage.from_system("Always respond in German even if some input data is in other languages."),
|
||||
ChatMessage.from_user("Tell me about {{location}}")
|
||||
]
|
||||
pipe.run(data={"prompt_builder": {"template_variables": {"location": location}, "template": messages}})
|
||||
|
||||
>> {'llm': {'replies': [ChatMessage(_role=<ChatRole.ASSISTANT: 'assistant'>,
|
||||
>> _content=[TextContent(text='Berlin ist die Hauptstadt Deutschlands und eine der
|
||||
>> bedeutendsten Städte Europas. Es ist bekannt für ihre reiche Geschichte,
|
||||
>> kulturelle Vielfalt und kreative Scene. \n\nDie Stadt hat eine bewegte
|
||||
>> Vergangenheit, die stark von der Teilung zwischen Ost- und Westberlin während
|
||||
>> des Kalten Krieges geprägt war. Die Berliner Mauer, die von 1961 bis 1989 die
|
||||
>> Stadt teilte, ist heute ein Symbol für die Wiedervereinigung und die Freiheit.')],
|
||||
>> _name=None, _meta={'model': 'gpt-4o-mini-2024-07-18', 'index': 0,
|
||||
>> 'finish_reason': 'stop', 'usage': {'completion_tokens': 260,
|
||||
>> 'prompt_tokens': 29, 'total_tokens': 289}})]}
|
||||
```
|
||||
|
||||
Using multiple models in one pipeline:
|
||||
|
||||
```python
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack_integrations.components.generators.cometapi import CometAPIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack import Pipeline
|
||||
|
||||
# Create a pipeline that uses different models for different tasks
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
# Use Claude for complex reasoning
|
||||
claude_llm = CometAPIChatGenerator(model="claude-3-5-sonnet-20241022")
|
||||
# Use GPT-4o-mini for simple tasks
|
||||
gpt_llm = CometAPIChatGenerator(model="gpt-4o-mini")
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("claude", claude_llm)
|
||||
pipe.add_component("gpt", gpt_llm)
|
||||
|
||||
# Feed the same prompt to both models
|
||||
pipe.connect("prompt_builder.prompt", "claude.messages")
|
||||
pipe.connect("prompt_builder.prompt", "gpt.messages")
|
||||
|
||||
messages = [ChatMessage.from_user("Explain quantum computing in simple terms.")]
|
||||
result = pipe.run(data={"prompt_builder": {"template": messages}})
|
||||
|
||||
print("Claude:", result["claude"]["replies"][0].text)
|
||||
print("GPT-4o-mini:", result["gpt"]["replies"][0].text)
|
||||
```
|
||||
|
||||
With tool calling:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.tools import ToolInvoker
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.tools import Tool
|
||||
from haystack_integrations.components.generators.cometapi import CometAPIChatGenerator
|
||||
|
||||
def weather(city: str) -> str:
|
||||
"""Get weather for a given city."""
|
||||
return f"The weather in {city} is sunny and 32°C"
|
||||
|
||||
tool = Tool(
|
||||
name="weather",
|
||||
description="Get weather for a given city",
|
||||
parameters={"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]},
|
||||
function=weather,
|
||||
)
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("generator", CometAPIChatGenerator(tools=[tool]))
|
||||
pipeline.add_component("tool_invoker", ToolInvoker(tools=[tool]))
|
||||
|
||||
pipeline.connect("generator", "tool_invoker")
|
||||
|
||||
results = pipeline.run(
|
||||
data={
|
||||
"generator": {
|
||||
"messages": [ChatMessage.from_user("What's the weather like in Paris?")],
|
||||
"generation_kwargs": {"tool_choice": "auto"},
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
print(results["tool_invoker"]["tool_messages"][0].tool_call_result.result)
|
||||
>> The weather in Paris is sunny and 32°C
|
||||
```
|
||||
+97
@@ -0,0 +1,97 @@
|
||||
---
|
||||
title: "DALLEImageGenerator"
|
||||
id: dalleimagegenerator
|
||||
slug: "/dalleimagegenerator"
|
||||
description: "Generate images using OpenAI's DALL-E model."
|
||||
---
|
||||
|
||||
# DALLEImageGenerator
|
||||
|
||||
Generate images using OpenAI's DALL-E model.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`PromptBuilder`](../builders/promptbuilder.mdx), flexible |
|
||||
| **Mandatory init variables** | `api_key`: An OpenAI API key. Can be set with `OPENAI_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `prompt`: A string containing the prompt for the model |
|
||||
| **Output variables** | `images`: A list of generated images <br /> <br />`revised_prompt`: A string containing the prompt that was used to generate the image, if there was any revision to the prompt made by OpenAI |
|
||||
| **API reference** | [Generators](/reference/generators-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/generators/openai_dalle.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `DALLEImageGenerator` component generates images using OpenAI's DALL-E model.
|
||||
|
||||
By default, the component uses `dall-e-3` model, standard picture quality, and 1024x1024 resolution. You can change these parameters using `model` (during component initialization), `quality`, and `size` (during component initialization or run) parameters.
|
||||
|
||||
`DALLEImageGenerator` needs an OpenAI key to work. It uses an `OPENAI_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`:
|
||||
|
||||
```
|
||||
image_generator = DALLEImageGenerator(api_key=Secret.from_token("<your-api-key>"))
|
||||
```
|
||||
|
||||
Check our [API reference](/reference/generators-api#dalleimagegenerator) for the detailed component parameters description, or the [OpenAI documentation](https://platform.openai.com/docs/api-reference/images/create) for the details on OpenAI API parameters.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.components.generators import DALLEImageGenerator
|
||||
|
||||
image_generator = DALLEImageGenerator()
|
||||
response = image_generator.run("Show me a picture of a black cat.")
|
||||
|
||||
print(response)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
In the following pipeline, we first set up a `PromptBuilder` that will structure the image description with a detailed template describing various artistic elements. The pipeline then passes this structured prompt into a `DALLEImageGenerator` to generate the image based on this detailed description.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.generators import DALLEImageGenerator
|
||||
from haystack.components.builders import PromptBuilder
|
||||
|
||||
prompt_builder = PromptBuilder(
|
||||
template="""Create a {style} image with the following details:
|
||||
|
||||
Main subject: {prompt}
|
||||
Artistic style: {art_style}
|
||||
Lighting: {lighting}
|
||||
Color palette: {colors}
|
||||
Composition: {composition}
|
||||
Additional details: {details}""",
|
||||
)
|
||||
|
||||
image_generator = DALLEImageGenerator()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("prompt_builder", prompt_builder)
|
||||
pipeline.add_component("image_generator", image_generator)
|
||||
|
||||
pipeline.connect("prompt_builder.prompt", "image_generator.prompt")
|
||||
|
||||
results = pipeline.run(
|
||||
{
|
||||
"prompt": "a mystical treehouse library",
|
||||
"style": "photorealistic",
|
||||
"art_style": "fantasy concept art with intricate details",
|
||||
"lighting": "dusk with warm lantern light glowing from within",
|
||||
"colors": "rich earth tones, deep greens, and golden accents",
|
||||
"composition": "wide angle view showing the entire structure nestled in an ancient oak tree",
|
||||
"details": "spiral staircases wrapping around branches, stained glass windows, floating books, and magical fireflies providing ambient illumination",
|
||||
},
|
||||
)
|
||||
|
||||
generated_images = results["image_generator"]["images"]
|
||||
revised_prompt = results["image_generator"]["revised_prompt"]
|
||||
|
||||
print(f"Generated image URL: {generated_images[0]}")
|
||||
print(f"Revised prompt: {revised_prompt}")
|
||||
```
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
---
|
||||
title: "External Integrations"
|
||||
id: external-integrations-generators
|
||||
slug: "/external-integrations-generators"
|
||||
description: "External integrations that enable RAG pipeline creation."
|
||||
---
|
||||
|
||||
# External Integrations
|
||||
|
||||
External integrations that enable RAG pipeline creation.
|
||||
|
||||
| Name | Description |
|
||||
| --- | --- |
|
||||
| [DeepL](https://haystack.deepset.ai/integrations/deepl) | Translate your text and documents using DeepL services. |
|
||||
| [fastRAG](https://haystack.deepset.ai/integrations/fastrag/) | Enables the creation of efficient and optimized retrieval augmented generative pipelines. |
|
||||
| [LM Format Enforcer](https://haystack.deepset.ai/integrations/lmformatenforcer) | Enforce JSON Schema / Regex output of your local models with `LMFormatEnforcerLocalGenerator`. |
|
||||
| [Titan](https://haystack.deepset.ai/integrations/titanml-takeoff) | Run local open-source LLMs from Meta, Mistral and Alphabet directly in your computer. |
|
||||
+238
@@ -0,0 +1,238 @@
|
||||
---
|
||||
title: "FallbackChatGenerator"
|
||||
id: fallbackchatgenerator
|
||||
slug: "/fallbackchatgenerator"
|
||||
description: "A ChatGenerator wrapper that tries multiple Chat Generators sequentially until one succeeds."
|
||||
---
|
||||
|
||||
# FallbackChatGenerator
|
||||
|
||||
A ChatGenerator wrapper that tries multiple Chat Generators sequentially until one succeeds.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `chat_generators`: A non-empty list of Chat Generator components to try in order |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects representing the chat |
|
||||
| **Output variables** | `replies`: Generated ChatMessage instances from the first successful generator <br /> <br />`meta`: Execution metadata including successful generator details |
|
||||
| **API reference** | [Generators](/reference/generators-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/generators/chat/fallback.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`FallbackChatGenerator` is a wrapper component that tries multiple Chat Generators sequentially until one succeeds. If a Generator fails, the component tries the next one in the list. This handles provider outages, rate limits, and other transient failures.
|
||||
|
||||
The component forwards all parameters to the underlying Chat Generators and returns the first successful result. When a Generator raises any exception, the component tries the next Generator. This includes timeout errors, rate limit errors (429), authentication errors (401), context length errors (400), server errors (500+), and any other exception.
|
||||
|
||||
The component returns execution metadata including which Generator succeeded, how many attempts were made, and which Generators failed. All parameters (`messages`, `generation_kwargs`, `tools`, `streaming_callback`) are forwarded to the underlying Generators.
|
||||
|
||||
Timeout enforcement is delegated to the underlying Chat Generators. To control latency, configure your Chat Generators with a `timeout` parameter. Chat Generators like OpenAI, Anthropic, and Cohere support timeout parameters that raise exceptions when exceeded.
|
||||
|
||||
### Monitoring and Telemetry
|
||||
|
||||
The `meta` dictionary in the output contains useful information for monitoring:
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import (
|
||||
FallbackChatGenerator,
|
||||
OpenAIChatGenerator,
|
||||
)
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
## Set up generators
|
||||
primary = OpenAIChatGenerator(model="gpt-4o")
|
||||
backup = OpenAIChatGenerator(model="gpt-4o-mini")
|
||||
generator = FallbackChatGenerator(chat_generators=[primary, backup])
|
||||
|
||||
## Run and inspect metadata
|
||||
result = generator.run(messages=[ChatMessage.from_user("Hello")])
|
||||
|
||||
meta = result["meta"]
|
||||
print(
|
||||
f"Successful generator index: {meta['successful_chat_generator_index']}",
|
||||
) # 0 for first, 1 for second, etc.
|
||||
print(
|
||||
f"Successful generator class: {meta['successful_chat_generator_class']}",
|
||||
) # e.g., "OpenAIChatGenerator"
|
||||
print(
|
||||
f"Total attempts made: {meta['total_attempts']}",
|
||||
) # How many Generators were tried
|
||||
print(
|
||||
f"Failed generators: {meta['failed_chat_generators']}",
|
||||
) # List of failed Generator names
|
||||
```
|
||||
|
||||
You can use this metadata to:
|
||||
|
||||
- Track which Generators are being used most frequently
|
||||
- Monitor failure rates for each Generator
|
||||
- Set up alerts when fallbacks occur
|
||||
- Adjust Generator ordering based on success rates
|
||||
|
||||
### Streaming
|
||||
|
||||
`FallbackChatGenerator` supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) through the `streaming_callback` parameter. The callback is passed directly to the underlying Generators.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage with fallback from a primary to a backup model:
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import FallbackChatGenerator, OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
## Create primary and backup generators
|
||||
primary = OpenAIChatGenerator(model="gpt-4o", timeout=30)
|
||||
backup = OpenAIChatGenerator(model="gpt-4o-mini", timeout=30)
|
||||
|
||||
## Wrap them in a FallbackChatGenerator
|
||||
generator = FallbackChatGenerator(chat_generators=[primary, backup])
|
||||
|
||||
## Use it like any other Chat Generator
|
||||
messages = [ChatMessage.from_user("What's Natural Language Processing? Be brief.")]
|
||||
result = generator.run(messages=messages)
|
||||
|
||||
print(result["replies"][0].text)
|
||||
print(f"Successful generator: {result['meta']['successful_chat_generator_class']}")
|
||||
print(f"Total attempts: {result['meta']['total_attempts']}")
|
||||
|
||||
>> Natural Language Processing (NLP) is a field of artificial intelligence that
|
||||
>> focuses on the interaction between computers and humans through natural language...
|
||||
>> Successful generator: OpenAIChatGenerator
|
||||
>> Total attempts: 1
|
||||
```
|
||||
|
||||
With multiple providers:
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import (
|
||||
FallbackChatGenerator,
|
||||
OpenAIChatGenerator,
|
||||
AzureOpenAIChatGenerator,
|
||||
)
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.utils import Secret
|
||||
|
||||
## Create generators from different providers
|
||||
openai_gen = OpenAIChatGenerator(
|
||||
model="gpt-4o-mini",
|
||||
api_key=Secret.from_env_var("OPENAI_API_KEY"),
|
||||
timeout=30,
|
||||
)
|
||||
|
||||
azure_gen = AzureOpenAIChatGenerator(
|
||||
azure_endpoint="<Your Azure endpoint>",
|
||||
api_key=Secret.from_env_var("AZURE_OPENAI_API_KEY"),
|
||||
azure_deployment="gpt-4o-mini",
|
||||
timeout=30,
|
||||
)
|
||||
|
||||
## Fallback will try OpenAI first, then Azure
|
||||
generator = FallbackChatGenerator(chat_generators=[openai_gen, azure_gen])
|
||||
|
||||
messages = [ChatMessage.from_user("Explain quantum computing briefly.")]
|
||||
result = generator.run(messages=messages)
|
||||
|
||||
print(result["replies"][0].text)
|
||||
```
|
||||
|
||||
With streaming:
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import (
|
||||
FallbackChatGenerator,
|
||||
OpenAIChatGenerator,
|
||||
)
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
primary = OpenAIChatGenerator(model="gpt-4o")
|
||||
backup = OpenAIChatGenerator(model="gpt-4o-mini")
|
||||
|
||||
generator = FallbackChatGenerator(chat_generators=[primary, backup])
|
||||
|
||||
messages = [ChatMessage.from_user("What's Natural Language Processing? Be brief.")]
|
||||
result = generator.run(
|
||||
messages=messages,
|
||||
streaming_callback=lambda chunk: print(chunk.content, end="", flush=True),
|
||||
)
|
||||
|
||||
print("\n", result["meta"])
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import (
|
||||
FallbackChatGenerator,
|
||||
OpenAIChatGenerator,
|
||||
)
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
## Create primary and backup generators with timeouts
|
||||
primary = OpenAIChatGenerator(model="gpt-4o", timeout=30)
|
||||
backup = OpenAIChatGenerator(model="gpt-4o-mini", timeout=30)
|
||||
|
||||
## Wrap in fallback
|
||||
fallback_generator = FallbackChatGenerator(chat_generators=[primary, backup])
|
||||
|
||||
## Build pipeline
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", fallback_generator)
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
## Run pipeline
|
||||
messages = [
|
||||
ChatMessage.from_system(
|
||||
"You are a helpful assistant that provides concise answers.",
|
||||
),
|
||||
ChatMessage.from_user("Tell me about {{location}}"),
|
||||
]
|
||||
|
||||
result = pipe.run(
|
||||
data={
|
||||
"prompt_builder": {
|
||||
"template": messages,
|
||||
"template_variables": {"location": "Paris"},
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
print(result["llm"]["replies"][0].text)
|
||||
print(f"Generator used: {result['llm']['meta']['successful_chat_generator_class']}")
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
If all Generators fail, `FallbackChatGenerator` raises a `RuntimeError` with details about which Generators failed and the last error encountered:
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import (
|
||||
FallbackChatGenerator,
|
||||
OpenAIChatGenerator,
|
||||
)
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.utils import Secret
|
||||
|
||||
## Create generators with invalid credentials to demonstrate error handling
|
||||
primary = OpenAIChatGenerator(api_key=Secret.from_token("invalid-key-1"))
|
||||
backup = OpenAIChatGenerator(api_key=Secret.from_token("invalid-key-2"))
|
||||
|
||||
generator = FallbackChatGenerator(chat_generators=[primary, backup])
|
||||
|
||||
try:
|
||||
result = generator.run(messages=[ChatMessage.from_user("Hello")])
|
||||
except RuntimeError as e:
|
||||
print(f"All generators failed: {e}")
|
||||
# Output: All 2 chat generators failed. Last error: ... Failed chat generators: [OpenAIChatGenerator, OpenAIChatGenerator]
|
||||
```
|
||||
+176
@@ -0,0 +1,176 @@
|
||||
---
|
||||
title: "GoogleAIGeminiChatGenerator"
|
||||
id: googleaigeminichatgenerator
|
||||
slug: "/googleaigeminichatgenerator"
|
||||
description: "This component enables chat completion using Google Gemini models."
|
||||
---
|
||||
|
||||
# GoogleAIGeminiChatGenerator
|
||||
|
||||
This component enables chat completion using Google Gemini models.
|
||||
|
||||
:::warning[Deprecation Notice]
|
||||
|
||||
This integration uses the deprecated google-generativeai SDK, which will lose support after August 2025.
|
||||
|
||||
We recommend switching to the new [GoogleGenAIChatGenerator](googlegenaichatgenerator.mdx) integration instead.
|
||||
:::
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: A Google AI Studio API key. Can be set with `GOOGLE_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects representing the chat |
|
||||
| **Output variables** | `replies`: A list of alternative replies of the model to the input chat |
|
||||
| **API reference** | [Google AI](/reference/integrations-google-ai) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/google_ai |
|
||||
|
||||
</div>
|
||||
|
||||
`GoogleAIGeminiChatGenerator` supports `gemini-2.5-pro-exp-03-25`, `gemini-2.0-flash`, `gemini-1.5-pro`, and `gemini-1.5-flash` models.
|
||||
|
||||
For available models, see https://ai.google.dev/gemini-api/docs/models/gemini.
|
||||
|
||||
### Parameters Overview
|
||||
|
||||
`GoogleAIGeminiChatGenerator` uses a Google Studio API key for authentication. You can write this key in an `api_key` parameter or as a `GOOGLE_API_KEY` environment variable (recommended).
|
||||
|
||||
To get an API key, visit the [Google AI Studio](https://aistudio.google.com/) website.
|
||||
|
||||
### Streaming
|
||||
|
||||
This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
To begin working with `GoogleAIGeminiChatGenerator`, install the `google-ai-haystack` package:
|
||||
|
||||
```shell
|
||||
pip install google-ai-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage:
|
||||
|
||||
```python
|
||||
import os
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.google_ai import GoogleAIGeminiChatGenerator
|
||||
|
||||
os.environ["GOOGLE_API_KEY"] = "<MY_API_KEY>"
|
||||
gemini_chat = GoogleAIGeminiChatGenerator()
|
||||
|
||||
messages = [ChatMessage.from_user("Tell me the name of a movie")]
|
||||
res = gemini_chat.run(messages)
|
||||
|
||||
print(res["replies"][0].text)
|
||||
>>> The Shawshank Redemption
|
||||
|
||||
messages += [res["replies"], ChatMessage.from_user("Who's the main actor?")]
|
||||
res = gemini_chat.run(messages)
|
||||
|
||||
print(res["replies"][0].text)
|
||||
>>> Tim Robbins
|
||||
```
|
||||
|
||||
When chatting with Gemini, you can also easily use function calls. First, define the function locally and convert into a [Tool](../../tools/tool.mdx):
|
||||
|
||||
```python
|
||||
from typing import Annotated
|
||||
from haystack.tools import create_tool_from_function
|
||||
|
||||
|
||||
## example function to get the current weather
|
||||
def get_current_weather(
|
||||
location: Annotated[
|
||||
str,
|
||||
"The city for which to get the weather, e.g. 'San Francisco'",
|
||||
] = "Munich",
|
||||
unit: Annotated[str, "The unit for the temperature, e.g. 'celsius'"] = "celsius",
|
||||
) -> str:
|
||||
return f"The weather in {location} is sunny. The temperature is 20 {unit}."
|
||||
|
||||
|
||||
tool = create_tool_from_function(get_current_weather)
|
||||
```
|
||||
|
||||
Create a new instance of `GoogleAIGeminiChatGenerator` to set the tools and a [ToolInvoker](../tools/toolinvoker.mdx) to invoke the tools.
|
||||
|
||||
```python
|
||||
import os
|
||||
from haystack_integrations.components.generators.google_ai import (
|
||||
GoogleAIGeminiChatGenerator,
|
||||
)
|
||||
from haystack.components.tools import ToolInvoker
|
||||
|
||||
os.environ["GOOGLE_API_KEY"] = "<MY_API_KEY>"
|
||||
|
||||
gemini_chat = GoogleAIGeminiChatGenerator(model="gemini-2.0-flash", tools=[tool])
|
||||
|
||||
tool_invoker = ToolInvoker(tools=[tool])
|
||||
```
|
||||
|
||||
And then ask a question:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
messages = [ChatMessage.from_user("What is the temperature in celsius in Berlin?")]
|
||||
res = gemini_chat.run(messages=messages)
|
||||
|
||||
print(res["replies"][0].tool_calls)
|
||||
>>> [ToolCall(tool_name='get_current_weather',
|
||||
>>> arguments={'unit': 'celsius', 'location': 'Berlin'}, id=None)]
|
||||
|
||||
tool_messages = tool_invoker.run(messages=replies)["tool_messages"]
|
||||
messages = user_message + replies + tool_messages
|
||||
|
||||
messages += res["replies"][0] + [ChatMessage.from_function(content=weather, name="get_current_weather")]
|
||||
|
||||
final_replies = gemini_chat.run(messages=messages)["replies"]
|
||||
print(final_replies[0].text)
|
||||
>>> The temperature in Berlin is 20 degrees Celsius.
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
import os
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack import Pipeline
|
||||
from haystack_integrations.components.generators.google_ai import GoogleAIGeminiChatGenerator
|
||||
|
||||
## no parameter init, we don't use any runtime template variables
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
|
||||
os.environ["GOOGLE_API_KEY"] = "<MY_API_KEY>"
|
||||
gemini_chat = GoogleAIGeminiChatGenerator()
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("gemini", gemini_chat)
|
||||
pipe.connect("prompt_builder.prompt", "gemini.messages")
|
||||
|
||||
location = "Rome"
|
||||
messages = [ChatMessage.from_user("Tell me briefly about {{location}} history")]
|
||||
res = pipe.run(data={"prompt_builder": {"template_variables":{"location": location}, "template": messages}})
|
||||
|
||||
print(res)
|
||||
|
||||
>>> - **753 B.C.:** Traditional date of the founding of Rome by Romulus and Remus.
|
||||
>>> - **509 B.C.:** Establishment of the Roman Republic, replacing the Etruscan monarchy.
|
||||
>>> - **492-264 B.C.:** Series of wars against neighboring tribes, resulting in the expansion of the Roman Republic's territory.
|
||||
>>> - **264-146 B.C.:** Three Punic Wars against Carthage, resulting in the destruction of Carthage and the Roman Republic becoming the dominant power in the Mediterranean.
|
||||
>>> - **133-73 B.C.:** Series of civil wars and slave revolts, leading to the rise of Julius Caesar.
|
||||
>>> - **49 B.C.:** Julius Caesar crosses the Rubicon River, starting the Roman Civil War.
|
||||
>>> - **44 B.C.:** Julius Caesar is assassinated, leading to the Second Triumvirate of Octavian, Mark Antony, and Lepidus.
|
||||
>>> - **31 B.C.:** Battle of Actium, where Octavian defeats Mark Antony and Cleopatra, becoming the sole ruler of Rome.
|
||||
>>> - **27 B.C.:** The Roman Republic is transformed into the Roman Empire, with Octavian becoming the first Roman emperor, known as Augustus.
|
||||
>>> - **1st century A.D.:** The Roman Empire reaches its greatest extent, stretching from Britain to Egypt.
|
||||
>>> - **3rd century A.D.:** The Roman Empire begins to decline, facing internal instability, invasions by Germanic tribes, and the rise of Christianity.
|
||||
>>> - **476 A.D.:** The last Western Roman emperor, Romulus Augustulus, is overthrown by the Germanic leader Odoacer, marking the end of the Roman Empire in the West.
|
||||
```
|
||||
+150
@@ -0,0 +1,150 @@
|
||||
---
|
||||
title: "GoogleAIGeminiGenerator"
|
||||
id: googleaigeminigenerator
|
||||
slug: "/googleaigeminigenerator"
|
||||
description: "This component enables text generation using the Google Gemini models."
|
||||
---
|
||||
|
||||
# GoogleAIGeminiGenerator
|
||||
|
||||
This component enables text generation using the Google Gemini models.
|
||||
|
||||
:::warning[Deprecation Notice]
|
||||
|
||||
This integration uses the deprecated google-generativeai SDK, which will lose support after August 2025.
|
||||
|
||||
We recommend switching to the new [GoogleGenAIChatGenerator](googlegenaichatgenerator.mdx) integration instead.
|
||||
:::
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`PromptBuilder`](../builders/promptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: A Google AI Studio API key. Can be set with `GOOGLE_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `parts`: A variadic list containing a mix of images, audio, video, and text to prompt Gemini |
|
||||
| **Output variables** | `replies`: A list of strings or dictionaries with all the replies generated by the model |
|
||||
| **API reference** | [Google AI](/reference/integrations-google-ai) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/google_ai |
|
||||
|
||||
</div>
|
||||
|
||||
`GoogleAIGeminiGenerator` supports `gemini-2.5-pro-exp-03-25`, `gemini-2.0-flash`, `gemini-1.5-pro`, and `gemini-1.5-flash` models.
|
||||
|
||||
For available models, see https://ai.google.dev/gemini-api/docs/models/gemini.
|
||||
|
||||
### Parameters Overview
|
||||
|
||||
`GoogleAIGeminiGenerator` uses a Google AI Studio API key for authentication. You can write this key in an `api_key` parameter or as a `GOOGLE_API_KEY` environment variable (recommended).
|
||||
|
||||
To get an API key, visit the [Google AI Studio](https://ai.google.dev/gemini-api/docs/api-key) website.
|
||||
|
||||
### Streaming
|
||||
|
||||
This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
Start by installing the `google-ai-haystack` package to use the `GoogleAIGeminiGenerator`:
|
||||
|
||||
```shell
|
||||
pip install google-ai-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage:
|
||||
|
||||
```python
|
||||
import os
|
||||
from haystack_integrations.components.generators.google_ai import GoogleAIGeminiGenerator
|
||||
|
||||
os.environ["GOOGLE_API_KEY"] = "<MY_API_KEY>"
|
||||
|
||||
gemini = GoogleAIGeminiGenerator(model="gemini-1.5-pro")
|
||||
res = gemini.run(parts = ["What is the most interesting thing you know?"])
|
||||
for answer in res["replies"]:
|
||||
print(answer)
|
||||
|
||||
>>> 1. **The Fermi Paradox:** This paradox questions why we haven't found any signs of extraterrestrial life, despite the vastness of the universe and the high probability of life existing elsewhere.
|
||||
>>> 2. **The Goldilocks Enigma:** This conundrum explores why Earth has such favorable conditions for life, despite the extreme conditions found in most of the universe. It raises questions about the rarity or commonality of Earth-like planets.
|
||||
>>> 3. **The Quantum Enigma:** Quantum mechanics, the study of the behavior of matter and energy at the atomic and subatomic level, presents many counterintuitive phenomena that challenge our understanding of reality. Questions about the nature of quantum entanglement, superposition, and the origin of quantum mechanics remain unsolved.
|
||||
>>> 4. **The Origin of Consciousness:** The emergence of consciousness from non-conscious matter is one of the biggest mysteries in science. How and why subjective experiences arise from physical processes in the brain remains a perplexing question.
|
||||
>>> 5. **The Nature of Dark Matter and Dark Energy:** Dark matter and dark energy are mysterious substances that make up most of the universe, but their exact nature and properties are still unknown. Understanding their role in the universe's expansion and evolution is a major cosmological challenge.
|
||||
>>> 6. **The Future of Artificial Intelligence:** The rapid development of Artificial Intelligence (AI) raises fundamental questions about the potential consequences and implications for society, including ethical issues, job displacement, and the long-term impact on human civilization.
|
||||
>>> 7. **The Search for Life Beyond Earth:** As we continue to explore our solar system and beyond, the search for life on other planets or moons is a captivating and ongoing endeavor. Discovering extraterrestrial life would have profound implications for our understanding of the universe and our place in it.
|
||||
>>> 8. **Time Travel:** The concept of time travel, whether forward or backward, remains a theoretical possibility that challenges our understanding of causality and the laws of physics. The implications and paradoxes associated with time travel have fascinated scientists and philosophers alike.
|
||||
>>> 9. **The Multiverse Theory:** The multiverse theory suggests the existence of multiple universes, each with its own set of physical laws and properties. This idea raises questions about the nature of reality, the role of chance and necessity, and the possibility of parallel universes.
|
||||
>>> 10. **The Fate of the Universe:** The ultimate fate of the universe is a subject of ongoing debate among cosmologists. Various theories, such as the Big Crunch, the Big Freeze, or the Big Rip, attempt to explain how the universe will end or evolve over time. Understanding the universe's destiny is a profound and awe-inspiring pursuit.
|
||||
```
|
||||
|
||||
This is a more advanced usage that also uses text and images as input:
|
||||
|
||||
```python
|
||||
import requests
|
||||
import os
|
||||
from haystack.dataclasses.byte_stream import ByteStream
|
||||
from haystack_integrations.components.generators.google_ai import GoogleAIGeminiGenerator
|
||||
|
||||
URLS = [
|
||||
"https://raw.githubusercontent.com/silvanocerza/robots/main/robot1.jpg",
|
||||
"https://raw.githubusercontent.com/silvanocerza/robots/main/robot2.jpg",
|
||||
"https://raw.githubusercontent.com/silvanocerza/robots/main/robot3.jpg",
|
||||
"https://raw.githubusercontent.com/silvanocerza/robots/main/robot4.jpg"
|
||||
]
|
||||
images = [
|
||||
ByteStream(data=requests.get(url).content, mime_type="image/jpeg")
|
||||
for url in URLS
|
||||
]
|
||||
|
||||
os.environ["GOOGLE_API_KEY"] = "<MY_API_KEY>"
|
||||
|
||||
gemini = GoogleAIGeminiGenerator(model="gemini-1.5-pro")
|
||||
result = gemini.run(parts = ["What can you tell me about this robots?", *images])
|
||||
for answer in result["replies"]:
|
||||
print(answer)
|
||||
|
||||
>>> The first image is of C-3PO and R2-D2 from the Star Wars franchise. C-3PO is a protocol droid, while R2-D2 is an astromech droid. They are both loyal companions to the heroes of the Star Wars saga.
|
||||
>>> The second image is of Maria from the 1927 film Metropolis. Maria is a robot who is created to be the perfect woman. She is beautiful, intelligent, and obedient. However, she is also soulless and lacks any real emotions.
|
||||
>>> The third image is of Gort from the 1951 film The Day the Earth Stood Still. Gort is a robot who is sent to Earth to warn humanity about the dangers of nuclear war. He is a powerful and intelligent robot, but he is also compassionate and understanding.
|
||||
>>> The fourth image is of Marvin from the 1977 film The Hitchhiker's Guide to the Galaxy. Marvin is a robot who is depressed and pessimistic. He is constantly complaining about everything, but he is also very intelligent and has a dry sense of humor.
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
In a RAG pipeline:
|
||||
|
||||
```python
|
||||
import os
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.components.builders import PromptBuilder
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.generators.google_ai import (
|
||||
GoogleAIGeminiGenerator,
|
||||
)
|
||||
|
||||
os.environ["GOOGLE_API_KEY"] = "<MY_API_KEY>"
|
||||
|
||||
docstore = InMemoryDocumentStore()
|
||||
|
||||
template = """
|
||||
Given the following information, answer the question.
|
||||
|
||||
Context:
|
||||
{% for document in documents %}
|
||||
{{ document.content }}
|
||||
{% endfor %}
|
||||
|
||||
Question: What's the official language of {{ country }}?
|
||||
"""
|
||||
pipe = Pipeline()
|
||||
|
||||
pipe.add_component("retriever", InMemoryBM25Retriever(document_store=docstore))
|
||||
pipe.add_component("prompt_builder", PromptBuilder(template=template))
|
||||
pipe.add_component("gemini", GoogleAIGeminiGenerator(model="gemini-pro"))
|
||||
pipe.connect("retriever", "prompt_builder.documents")
|
||||
pipe.connect("prompt_builder", "gemini")
|
||||
|
||||
pipe.run({"prompt_builder": {"country": "France"}})
|
||||
```
|
||||
+277
@@ -0,0 +1,277 @@
|
||||
---
|
||||
title: "GoogleGenAIChatGenerator"
|
||||
id: googlegenaichatgenerator
|
||||
slug: "/googlegenaichatgenerator"
|
||||
description: "This component enables chat completion using Google Gemini models through Google Gen AI SDK."
|
||||
---
|
||||
|
||||
# GoogleGenAIChatGenerator
|
||||
|
||||
This component enables chat completion using Google Gemini models through Google Gen AI SDK.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: A Google API key. Can be set with `GOOGLE_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects representing the chat |
|
||||
| **Output variables** | `replies`: A list of alternative replies of the model to the input chat |
|
||||
| **API reference** | [Google GenAI](/reference/integrations-google-genai) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/google_genai |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`GoogleGenAIChatGenerator` supports Gemini generative models, such as
|
||||
`gemini-3.1-flash-lite-preview`, `gemini-3.1-pro-preview`, `gemini-3-flash-preview`, `gemini-2.5-flash`, `gemini-2.5-pro`, and `gemini-2.5-flash-lite`.
|
||||
|
||||
### Tool Support
|
||||
|
||||
`GoogleGenAIChatGenerator` supports function calling through the `tools` parameter, which accepts flexible tool configurations:
|
||||
|
||||
- **A list of Tool objects**: Pass individual tools as a list
|
||||
- **A single Toolset**: Pass an entire Toolset directly
|
||||
- **Mixed Tools and Toolsets**: Combine multiple Toolsets with standalone tools in a single list
|
||||
|
||||
This allows you to organize related tools into logical groups while also including standalone tools as needed.
|
||||
|
||||
```python
|
||||
from haystack.tools import Tool, Toolset
|
||||
from haystack_integrations.components.generators.google_genai import GoogleGenAIChatGenerator
|
||||
|
||||
# Create individual tools
|
||||
weather_tool = Tool(name="weather", description="Get weather info", ...)
|
||||
news_tool = Tool(name="news", description="Get latest news", ...)
|
||||
|
||||
# Group related tools into a toolset
|
||||
math_toolset = Toolset([add_tool, subtract_tool, multiply_tool])
|
||||
|
||||
# Pass mixed tools and toolsets to the generator
|
||||
generator = GoogleGenAIChatGenerator(
|
||||
tools=[math_toolset, weather_tool, news_tool] # Mix of Toolset and Tool objects
|
||||
)
|
||||
```
|
||||
|
||||
For more details on working with tools, see the [Tool](../../tools/tool.mdx) and [Toolset](../../tools/toolset.mdx) documentation.
|
||||
|
||||
### Streaming
|
||||
|
||||
This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
### Authentication
|
||||
|
||||
Google Gen AI is compatible with both the Gemini Developer API and the Vertex AI API.
|
||||
|
||||
To use this component with the Gemini Developer API and get an API key, visit [Google AI Studio](https://aistudio.google.com/).
|
||||
To use this component with the Vertex AI API, visit [Google Cloud > Vertex AI](https://cloud.google.com/vertex-ai).
|
||||
|
||||
The component uses a `GOOGLE_API_KEY` or `GEMINI_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with a [Secret](../../concepts/secret-management.mdx) and `Secret.from_token` static method:
|
||||
|
||||
```python
|
||||
embedder = GoogleGenAITextEmbedder(api_key=Secret.from_token("<your-api-key>"))
|
||||
```
|
||||
|
||||
The following examples show how to use the component with the Gemini Developer API and the Vertex AI API.
|
||||
|
||||
#### Gemini Developer API (API Key Authentication)
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.google_genai import (
|
||||
GoogleGenAIChatGenerator,
|
||||
)
|
||||
|
||||
## set the environment variable (GOOGLE_API_KEY or GEMINI_API_KEY)
|
||||
chat_generator = GoogleGenAIChatGenerator()
|
||||
```
|
||||
|
||||
#### Vertex AI (Application Default Credentials)
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.google_genai import (
|
||||
GoogleGenAIChatGenerator,
|
||||
)
|
||||
|
||||
## Using Application Default Credentials (requires gcloud auth setup)
|
||||
chat_generator = GoogleGenAIChatGenerator(
|
||||
api="vertex",
|
||||
vertex_ai_project="my-project",
|
||||
vertex_ai_location="us-central1",
|
||||
)
|
||||
```
|
||||
|
||||
#### Vertex AI (API Key Authentication)
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.google_genai import (
|
||||
GoogleGenAIChatGenerator,
|
||||
)
|
||||
|
||||
## set the environment variable (GOOGLE_API_KEY or GEMINI_API_KEY)
|
||||
chat_generator = GoogleGenAIChatGenerator(api="vertex")
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
To start using this integration, install the package with:
|
||||
|
||||
```shell
|
||||
pip install google-genai-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.dataclasses.chat_message import ChatMessage
|
||||
from haystack_integrations.components.generators.google_genai import (
|
||||
GoogleGenAIChatGenerator,
|
||||
)
|
||||
|
||||
## Initialize the chat generator
|
||||
chat_generator = GoogleGenAIChatGenerator()
|
||||
|
||||
## Generate a response
|
||||
messages = [ChatMessage.from_user("Tell me about movie Shawshank Redemption")]
|
||||
response = chat_generator.run(messages=messages)
|
||||
print(response["replies"][0].text)
|
||||
```
|
||||
|
||||
With multimodal inputs:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage, ImageContent
|
||||
from haystack_integrations.components.generators.google_genai import (
|
||||
GoogleGenAIChatGenerator,
|
||||
)
|
||||
|
||||
llm = GoogleGenAIChatGenerator()
|
||||
|
||||
image = ImageContent.from_file_path("apple.jpg")
|
||||
user_message = ChatMessage.from_user(
|
||||
content_parts=["What does the image show? Max 5 words.", image],
|
||||
)
|
||||
|
||||
response = llm.run([user_message])["replies"][0].text
|
||||
print(response)
|
||||
|
||||
# Red apple on straw.
|
||||
```
|
||||
|
||||
You can also easily use function calls. First, define the function locally and convert into a [Tool](https://www.notion.so/docs/tool):
|
||||
|
||||
```python
|
||||
from typing import Annotated
|
||||
from haystack.tools import create_tool_from_function
|
||||
|
||||
|
||||
## example function to get the current weather
|
||||
def get_current_weather(
|
||||
location: Annotated[
|
||||
str,
|
||||
"The city for which to get the weather, e.g. 'San Francisco'",
|
||||
] = "Munich",
|
||||
unit: Annotated[str, "The unit for the temperature, e.g. 'celsius'"] = "celsius",
|
||||
) -> str:
|
||||
return f"The weather in {location} is sunny. The temperature is 20 {unit}."
|
||||
|
||||
|
||||
tool = create_tool_from_function(get_current_weather)
|
||||
```
|
||||
|
||||
Create a new instance of `GoogleGenAIChatGenerator` to set the tools and a [ToolInvoker](https://www.notion.so/docs/toolinvoker) to invoke the tools.
|
||||
|
||||
```python
|
||||
import os
|
||||
from haystack_integrations.components.generators.google_genai import (
|
||||
GoogleGenAIChatGenerator,
|
||||
)
|
||||
from haystack.components.tools import ToolInvoker
|
||||
|
||||
os.environ["GOOGLE_API_KEY"] = "<MY_API_KEY>"
|
||||
|
||||
genai_chat = GoogleGenAIChatGenerator(tools=[tool])
|
||||
|
||||
tool_invoker = ToolInvoker(tools=[tool])
|
||||
```
|
||||
|
||||
And then ask a question:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
messages = [ChatMessage.from_user("What is the temperature in celsius in Berlin?")]
|
||||
res = genai_chat.run(messages=messages)
|
||||
|
||||
print(res["replies"][0].tool_calls)
|
||||
>>> [ToolCall(tool_name='get_current_weather',
|
||||
>>> arguments={'unit': 'celsius', 'location': 'Berlin'}, id=None)]
|
||||
|
||||
tool_messages = tool_invoker.run(messages=replies)["tool_messages"]
|
||||
messages = user_message + replies + tool_messages
|
||||
|
||||
messages += res["replies"][0] + [ChatMessage.from_function(content=weather, name="get_current_weather")]
|
||||
|
||||
final_replies = genai_chat.run(messages=messages)["replies"]
|
||||
print(final_replies[0].text)
|
||||
>>> The temperature in Berlin is 20 degrees Celsius.
|
||||
```
|
||||
|
||||
#### With Streaming
|
||||
|
||||
```python
|
||||
from haystack.dataclasses.chat_message import ChatMessage
|
||||
from haystack.dataclasses import StreamingChunk
|
||||
from haystack_integrations.components.generators.google_genai import (
|
||||
GoogleGenAIChatGenerator,
|
||||
)
|
||||
|
||||
|
||||
def streaming_callback(chunk: StreamingChunk):
|
||||
print(chunk.content, end="", flush=True)
|
||||
|
||||
|
||||
## Initialize with streaming callback
|
||||
chat_generator = GoogleGenAIChatGenerator(streaming_callback=streaming_callback)
|
||||
|
||||
## Generate a streaming response
|
||||
messages = [ChatMessage.from_user("Write a short story")]
|
||||
response = chat_generator.run(messages=messages)
|
||||
## Text will stream in real-time through the callback
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
import os
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack import Pipeline
|
||||
from haystack_integrations.components.generators.google_genai import (
|
||||
GoogleGenAIChatGenerator,
|
||||
)
|
||||
|
||||
## no parameter init, we don't use any runtime template variables
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
|
||||
os.environ["GOOGLE_API_KEY"] = "<MY_API_KEY>"
|
||||
genai_chat = GoogleGenAIChatGenerator()
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("genai", genai_chat)
|
||||
pipe.connect("prompt_builder.prompt", "genai.messages")
|
||||
|
||||
location = "Rome"
|
||||
messages = [ChatMessage.from_user("Tell me briefly about {{location}} history")]
|
||||
res = pipe.run(
|
||||
data={
|
||||
"prompt_builder": {
|
||||
"template_variables": {"location": location},
|
||||
"template": messages,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
print(res)
|
||||
```
|
||||
+206
@@ -0,0 +1,206 @@
|
||||
---
|
||||
title: "Choosing the Right Generator"
|
||||
id: choosing-the-right-generator
|
||||
slug: "/choosing-the-right-generator"
|
||||
description: "This page provides information on choosing the right Generator for interacting with Generative Language Models in Haystack. It explains the distinction between Generators and ChatGenerators, discusses using proprietary and open models from various providers, and explores options for using open models on-premise."
|
||||
---
|
||||
|
||||
# Choosing the Right Generator
|
||||
|
||||
This page provides information on choosing the right Generator for interacting with Generative Language Models in Haystack. It explains the distinction between Generators and ChatGenerators, discusses using proprietary and open models from various providers, and explores options for using open models on-premise.
|
||||
|
||||
In Haystack, Generators are the main interface for interacting with Generative Language Models.
|
||||
This guide aims to simplify the process of choosing the right Generator based on your preferences and computing resources. This guide does not focus on selecting a specific model itself but rather a model type and a Haystack Generator: as you will see, in several cases, you have different options to use the same model.
|
||||
|
||||
## Generators vs ChatGenerators
|
||||
|
||||
The first distinction we are talking about is between Generators and ChatGenerators, for example, OpenAIGenerator and OpenAIChatGenerator, HuggingFaceAPIGenerator and HuggingFaceAPIChatGenerator, and so on.
|
||||
|
||||
- **Generators** are components that expect a prompt (a string) and return the generated text in “replies”.
|
||||
- **ChatGenerators** support the [ChatMessage data class](../../../concepts/data-classes/chatmessage.mdx) out of the box. They expect a list of Chat Messages and return a Chat Message in “replies”.
|
||||
|
||||
:::warning
|
||||
ChatGenerators are way more powerful than Generators: they for example support Function Calling and Multimodal inputs.
|
||||
|
||||
We recommend using ChatGenerators. Generators might be removed in a future major version of Haystack.
|
||||
:::
|
||||
|
||||
## Streaming Support
|
||||
|
||||
Streaming refers to outputting LLM responses word by word rather than waiting for the entire response to be generated before outputting everything at once.
|
||||
|
||||
You can check which Generators have streaming support on the [Generators overview page](../../generators.mdx).
|
||||
|
||||
When you enable streaming, the generator calls your `streaming_callback` for every `StreamingChunk`. Each chunk represents exactly one of the following:
|
||||
|
||||
- **Tool calls**: The model is building a tool/function call. Read `chunk.tool_calls`.
|
||||
- **Tool result**: A tool finished and returned output. Read `chunk.tool_call_result`.
|
||||
- **Text tokens**: Normal assistant text. Read `chunk.content`.
|
||||
- **Reasoning tokens**: Extended thinking output (for models that support it). Read `chunk.reasoning`.
|
||||
|
||||
Only one of these fields appears per chunk. Use `chunk.start` and `chunk.finish_reason` to detect boundaries. Use `chunk.index` and `chunk.component_info` for tracing.
|
||||
|
||||
For providers that support multiple candidates, set `n=1` to stream.
|
||||
|
||||
:::info[Parameter Details]
|
||||
|
||||
Check out the parameter details in our [API Reference for StreamingChunk](/reference/data-classes-api#streamingchunk).
|
||||
:::
|
||||
|
||||
The simplest way is to use the built-in `print_streaming_chunk` function. It handles all chunk types and prints formatted output to stdout:
|
||||
|
||||
```python
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
|
||||
generator = SomeGenerator(streaming_callback=print_streaming_chunk)
|
||||
# For ChatGenerators, pass a list[ChatMessage]. For text generators, pass a prompt string.
|
||||
```
|
||||
|
||||
### Custom Callback
|
||||
|
||||
If you need custom rendering, write your own callback. Handle the four chunk types in order:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import StreamingChunk
|
||||
|
||||
|
||||
def my_streaming_callback(chunk: StreamingChunk) -> None:
|
||||
if chunk.start and chunk.index and chunk.index > 0:
|
||||
print("\n\n", flush=True, end="")
|
||||
|
||||
# Tool Call streaming
|
||||
if chunk.tool_calls:
|
||||
for tool_call in chunk.tool_calls:
|
||||
if chunk.start:
|
||||
if chunk.index and tool_call.index > chunk.index:
|
||||
print("\n\n", flush=True, end="")
|
||||
print(
|
||||
f">>> Tool Call: {tool_call.tool_name}\n>>> Arguments: ",
|
||||
flush=True,
|
||||
end="",
|
||||
)
|
||||
if tool_call.arguments:
|
||||
print(tool_call.arguments, flush=True, end="")
|
||||
|
||||
# Tool Result streaming
|
||||
if chunk.tool_call_result:
|
||||
print(f">>> Tool Result\n{chunk.tool_call_result.result}", flush=True, end="")
|
||||
|
||||
# Text streaming
|
||||
if chunk.content:
|
||||
if chunk.start:
|
||||
print(">>> Assistant\n", flush=True, end="")
|
||||
print(chunk.content, flush=True, end="")
|
||||
|
||||
# Reasoning streaming
|
||||
if chunk.reasoning:
|
||||
if chunk.start:
|
||||
print(">>> Reasoning\n", flush=True, end="")
|
||||
print(chunk.reasoning.reasoning_text, flush=True, end="")
|
||||
|
||||
if chunk.finish_reason is not None:
|
||||
print("\n\n", flush=True, end="")
|
||||
```
|
||||
|
||||
### Agents and Tools
|
||||
|
||||
Agents and `ToolInvoker` forward your `streaming_callback`. They also emit a final tool-result chunk with a `finish_reason` so UIs can close the “tool phase” cleanly before assistant text resumes. The default `print_streaming_chunk` formats this for you.
|
||||
|
||||
## Proprietary vs Open-weights Models
|
||||
|
||||
Before choosing a Generator, it helps to know which type of model you want to use.
|
||||
|
||||
### Proprietary Models
|
||||
|
||||
Using proprietary models is a quick way to start with Generative Language Models. The typical approach involves calling these hosted models using an API Key. You are paying based on the number of tokens, both sent and generated.
|
||||
You don’t need significant resources on your local machine, as the computation is executed on the provider’s infrastructure. When using these models, your data exits your machine and is transmitted to the model provider.
|
||||
|
||||
### Open-weights Models
|
||||
|
||||
When discussing open (weights) models, we're referring to models with public weights that anyone can deploy on their infrastructure. The datasets used for training are shared less frequently. One could choose to use an open model for several reasons, including more transparency and control of the model.
|
||||
|
||||
:::info[Commercial Use]
|
||||
|
||||
Not all open models are suitable for commercial use. We advise thoroughly reviewing the license, typically available on Hugging Face, before considering their adoption.
|
||||
:::
|
||||
|
||||
Even if the model is open, you might still want to rely on model providers to use it, mostly because you want someone else to host the model and take care of the infrastructural aspects. In these scenarios, your data transitions from your machine to the provider facilitating the model.
|
||||
|
||||
## Where the Model runs
|
||||
|
||||
Where the model runs is a separate decision from the proprietary-vs-open one: a proprietary model is always provider-hosted, but an open-weights model can be served in any of the ways described below.
|
||||
|
||||
The Generator you pick is mostly determined by where the model runs and which API you call. The costs associated with these solutions can vary. Depending on the solution you choose, you pay for the tokens consumed, both sent and generated or for the hosting of the model, often billed per hour.
|
||||
|
||||
### Provider-hosted APIs
|
||||
|
||||
With provider-hosted APIs, you leverage an instance of the model shared with other users, with payment typically based on consumed tokens, both sent and generated.
|
||||
|
||||
#### Single-vendor APIs
|
||||
|
||||
These providers host their own models behind a dedicated API. Haystack supports the models offered by a variety of providers: OpenAI, Azure, Google, Cohere, and Mistral, with more being added constantly.
|
||||
|
||||
#### Multi-model Gateways
|
||||
|
||||
Several providers expose many models through a single API, so one Generator lets you switch between models from different vendors. Some of these providers focus on open-weights models, while others also include proprietary ones:
|
||||
|
||||
- [Amazon Bedrock](../amazonbedrockgenerator.mdx) provides access to proprietary models from the Amazon Titan family, AI21 Labs, Anthropic, and Cohere, plus several open models, such as Llama from Meta.
|
||||
- [Hugging Face Inference Providers](https://huggingface.co/docs/inference-providers/index), available through the Hugging Face API Generators, give access to hundreds of LLMs from different providers through a unified interface.
|
||||
- [AIMLAPI](../aimllapichatgenerator.mdx), [Comet API](../cometapichatgenerator.mdx), [NVIDIA](../nvidiachatgenerator.mdx), [OpenRouter](../openrouterchatgenerator.mdx), [STACKIT](../stackitchatgenerator.mdx), [Together AI](../togetheraichatgenerator.mdx), and [WatsonX](../watsonxchatgenerator.mdx) each have a dedicated Haystack integration.
|
||||
- DeepInfra, Fireworks, and other cloud providers offer OpenAI-compatible interfaces and can be used through the OpenAI Generators.
|
||||
|
||||
Here is an example using DeepInfra and [`OpenAIChatGenerator`](../openaichatgenerator.mdx):
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.utils import Secret
|
||||
|
||||
generator = OpenAIChatGenerator(
|
||||
api_key=Secret.from_env_var("ENVVAR_WITH_API_KEY"),
|
||||
api_base_url="https://api.deepinfra.com/v1",
|
||||
model="Qwen/Qwen3.6-35B-A3B",
|
||||
)
|
||||
|
||||
generator.run(messages=[ChatMessage.from_user("What is the best French cheese?")])
|
||||
```
|
||||
|
||||
### Dedicated Cloud Instances
|
||||
|
||||
In this case, a private instance of the model is deployed by the provider, and you typically pay per hour.
|
||||
|
||||
Here are the components that support this in Haystack:
|
||||
|
||||
- Amazon [SagemakerGenerator](../sagemakergenerator.mdx)
|
||||
- HuggingFace API Generators, when used to query [HuggingFace Inference endpoints](https://huggingface.co/inference-endpoints).
|
||||
|
||||
### Provider-hosted API vs Dedicated Cloud Instance
|
||||
|
||||
**Why choose a provider-hosted API:**
|
||||
|
||||
- Cost Savings: Access cost-effective solutions especially suitable for users with varying usage patterns or limited budgets.
|
||||
- Ease of Use: Setup and maintenance are simplified as the provider manages the infrastructure and updates, making it user-friendly.
|
||||
|
||||
**Why choose a dedicated cloud instance:**
|
||||
|
||||
- Dedicated Resources: Ensure consistent performance with dedicated resources for your instance and avoid any impact from other users.
|
||||
- Scalability: Scale resources based on requirements while ensuring optimal performance during peak times and cost savings during off-peak hours.
|
||||
- Predictable Costs: Billing per hour leads to more predictable costs, especially when there is a clear understanding of usage patterns.
|
||||
|
||||
### Self-hosted / On-premise
|
||||
|
||||
On-premise models mean that you host open models on your machine or infrastructure. This is ideal for local experimentation, and also suitable in production scenarios where data privacy concerns prevent sending data to external providers, provided you have ample computational resources.
|
||||
|
||||
#### Local Experimentation
|
||||
|
||||
- GPU: [`HuggingFaceLocalChatGenerator`](../huggingfacelocalchatgenerator.mdx) is based on the Hugging Face Transformers library. This is good for experimentation when you have some GPU resources (for example, in Colab). If GPU resources are limited, alternative quantization options like bitsandbytes, GPTQ, and AWQ are supported. For more performant solutions in production use cases, refer to the options below.
|
||||
- CPU (+ GPU if available): [`LlamaCppChatGenerator`](../llamacppchatgenerator.mdx) uses the Llama.cpp library – a project written in C/C++ for efficient inference of LLMs. In particular, it employs the quantized GGUF format, suitable for running these models on standard machines (even without GPUs). If GPU resources are available, some model layers can be offloaded to GPU for enhanced speed.
|
||||
- CPU (+ GPU if available): [`OllamaChatGenerator`](../ollamachatgenerator.mdx) is based on the Ollama project, acting like Docker for LLMs. It provides a simple way to package and deploy these models. Internally based on the Llama.cpp library, it offers a more streamlined process for running on various platforms.
|
||||
|
||||
#### Serving LLMs in Production
|
||||
|
||||
The following solutions are suitable if you want to run Language Models in production and have GPU resources available. They use innovative techniques for fast inference and efficient handling of numerous concurrent requests.
|
||||
|
||||
- vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs. Haystack supports vLLM through [vLLMChatGenerator](../vllmchatgenerator.mdx).
|
||||
- SGLang is a similar high-performance LLM serving framework. Haystack supports it through the OpenAI Generators.
|
||||
- Hugging Face API Generators, when used to query a TGI instance deployed on-premise. Hugging Face Text Generation Inference is a toolkit for efficiently deploying and serving LLMs. **This project is now in maintenance mode**.
|
||||
+124
@@ -0,0 +1,124 @@
|
||||
---
|
||||
title: "Function Calling"
|
||||
id: function-calling
|
||||
slug: "/function-calling"
|
||||
description: "Learn about function calling and how to use it in Haystack."
|
||||
---
|
||||
|
||||
# Function Calling
|
||||
|
||||
Learn about function calling and how to use it in Haystack.
|
||||
|
||||
Function calling is a powerful feature that significantly enhances the capabilities of Large Language Models (LLMs). It enables better functionality, immediate data access, and interaction, and sets up for integration with external APIs and services. Function calling turns LLMs into adaptable tools for various use case scenarios.
|
||||
|
||||
## Use Cases
|
||||
|
||||
Function calling is useful for a variety of purposes, but two main points are particularly notable:
|
||||
|
||||
1. **Enhanced LLM Functionality**: Function calling enhances the capabilities of LLMs beyond just text generation. It allows to convert human-generated prompts into precise function invocation descriptors. These descriptors can then be used by connected LLM frameworks to perform computations, manipulate data, and interact with external APIs. This expansion of functionality makes LLMs adaptable tools for a wide array of tasks and industries.
|
||||
2. **Real-Time Data Access and Interaction**: Function calling lets LLMs create function calls that access and interact with real-time data. This is necessary for apps that need current data, like news, weather, or financial market updates. By giving access to the latest information, this feature greatly improves the usefulness and trustworthiness of LLMs in changing and time-critical situations.
|
||||
|
||||
:::note[Important Note]
|
||||
|
||||
The model doesn't actually call the function. Function calling returns the name of a function and the arguments to invoke it. The actual invocation is performed by your code (or by a Haystack component such as `ToolInvoker` or `Agent`).
|
||||
:::
|
||||
|
||||
## Example
|
||||
|
||||
Let's walk through function calling in Haystack in three steps: define a tool, let the LLM pick it, and then actually invoke it.
|
||||
|
||||
:::tip[Real-world usage]
|
||||
|
||||
We split function calling into separate steps below for clarity. In real applications, the [`Agent`](../../agents-1/agent.mdx) component handles the full loop for you. See [Using the Agent component](#using-the-agent-component) at the end of this section.
|
||||
:::
|
||||
|
||||
### 1. Define a Tool
|
||||
|
||||
The simplest way to expose a Python function to an LLM is the `@tool` decorator. Type hints (and `Annotated` metadata) are used to automatically build the JSON schema the LLM needs.
|
||||
|
||||
```python
|
||||
from typing import Annotated, Literal
|
||||
from haystack.tools import tool
|
||||
|
||||
|
||||
@tool
|
||||
def get_weather(
|
||||
city: Annotated[str, "The city to get the weather for"],
|
||||
unit: Annotated[Literal["celsius", "fahrenheit"], "Temperature unit"] = "celsius",
|
||||
):
|
||||
"""Get the current weather for a city."""
|
||||
# In a real application, this would call a weather API.
|
||||
return {"city": city, "temperature": 18, "unit": unit, "condition": "Partly Cloudy"}
|
||||
```
|
||||
|
||||
This produces a [`Tool`](../../../tools/tool.mdx) instance you can pass directly to a ChatGenerator. If you prefer to build it explicitly, you can also instantiate `Tool(...)` yourself.
|
||||
|
||||
### 2. Let the LLM Pick the Tool
|
||||
|
||||
Pass the tool to a ChatGenerator and run it on a user message. The model decides whether to call the tool and, if so, with which arguments. The result lives on `replies[0].tool_calls`.
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
generator = OpenAIChatGenerator(tools=[get_weather])
|
||||
response = generator.run(
|
||||
messages=[ChatMessage.from_user("What's the weather like in Berlin?")],
|
||||
)
|
||||
|
||||
print(response["replies"][0].tool_calls)
|
||||
# >> [ToolCall(tool_name='get_weather', arguments={'city': 'Berlin', 'unit': 'celsius'}, id='call_...')]
|
||||
```
|
||||
|
||||
At this point the model has only *requested* a call. Nothing has been executed yet.
|
||||
|
||||
### 3. Actually Invoke the Tool
|
||||
|
||||
To execute the requested tool calls, use the [`ToolInvoker`](../../tools/toolinvoker.mdx) component. It takes a list of [`ChatMessage`](../../../concepts/data-classes/chatmessage.mdx) objects containing tool calls, runs the corresponding tools, and returns new `ChatMessage` objects (with role `tool`) carrying the results as `ToolCallResult`s. To get a final natural-language answer, feed those results back to the ChatGenerator.
|
||||
|
||||
```python
|
||||
from haystack.components.tools import ToolInvoker
|
||||
|
||||
invoker = ToolInvoker(tools=[get_weather])
|
||||
|
||||
# Run the tools requested by the assistant.
|
||||
tool_messages = invoker.run(messages=response["replies"])["tool_messages"]
|
||||
|
||||
# Send the full conversation back to the model for the final reply.
|
||||
messages = [
|
||||
ChatMessage.from_user("What's the weather like in Berlin?"),
|
||||
*response["replies"], # assistant message with the tool call
|
||||
*tool_messages, # tool result messages
|
||||
]
|
||||
final = generator.run(messages=messages)
|
||||
print(final["replies"][0].text)
|
||||
# The weather in Berlin is partly cloudy with a temperature of 18°C.
|
||||
```
|
||||
|
||||
#### Using the Agent component
|
||||
|
||||
In real applications, you typically don't manage the loop yourself: the [`Agent`](../../agents-1/agent.mdx) component does it for you. Internally, it wraps a ChatGenerator and a `ToolInvoker` and iterates until the model produces a final answer.
|
||||
|
||||
```python
|
||||
from haystack.components.agents import Agent
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
agent = Agent(
|
||||
chat_generator=OpenAIChatGenerator(),
|
||||
tools=[get_weather],
|
||||
)
|
||||
|
||||
response = agent.run(
|
||||
messages=[ChatMessage.from_user("What's the weather like in Berlin?")],
|
||||
)
|
||||
print(response["messages"][-1].text)
|
||||
# The weather in Berlin is partly cloudy with a temperature of 18°C.
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
- [`Tool`](../../../tools/tool.mdx), [`Toolset`](../../../tools/toolset.mdx), and [`ComponentTool`](../../../tools/componenttool.mdx) cover the building blocks for defining tools, grouping them, and exposing existing Haystack components as tools.
|
||||
- To connect an LLM to external services or other Haystack applications, the recommended approach is the [Model Context Protocol (MCP)](https://modelcontextprotocol.io/). Haystack provides [`MCPTool`](../../../tools/mcptool.mdx) and [`MCPToolset`](../../../tools/mcptoolset.mdx) for connecting to MCP servers over Streamable HTTP or stdio. Install them with `pip install mcp-haystack`.
|
||||
|
||||
:notebook: **Tutorial:** [Building a Chat Application with Function Calling](https://haystack.deepset.ai/tutorials/40_building_chat_application_with_function_calling)
|
||||
+225
@@ -0,0 +1,225 @@
|
||||
---
|
||||
title: "HuggingFaceAPIChatGenerator"
|
||||
id: huggingfaceapichatgenerator
|
||||
slug: "/huggingfaceapichatgenerator"
|
||||
description: "This generator enables chat completion using various Hugging Face APIs."
|
||||
---
|
||||
|
||||
# HuggingFaceAPIChatGenerator
|
||||
|
||||
This generator enables chat completion using various Hugging Face APIs.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`ChatPromptBuilder`](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_type`: The type of Hugging Face API to use <br /> <br />`api_params`: A dictionary with one of the following keys: <br /> <br />- `model`: Hugging Face model ID. Required when `api_type` is `SERVERLESS_INFERENCE_API`.**OR** - `url`: URL of the inference endpoint. Required when `api_type` is `INFERENCE_ENDPOINTS` or `TEXT_EMBEDDINGS_INFERENCE`.`token`: The Hugging Face API token. Can be set with `HF_API_TOKEN` or `HF_TOKEN` env var. |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects representing the chat |
|
||||
| **Output variables** | `replies`: A list of replies of the LLM to the input chat |
|
||||
| **API reference** | [Generators](/reference/generators-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/generators/chat/hugging_face_api.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`HuggingFaceAPIChatGenerator` can be used to generate chat completions using different Hugging Face APIs:
|
||||
|
||||
- [Serverless Inference API (Inference Providers)](https://huggingface.co/docs/inference-providers) - free tier available
|
||||
- [Paid Inference Endpoints](https://huggingface.co/inference-endpoints)
|
||||
- [Self-hosted Text Generation Inference](https://github.com/huggingface/text-generation-inference)
|
||||
|
||||
This component's main input is a list of `ChatMessage` objects. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata. For more information, check out our [`ChatMessage` docs](../../concepts/data-classes/chatmessage.mdx).
|
||||
|
||||
:::info
|
||||
This component is designed for chat completion, so it expects a list of messages, not a single string. If you want to use Hugging Face APIs for simple text generation (such as translation or summarization tasks) or don't want to use the `ChatMessage` object, use [`HuggingFaceAPIGenerator`](huggingfaceapigenerator.mdx) instead.
|
||||
:::
|
||||
|
||||
The component uses a `HF_API_TOKEN` environment variable by default. Otherwise, you can pass a Hugging Face API token at initialization with `token` – see code examples below.
|
||||
The token is needed:
|
||||
|
||||
- If you use the Serverless Inference API, or
|
||||
- If you use the Inference Endpoints.
|
||||
|
||||
### Streaming
|
||||
|
||||
This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
#### Using Serverless Inference API (Inference Providers) - Free Tier Available
|
||||
|
||||
This API allows you to quickly experiment with many models hosted on the Hugging Face Hub, offloading the inference to Hugging Face servers. It's rate-limited and not meant for production.
|
||||
|
||||
To use this API, you need a [free Hugging Face token](https://huggingface.co/settings/tokens).
|
||||
The Generator expects the `model` in `api_params`. It's also recommended to specify a `provider` for better performance and reliability.
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import HuggingFaceAPIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.utils import Secret
|
||||
from haystack.utils.hf import HFGenerationAPIType
|
||||
|
||||
messages = [
|
||||
ChatMessage.from_system("\\nYou are a helpful, respectful and honest assistant"),
|
||||
ChatMessage.from_user("What's Natural Language Processing?"),
|
||||
]
|
||||
|
||||
## the api_type can be expressed using the HFGenerationAPIType enum or as a string
|
||||
api_type = HFGenerationAPIType.SERVERLESS_INFERENCE_API
|
||||
api_type = "serverless_inference_api" # this is equivalent to the above
|
||||
|
||||
generator = HuggingFaceAPIChatGenerator(
|
||||
api_type=api_type,
|
||||
api_params={"model": "Qwen/Qwen2.5-7B-Instruct", "provider": "together"},
|
||||
token=Secret.from_env_var("HF_API_TOKEN"),
|
||||
)
|
||||
|
||||
result = generator.run(messages)
|
||||
print(result)
|
||||
```
|
||||
|
||||
#### Using Paid Inference Endpoints
|
||||
|
||||
In this case, a private instance of the model is deployed by Hugging Face, and you typically pay per hour.
|
||||
|
||||
To understand how to spin up an Inference Endpoint, visit [Hugging Face documentation](https://huggingface.co/inference-endpoints/dedicated).
|
||||
|
||||
Additionally, in this case, you need to provide your Hugging Face token.
|
||||
The Generator expects the `url` of your endpoint in `api_params`.
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import HuggingFaceAPIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.utils import Secret
|
||||
|
||||
messages = [
|
||||
ChatMessage.from_system("\\nYou are a helpful, respectful and honest assistant"),
|
||||
ChatMessage.from_user("What's Natural Language Processing?"),
|
||||
]
|
||||
|
||||
generator = HuggingFaceAPIChatGenerator(
|
||||
api_type="inference_endpoints",
|
||||
api_params={"url": "<your-inference-endpoint-url>"},
|
||||
token=Secret.from_env_var("HF_API_TOKEN"),
|
||||
)
|
||||
|
||||
result = generator.run(messages)
|
||||
print(result)
|
||||
```
|
||||
|
||||
#### Using Serverless Inference API (Inference Providers) with Text+Image Input
|
||||
|
||||
You can also use this component with multimodal models that support both text and image input:
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import HuggingFaceAPIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage, ImageContent
|
||||
from haystack.utils import Secret
|
||||
from haystack.utils.hf import HFGenerationAPIType
|
||||
|
||||
## Create an image from file path, URL, or base64
|
||||
image = ImageContent.from_file_path("path/to/your/image.jpg")
|
||||
|
||||
## Create a multimodal message with both text and image
|
||||
messages = [
|
||||
ChatMessage.from_user(content_parts=["Describe this image in detail", image]),
|
||||
]
|
||||
|
||||
generator = HuggingFaceAPIChatGenerator(
|
||||
api_type=HFGenerationAPIType.SERVERLESS_INFERENCE_API,
|
||||
api_params={
|
||||
"model": "Qwen/Qwen2.5-VL-7B-Instruct", # Vision Language Model
|
||||
"provider": "hyperbolic",
|
||||
},
|
||||
token=Secret.from_token("<your-api-key>"),
|
||||
)
|
||||
|
||||
result = generator.run(messages)
|
||||
print(result)
|
||||
```
|
||||
|
||||
#### Using Self-Hosted Text Generation Inference (TGI)
|
||||
|
||||
[Hugging Face Text Generation Inference](https://github.com/huggingface/text-generation-inference) is a toolkit for efficiently deploying and serving LLMs.
|
||||
|
||||
While it powers the most recent versions of Serverless Inference API and Inference Endpoints, it can be used easily on-premise through Docker.
|
||||
|
||||
For example, you can run a TGI container as follows:
|
||||
|
||||
```shell
|
||||
model=HuggingFaceH4/zephyr-7b-beta
|
||||
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
|
||||
|
||||
docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:1.4 --model-id $model
|
||||
```
|
||||
|
||||
For more information, refer to the [official TGI repository](https://github.com/huggingface/text-generation-inference).
|
||||
|
||||
The Generator expects the `url` of your TGI instance in `api_params`.
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import HuggingFaceAPIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
messages = [
|
||||
ChatMessage.from_system("\\nYou are a helpful, respectful and honest assistant"),
|
||||
ChatMessage.from_user("What's Natural Language Processing?"),
|
||||
]
|
||||
|
||||
generator = HuggingFaceAPIChatGenerator(
|
||||
api_type="text_generation_inference",
|
||||
api_params={"url": "http://localhost:8080"},
|
||||
)
|
||||
|
||||
result = generator.run(messages)
|
||||
print(result)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import HuggingFaceAPIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack import Pipeline
|
||||
from haystack.utils import Secret
|
||||
from haystack.utils.hf import HFGenerationAPIType
|
||||
|
||||
## no parameter init, we don't use any runtime template variables
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
llm = HuggingFaceAPIChatGenerator(
|
||||
api_type=HFGenerationAPIType.SERVERLESS_INFERENCE_API,
|
||||
api_params={"model": "Qwen/Qwen2.5-7B-Instruct", "provider": "together"},
|
||||
token=Secret.from_env_var("HF_API_TOKEN"),
|
||||
)
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
location = "Berlin"
|
||||
messages = [
|
||||
ChatMessage.from_system(
|
||||
"Always respond in German even if some input data is in other languages.",
|
||||
),
|
||||
ChatMessage.from_user("Tell me about {{location}}"),
|
||||
]
|
||||
result = pipe.run(
|
||||
data={
|
||||
"prompt_builder": {
|
||||
"template_variables": {"location": location},
|
||||
"template": messages,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
print(result)
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Build with Google Gemma: chat and RAG](https://haystack.deepset.ai/cookbook/gemma_chat_rag)
|
||||
+190
@@ -0,0 +1,190 @@
|
||||
---
|
||||
title: "HuggingFaceAPIGenerator"
|
||||
id: huggingfaceapigenerator
|
||||
slug: "/huggingfaceapigenerator"
|
||||
description: "This generator enables text generation using various Hugging Face APIs."
|
||||
---
|
||||
|
||||
# HuggingFaceAPIGenerator
|
||||
|
||||
This generator enables text generation using various Hugging Face APIs.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`PromptBuilder`](../builders/promptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_type`: The type of Hugging Face API to use <br /> <br />`api_params`: A dictionary with one of the following keys: <br /> <br />- `model`: Hugging Face model ID. Required when `api_type` is `SERVERLESS_INFERENCE_API`.**OR** - `url`: URL of the inference endpoint. Required when `api_type` is `INFERENCE_ENDPOINTS` or `TEXT_EMBEDDINGS_INFERENCE`.`token`: The Hugging Face API token. Can be set with `HF_API_TOKEN` or `HF_TOKEN` env var. |
|
||||
| **Mandatory run variables** | `prompt`: A string containing the prompt for the LLM |
|
||||
| **Output variables** | `replies`: A list of strings with all the replies generated by the LLM <br /> <br />`meta`: A list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and others |
|
||||
| **API reference** | [Generators](/reference/generators-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/generators/hugging_face_api.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`HuggingFaceAPIGenerator` can be used to generate text using different Hugging Face APIs:
|
||||
|
||||
- [Paid Inference Endpoints](https://huggingface.co/inference-endpoints)
|
||||
- [Self-hosted Text Generation Inference](https://github.com/huggingface/text-generation-inference)
|
||||
|
||||
:::note[Important Note]
|
||||
|
||||
As of July 2025, the Hugging Face Inference API no longer offers generative models through the `text_generation` endpoint. Generative models are now only available through providers supporting the `chat_completion` endpoint. As a result, this component might no longer work with the Hugging Face Inference API.
|
||||
|
||||
Use the [`HuggingFaceAPIChatGenerator`](huggingfaceapichatgenerator.mdx) component instead, which supports the `chat_completion` endpoint and works with the free Serverless Inference API.
|
||||
:::
|
||||
|
||||
:::info
|
||||
This component is designed for text generation, not for chat. If you want to use these LLMs for chat, use [`HuggingFaceAPIChatGenerator`](huggingfaceapichatgenerator.mdx) instead.
|
||||
:::
|
||||
|
||||
The component uses a `HF_API_TOKEN` environment variable by default. Otherwise, you can pass a Hugging Face API token at initialization with `token` – see code examples below.
|
||||
The token is needed when you use the Inference Endpoints.
|
||||
|
||||
### Streaming
|
||||
|
||||
This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
#### Using Paid Inference Endpoints
|
||||
|
||||
In this case, a private instance of the model is deployed by Hugging Face, and you typically pay per hour.
|
||||
|
||||
To understand how to spin up an Inference Endpoint, visit [Hugging Face documentation](https://huggingface.co/inference-endpoints/dedicated).
|
||||
|
||||
Additionally, in this case, you need to provide your Hugging Face token.
|
||||
The Generator expects the `url` of your endpoint in `api_params`.
|
||||
|
||||
```python
|
||||
from haystack.components.generators import HuggingFaceAPIGenerator
|
||||
from haystack.utils import Secret
|
||||
|
||||
generator = HuggingFaceAPIGenerator(
|
||||
api_type="inference_endpoints",
|
||||
api_params={"url": "<your-inference-endpoint-url>"},
|
||||
token=Secret.from_token("<your-api-key>"),
|
||||
)
|
||||
|
||||
result = generator.run(prompt="What's Natural Language Processing?")
|
||||
print(result)
|
||||
```
|
||||
|
||||
#### Using Self-Hosted Text Generation Inference (TGI)
|
||||
|
||||
[Hugging Face Text Generation Inference](https://github.com/huggingface/text-generation-inference) is a toolkit for efficiently deploying and serving LLMs.
|
||||
|
||||
While it powers the most recent versions of Serverless Inference API and Inference Endpoints, it can be used easily on-premise through Docker.
|
||||
|
||||
For example, you can run a TGI container as follows:
|
||||
|
||||
```shell
|
||||
model=mistralai/Mistral-7B-v0.1
|
||||
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
|
||||
|
||||
docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:1.4 --model-id $model
|
||||
```
|
||||
|
||||
For more information, refer to the [official TGI repository](https://github.com/huggingface/text-generation-inference).
|
||||
|
||||
The Generator expects the `url` of your TGI instance in `api_params`.
|
||||
|
||||
```python
|
||||
from haystack.components.generators import HuggingFaceAPIGenerator
|
||||
|
||||
generator = HuggingFaceAPIGenerator(
|
||||
api_type="text_generation_inference",
|
||||
api_params={"url": "http://localhost:8080"},
|
||||
)
|
||||
|
||||
result = generator.run(prompt="What's Natural Language Processing?")
|
||||
print(result)
|
||||
```
|
||||
|
||||
#### Using the Free Serverless Inference API (Not Recommended)
|
||||
|
||||
:::warning
|
||||
This example might not work as the Hugging Face Inference API no longer offers models that support the `text_generation` endpoint. Use the [`HuggingFaceAPIChatGenerator`](huggingfaceapichatgenerator.mdx) for generative models through the `chat_completion` endpoint.
|
||||
|
||||
:::
|
||||
|
||||
Formerly known as (free) Hugging Face Inference API, this API allows you to quickly experiment with many models hosted on the Hugging Face Hub, offloading the inference to Hugging Face servers. It's rate-limited and not meant for production.
|
||||
|
||||
To use this API, you need a [free Hugging Face token](https://huggingface.co/settings/tokens).
|
||||
The Generator expects the `model` in `api_params`.
|
||||
|
||||
```python
|
||||
from haystack.components.generators import HuggingFaceAPIGenerator
|
||||
from haystack.utils import Secret
|
||||
|
||||
generator = HuggingFaceAPIGenerator(
|
||||
api_type="serverless_inference_api",
|
||||
api_params={"model": "HuggingFaceH4/zephyr-7b-beta"},
|
||||
token=Secret.from_token("<your-api-key>"),
|
||||
)
|
||||
|
||||
result = generator.run(prompt="What's Natural Language Processing?")
|
||||
print(result)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.components.builders.prompt_builder import PromptBuilder
|
||||
from haystack.components.generators import HuggingFaceAPIGenerator
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack import Document
|
||||
|
||||
docstore = InMemoryDocumentStore()
|
||||
docstore.write_documents(
|
||||
[
|
||||
Document(content="Rome is the capital of Italy"),
|
||||
Document(content="Paris is the capital of France"),
|
||||
],
|
||||
)
|
||||
|
||||
query = "What is the capital of France?"
|
||||
|
||||
template = """
|
||||
Given the following information, answer the question.
|
||||
|
||||
Context:
|
||||
{% for document in documents %}
|
||||
{{ document.content }}
|
||||
{% endfor %}
|
||||
|
||||
Question: {{ query }}?
|
||||
"""
|
||||
|
||||
generator = HuggingFaceAPIGenerator(
|
||||
api_type="inference_endpoints",
|
||||
api_params={"url": "<your-inference-endpoint-url>"},
|
||||
token=Secret.from_token("<your-api-key>"),
|
||||
)
|
||||
|
||||
pipe = Pipeline()
|
||||
|
||||
pipe.add_component("retriever", InMemoryBM25Retriever(document_store=docstore))
|
||||
pipe.add_component("prompt_builder", PromptBuilder(template=template))
|
||||
pipe.add_component("llm", generator)
|
||||
pipe.connect("retriever", "prompt_builder.documents")
|
||||
pipe.connect("prompt_builder", "llm")
|
||||
|
||||
res = pipe.run({"prompt_builder": {"query": query}, "retriever": {"query": query}})
|
||||
|
||||
print(res)
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbooks:
|
||||
|
||||
- [Multilingual RAG from a podcast with Whisper, Qdrant and Mistral](https://haystack.deepset.ai/cookbook/multilingual_rag_podcast)
|
||||
- [Information Extraction with Raven](https://haystack.deepset.ai/cookbook/information_extraction_raven)
|
||||
- [Web QA with Mixtral-8x7B-Instruct-v0.1](https://haystack.deepset.ai/cookbook/mixtral-8x7b-for-web-qa)
|
||||
+92
@@ -0,0 +1,92 @@
|
||||
---
|
||||
title: "HuggingFaceLocalChatGenerator"
|
||||
id: huggingfacelocalchatgenerator
|
||||
slug: "/huggingfacelocalchatgenerator"
|
||||
description: "Provides an interface for chat completion using a Hugging Face model that runs locally."
|
||||
---
|
||||
|
||||
# HuggingFaceLocalChatGenerator
|
||||
|
||||
Provides an interface for chat completion using a Hugging Face model that runs locally.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `token`: The Hugging Face API token. Can be set with `HF_API_TOKEN` or `HF_TOKEN` env var. |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects representing the chat |
|
||||
| **Output variables** | `replies`: A list of strings with all the replies generated by the LLM |
|
||||
| **API reference** | [Generators](/reference/generators-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/generators/chat/hugging_face_local.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
Keep in mind that if LLMs run locally, you may need a powerful machine to run them. This depends strongly on the model you select and its parameter count.
|
||||
|
||||
:::info
|
||||
This component is designed for chat completion, not for text generation. If you want to use Hugging Face LLMs for text generation, use [`HuggingFaceLocalGenerator`](huggingfacelocalgenerator.mdx) instead.
|
||||
:::
|
||||
|
||||
For remote file authorization, this component uses a `HF_API_TOKEN` environment variable by default. Otherwise, you can pass a Hugging Face API token at initialization with `token`:
|
||||
|
||||
```python
|
||||
local_generator = HuggingFaceLocalChatGenerator(
|
||||
token=Secret.from_token("<your-api-key>"),
|
||||
)
|
||||
```
|
||||
|
||||
### Streaming
|
||||
|
||||
This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import HuggingFaceLocalChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
generator = HuggingFaceLocalChatGenerator(model="HuggingFaceH4/zephyr-7b-beta")
|
||||
messages = [ChatMessage.from_user("What's Natural Language Processing? Be brief.")]
|
||||
print(generator.run(messages))
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders.prompt_builder import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import HuggingFaceLocalChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.utils import Secret
|
||||
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
llm = HuggingFaceLocalChatGenerator(
|
||||
model="HuggingFaceH4/zephyr-7b-beta",
|
||||
token=Secret.from_env_var("HF_API_TOKEN"),
|
||||
)
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
location = "Berlin"
|
||||
messages = [
|
||||
ChatMessage.from_system(
|
||||
"Always respond in German even if some input data is in other languages.",
|
||||
),
|
||||
ChatMessage.from_user("Tell me about {{location}}"),
|
||||
]
|
||||
pipe.run(
|
||||
data={
|
||||
"prompt_builder": {
|
||||
"template_variables": {"location": location},
|
||||
"template": messages,
|
||||
},
|
||||
},
|
||||
)
|
||||
```
|
||||
+123
@@ -0,0 +1,123 @@
|
||||
---
|
||||
title: "HuggingFaceLocalGenerator"
|
||||
id: huggingfacelocalgenerator
|
||||
slug: "/huggingfacelocalgenerator"
|
||||
description: "`HuggingFaceLocalGenerator` provides an interface to generate text using a Hugging Face model that runs locally."
|
||||
---
|
||||
|
||||
# HuggingFaceLocalGenerator
|
||||
|
||||
`HuggingFaceLocalGenerator` provides an interface to generate text using a Hugging Face model that runs locally.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`PromptBuilder`](../builders/promptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `token`: The Hugging Face API token. Can be set with `HF_API_TOKEN` or `HF_TOKEN` env var. |
|
||||
| **Mandatory run variables** | `prompt`: A string containing the prompt for the LLM |
|
||||
| **Output variables** | `replies`: A list of strings with all the replies generated by the LLM |
|
||||
| **API reference** | [Generators](/reference/generators-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/generators/hugging_face_local.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
Keep in mind that if LLMs run locally, you may need a powerful machine to run them. This depends strongly on the model you select and its parameter count.
|
||||
|
||||
:::info[Looking for chat completion?]
|
||||
|
||||
This component is designed for text generation, not for chat. If you want to use Hugging Face LLMs for chat, consider using [`HuggingFaceLocalChatGenerator`](huggingfacelocalchatgenerator.mdx) instead.
|
||||
:::
|
||||
|
||||
For remote files authorization, this component uses a `HF_API_TOKEN` environment variable by default. Otherwise, you can pass a Hugging Face API token at initialization with `token`:
|
||||
|
||||
```python
|
||||
local_generator = HuggingFaceLocalGenerator(token=Secret.from_token("<your-api-key>"))
|
||||
```
|
||||
|
||||
### Streaming
|
||||
|
||||
This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.components.generators import HuggingFaceLocalGenerator
|
||||
|
||||
generator = HuggingFaceLocalGenerator(
|
||||
model="google/flan-t5-large",
|
||||
task="text2text-generation",
|
||||
generation_kwargs={
|
||||
"max_new_tokens": 100,
|
||||
"temperature": 0.9,
|
||||
},
|
||||
)
|
||||
|
||||
print(generator.run("Who is the best American actor?"))
|
||||
## {'replies': ['john wayne']}
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.components.builders.prompt_builder import PromptBuilder
|
||||
from haystack.components.generators import HuggingFaceLocalGenerator
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack import Document
|
||||
|
||||
docstore = InMemoryDocumentStore()
|
||||
docstore.write_documents(
|
||||
[
|
||||
Document(content="Rome is the capital of Italy"),
|
||||
Document(content="Paris is the capital of France"),
|
||||
],
|
||||
)
|
||||
|
||||
generator = HuggingFaceLocalGenerator(
|
||||
model="google/flan-t5-large",
|
||||
task="text2text-generation",
|
||||
generation_kwargs={
|
||||
"max_new_tokens": 100,
|
||||
"temperature": 0.9,
|
||||
},
|
||||
)
|
||||
|
||||
query = "What is the capital of France?"
|
||||
|
||||
template = """
|
||||
Given the following information, answer the question.
|
||||
|
||||
Context:
|
||||
{% for document in documents %}
|
||||
{{ document.content }}
|
||||
{% endfor %}
|
||||
|
||||
Question: {{ query }}?
|
||||
"""
|
||||
pipe = Pipeline()
|
||||
|
||||
pipe.add_component("retriever", InMemoryBM25Retriever(document_store=docstore))
|
||||
pipe.add_component("prompt_builder", PromptBuilder(template=template))
|
||||
pipe.add_component("llm", generator)
|
||||
pipe.connect("retriever", "prompt_builder.documents")
|
||||
pipe.connect("prompt_builder", "llm")
|
||||
|
||||
res = pipe.run({"prompt_builder": {"query": query}, "retriever": {"query": query}})
|
||||
|
||||
print(res)
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbooks:
|
||||
|
||||
- [Use Zephyr 7B Beta with Hugging Face for RAG](https://haystack.deepset.ai/cookbook/zephyr-7b-beta-for-rag)
|
||||
- [Information Extraction with Gorilla](https://haystack.deepset.ai/cookbook/information-extraction-gorilla)
|
||||
- [RAG on the Oscars using Llama 3.1 models](https://haystack.deepset.ai/cookbook/llama3_rag)
|
||||
- [Agentic RAG with Llama 3.2 3B](https://haystack.deepset.ai/cookbook/llama32_agentic_rag)
|
||||
+324
@@ -0,0 +1,324 @@
|
||||
---
|
||||
title: "LlamaCppChatGenerator"
|
||||
id: llamacppchatgenerator
|
||||
slug: "/llamacppchatgenerator"
|
||||
description: "`LlamaCppGenerator` enables chat completion using an LLM running on Llama.cpp."
|
||||
---
|
||||
|
||||
# LlamaCppChatGenerator
|
||||
|
||||
`LlamaCppGenerator` enables chat completion using an LLM running on Llama.cpp.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`ChatPromptBuilder`](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `model`: The path of the model to use |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) instances representing the input messages |
|
||||
| **Output variables** | `replies`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) instances with all the replies generated by the LLM |
|
||||
| **API reference** | [Llama.cpp](/reference/integrations-llama-cpp) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/llama_cpp |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
[Llama.cpp](https://github.com/ggml-org/llama.cpp) is a library written in C/C++ for efficient inference of Large Language Models. It leverages the efficient quantized GGUF format, dramatically reducing memory requirements and accelerating inference. This means it is possible to run LLMs efficiently on standard machines (even without GPUs).
|
||||
|
||||
`Llama.cpp` uses the quantized binary file of the LLM in GGUF format, which can be downloaded from [Hugging Face](https://huggingface.co/models?library=gguf). `LlamaCppChatGenerator` supports models running on `Llama.cpp` by taking the path to the locally saved GGUF file as `model` parameter at initialization.
|
||||
|
||||
### Tool Support
|
||||
|
||||
`LlamaCppChatGenerator` supports function calling through the `tools` parameter, which accepts flexible tool configurations:
|
||||
|
||||
- **A list of Tool objects**: Pass individual tools as a list
|
||||
- **A single Toolset**: Pass an entire Toolset directly
|
||||
- **Mixed Tools and Toolsets**: Combine multiple Toolsets with standalone tools in a single list
|
||||
|
||||
This allows you to organize related tools into logical groups while also including standalone tools as needed.
|
||||
|
||||
```python
|
||||
from haystack.tools import Tool, Toolset
|
||||
from haystack_integrations.components.generators.llama_cpp import LlamaCppChatGenerator
|
||||
|
||||
# Create individual tools
|
||||
weather_tool = Tool(name="weather", description="Get weather info", ...)
|
||||
news_tool = Tool(name="news", description="Get latest news", ...)
|
||||
|
||||
# Group related tools into a toolset
|
||||
math_toolset = Toolset([add_tool, subtract_tool, multiply_tool])
|
||||
|
||||
# Pass mixed tools and toolsets to the generator
|
||||
generator = LlamaCppChatGenerator(
|
||||
model="/path/to/model.gguf",
|
||||
tools=[math_toolset, weather_tool, news_tool] # Mix of Toolset and Tool objects
|
||||
)
|
||||
```
|
||||
|
||||
For more details on working with tools, see the [Tool](../../tools/tool.mdx) and [Toolset](../../tools/toolset.mdx) documentation.
|
||||
|
||||
## Installation
|
||||
|
||||
Install the `llama-cpp-haystack` package to use this integration:
|
||||
|
||||
```shell
|
||||
pip install llama-cpp-haystack
|
||||
```
|
||||
|
||||
### Using a different compute backend
|
||||
|
||||
The default installation behavior is to build `llama.cpp` for CPU on Linux and Windows and use Metal on MacOS. To use other compute backends:
|
||||
|
||||
1. Follow instructions on the [llama.cpp installation page](https://github.com/abetlen/llama-cpp-python#installation) to install [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) for your preferred compute backend.
|
||||
2. Install [llama-cpp-haystack](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/llama_cpp) using the command above.
|
||||
|
||||
For example, to use `llama-cpp-haystack` with the **cuBLAS backend**, you have to run the following commands:
|
||||
|
||||
```shell
|
||||
export GGML_CUDA=1
|
||||
CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python
|
||||
pip install llama-cpp-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
1. Download the GGUF version of the desired LLM. The GGUF versions of popular models can be downloaded from [Hugging Face](https://huggingface.co/models?library=gguf).
|
||||
2. Initialize `LlamaCppChatGenerator` with the path to the GGUF file and specify the required model and text generation parameters:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.llama_cpp import LlamaCppChatGenerator
|
||||
|
||||
generator = LlamaCppChatGenerator(
|
||||
model="/content/openchat-3.5-1210.Q3_K_S.gguf",
|
||||
n_ctx=512,
|
||||
n_batch=128,
|
||||
model_kwargs={"n_gpu_layers": -1},
|
||||
generation_kwargs={"max_tokens": 128, "temperature": 0.1},
|
||||
)
|
||||
generator.warm_up()
|
||||
messages = [ChatMessage.from_user("Who is the best American actor?")]
|
||||
result = generator.run(messages)
|
||||
```
|
||||
|
||||
### Passing additional model parameters
|
||||
|
||||
The `model`, `n_ctx`, `n_batch` arguments have been exposed for convenience and can be directly passed to the Generator during initialization as keyword arguments. Note that `model` translates to `llama.cpp`'s `model_path` parameter.
|
||||
|
||||
The `model_kwargs` parameter can pass additional arguments when initializing the model. In case of duplication, these parameters override the `model`, `n_ctx`, and `n_batch` initialization parameters.
|
||||
|
||||
See [Llama.cpp's LLM documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.__init__) for more information on the available model arguments.
|
||||
|
||||
**Note**: Llama.cpp automatically extracts the `chat_template` from the model metadata for applying formatting to ChatMessages. You can override the `chat_template` used by passing in a custom `chat_handler` or `chat_format` as a model parameter.
|
||||
|
||||
For example, to offload the model to GPU during initialization:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.llama_cpp import LlamaCppChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
generator = LlamaCppChatGenerator(
|
||||
model="/content/openchat-3.5-1210.Q3_K_S.gguf",
|
||||
n_ctx=512,
|
||||
n_batch=128,
|
||||
model_kwargs={"n_gpu_layers": -1},
|
||||
)
|
||||
messages = [ChatMessage.from_user("Who is the best American actor?")]
|
||||
result = generator.run(messages, generation_kwargs={"max_tokens": 128})
|
||||
generated_reply = result["replies"][0].text
|
||||
print(generated_reply)
|
||||
```
|
||||
|
||||
### Passing text generation parameters
|
||||
|
||||
The `generation_kwargs` parameter can pass additional generation arguments like `max_tokens`, `temperature`, `top_k`, `top_p`, and others to the model during inference.
|
||||
|
||||
See [Llama.cpp's Chat Completion API documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.create_chat_completion) for more information on the available generation arguments.
|
||||
|
||||
**Note**: JSON mode, Function Calling, and Tools are all supported as `generation_kwargs`. Please see the [llama-cpp-python GitHub README](https://github.com/abetlen/llama-cpp-python?tab=readme-ov-file#json-and-json-schema-mode) for more information on how to use them.
|
||||
|
||||
For example, to set the `max_tokens` and `temperature`:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.llama_cpp import LlamaCppChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
generator = LlamaCppChatGenerator(
|
||||
model="/content/openchat-3.5-1210.Q3_K_S.gguf",
|
||||
n_ctx=512,
|
||||
n_batch=128,
|
||||
generation_kwargs={"max_tokens": 128, "temperature": 0.1},
|
||||
)
|
||||
messages = [ChatMessage.from_user("Who is the best American actor?")]
|
||||
result = generator.run(messages)
|
||||
```
|
||||
|
||||
### With multimodal (image + text) inputs
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage, ImageContent
|
||||
from haystack_integrations.components.generators.llama_cpp import LlamaCppChatGenerator
|
||||
|
||||
# Initialize with multimodal support
|
||||
llm = LlamaCppChatGenerator(
|
||||
model="llava-v1.5-7b-q4_0.gguf",
|
||||
chat_handler_name="Llava15ChatHandler", # Use llava-1-5 handler
|
||||
model_clip_path="mmproj-model-f16.gguf", # CLIP model
|
||||
n_ctx=4096, # Larger context for image processing
|
||||
)
|
||||
|
||||
image = ImageContent.from_file_path("apple.jpg")
|
||||
user_message = ChatMessage.from_user(
|
||||
content_parts=["What does the image show? Max 5 words.", image],
|
||||
)
|
||||
|
||||
response = llm.run([user_message])["replies"][0].text
|
||||
print(response)
|
||||
|
||||
# Red apple on straw.
|
||||
```
|
||||
|
||||
The `generation_kwargs` can also be passed to the `run` method of the generator directly:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.llama_cpp import LlamaCppChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
generator = LlamaCppChatGenerator(
|
||||
model="/content/openchat-3.5-1210.Q3_K_S.gguf",
|
||||
n_ctx=512,
|
||||
n_batch=128,
|
||||
)
|
||||
messages = [ChatMessage.from_user("Who is the best American actor?")]
|
||||
result = generator.run(
|
||||
messages,
|
||||
generation_kwargs={"max_tokens": 128, "temperature": 0.1},
|
||||
)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
We use the `LlamaCppChatGenerator` in a Retrieval Augmented Generation pipeline on the [Simple Wikipedia](https://huggingface.co/datasets/pszemraj/simple_wikipedia) Dataset from Hugging Face and generate answers using the [OpenChat-3.5](https://huggingface.co/openchat/openchat-3.5-1210) LLM.
|
||||
|
||||
Load the dataset:
|
||||
|
||||
```python
|
||||
## Install HuggingFace Datasets using "pip install datasets"
|
||||
from datasets import load_dataset
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.builders.answer_builder import AnswerBuilder
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.embedders import (
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
SentenceTransformersTextEmbedder,
|
||||
)
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
## Import LlamaCppChatGenerator
|
||||
from haystack_integrations.components.generators.llama_cpp import LlamaCppChatGenerator
|
||||
|
||||
## Load first 100 rows of the Simple Wikipedia Dataset from HuggingFace
|
||||
dataset = load_dataset("pszemraj/simple_wikipedia", split="validation[:100]")
|
||||
|
||||
docs = [
|
||||
Document(
|
||||
content=doc["text"],
|
||||
meta={
|
||||
"title": doc["title"],
|
||||
"url": doc["url"],
|
||||
},
|
||||
)
|
||||
for doc in dataset
|
||||
]
|
||||
```
|
||||
|
||||
Index the documents to the `InMemoryDocumentStore` using the `SentenceTransformersDocumentEmbedder` and `DocumentWriter`:
|
||||
|
||||
```python
|
||||
doc_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
## Install sentence transformers using "pip install sentence-transformers"
|
||||
doc_embedder = SentenceTransformersDocumentEmbedder(
|
||||
model="sentence-transformers/all-MiniLM-L6-v2",
|
||||
)
|
||||
|
||||
## Indexing Pipeline
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component(instance=doc_embedder, name="DocEmbedder")
|
||||
indexing_pipeline.add_component(
|
||||
instance=DocumentWriter(document_store=doc_store),
|
||||
name="DocWriter",
|
||||
)
|
||||
indexing_pipeline.connect("DocEmbedder", "DocWriter")
|
||||
|
||||
indexing_pipeline.run({"DocEmbedder": {"documents": docs}})
|
||||
```
|
||||
|
||||
Create the RAG pipeline and add the `LlamaCppChatGenerator` to it:
|
||||
|
||||
```python
|
||||
system_message = ChatMessage.from_system(
|
||||
"""
|
||||
Answer the question using the provided context.
|
||||
Context:
|
||||
{% for doc in documents %}
|
||||
{{ doc.content }}
|
||||
{% endfor %}
|
||||
""",
|
||||
)
|
||||
user_message = ChatMessage.from_user("Question: {{question}}")
|
||||
assistent_message = ChatMessage.from_assistant("Answer: ")
|
||||
|
||||
chat_template = [system_message, user_message, assistent_message]
|
||||
|
||||
rag_pipeline = Pipeline()
|
||||
|
||||
text_embedder = SentenceTransformersTextEmbedder(
|
||||
model="sentence-transformers/all-MiniLM-L6-v2",
|
||||
)
|
||||
|
||||
## Load the LLM using LlamaCppChatGenerator
|
||||
model_path = "openchat-3.5-1210.Q3_K_S.gguf"
|
||||
generator = LlamaCppChatGenerator(model=model_path, n_ctx=4096, n_batch=128)
|
||||
|
||||
rag_pipeline.add_component(
|
||||
instance=text_embedder,
|
||||
name="text_embedder",
|
||||
)
|
||||
rag_pipeline.add_component(
|
||||
instance=InMemoryEmbeddingRetriever(document_store=doc_store, top_k=3),
|
||||
name="retriever",
|
||||
)
|
||||
rag_pipeline.add_component(
|
||||
instance=ChatPromptBuilder(template=chat_template),
|
||||
name="prompt_builder",
|
||||
)
|
||||
rag_pipeline.add_component(instance=generator, name="llm")
|
||||
rag_pipeline.add_component(instance=AnswerBuilder(), name="answer_builder")
|
||||
|
||||
rag_pipeline.connect("text_embedder", "retriever")
|
||||
rag_pipeline.connect("retriever", "prompt_builder.documents")
|
||||
rag_pipeline.connect("prompt_builder", "llm")
|
||||
rag_pipeline.connect("llm", "answer_builder")
|
||||
rag_pipeline.connect("retriever", "answer_builder.documents")
|
||||
```
|
||||
|
||||
Run the pipeline:
|
||||
|
||||
```python
|
||||
question = "Which year did the Joker movie release?"
|
||||
result = rag_pipeline.run(
|
||||
{
|
||||
"text_embedder": {"text": question},
|
||||
"prompt_builder": {"question": question},
|
||||
"llm": {"generation_kwargs": {"max_tokens": 128, "temperature": 0.1}},
|
||||
"answer_builder": {"query": question},
|
||||
},
|
||||
)
|
||||
|
||||
generated_answer = result["answer_builder"]["answers"][0]
|
||||
print(generated_answer.data)
|
||||
## The Joker movie was released on October 4, 2019.
|
||||
```
|
||||
+257
@@ -0,0 +1,257 @@
|
||||
---
|
||||
title: "LlamaCppGenerator"
|
||||
id: llamacppgenerator
|
||||
slug: "/llamacppgenerator"
|
||||
description: "`LlamaCppGenerator` provides an interface to generate text using an LLM running on Llama.cpp."
|
||||
---
|
||||
|
||||
# LlamaCppGenerator
|
||||
|
||||
`LlamaCppGenerator` provides an interface to generate text using an LLM running on Llama.cpp.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`PromptBuilder`](../builders/promptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `model`: The path of the model to use |
|
||||
| **Mandatory run variables** | `prompt`: A string containing the prompt for the LLM |
|
||||
| **Output variables** | `replies`: A list of strings with all the replies generated by the LLM <br /> <br />`meta`: A list of dictionaries with the metadata associated with each reply, such as token count and others |
|
||||
| **API reference** | [Llama.cpp](/reference/integrations-llama-cpp) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/llama_cpp |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
[Llama.cpp](https://github.com/ggml-org/llama.cpp) is a library written in C/C++ for efficient inference of Large Language Models. It leverages the efficient quantized GGUF format, dramatically reducing memory requirements and accelerating inference. This means it is possible to run LLMs efficiently on standard machines (even without GPUs).
|
||||
|
||||
`Llama.cpp` uses the quantized binary file of the LLM in GGUF format that can be downloaded from [Hugging Face](https://huggingface.co/models?library=gguf). `LlamaCppGenerator` supports models running on `Llama.cpp` by taking the path to the locally saved GGUF file as `model` parameter at initialization.
|
||||
|
||||
## Installation
|
||||
|
||||
Install the `llama-cpp-haystack` package:
|
||||
|
||||
```bash
|
||||
pip install llama-cpp-haystack
|
||||
```
|
||||
|
||||
### Using a different compute backend
|
||||
|
||||
The default installation behavior is to build `llama.cpp` for CPU on Linux and Windows and use Metal on MacOS. To use other compute backends:
|
||||
|
||||
1. Follow instructions on the [llama.cpp installation page](https://github.com/abetlen/llama-cpp-python#installation) to install [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) for your preferred compute backend.
|
||||
2. Install [llama-cpp-haystack](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/llama_cpp) using the command above.
|
||||
|
||||
For example, to use `llama-cpp-haystack` with the **cuBLAS backend**, you have to run the following commands:
|
||||
|
||||
```bash
|
||||
export GGML_CUDA=1
|
||||
CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python
|
||||
pip install llama-cpp-haystack
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
1. You need to download the GGUF version of the desired LLM. The GGUF versions of popular models can be downloaded from [Hugging Face](https://huggingface.co/models?library=gguf).
|
||||
2. Initialize a `LlamaCppGenerator` with the path to the GGUF file and also specify the required model and text generation parameters:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.llama_cpp import LlamaCppGenerator
|
||||
|
||||
generator = LlamaCppGenerator(
|
||||
model="/content/openchat-3.5-1210.Q3_K_S.gguf",
|
||||
n_ctx=512,
|
||||
n_batch=128,
|
||||
model_kwargs={"n_gpu_layers": -1},
|
||||
generation_kwargs={"max_tokens": 128, "temperature": 0.1},
|
||||
)
|
||||
generator.warm_up()
|
||||
prompt = f"Who is the best American actor?"
|
||||
result = generator.run(prompt)
|
||||
```
|
||||
|
||||
### Passing additional model parameters
|
||||
|
||||
The `model`, `n_ctx`, `n_batch` arguments have been exposed for convenience and can be directly passed to the Generator during initialization as keyword arguments. Note that `model` translates to `llama.cpp`'s `model_path` parameter.
|
||||
|
||||
The `model_kwargs` parameter can pass additional arguments when initializing the model. In case of duplication, these parameters override the `model`, `n_ctx`, and `n_batch` initialization parameters.
|
||||
|
||||
See [Llama.cpp's LLM documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.__init__) for more information on the available model arguments.
|
||||
|
||||
For example, to offload the model to GPU during initialization:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.llama_cpp import LlamaCppGenerator
|
||||
|
||||
generator = LlamaCppGenerator(
|
||||
model="/content/openchat-3.5-1210.Q3_K_S.gguf",
|
||||
n_ctx=512,
|
||||
n_batch=128,
|
||||
model_kwargs={"n_gpu_layers": -1},
|
||||
)
|
||||
prompt = f"Who is the best American actor?"
|
||||
result = generator.run(prompt, generation_kwargs={"max_tokens": 128})
|
||||
generated_text = result["replies"][0]
|
||||
print(generated_text)
|
||||
```
|
||||
|
||||
### Passing text generation parameters
|
||||
|
||||
The `generation_kwargs` parameter can pass additional generation arguments like `max_tokens`, `temperature`, `top_k`, `top_p`, and others to the model during inference.
|
||||
|
||||
See [Llama.cpp's Completion API documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.create_completion) for more information on the available generation arguments.
|
||||
|
||||
For example, to set the `max_tokens` and `temperature`:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.llama_cpp import LlamaCppGenerator
|
||||
|
||||
generator = LlamaCppGenerator(
|
||||
model="/content/openchat-3.5-1210.Q3_K_S.gguf",
|
||||
n_ctx=512,
|
||||
n_batch=128,
|
||||
generation_kwargs={"max_tokens": 128, "temperature": 0.1},
|
||||
)
|
||||
prompt = f"Who is the best American actor?"
|
||||
result = generator.run(prompt)
|
||||
```
|
||||
|
||||
The `generation_kwargs` can also be passed to the `run` method of the generator directly:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.llama_cpp import LlamaCppGenerator
|
||||
|
||||
generator = LlamaCppGenerator(
|
||||
model="/content/openchat-3.5-1210.Q3_K_S.gguf",
|
||||
n_ctx=512,
|
||||
n_batch=128,
|
||||
)
|
||||
prompt = f"Who is the best American actor?"
|
||||
result = generator.run(
|
||||
prompt,
|
||||
generation_kwargs={"max_tokens": 128, "temperature": 0.1},
|
||||
)
|
||||
```
|
||||
|
||||
### Using in a Pipeline
|
||||
|
||||
We use the `LlamaCppGenerator` in a Retrieval Augmented Generation pipeline on the [Simple Wikipedia](https://huggingface.co/datasets/pszemraj/simple_wikipedia) Dataset from HuggingFace and generate answers using the [OpenChat-3.5](https://huggingface.co/openchat/openchat-3.5-1210) LLM.
|
||||
|
||||
Load the dataset:
|
||||
|
||||
```python
|
||||
## Install HuggingFace Datasets using "pip install datasets"
|
||||
from datasets import load_dataset
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.builders.answer_builder import AnswerBuilder
|
||||
from haystack.components.builders.prompt_builder import PromptBuilder
|
||||
from haystack.components.embedders import (
|
||||
SentenceTransformersDocumentEmbedder,
|
||||
SentenceTransformersTextEmbedder,
|
||||
)
|
||||
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
## Import LlamaCppGenerator
|
||||
from haystack_integrations.components.generators.llama_cpp import LlamaCppGenerator
|
||||
|
||||
## Load first 100 rows of the Simple Wikipedia Dataset from HuggingFace
|
||||
dataset = load_dataset("pszemraj/simple_wikipedia", split="validation[:100]")
|
||||
|
||||
docs = [
|
||||
Document(
|
||||
content=doc["text"],
|
||||
meta={
|
||||
"title": doc["title"],
|
||||
"url": doc["url"],
|
||||
},
|
||||
)
|
||||
for doc in dataset
|
||||
]
|
||||
```
|
||||
|
||||
Index the documents to the `InMemoryDocumentStore` using the `SentenceTransformersDocumentEmbedder` and `DocumentWriter`:
|
||||
|
||||
```python
|
||||
doc_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
|
||||
doc_embedder = SentenceTransformersDocumentEmbedder(
|
||||
model="sentence-transformers/all-MiniLM-L6-v2",
|
||||
)
|
||||
|
||||
## Indexing Pipeline
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component(instance=doc_embedder, name="DocEmbedder")
|
||||
indexing_pipeline.add_component(
|
||||
instance=DocumentWriter(document_store=doc_store),
|
||||
name="DocWriter",
|
||||
)
|
||||
indexing_pipeline.connect(connect_from="DocEmbedder", connect_to="DocWriter")
|
||||
|
||||
indexing_pipeline.run({"DocEmbedder": {"documents": docs}})
|
||||
```
|
||||
|
||||
Create the Retrieval Augmented Generation (RAG) pipeline and add the `LlamaCppGenerator` to it:
|
||||
|
||||
```python
|
||||
## Prompt Template for the https://huggingface.co/openchat/openchat-3.5-1210 LLM
|
||||
prompt_template = """GPT4 Correct User: Answer the question using the provided context.
|
||||
Question: {{question}}
|
||||
Context:
|
||||
{% for doc in documents %}
|
||||
{{ doc.content }}
|
||||
{% endfor %}
|
||||
<|end_of_turn|>
|
||||
GPT4 Correct Assistant:
|
||||
"""
|
||||
|
||||
rag_pipeline = Pipeline()
|
||||
|
||||
text_embedder = SentenceTransformersTextEmbedder(
|
||||
model="sentence-transformers/all-MiniLM-L6-v2",
|
||||
)
|
||||
|
||||
## Load the LLM using LlamaCppGenerator
|
||||
model_path = "openchat-3.5-1210.Q3_K_S.gguf"
|
||||
generator = LlamaCppGenerator(model=model_path, n_ctx=4096, n_batch=128)
|
||||
|
||||
rag_pipeline.add_component(
|
||||
instance=text_embedder,
|
||||
name="text_embedder",
|
||||
)
|
||||
rag_pipeline.add_component(
|
||||
instance=InMemoryEmbeddingRetriever(document_store=doc_store, top_k=3),
|
||||
name="retriever",
|
||||
)
|
||||
rag_pipeline.add_component(
|
||||
instance=PromptBuilder(template=prompt_template),
|
||||
name="prompt_builder",
|
||||
)
|
||||
rag_pipeline.add_component(instance=generator, name="llm")
|
||||
rag_pipeline.add_component(instance=AnswerBuilder(), name="answer_builder")
|
||||
|
||||
rag_pipeline.connect("text_embedder", "retriever")
|
||||
rag_pipeline.connect("retriever", "prompt_builder.documents")
|
||||
rag_pipeline.connect("prompt_builder", "llm")
|
||||
rag_pipeline.connect("llm.replies", "answer_builder.replies")
|
||||
rag_pipeline.connect("retriever", "answer_builder.documents")
|
||||
```
|
||||
|
||||
Run the pipeline:
|
||||
|
||||
```python
|
||||
question = "Which year did the Joker movie release?"
|
||||
result = rag_pipeline.run(
|
||||
{
|
||||
"text_embedder": {"text": question},
|
||||
"prompt_builder": {"question": question},
|
||||
"llm": {"generation_kwargs": {"max_tokens": 128, "temperature": 0.1}},
|
||||
"answer_builder": {"query": question},
|
||||
},
|
||||
)
|
||||
|
||||
generated_answer = result["answer_builder"]["answers"][0]
|
||||
print(generated_answer.data)
|
||||
## The Joker movie was released on October 4, 2019.
|
||||
```
|
||||
+150
@@ -0,0 +1,150 @@
|
||||
---
|
||||
title: "LlamaStackChatGenerator"
|
||||
id: llamastackchatgenerator
|
||||
slug: "/llamastackchatgenerator"
|
||||
description: "This component enables chat completions using any model made available by inference providers on a Llama Stack server."
|
||||
---
|
||||
|
||||
# LlamaStackChatGenerator
|
||||
|
||||
This component enables chat completions using any model made available by inference providers on a Llama Stack server.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `model`: The name of the model to use for chat completion. <br />This depends on the inference provider used for the Llama Stack Server. |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects representing the chat |
|
||||
| **Output variables** | `replies`: A list of alternative replies of the model to the input chat |
|
||||
| **API reference** | [Llama Stack](/reference/integrations-llama-stack) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/blob/main/integrations/llama_stack |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
[Llama Stack](https://llama-stack.readthedocs.io/en/latest/index.html) provides building blocks and unified APIs to streamline the development of AI applications across various environments.
|
||||
|
||||
The `LlamaStackChatGenerator` enables you to access any LLMs exposed by inference providers hosted on a Llama Stack server. It abstracts away the underlying provider details, allowing you to reuse the same client-side code regardless of the inference backend. For a list of supported providers and configuration options, refer to the [Llama Stack documentation](https://llama-stack.readthedocs.io/en/latest/providers/inference/index.html).
|
||||
|
||||
This component uses the same `ChatMessage` format as other Haystack Chat Generators for structured input and output. For more information, see the [ChatMessage documentation](../../concepts/data-classes/chatmessage.mdx).
|
||||
|
||||
### Tool Support
|
||||
|
||||
`LlamaStackChatGenerator` supports function calling through the `tools` parameter, which accepts flexible tool configurations:
|
||||
|
||||
- **A list of Tool objects**: Pass individual tools as a list
|
||||
- **A single Toolset**: Pass an entire Toolset directly
|
||||
- **Mixed Tools and Toolsets**: Combine multiple Toolsets with standalone tools in a single list
|
||||
|
||||
This allows you to organize related tools into logical groups while also including standalone tools as needed.
|
||||
|
||||
```python
|
||||
from haystack.tools import Tool, Toolset
|
||||
from haystack_integrations.components.generators.llama_stack import LlamaStackChatGenerator
|
||||
|
||||
# Create individual tools
|
||||
weather_tool = Tool(name="weather", description="Get weather info", ...)
|
||||
news_tool = Tool(name="news", description="Get latest news", ...)
|
||||
|
||||
# Group related tools into a toolset
|
||||
math_toolset = Toolset([add_tool, subtract_tool, multiply_tool])
|
||||
|
||||
# Pass mixed tools and toolsets to the generator
|
||||
generator = LlamaStackChatGenerator(
|
||||
model="ollama/llama3.2:3b",
|
||||
tools=[math_toolset, weather_tool, news_tool] # Mix of Toolset and Tool objects
|
||||
)
|
||||
```
|
||||
|
||||
For more details on working with tools, see the [Tool](../../tools/tool.mdx) and [Toolset](../../tools/toolset.mdx) documentation.
|
||||
|
||||
## Initialization
|
||||
|
||||
To use this integration, you must have:
|
||||
|
||||
- A running instance of a Llama Stack server (local or remote)
|
||||
- A valid model name supported by your selected inference provider
|
||||
|
||||
Then initialize the `LlamaStackChatGenerator` by specifying the `model` name or ID. The value depends on the inference provider running on your server.
|
||||
|
||||
**Examples:**
|
||||
|
||||
- For Ollama: `model="ollama/llama3.2:3b"`
|
||||
- For vLLM: `model="meta-llama/Llama-3.2-3B"`
|
||||
|
||||
**Note:** Switching the inference provider only requires updating the model name.
|
||||
|
||||
### Streaming
|
||||
|
||||
This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
To start using this integration, install the package with:
|
||||
|
||||
```shell
|
||||
pip install llama-stack-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
import os
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.llama_stack import (
|
||||
LlamaStackChatGenerator,
|
||||
)
|
||||
|
||||
client = LlamaStackChatGenerator(model="ollama/llama3.2:3b")
|
||||
response = client.run([ChatMessage.from_user("What are Agentic Pipelines? Be brief.")])
|
||||
print(response["replies"])
|
||||
```
|
||||
|
||||
#### With Streaming
|
||||
|
||||
```python
|
||||
import os
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.llama_stack import (
|
||||
LlamaStackChatGenerator,
|
||||
)
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
|
||||
client = LlamaStackChatGenerator(
|
||||
model="ollama/llama3.2:3b",
|
||||
streaming_callback=print_streaming_chunk,
|
||||
)
|
||||
response = client.run([ChatMessage.from_user("What are Agentic Pipelines? Be brief.")])
|
||||
print(response["replies"])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.llama_stack import (
|
||||
LlamaStackChatGenerator,
|
||||
)
|
||||
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
llm = LlamaStackChatGenerator(model="ollama/llama3.2:3b")
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
pipe.connect("builder.prompt", "llm.messages")
|
||||
|
||||
messages = [
|
||||
ChatMessage.from_system("Give brief answers."),
|
||||
ChatMessage.from_user("Tell me about {{city}}"),
|
||||
]
|
||||
|
||||
response = pipe.run(
|
||||
data={"builder": {"template": messages, "template_variables": {"city": "Berlin"}}},
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
+206
@@ -0,0 +1,206 @@
|
||||
---
|
||||
title: "MetaLlamaChatGenerator"
|
||||
id: metallamachatgenerator
|
||||
slug: "/metallamachatgenerator"
|
||||
description: "This component enables chat completion with any model hosted available with Meta Llama API."
|
||||
---
|
||||
|
||||
# MetaLlamaChatGenerator
|
||||
|
||||
This component enables chat completion with any model hosted available with Meta Llama API.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: A Meta Llama API key. Can be set with `LLAMA_API_KEY` env variable or passed to `init()` method. |
|
||||
| **Mandatory run variables** | `messages`: A list of [ChatMessage](../../concepts/data-classes/chatmessage.mdx) objects |
|
||||
| **Output variables** | `replies`: A list of [ChatMessage](../../concepts/data-classes/chatmessage.mdx) objects |
|
||||
| **API reference** | [Meta Llama API](/reference/integrations-meta-llama) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/meta_llama |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `MetaLlamaChatGenerator` enables you to use multiple Meta Llama models by making chat completion calls to the Meta [Llama API](https://llama.developer.meta.com/?utm_source=partner-haystack&utm_medium=website). The default model is `Llama-4-Scout-17B-16E-Instruct-FP8`.
|
||||
|
||||
Currently available models are:
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | | | | |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| Model ID | Input context length | Output context length | Input Modalities | Output Modalities |
|
||||
| `Llama-4-Scout-17B-16E-Instruct-FP8` | 128k | 4028 | Text, Image | Text |
|
||||
| `Llama-4-Maverick-17B-128E-Instruct-FP8` | 128k | 4028 | Text, Image | Text |
|
||||
| `Llama-3.3-70B-Instruct` | 128k | 4028 | Text | Text |
|
||||
| `Llama-3.3-8B-Instruct` | 128k | 4028 | Text | Text |
|
||||
|
||||
</div>
|
||||
This component uses the same `ChatMessage` format as other Haystack Chat Generators for structured input and output. For more information, see the [ChatMessage documentation](../../concepts/data-classes/chatmessage.mdx).
|
||||
|
||||
### Tool Support
|
||||
|
||||
`MetaLlamaChatGenerator` supports function calling through the `tools` parameter, which accepts flexible tool configurations:
|
||||
|
||||
- **A list of Tool objects**: Pass individual tools as a list
|
||||
- **A single Toolset**: Pass an entire Toolset directly
|
||||
- **Mixed Tools and Toolsets**: Combine multiple Toolsets with standalone tools in a single list
|
||||
|
||||
This allows you to organize related tools into logical groups while also including standalone tools as needed.
|
||||
|
||||
```python
|
||||
from haystack.tools import Tool, Toolset
|
||||
from haystack_integrations.components.generators.meta_llama import MetaLlamaChatGenerator
|
||||
|
||||
# Create individual tools
|
||||
weather_tool = Tool(name="weather", description="Get weather info", ...)
|
||||
news_tool = Tool(name="news", description="Get latest news", ...)
|
||||
|
||||
# Group related tools into a toolset
|
||||
math_toolset = Toolset([add_tool, subtract_tool, multiply_tool])
|
||||
|
||||
# Pass mixed tools and toolsets to the generator
|
||||
generator = MetaLlamaChatGenerator(
|
||||
tools=[math_toolset, weather_tool, news_tool] # Mix of Toolset and Tool objects
|
||||
)
|
||||
```
|
||||
|
||||
For more details on working with tools, see the [Tool](../../tools/tool.mdx) and [Toolset](../../tools/toolset.mdx) documentation.
|
||||
|
||||
### Initialization
|
||||
|
||||
To use this integration, you must have a Meta Llama API key. You can provide it with the `LLAMA_API_KEY` environment variable or by using a [Secret](../../concepts/secret-management.mdx).
|
||||
|
||||
Then, install the `meta-llama-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install meta-llama-haystack
|
||||
```
|
||||
|
||||
### Streaming
|
||||
|
||||
`MetaLlamaChatGenerator` supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) responses from the LLM, allowing tokens to be emitted as they are generated. To enable streaming, pass a callable to the `streaming_callback` parameter during initialization.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.meta_llama import (
|
||||
MetaLlamaChatGenerator,
|
||||
)
|
||||
|
||||
llm = MetaLlamaChatGenerator()
|
||||
response = llm.run([ChatMessage.from_user("What are Agentic Pipelines? Be brief.")])
|
||||
print(response["replies"][0].text)
|
||||
```
|
||||
|
||||
With streaming and model routing:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.meta_llama import (
|
||||
MetaLlamaChatGenerator,
|
||||
)
|
||||
|
||||
llm = MetaLlamaChatGenerator(
|
||||
model="Llama-3.3-8B-Instruct",
|
||||
streaming_callback=lambda chunk: print(chunk.content, end="", flush=True),
|
||||
)
|
||||
|
||||
response = llm.run([ChatMessage.from_user("What are Agentic Pipelines? Be brief.")])
|
||||
|
||||
## check the model used for the response
|
||||
print("\n\n Model used: ", response["replies"][0].meta["model"])
|
||||
```
|
||||
|
||||
With multimodal inputs:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage, ImageContent
|
||||
from haystack_integrations.components.generators.meta_llama import (
|
||||
MetaLlamaChatGenerator,
|
||||
)
|
||||
|
||||
llm = MetaLlamaChatGenerator(model="Llama-4-Scout-17B-16E-Instruct-FP8")
|
||||
|
||||
image = ImageContent.from_file_path("apple.jpg")
|
||||
user_message = ChatMessage.from_user(
|
||||
content_parts=["What does the image show? Max 5 words.", image],
|
||||
)
|
||||
|
||||
response = llm.run([user_message])["replies"][0].text
|
||||
print(response)
|
||||
|
||||
# Red apple on straw.
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
## To run this example, you will need to set a `LLAMA_API_KEY` environment variable.
|
||||
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.utils import Secret
|
||||
|
||||
from haystack_integrations.components.generators.meta_llama import (
|
||||
MetaLlamaChatGenerator,
|
||||
)
|
||||
|
||||
## Write documents to InMemoryDocumentStore
|
||||
document_store = InMemoryDocumentStore()
|
||||
document_store.write_documents(
|
||||
[
|
||||
Document(content="My name is Jean and I live in Paris."),
|
||||
Document(content="My name is Mark and I live in Berlin."),
|
||||
Document(content="My name is Giorgio and I live in Rome."),
|
||||
],
|
||||
)
|
||||
|
||||
## Build a RAG pipeline
|
||||
prompt_template = [
|
||||
ChatMessage.from_user(
|
||||
"Given these documents, answer the question.\n"
|
||||
"Documents:\n{% for doc in documents %}{{ doc.content }}{% endfor %}\n"
|
||||
"Question: {{question}}\n"
|
||||
"Answer:",
|
||||
),
|
||||
]
|
||||
|
||||
## Define required variables explicitly
|
||||
prompt_builder = ChatPromptBuilder(
|
||||
template=prompt_template,
|
||||
required_variables={"question", "documents"},
|
||||
)
|
||||
|
||||
retriever = InMemoryBM25Retriever(document_store=document_store)
|
||||
llm = MetaLlamaChatGenerator(
|
||||
api_key=Secret.from_env_var("LLAMA_API_KEY"),
|
||||
streaming_callback=print_streaming_chunk,
|
||||
)
|
||||
|
||||
rag_pipeline = Pipeline()
|
||||
rag_pipeline.add_component("retriever", retriever)
|
||||
rag_pipeline.add_component("prompt_builder", prompt_builder)
|
||||
rag_pipeline.add_component("llm", llm)
|
||||
rag_pipeline.connect("retriever", "prompt_builder.documents")
|
||||
rag_pipeline.connect("prompt_builder", "llm.messages")
|
||||
|
||||
## Ask a question
|
||||
question = "Who lives in Paris?"
|
||||
rag_pipeline.run(
|
||||
{
|
||||
"retriever": {"query": question},
|
||||
"prompt_builder": {"question": question},
|
||||
},
|
||||
)
|
||||
```
|
||||
+176
@@ -0,0 +1,176 @@
|
||||
---
|
||||
title: "MistralChatGenerator"
|
||||
id: mistralchatgenerator
|
||||
slug: "/mistralchatgenerator"
|
||||
description: "This component enables chat completion using Mistral’s text generation models."
|
||||
---
|
||||
|
||||
# MistralChatGenerator
|
||||
|
||||
This component enables chat completion using Mistral’s text generation models.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: The Mistral API key. Can be set with `MISTRAL_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `messages` A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects |
|
||||
| **Output variables** | `replies`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects <br /> <br />`meta`: A list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and so on |
|
||||
| **API reference** | [Mistral](/reference/integrations-mistral) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/mistral |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
This integration supports Mistral’s models provided through the generative endpoint. For a full list of available models, check out the [Mistral documentation](https://docs.mistral.ai/platform/endpoints/#generative-endpoints).
|
||||
|
||||
`MistralChatGenerator` needs a Mistral API key to work. You can write this key in:
|
||||
|
||||
- The `api_key` init parameter using [Secret API](../../concepts/secret-management.mdx)
|
||||
- The `MISTRAL_API_KEY` environment variable (recommended)
|
||||
|
||||
Currently, available models are:
|
||||
|
||||
- `mistral-tiny` (default)
|
||||
- `mistral-small`
|
||||
- `mistral-medium`(soon to be deprecated)
|
||||
- `mistral-large-latest`
|
||||
- `codestral-latest`
|
||||
|
||||
This component needs a list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects to operate. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata.
|
||||
|
||||
Refer to the [Mistral API documentation](https://docs.mistral.ai/api/#operation/createChatCompletion) for more details on the parameters supported by the Mistral API, which you can provide with `generation_kwargs` when running the component.
|
||||
|
||||
### Tool Support
|
||||
|
||||
`MistralChatGenerator` supports function calling through the `tools` parameter, which accepts flexible tool configurations:
|
||||
|
||||
- **A list of Tool objects**: Pass individual tools as a list
|
||||
- **A single Toolset**: Pass an entire Toolset directly
|
||||
- **Mixed Tools and Toolsets**: Combine multiple Toolsets with standalone tools in a single list
|
||||
|
||||
This allows you to organize related tools into logical groups while also including standalone tools as needed.
|
||||
|
||||
```python
|
||||
from haystack.tools import Tool, Toolset
|
||||
from haystack_integrations.components.generators.mistral import MistralChatGenerator
|
||||
|
||||
# Create individual tools
|
||||
weather_tool = Tool(name="weather", description="Get weather info", ...)
|
||||
news_tool = Tool(name="news", description="Get latest news", ...)
|
||||
|
||||
# Group related tools into a toolset
|
||||
math_toolset = Toolset([add_tool, subtract_tool, multiply_tool])
|
||||
|
||||
# Pass mixed tools and toolsets to the generator
|
||||
generator = MistralChatGenerator(
|
||||
tools=[math_toolset, weather_tool, news_tool] # Mix of Toolset and Tool objects
|
||||
)
|
||||
```
|
||||
|
||||
For more details on working with tools, see the [Tool](../../tools/tool.mdx) and [Toolset](../../tools/toolset.mdx) documentation.
|
||||
|
||||
### Streaming
|
||||
|
||||
This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `mistral-haystack` package to use the `MistralChatGenerator`:
|
||||
|
||||
```shell
|
||||
pip install mistral-haystack
|
||||
```
|
||||
|
||||
#### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.mistral import MistralChatGenerator
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.utils import Secret
|
||||
|
||||
generator = MistralChatGenerator(
|
||||
api_key=Secret.from_env_var("MISTRAL_API_KEY"),
|
||||
streaming_callback=print_streaming_chunk,
|
||||
)
|
||||
message = ChatMessage.from_user("What's Natural Language Processing? Be brief.")
|
||||
print(generator.run([message]))
|
||||
```
|
||||
|
||||
With multimodal inputs:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage, ImageContent
|
||||
from haystack_integrations.components.generators.mistral import MistralChatGenerator
|
||||
|
||||
llm = MistralChatGenerator(model="pixtral-12b-2409")
|
||||
|
||||
image = ImageContent.from_file_path("apple.jpg")
|
||||
user_message = ChatMessage.from_user(
|
||||
content_parts=["What does the image show? Max 5 words.", image],
|
||||
)
|
||||
|
||||
response = llm.run([user_message])["replies"][0].text
|
||||
print(response)
|
||||
|
||||
# Red apple on straw.
|
||||
```
|
||||
|
||||
#### In a Pipeline
|
||||
|
||||
Below is an example RAG Pipeline where we answer questions based on the URL contents. We add the contents of the URL into our `messages` in the `ChatPromptBuilder` and generate an answer with the `MistralChatGenerator`.
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
from haystack.components.fetchers import LinkContentFetcher
|
||||
from haystack.components.converters import HTMLToDocument
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
from haystack_integrations.components.generators.mistral import MistralChatGenerator
|
||||
|
||||
fetcher = LinkContentFetcher()
|
||||
converter = HTMLToDocument()
|
||||
prompt_builder = ChatPromptBuilder(variables=["documents"])
|
||||
llm = MistralChatGenerator(
|
||||
streaming_callback=print_streaming_chunk,
|
||||
model="mistral-small",
|
||||
)
|
||||
|
||||
message_template = """Answer the following question based on the contents of the article: {{query}}\n
|
||||
Article: {{documents[0].content}} \n
|
||||
"""
|
||||
messages = [ChatMessage.from_user(message_template)]
|
||||
|
||||
rag_pipeline = Pipeline()
|
||||
rag_pipeline.add_component(name="fetcher", instance=fetcher)
|
||||
rag_pipeline.add_component(name="converter", instance=converter)
|
||||
rag_pipeline.add_component("prompt_builder", prompt_builder)
|
||||
rag_pipeline.add_component("llm", llm)
|
||||
|
||||
rag_pipeline.connect("fetcher.streams", "converter.sources")
|
||||
rag_pipeline.connect("converter.documents", "prompt_builder.documents")
|
||||
rag_pipeline.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
question = "What are the capabilities of Mixtral?"
|
||||
|
||||
result = rag_pipeline.run(
|
||||
{
|
||||
"fetcher": {"urls": ["https://mistral.ai/news/mixtral-of-experts"]},
|
||||
"prompt_builder": {
|
||||
"template_variables": {"query": question},
|
||||
"template": messages,
|
||||
},
|
||||
"llm": {"generation_kwargs": {"max_tokens": 165}},
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Web QA with Mixtral-8x7B-Instruct-v0.1](https://haystack.deepset.ai/cookbook/mixtral-8x7b-for-web-qa)
|
||||
+163
@@ -0,0 +1,163 @@
|
||||
---
|
||||
title: "NvidiaChatGenerator"
|
||||
id: nvidiachatgenerator
|
||||
slug: "/nvidiachatgenerator"
|
||||
description: "This Generator enables chat completion using NVIDIA-hosted models."
|
||||
---
|
||||
|
||||
# NvidiaChatGenerator
|
||||
|
||||
This Generator enables chat completion using NVIDIA-hosted models.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: API key for the NVIDIA NIM. Can be set with `NVIDIA_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `messages`: A list of [ChatMessage](../../concepts/data-classes/chatmessage.mdx) objects |
|
||||
| **Output variables** | `replies`: A list of [ChatMessage](../../concepts/data-classes/chatmessage.mdx) objects |
|
||||
| **API reference** | [NVIDIA API](https://build.nvidia.com/models) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/nvidia |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`NvidiaChatGenerator` enables chat completions using NVIDIA generative models via the NVIDIA API. It is compatible with the [ChatMessage](../../concepts/data-classes/chatmessage.mdx) format for both input and output, ensuring seamless integration in chat-based pipelines.
|
||||
|
||||
You can use LLMs self-hosted with NVIDIA NIM or models hosted on the [NVIDIA API Catalog](https://build.nvidia.com/explore/discover). The default model for this component is `meta/llama-3.1-8b-instruct`.
|
||||
|
||||
To use this integration, you must have an NVIDIA API key. You can provide it with the `NVIDIA_API_KEY` environment variable or by using a [Secret](../../concepts/secret-management.mdx).
|
||||
|
||||
### Tool support
|
||||
|
||||
`NvidiaChatGenerator` supports function calling through the `tools` parameter, which accepts flexible tool configurations:
|
||||
|
||||
- **A list of Tool objects**: Pass individual tools as a list
|
||||
- **A single Toolset**: Pass an entire Toolset directly
|
||||
- **Mixed Tools and Toolsets**: Combine multiple Toolsets with standalone tools in a single list
|
||||
|
||||
This allows you to organize related tools into logical groups while also including standalone tools as needed.
|
||||
|
||||
```python
|
||||
from haystack.tools import Tool, Toolset
|
||||
from haystack_integrations.components.generators.nvidia import NvidiaChatGenerator
|
||||
|
||||
# Create individual tools
|
||||
weather_tool = Tool(name="weather", description="Get weather info", ...)
|
||||
news_tool = Tool(name="news", description="Get latest news", ...)
|
||||
|
||||
# Group related tools into a toolset
|
||||
math_toolset = Toolset([add_tool, subtract_tool, multiply_tool])
|
||||
|
||||
# Pass mixed tools and toolsets to the generator
|
||||
generator = NvidiaChatGenerator(
|
||||
tools=[math_toolset, weather_tool, news_tool] # Mix of Toolset and Tool objects
|
||||
)
|
||||
```
|
||||
|
||||
For more details on working with tools, refer to the [Tool](../../tools/tool.mdx) and [Toolset](../../tools/toolset.mdx) documentation.
|
||||
|
||||
### Streaming
|
||||
|
||||
This generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) responses from the LLM. To enable streaming, pass a callable to the `streaming_callback` parameter during initialization.
|
||||
|
||||
## Usage
|
||||
|
||||
To start using `NvidiaChatGenerator`, install the `nvidia-haystack` package:
|
||||
|
||||
```shell
|
||||
pip install nvidia-haystack
|
||||
```
|
||||
|
||||
You can use `NvidiaChatGenerator` with all the LLMs available in the [NVIDIA API Catalog](https://docs.api.nvidia.com/nim/reference) or with a model deployed using NVIDIA NIM. For more information, refer to the [NVIDIA NIM for LLMs Playbook](https://developer.nvidia.com/docs/nemo-microservices/inference/playbooks/nmi_playbook.html).
|
||||
|
||||
### On its own
|
||||
|
||||
To use LLMs from the NVIDIA API Catalog, specify the `api_url` if needed (the default is `https://integrate.api.nvidia.com/v1`) and your API key. You can get your API key from the [NVIDIA API Catalog](https://build.nvidia.com/explore/discover).
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.components.generators.nvidia import NvidiaChatGenerator
|
||||
|
||||
generator = NvidiaChatGenerator(
|
||||
model="meta/llama-3.1-8b-instruct",
|
||||
api_key=Secret.from_env_var("NVIDIA_API_KEY"),
|
||||
)
|
||||
|
||||
messages = [ChatMessage.from_user("What's Natural Language Processing? Be brief.")]
|
||||
result = generator.run(messages)
|
||||
print(result["replies"])
|
||||
print(result["meta"])
|
||||
```
|
||||
|
||||
With multimodal inputs:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage, ImageContent
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.components.generators.nvidia import NvidiaChatGenerator
|
||||
|
||||
llm = NvidiaChatGenerator(
|
||||
model="meta/llama-3.2-11b-vision-instruct",
|
||||
api_key=Secret.from_env_var("NVIDIA_API_KEY"),
|
||||
)
|
||||
|
||||
image = ImageContent.from_file_path("apple.jpg")
|
||||
user_message = ChatMessage.from_user(
|
||||
content_parts=[
|
||||
"What does the image show? Max 5 words.",
|
||||
image,
|
||||
],
|
||||
)
|
||||
|
||||
response = llm.run([user_message])["replies"][0].text
|
||||
print(response)
|
||||
# Red apple on straw.
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.utils import Secret
|
||||
from haystack_integrations.components.generators.nvidia import NvidiaChatGenerator
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", ChatPromptBuilder())
|
||||
pipe.add_component(
|
||||
"llm",
|
||||
NvidiaChatGenerator(
|
||||
model="meta/llama-3.1-8b-instruct",
|
||||
api_key=Secret.from_env_var("NVIDIA_API_KEY"),
|
||||
),
|
||||
)
|
||||
pipe.connect("prompt_builder", "llm")
|
||||
|
||||
country = "Germany"
|
||||
system_message = ChatMessage.from_system(
|
||||
"You are an assistant giving out valuable information to language learners.",
|
||||
)
|
||||
messages = [
|
||||
system_message,
|
||||
ChatMessage.from_user("What's the official language of {{ country }}?"),
|
||||
]
|
||||
|
||||
res = pipe.run(
|
||||
data={
|
||||
"prompt_builder": {
|
||||
"template_variables": {"country": country},
|
||||
"template": messages,
|
||||
},
|
||||
},
|
||||
)
|
||||
print(res)
|
||||
```
|
||||
|
||||
## Related
|
||||
|
||||
- Cookbook: [Haystack RAG Pipeline with Self-Deployed AI models using NVIDIA NIMs](https://haystack.deepset.ai/cookbook/rag-with-nims)
|
||||
+147
@@ -0,0 +1,147 @@
|
||||
---
|
||||
title: "NvidiaGenerator"
|
||||
id: nvidiagenerator
|
||||
slug: "/nvidiagenerator"
|
||||
description: "This Generator enables text generation using NVIDIA-hosted models."
|
||||
---
|
||||
|
||||
# NvidiaGenerator
|
||||
|
||||
This Generator enables text generation using NVIDIA-hosted models.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`PromptBuilder`](../builders/promptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: API key for the NVIDIA NIM. Can be set with `NVIDIA_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `prompt`: A string containing the prompt for the LLM |
|
||||
| **Output variables** | `replies`: A list of strings with all the replies generated by the LLM <br /> <br />`meta`: A list of dictionaries with the metadata associated with each reply, such as token count and others |
|
||||
| **API reference** | [NVIDIA](/reference/integrations-nvidia) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/nvidia |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`NvidiaGenerator` provides an interface for generating text using LLMs self-hosted with NVIDIA NIM or models hosted on the [NVIDIA API Catalog](https://build.nvidia.com/explore/discover).
|
||||
|
||||
## Usage
|
||||
|
||||
To start using `NvidiaGenerator`, install the `nvidia-haystack` package:
|
||||
|
||||
```shell
|
||||
pip install nvidia-haystack
|
||||
```
|
||||
|
||||
You can use `NvidiaGenerator` with all the LLMs available in the [NVIDIA API Catalog](https://docs.api.nvidia.com/nim/reference) or with a model deployed using NVIDIA NIM. For more information, refer to the [NVIDIA NIM for LLMs Playbook](https://developer.nvidia.com/docs/nemo-microservices/inference/playbooks/nmi_playbook.html).
|
||||
|
||||
### On its own
|
||||
|
||||
To use LLMs from the NVIDIA API Catalog, specify the `api_url` and your API key. You can get your API key from the [NVIDIA API Catalog](https://build.nvidia.com/explore/discover).
|
||||
|
||||
`NvidiaGenerator` uses the `NVIDIA_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with the `api_key` parameter:
|
||||
|
||||
```python
|
||||
from haystack.utils.auth import Secret
|
||||
from haystack_integrations.components.generators.nvidia import NvidiaGenerator
|
||||
|
||||
generator = NvidiaGenerator(
|
||||
model="meta/llama-3.1-70b-instruct",
|
||||
api_url="https://integrate.api.nvidia.com/v1",
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
model_arguments={
|
||||
"temperature": 0.2,
|
||||
"top_p": 0.7,
|
||||
"max_tokens": 1024,
|
||||
},
|
||||
)
|
||||
result = generator.run(prompt="What is the answer?")
|
||||
print(result["replies"])
|
||||
print(result["meta"])
|
||||
```
|
||||
|
||||
To use a locally deployed model, set the `api_url` to your localhost and set `api_key` to `None`:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.nvidia import NvidiaGenerator
|
||||
|
||||
generator = NvidiaGenerator(
|
||||
model="meta/llama-3.1-8b-instruct",
|
||||
api_url="http://localhost:9999/v1",
|
||||
api_key=None,
|
||||
model_arguments={
|
||||
"temperature": 0.2,
|
||||
},
|
||||
)
|
||||
result = generator.run(prompt="What is the answer?")
|
||||
print(result["replies"])
|
||||
print(result["meta"])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
The following example shows a RAG pipeline:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline, Document
|
||||
from haystack.utils.auth import Secret
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.components.builders.prompt_builder import PromptBuilder
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.generators.nvidia import NvidiaGenerator
|
||||
|
||||
docstore = InMemoryDocumentStore()
|
||||
docstore.write_documents(
|
||||
[
|
||||
Document(content="Rome is the capital of Italy"),
|
||||
Document(content="Paris is the capital of France"),
|
||||
],
|
||||
)
|
||||
|
||||
query = "What is the capital of France?"
|
||||
|
||||
template = """
|
||||
Given the following information, answer the question.
|
||||
|
||||
Context:
|
||||
{% for document in documents %}
|
||||
{{ document.content }}
|
||||
{% endfor %}
|
||||
|
||||
Question: {{ query }}?
|
||||
"""
|
||||
|
||||
pipe = Pipeline()
|
||||
|
||||
pipe.add_component("retriever", InMemoryBM25Retriever(document_store=docstore))
|
||||
pipe.add_component("prompt_builder", PromptBuilder(template=template))
|
||||
pipe.add_component(
|
||||
"llm",
|
||||
NvidiaGenerator(
|
||||
model="meta/llama-3.1-70b-instruct",
|
||||
api_url="https://integrate.api.nvidia.com/v1",
|
||||
api_key=Secret.from_token("<your-api-key>"),
|
||||
model_arguments={
|
||||
"temperature": 0.2,
|
||||
"top_p": 0.7,
|
||||
"max_tokens": 1024,
|
||||
},
|
||||
),
|
||||
)
|
||||
pipe.connect("retriever", "prompt_builder.documents")
|
||||
pipe.connect("prompt_builder", "llm")
|
||||
|
||||
res = pipe.run(
|
||||
{
|
||||
"prompt_builder": {"query": query},
|
||||
"retriever": {"query": query},
|
||||
},
|
||||
)
|
||||
|
||||
print(res)
|
||||
```
|
||||
|
||||
## Related
|
||||
|
||||
- Cookbook: [Haystack RAG Pipeline with Self-Deployed AI models using NVIDIA NIMs](https://haystack.deepset.ai/cookbook/rag-with-nims)
|
||||
+279
@@ -0,0 +1,279 @@
|
||||
---
|
||||
title: "OllamaChatGenerator"
|
||||
id: ollamachatgenerator
|
||||
slug: "/ollamachatgenerator"
|
||||
description: "This component enables chat completion using an LLM running on Ollama."
|
||||
---
|
||||
|
||||
# OllamaChatGenerator
|
||||
|
||||
This component enables chat completion using an LLM running on Ollama.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects representing the chat |
|
||||
| **Output variables** | `replies`: A list of LLM’s alternative replies |
|
||||
| **API reference** | [Ollama](/reference/integrations-ollama) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/ollama |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
[Ollama](https://github.com/jmorganca/ollama) is a project focused on running LLMs locally. Internally, it uses the quantized GGUF format by default. This means it is possible to run LLMs on standard machines (even without GPUs) without having to handle complex installation procedures.
|
||||
|
||||
`OllamaChatGenerator` supports models running on Ollama, such as `llama2` and `mixtral`. Find the full list of supported models [here](https://ollama.ai/library).
|
||||
|
||||
`OllamaChatGenerator` needs a `model` name and a `url` to work. By default, it uses `"orca-mini"` model and `"http://localhost:11434"` url.
|
||||
|
||||
The way to operate with `OllamaChatGenerator` is by using `ChatMessage` objects. [ChatMessage](../../concepts/data-classes/chatmessage.mdx) is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata. See the [usage](#usage) section for an example.
|
||||
|
||||
### Tool Support
|
||||
|
||||
`OllamaChatGenerator` supports function calling through the `tools` parameter, which accepts flexible tool configurations:
|
||||
|
||||
- **A list of Tool objects**: Pass individual tools as a list
|
||||
- **A single Toolset**: Pass an entire Toolset directly
|
||||
- **Mixed Tools and Toolsets**: Combine multiple Toolsets with standalone tools in a single list
|
||||
|
||||
This allows you to organize related tools into logical groups while also including standalone tools as needed.
|
||||
|
||||
```python
|
||||
from haystack.tools import Tool, Toolset
|
||||
from haystack_integrations.components.generators.ollama import OllamaChatGenerator
|
||||
|
||||
# Create individual tools
|
||||
weather_tool = Tool(name="weather", description="Get weather info", ...)
|
||||
news_tool = Tool(name="news", description="Get latest news", ...)
|
||||
|
||||
# Group related tools into a toolset
|
||||
math_toolset = Toolset([add_tool, subtract_tool, multiply_tool])
|
||||
|
||||
# Pass mixed tools and toolsets to the generator
|
||||
generator = OllamaChatGenerator(
|
||||
model="llama2",
|
||||
tools=[math_toolset, weather_tool, news_tool] # Mix of Toolset and Tool objects
|
||||
)
|
||||
```
|
||||
|
||||
For more details on working with tools, see the [Tool](../../tools/tool.mdx) and [Toolset](../../tools/toolset.mdx) documentation.
|
||||
|
||||
### Streaming
|
||||
|
||||
You can stream output as it’s generated. Pass a callback to `streaming_callback`. Use the built-in `print_streaming_chunk` to print text tokens and tool events (tool calls and tool results).
|
||||
|
||||
```python
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
|
||||
## Configure any `Generator` or `ChatGenerator` with a streaming callback
|
||||
component = SomeGeneratorOrChatGenerator(streaming_callback=print_streaming_chunk)
|
||||
|
||||
## If this is a `ChatGenerator`, pass a list of messages:
|
||||
## from haystack.dataclasses import ChatMessage
|
||||
## component.run([ChatMessage.from_user("Your question here")])
|
||||
|
||||
## If this is a (non-chat) `Generator`, pass a prompt:
|
||||
## component.run({"prompt": "Your prompt here"})
|
||||
```
|
||||
|
||||
:::info
|
||||
Streaming works only with a single response. If a provider supports multiple candidates, set `n=1`.
|
||||
:::
|
||||
|
||||
See our [Streaming Support](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) docs to learn more how `StreamingChunk` works and how to write a custom callback.
|
||||
|
||||
Give preference to `print_streaming_chunk` by default. Write a custom callback only if you need a specific transport (for example, SSE/WebSocket) or custom UI formatting.
|
||||
|
||||
### Streaming with Tools
|
||||
|
||||
You can combine streaming with tool calling. Pass both `tools` and `streaming_callback`; when the model decides to invoke a tool, the streamed chunks carry tool-call deltas instead of text tokens, and the final reconstructed `ChatMessage` exposes the resolved `tool_calls` list on `replies[0]`.
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.dataclasses.streaming_chunk import StreamingChunk
|
||||
from haystack.tools import create_tool_from_function
|
||||
from haystack_integrations.components.generators.ollama import OllamaChatGenerator
|
||||
|
||||
|
||||
def get_weather(city: str) -> str:
|
||||
"""Get current weather for a city."""
|
||||
return f"Sunny, 22°C in {city}"
|
||||
|
||||
|
||||
def callback(chunk: StreamingChunk) -> None:
|
||||
if chunk.tool_calls:
|
||||
print(f"[tool delta] {chunk.tool_calls}")
|
||||
elif chunk.content:
|
||||
print(chunk.content, end="", flush=True)
|
||||
|
||||
|
||||
generator = OllamaChatGenerator(
|
||||
model="llama3.1:8b",
|
||||
generation_kwargs={"temperature": 0.0},
|
||||
tools=[create_tool_from_function(get_weather)],
|
||||
streaming_callback=callback,
|
||||
)
|
||||
|
||||
response = generator.run(
|
||||
messages=[ChatMessage.from_user(
|
||||
"What's the weather in Berlin? Use the get_weather tool."
|
||||
)]
|
||||
)
|
||||
|
||||
# Final reconstructed message: tool_calls populated, text is None
|
||||
assistant_message = response["replies"][0]
|
||||
print(assistant_message.tool_calls)
|
||||
# -> [ToolCall(tool_name='get_weather', arguments={'city': 'Berlin'}, ...)]
|
||||
```
|
||||
|
||||
You can use the built-in `print_streaming_chunk` callback (which handles both text tokens and tool events) instead of writing your own.
|
||||
|
||||
## Usage
|
||||
|
||||
1. You need a running instance of Ollama. The installation instructions are [in the Ollama GitHub repository](https://github.com/jmorganca/ollama).
|
||||
A fast way to run Ollama is using Docker:
|
||||
|
||||
```bash
|
||||
docker run -d -p 11434:11434 --name ollama ollama/ollama:latest
|
||||
```
|
||||
|
||||
2. You need to download or pull the desired LLM. The model library is available on the [Ollama website](https://ollama.ai/library).
|
||||
If you are using Docker, you can, for example, pull the Zephyr model:
|
||||
|
||||
```bash
|
||||
docker exec ollama ollama pull zephyr
|
||||
```
|
||||
|
||||
If you already installed Ollama in your system, you can execute:
|
||||
|
||||
```bash
|
||||
ollama pull zephyr
|
||||
```
|
||||
|
||||
:::tip[Choose a specific version of a model]
|
||||
|
||||
You can also specify a tag to choose a specific (quantized) version of your model. The available tags are shown in the model card of the Ollama models library. This is an [example](https://ollama.ai/library/zephyr/tags) for Zephyr.
|
||||
In this case, simply run
|
||||
|
||||
```shell
|
||||
# ollama pull model:tag
|
||||
ollama pull zephyr:7b-alpha-q3_K_S
|
||||
```
|
||||
:::
|
||||
|
||||
3. You also need to install the `ollama-haystack` package:
|
||||
|
||||
```bash
|
||||
pip install ollama-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.ollama import OllamaChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
generator = OllamaChatGenerator(model="zephyr",
|
||||
url = "http://localhost:11434",
|
||||
generation_kwargs={
|
||||
"num_predict": 100,
|
||||
"temperature": 0.9,
|
||||
})
|
||||
|
||||
messages = [ChatMessage.from_system("\nYou are a helpful, respectful and honest assistant"),
|
||||
ChatMessage.from_user("What's Natural Language Processing?")]
|
||||
|
||||
print(generator.run(messages=messages))
|
||||
>> {
|
||||
"replies": [
|
||||
ChatMessage(
|
||||
_role=<ChatRole.ASSISTANT: 'assistant'>,
|
||||
_content=[
|
||||
TextContent(
|
||||
text=(
|
||||
"Natural Language Processing (NLP) is a subfield of "
|
||||
"Artificial Intelligence that deals with understanding, "
|
||||
"interpreting, and generating human language in a meaningful "
|
||||
"way. It enables tasks such as language translation, sentiment "
|
||||
"analysis, and text summarization."
|
||||
)
|
||||
)
|
||||
],
|
||||
_name=None,
|
||||
_meta={
|
||||
"model": "zephyr",...
|
||||
}
|
||||
)
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
With multimodal inputs:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage, ImageContent
|
||||
from haystack_integrations.components.generators.ollama import OllamaChatGenerator
|
||||
|
||||
llm = OllamaChatGenerator(model="llava", url="http://localhost:11434")
|
||||
|
||||
image = ImageContent.from_file_path("apple.jpg")
|
||||
user_message = ChatMessage.from_user(
|
||||
content_parts=["What does the image show? Max 5 words.", image],
|
||||
)
|
||||
|
||||
response = llm.run([user_message])["replies"][0].text
|
||||
print(response)
|
||||
|
||||
# Red apple on straw.
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
```python
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack_integrations.components.generators.ollama import OllamaChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack import Pipeline
|
||||
|
||||
## no parameter init, we don't use any runtime template variables
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
generator = OllamaChatGenerator(model="zephyr",
|
||||
url = "http://localhost:11434",
|
||||
generation_kwargs={
|
||||
"temperature": 0.9,
|
||||
})
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", generator)
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
location = "Berlin"
|
||||
messages = [ChatMessage.from_system("Always respond in Spanish even if some input data is in other languages."),
|
||||
ChatMessage.from_user("Tell me about {{location}}")]
|
||||
print(pipe.run(data={"prompt_builder": {"template_variables":{"location": location}, "template": messages}}))
|
||||
|
||||
>> {
|
||||
"llm": {
|
||||
"replies": [
|
||||
ChatMessage(
|
||||
_role=<ChatRole.ASSISTANT: 'assistant'>,
|
||||
_content=[
|
||||
TextContent(
|
||||
text=(
|
||||
"Berlín es la capital y la mayor ciudad de Alemania. "
|
||||
"Está ubicada en el estado federado de Berlín, y tiene más..."
|
||||
)
|
||||
)
|
||||
],
|
||||
_name=None,
|
||||
_meta={
|
||||
"model": "zephyr",...
|
||||
}
|
||||
)
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
+154
@@ -0,0 +1,154 @@
|
||||
---
|
||||
title: "OllamaGenerator"
|
||||
id: ollamagenerator
|
||||
slug: "/ollamagenerator"
|
||||
description: "A component that provides an interface to generate text using an LLM running on Ollama."
|
||||
---
|
||||
|
||||
# OllamaGenerator
|
||||
|
||||
A component that provides an interface to generate text using an LLM running on Ollama.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`PromptBuilder`](../builders/promptbuilder.mdx) |
|
||||
| **Mandatory run variables** | `prompt`: A string containing the prompt for the LLM |
|
||||
| **Output variables** | `replies`: A list of strings with all the replies generated by the LLM <br /> <br />`meta`: A list of dictionaries with the metadata associated with each reply, such as token count and others |
|
||||
| **API reference** | [Ollama](/reference/integrations-ollama) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/ollama |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`OllamaGenerator` provides an interface to generate text using an LLM running on Ollama.
|
||||
|
||||
`OllamaGenerator` needs a `model` name and a `url` to work. By default, it uses `"orca-mini"` model and `"http://localhost:11434"` url.
|
||||
|
||||
[Ollama](https://github.com/jmorganca/ollama) is a project focused on running LLMs locally. Internally, it uses the quantized GGUF format by default. This means it is possible to run LLMs on standard machines (even without GPUs) without having to go through complex installation procedures.
|
||||
|
||||
### Streaming
|
||||
|
||||
This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
1. You need a running instance of Ollama. You can find the installation instructions [here](https://github.com/jmorganca/ollama).
|
||||
A fast way to run Ollama is using Docker:
|
||||
|
||||
```shell
|
||||
docker run -d -p 11434:11434 --name ollama ollama/ollama:latest
|
||||
```
|
||||
|
||||
2. You need to download or pull the desired LLM. The model library is available on the [Ollama website](https://ollama.ai/library).
|
||||
If you are using Docker, you can, for example, pull the Zephyr model:
|
||||
|
||||
```shell
|
||||
docker exec ollama ollama pull zephyr
|
||||
```
|
||||
|
||||
If you have already installed Ollama in your system, you can execute:
|
||||
|
||||
```shell
|
||||
ollama pull zephyr
|
||||
```
|
||||
|
||||
:::tip[Choose a specific version of a model]
|
||||
|
||||
You can also specify a tag to choose a specific (quantized) version of your model. The available tags are shown in the model card of the Ollama models library. This is an [example](https://ollama.ai/library/zephyr/tags) for Zephyr.
|
||||
In this case, simply run
|
||||
|
||||
```shell
|
||||
# ollama pull model:tag
|
||||
ollama pull zephyr:7b-alpha-q3_K_S
|
||||
```
|
||||
:::
|
||||
|
||||
3. You also need to install the `ollama-haystack` package:
|
||||
|
||||
```shell
|
||||
pip install ollama-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
Here's how the `OllamaGenerator` would work just on its own:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.ollama import OllamaGenerator
|
||||
|
||||
generator = OllamaGenerator(
|
||||
model="zephyr",
|
||||
url="http://localhost:11434",
|
||||
generation_kwargs={
|
||||
"num_predict": 100,
|
||||
"temperature": 0.9,
|
||||
},
|
||||
)
|
||||
|
||||
print(generator.run("Who is the best American actor?"))
|
||||
|
||||
## {'replies': ['I do not have the ability to form opinions or preferences.
|
||||
## However, some of the most acclaimed american actors in recent years include
|
||||
## denzel washington, tom hanks, leonardo dicaprio, matthew mcconaughey...'],
|
||||
## 'meta': [{'model': 'zephyr', ...}]}
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.ollama import OllamaGenerator
|
||||
|
||||
from haystack import Pipeline, Document
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.components.builders.prompt_builder import PromptBuilder
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
template = """
|
||||
Given the following information, answer the question.
|
||||
|
||||
Context:
|
||||
{% for document in documents %}
|
||||
{{ document.content }}
|
||||
{% endfor %}
|
||||
|
||||
Question: {{ query }}?
|
||||
"""
|
||||
|
||||
docstore = InMemoryDocumentStore()
|
||||
docstore.write_documents(
|
||||
[
|
||||
Document(content="I really like summer"),
|
||||
Document(content="My favorite sport is soccer"),
|
||||
Document(content="I don't like reading sci-fi books"),
|
||||
Document(content="I don't like crowded places"),
|
||||
],
|
||||
)
|
||||
|
||||
generator = OllamaGenerator(
|
||||
model="zephyr",
|
||||
url="http://localhost:11434",
|
||||
generation_kwargs={
|
||||
"num_predict": 100,
|
||||
"temperature": 0.9,
|
||||
},
|
||||
)
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("retriever", InMemoryBM25Retriever(document_store=docstore))
|
||||
pipe.add_component("prompt_builder", PromptBuilder(template=template))
|
||||
pipe.add_component("llm", generator)
|
||||
pipe.connect("retriever", "prompt_builder.documents")
|
||||
pipe.connect("prompt_builder", "llm")
|
||||
|
||||
result = pipe.run({"prompt_builder": {"query": query}, "retriever": {"query": query}})
|
||||
|
||||
print(result)
|
||||
|
||||
## {'llm': {'replies': ['Based on the provided context, it seems that you enjoy
|
||||
## soccer and summer. Unfortunately, there is no direct information given about
|
||||
## what else you enjoy...'],
|
||||
## 'meta': [{'model': 'zephyr', ...]}}
|
||||
```
|
||||
+274
@@ -0,0 +1,274 @@
|
||||
---
|
||||
title: "OpenAIChatGenerator"
|
||||
id: openaichatgenerator
|
||||
slug: "/openaichatgenerator"
|
||||
description: "`OpenAIChatGenerator` enables chat completion using OpenAI’s large language models (LLMs)."
|
||||
---
|
||||
|
||||
# OpenAIChatGenerator
|
||||
|
||||
`OpenAIChatGenerator` enables chat completion using OpenAI's large language models (LLMs).
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: An OpenAI API key. Can be set with `OPENAI_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects representing the chat |
|
||||
| **Output variables** | `replies`: A list of alternative replies of the LLM to the input chat |
|
||||
| **API reference** | [Generators](/reference/generators-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/generators/chat/openai.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`OpenAIChatGenerator` supports OpenAI models starting from gpt-3.5-turbo and later (gpt-4, gpt-4-turbo, and so on).
|
||||
|
||||
`OpenAIChatGenerator` needs an OpenAI key to work. It uses an ` OPENAI_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`:
|
||||
|
||||
```python
|
||||
generator = OpenAIChatGenerator(model="gpt-4o-mini")
|
||||
```
|
||||
|
||||
Then, the component needs a list of `ChatMessage` objects to operate. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata. See the [usage](#usage) section for an example.
|
||||
|
||||
You can pass any chat completion parameters valid for the `openai.ChatCompletion.create` method directly to `OpenAIChatGenerator` using the `generation_kwargs` parameter, both at initialization and to `run()` method. For more details on the parameters supported by the OpenAI API, refer to the [OpenAI documentation](https://platform.openai.com/docs/api-reference/chat).
|
||||
|
||||
`OpenAIChatGenerator` can support custom deployments of your OpenAI models through the `api_base_url` init parameter.
|
||||
|
||||
### Structured Output
|
||||
|
||||
`OpenAIChatGenerator` supports structured output generation, allowing you to receive responses in a predictable format. You can use Pydantic models or JSON schemas to define the structure of the output through the `response_format` parameter in `generation_kwargs`.
|
||||
|
||||
This is useful when you need to extract structured data from text or generate responses that match a specific format.
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
class NobelPrizeInfo(BaseModel):
|
||||
recipient_name: str
|
||||
award_year: int
|
||||
category: str
|
||||
achievement_description: str
|
||||
nationality: str
|
||||
|
||||
client = OpenAIChatGenerator(
|
||||
model="gpt-4o-2024-08-06",
|
||||
generation_kwargs={"response_format": NobelPrizeInfo}
|
||||
)
|
||||
|
||||
response = client.run(messages=[
|
||||
ChatMessage.from_user(
|
||||
"In 2021, American scientist David Julius received the Nobel Prize in"
|
||||
" Physiology or Medicine for his groundbreaking discoveries on how the human body"
|
||||
" senses temperature and touch."
|
||||
)
|
||||
])
|
||||
print(response["replies"][0].text)
|
||||
|
||||
>> {"recipient_name":"David Julius","award_year":2021,"category":"Physiology or Medicine",
|
||||
>> "achievement_description":"David Julius was awarded for his transformative findings
|
||||
>> regarding the molecular mechanisms underlying the human body's sense of temperature
|
||||
>> and touch. Through innovative experiments, he identified specific receptors responsible
|
||||
>> for detecting heat and mechanical stimuli, ranging from gentle touch to pain-inducing
|
||||
>> pressure.","nationality":"American"}
|
||||
```
|
||||
|
||||
:::info[Model Compatibility and Limitations]
|
||||
|
||||
- Pydantic models and JSON schemas are supported for latest models starting from `gpt-4o-2024-08-06`.
|
||||
- Older models only support basic JSON mode through `{"type": "json_object"}`. For details, see [OpenAI JSON mode documentation](https://platform.openai.com/docs/guides/structured-outputs#json-mode).
|
||||
- Streaming limitation: When using streaming with structured outputs, you must provide a JSON schema instead of a Pydantic model for `response_format`.
|
||||
- For complete information, check the [OpenAI Structured Outputs documentation](https://platform.openai.com/docs/guides/structured-outputs).
|
||||
:::
|
||||
|
||||
### Streaming
|
||||
|
||||
You can stream output as it’s generated. Pass a callback to `streaming_callback`. Use the built-in `print_streaming_chunk` to print text tokens and tool events (tool calls and tool results).
|
||||
|
||||
```python
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
|
||||
## Configure any `Generator` or `ChatGenerator` with a streaming callback
|
||||
component = SomeGeneratorOrChatGenerator(streaming_callback=print_streaming_chunk)
|
||||
|
||||
## If this is a `ChatGenerator`, pass a list of messages:
|
||||
## from haystack.dataclasses import ChatMessage
|
||||
## component.run([ChatMessage.from_user("Your question here")])
|
||||
|
||||
## If this is a (non-chat) `Generator`, pass a prompt:
|
||||
## component.run({"prompt": "Your prompt here"})
|
||||
```
|
||||
|
||||
:::info
|
||||
Streaming works only with a single response. If a provider supports multiple candidates, set `n=1`.
|
||||
:::
|
||||
|
||||
See our [Streaming Support](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) docs to learn more how `StreamingChunk` works and how to write a custom callback.
|
||||
|
||||
Give preference to `print_streaming_chunk` by default. Write a custom callback only if you need a specific transport (for example, SSE/WebSocket) or custom UI formatting.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
|
||||
client = OpenAIChatGenerator()
|
||||
response = client.run(
|
||||
[ChatMessage.from_user("What's Natural Language Processing? Be brief.")]
|
||||
)
|
||||
print(response)
|
||||
|
||||
>> {'replies': [ChatMessage(_role=<ChatRole.ASSISTANT: 'assistant'>, _content=
|
||||
>> [TextContent(text='Natural Language Processing (NLP) is a field of artificial
|
||||
>> intelligence that focuses on the interaction between computers and humans through
|
||||
>> natural language. It involves enabling machines to understand, interpret, and
|
||||
>> generate human language in a meaningful way, facilitating tasks such as
|
||||
>> language translation, sentiment analysis, and text summarization.')],
|
||||
>> _name=None, _meta={'model': 'gpt-4o-mini-2024-07-18', 'index': 0,
|
||||
>> 'finish_reason': 'stop', 'usage': {'completion_tokens': 59, 'prompt_tokens': 15,
|
||||
>> 'total_tokens': 74, 'completion_tokens_details': {'accepted_prediction_tokens':
|
||||
>> 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0},
|
||||
>> 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}})]}
|
||||
```
|
||||
|
||||
With streaming:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
|
||||
client = OpenAIChatGenerator(streaming_callback=lambda chunk: print(chunk.content, end="", flush=True))
|
||||
response = client.run(
|
||||
[ChatMessage.from_user("What's Natural Language Processing? Be brief.")]
|
||||
)
|
||||
print(response)
|
||||
|
||||
>> Natural Language Processing (NLP) is a field of artificial intelligence that
|
||||
>> focuses on the interaction between computers and humans through natural language.
|
||||
>> It involves enabling machines to understand, interpret, and generate human
|
||||
>> language in a way that is both meaningful and useful. NLP encompasses various
|
||||
>> tasks, including speech recognition, language translation, sentiment analysis,
|
||||
>> and text summarization.{'replies': [ChatMessage(_role=<ChatRole.ASSISTANT:
|
||||
>> 'assistant'>, _content=[TextContent(text='Natural Language Processing (NLP) is a
|
||||
>> field of artificial intelligence that focuses on the interaction between computers
|
||||
>> and humans through natural language. It involves enabling machines to understand,
|
||||
>> interpret, and generate human language in a way that is both meaningful and
|
||||
>> useful. NLP encompasses various tasks, including speech recognition, language
|
||||
>> translation, sentiment analysis, and text summarization.')], _name=None, _meta={'
|
||||
>> model': 'gpt-4o-mini-2024-07-18', 'index': 0, 'finish_reason': 'stop',
|
||||
>> 'completion_start_time': '2025-05-15T13:32:16.572912', 'usage': None})]}
|
||||
```
|
||||
|
||||
With multimodal inputs:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage, ImageContent
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
|
||||
llm = OpenAIChatGenerator(model="gpt-4o-mini")
|
||||
|
||||
image = ImageContent.from_file_path("apple.jpg", detail="low")
|
||||
user_message = ChatMessage.from_user(content_parts=[
|
||||
"What does the image show? Max 5 words.",
|
||||
image
|
||||
])
|
||||
|
||||
response = llm.run([user_message])["replies"][0].text
|
||||
print(response)
|
||||
|
||||
>>> Red apple on straw.
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
```python
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack import Pipeline
|
||||
from haystack.utils import Secret
|
||||
|
||||
## no parameter init, we don't use any runtime template variables
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
llm = OpenAIChatGenerator(api_key=Secret.from_env_var("OPENAI_API_KEY"), model="gpt-4o-mini")
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
location = "Berlin"
|
||||
messages = [ChatMessage.from_system("Always respond in German even if some input data is in other languages."),
|
||||
ChatMessage.from_user("Tell me about {{location}}")]
|
||||
pipe.run(data={"prompt_builder": {"template_variables":{"location": location}, "template": messages}})
|
||||
|
||||
>> {'llm': {'replies': [ChatMessage(_role=<ChatRole.ASSISTANT: 'assistant'>,
|
||||
>> _content=[TextContent(text='Berlin ist die Hauptstadt Deutschlands und eine der
|
||||
>> bedeutendsten Städte Europas. Es ist bekannt für ihre reiche Geschichte,
|
||||
>> kulturelle Vielfalt und kreative Scene. \n\nDie Stadt hat eine bewegte
|
||||
>> Vergangenheit, die stark von der Teilung zwischen Ost- und Westberlin während
|
||||
>> des Kalten Krieges geprägt war. Die Berliner Mauer, die von 1961 bis 1989 die
|
||||
>> Stadt teilte, ist heute ein Symbol für die Wiedervereinigung und die Freiheit.
|
||||
>> \n\nBerlin bietet eine Fülle von Sehenswürdigkeiten, darunter das Brandenburger
|
||||
>> Tor, den Reichstag, die Museumsinsel und den Alexanderplatz. Die Stadt ist auch
|
||||
>> für ihre lebendige Kunst- und Musikszene bekannt, mit zahlreichen Galerien,
|
||||
>> Theatern und Clubs. ')], _name=None, _meta={'model': 'gpt-4o-mini-2024-07-18',
|
||||
>> 'index': 0, 'finish_reason': 'stop', 'usage': {'completion_tokens': 260,
|
||||
>> 'prompt_tokens': 29, 'total_tokens': 289, 'completion_tokens_details':
|
||||
>> {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0,
|
||||
>> 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0,
|
||||
>> 'cached_tokens': 0}}})]}}
|
||||
```
|
||||
|
||||
### In YAML
|
||||
|
||||
This is the YAML representation of the pipeline shown above. It dynamically constructs a prompt and generates an answer using a chat model.
|
||||
|
||||
```yaml
|
||||
components:
|
||||
llm:
|
||||
init_parameters:
|
||||
api_base_url: null
|
||||
api_key:
|
||||
env_vars:
|
||||
- OPENAI_API_KEY
|
||||
strict: true
|
||||
type: env_var
|
||||
generation_kwargs: {}
|
||||
http_client_kwargs: null
|
||||
max_retries: null
|
||||
model: gpt-4o-mini
|
||||
organization: null
|
||||
streaming_callback: null
|
||||
timeout: null
|
||||
tools: null
|
||||
tools_strict: false
|
||||
type: haystack.components.generators.chat.openai.OpenAIChatGenerator
|
||||
prompt_builder:
|
||||
init_parameters:
|
||||
required_variables: null
|
||||
template: null
|
||||
variables: null
|
||||
type: haystack.components.builders.chat_prompt_builder.ChatPromptBuilder
|
||||
connection_type_validation: true
|
||||
connections:
|
||||
- receiver: llm.messages
|
||||
sender: prompt_builder.prompt
|
||||
max_runs_per_component: 100
|
||||
metadata: {}
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
:notebook: Tutorial: [Building a Chat Application with Function Calling](https://haystack.deepset.ai/tutorials/40_building_chat_application_with_function_calling)
|
||||
|
||||
🧑🍳 Cookbook: [Function Calling with OpenAIChatGenerator](https://haystack.deepset.ai/cookbook/function_calling_with_openaichatgenerator)
|
||||
+196
@@ -0,0 +1,196 @@
|
||||
---
|
||||
title: "OpenAIGenerator"
|
||||
id: openaigenerator
|
||||
slug: "/openaigenerator"
|
||||
description: "`OpenAIGenerator` enables text generation using OpenAI's large language models (LLMs)."
|
||||
---
|
||||
|
||||
# OpenAIGenerator
|
||||
|
||||
`OpenAIGenerator` enables text generation using OpenAI's large language models (LLMs).
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`PromptBuilder`](../builders/promptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: An OpenAI API key. Can be set with `OPENAI_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `prompt`: A string containing the prompt for the LLM |
|
||||
| **Output variables** | `replies`: A list of strings with all the replies generated by the LLM <br /> <br />`meta`: A list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and so on |
|
||||
| **API reference** | [Generators](/reference/generators-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/generators/openai.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`OpenAIGenerator` supports OpenAI models starting from gpt-3.5-turbo and later (gpt-4, gpt-4-turbo, and so on).
|
||||
|
||||
`OpenAIGenerator` needs an OpenAI key to work. It uses an `OPENAI_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key`:
|
||||
|
||||
```
|
||||
generator = OpenAIGenerator(api_key=Secret.from_token("<your-api-key>"), model="gpt-4o-mini")
|
||||
```
|
||||
|
||||
Then, the component needs a prompt to operate, but you can pass any text generation parameters valid for the `openai.ChatCompletion.create` method directly to this component using the `generation_kwargs` parameter, both at initialization and to `run()` method. For more details on the parameters supported by the OpenAI API, refer to the [OpenAI documentation](https://platform.openai.com/docs/api-reference/chat).
|
||||
|
||||
`OpenAIGenerator` supports custom deployments of your OpenAI models through the `api_base_url` init parameter.
|
||||
|
||||
### Streaming
|
||||
|
||||
`OpenAIGenerator` supports streaming the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter. Note that streaming the tokens is only compatible with generating a single response, so `n` must be set to 1 for streaming to work.
|
||||
|
||||
:::info
|
||||
This component is designed for text generation, not for chat. If you want to use OpenAI LLMs for chat, use [`OpenAIChatGenerator`](openaichatgenerator.mdx) instead.
|
||||
:::
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage:
|
||||
|
||||
```python
|
||||
from haystack.components.generators import OpenAIGenerator
|
||||
from haystack.utils import Secret
|
||||
|
||||
client = OpenAIGenerator(model="gpt-4", api_key=Secret.from_token("<your-api-key>"))
|
||||
response = client.run("What's Natural Language Processing? Be brief.")
|
||||
print(response)
|
||||
|
||||
>>> {'replies': ['Natural Language Processing, often abbreviated as NLP, is a field
|
||||
of artificial intelligence that focuses on the interaction between computers
|
||||
and humans through natural language. The primary aim of NLP is to enable
|
||||
computers to understand, interpret, and generate human language in a valuable way.'],
|
||||
'meta': [{'model': 'gpt-4-0613', 'index': 0, 'finish_reason':
|
||||
'stop', 'usage': {'prompt_tokens': 16, 'completion_tokens': 53,
|
||||
'total_tokens': 69}}]}
|
||||
```
|
||||
|
||||
With streaming:
|
||||
|
||||
```python
|
||||
from haystack.components.generators import OpenAIGenerator
|
||||
from haystack.utils import Secret
|
||||
|
||||
client = OpenAIGenerator(streaming_callback=lambda chunk: print(chunk.content, end="", flush=True))
|
||||
response = client.run("What's Natural Language Processing? Be brief.")
|
||||
print(response)
|
||||
|
||||
>>> Natural Language Processing (NLP) is a branch of artificial
|
||||
intelligence that focuses on the interaction between computers and human
|
||||
language. It involves enabling computers to understand, interpret,and respond
|
||||
to natural human language in a way that is both meaningful and useful.
|
||||
>>> {'replies': ['Natural Language Processing (NLP) is a branch of artificial
|
||||
intelligence that focuses on the interaction between computers and human
|
||||
language. It involves enabling computers to understand, interpret,and respond
|
||||
to natural human language in a way that is both meaningful and useful.'],
|
||||
'meta': [{'model': 'gpt-4o-mini', 'index': 0, 'finish_reason':
|
||||
'stop', 'usage': {'prompt_tokens': 16, 'completion_tokens': 49,
|
||||
'total_tokens': 65}}]}
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
Here's an example of RAG Pipeline:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.components.builders.prompt_builder import PromptBuilder
|
||||
from haystack.components.generators import OpenAIGenerator
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack import Document
|
||||
from haystack.utils import Secret
|
||||
|
||||
docstore = InMemoryDocumentStore()
|
||||
docstore.write_documents([Document(content="Rome is the capital of Italy"), Document(content="Paris is the capital of France")])
|
||||
|
||||
query = "What is the capital of France?"
|
||||
|
||||
template = """
|
||||
Given the following information, answer the question.
|
||||
|
||||
Context:
|
||||
{% for document in documents %}
|
||||
{{ document.content }}
|
||||
{% endfor %}
|
||||
|
||||
Question: {{ query }}?
|
||||
"""
|
||||
pipe = Pipeline()
|
||||
|
||||
pipe.add_component("retriever", InMemoryBM25Retriever(document_store=docstore))
|
||||
pipe.add_component("prompt_builder", PromptBuilder(template=template))
|
||||
pipe.add_component("llm", OpenAIGenerator(api_key=Secret.from_env_var("OPENAI_API_KEY")))
|
||||
pipe.connect("retriever", "prompt_builder.documents")
|
||||
pipe.connect("prompt_builder", "llm")
|
||||
|
||||
res=pipe.run({
|
||||
"prompt_builder": {
|
||||
"query": query
|
||||
},
|
||||
"retriever": {
|
||||
"query": query
|
||||
}
|
||||
})
|
||||
|
||||
print(res)
|
||||
```
|
||||
|
||||
### In YAML
|
||||
|
||||
This is the YAML representation of the RAG pipeline shown above. It retrieves documents based on a query, constructs a prompt using a template, and generates an answer using a chat model.
|
||||
|
||||
```yaml
|
||||
components:
|
||||
llm:
|
||||
init_parameters:
|
||||
api_base_url: null
|
||||
api_key:
|
||||
env_vars:
|
||||
- OPENAI_API_KEY
|
||||
strict: true
|
||||
type: env_var
|
||||
generation_kwargs: {}
|
||||
http_client_kwargs: null
|
||||
max_retries: null
|
||||
model: gpt-5-mini
|
||||
organization: null
|
||||
streaming_callback: null
|
||||
system_prompt: null
|
||||
timeout: null
|
||||
type: haystack.components.generators.openai.OpenAIGenerator
|
||||
prompt_builder:
|
||||
init_parameters:
|
||||
required_variables: null
|
||||
template: "\nGiven the following information, answer the question.\n\nContext:\n\
|
||||
{% for document in documents %}\n {{ document.content }}\n{% endfor %}\n\n\
|
||||
Question: {{ query }}?\n"
|
||||
variables: null
|
||||
type: haystack.components.builders.prompt_builder.PromptBuilder
|
||||
retriever:
|
||||
init_parameters:
|
||||
document_store:
|
||||
init_parameters:
|
||||
bm25_algorithm: BM25L
|
||||
bm25_parameters: {}
|
||||
bm25_tokenization_regex: (?u)\b\w+\b
|
||||
embedding_similarity_function: dot_product
|
||||
index: 64e4f9ab-87fb-47fd-b390-dabcfda61447
|
||||
return_embedding: true
|
||||
type: haystack.document_stores.in_memory.document_store.InMemoryDocumentStore
|
||||
filter_policy: replace
|
||||
filters: null
|
||||
scale_score: false
|
||||
top_k: 10
|
||||
type: haystack.components.retrievers.in_memory.bm25_retriever.InMemoryBM25Retriever
|
||||
connection_type_validation: true
|
||||
connections:
|
||||
- receiver: prompt_builder.documents
|
||||
sender: retriever.documents
|
||||
- receiver: llm.prompt
|
||||
sender: prompt_builder.prompt
|
||||
max_runs_per_component: 100
|
||||
metadata: {}
|
||||
```
|
||||
+306
@@ -0,0 +1,306 @@
|
||||
---
|
||||
title: "OpenAIResponsesChatGenerator"
|
||||
id: openairesponseschatgenerator
|
||||
slug: "/openairesponseschatgenerator"
|
||||
description: "`OpenAIResponsesChatGenerator` enables chat completion using OpenAI's Responses API with support for reasoning models."
|
||||
---
|
||||
|
||||
# OpenAIResponsesChatGenerator
|
||||
|
||||
`OpenAIResponsesChatGenerator` enables chat completion using OpenAI's Responses API with support for reasoning models.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: An OpenAI API key. Can be set with `OPENAI_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects representing the chat |
|
||||
| **Output variables** | `replies`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects containing the generated responses |
|
||||
| **API reference** | [Generators](/reference/generators-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/generators/chat/openai_responses.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`OpenAIResponsesChatGenerator` uses OpenAI's Responses API to generate chat completions. It supports gpt-4 and o-series models (reasoning models like o1, o3-mini). The default model is `gpt-5-mini`.
|
||||
|
||||
The Responses API is designed for reasoning-capable models and supports features like reasoning summaries, multi-turn conversations with previous response IDs, and structured outputs.
|
||||
|
||||
The component requires a list of `ChatMessage` objects to operate. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`), and optional metadata. See the [usage](#usage) section for examples.
|
||||
|
||||
You can pass any parameters valid for the OpenAI Responses API directly to `OpenAIResponsesChatGenerator` using the `generation_kwargs` parameter, both at initialization and to the `run()` method. For more details on the parameters supported by the OpenAI API, refer to the [OpenAI Responses API documentation](https://platform.openai.com/docs/api-reference/responses).
|
||||
|
||||
`OpenAIResponsesChatGenerator` can support custom deployments of your OpenAI models through the `api_base_url` init parameter.
|
||||
|
||||
### Authentication
|
||||
|
||||
`OpenAIResponsesChatGenerator` needs an OpenAI key to work. It uses an `OPENAI_API_KEY` environment variable by default. Otherwise, you can pass an API key at initialization with `api_key` using a [`Secret`](../../concepts/secret-management.mdx):
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import OpenAIResponsesChatGenerator
|
||||
from haystack.utils import Secret
|
||||
|
||||
generator = OpenAIResponsesChatGenerator(api_key=Secret.from_token("<your-api-key>"))
|
||||
```
|
||||
|
||||
### Reasoning Support
|
||||
|
||||
One of the key features of the Responses API is support for reasoning models. You can configure reasoning behavior using the `reasoning` parameter in `generation_kwargs`:
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import OpenAIResponsesChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
client = OpenAIResponsesChatGenerator(
|
||||
generation_kwargs={"reasoning": {"effort": "medium", "summary": "auto"}},
|
||||
)
|
||||
|
||||
messages = [
|
||||
ChatMessage.from_user(
|
||||
"What's the most efficient sorting algorithm for nearly sorted data?",
|
||||
),
|
||||
]
|
||||
response = client.run(messages)
|
||||
print(response)
|
||||
```
|
||||
|
||||
The `reasoning` parameter accepts:
|
||||
- `effort`: Level of reasoning effort - `"low"`, `"medium"`, or `"high"`
|
||||
- `summary`: How to generate reasoning summaries - `"auto"` or `"generate_summary": True/False`
|
||||
|
||||
:::note
|
||||
OpenAI does not return the actual reasoning tokens, but you can view the summary if enabled. For more details, see the [OpenAI Reasoning documentation](https://platform.openai.com/docs/guides/reasoning).
|
||||
:::
|
||||
|
||||
### Multi-turn Conversations
|
||||
|
||||
The Responses API supports multi-turn conversations using `previous_response_id`. You can pass the response ID from a previous turn to maintain conversation context:
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import OpenAIResponsesChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
client = OpenAIResponsesChatGenerator()
|
||||
|
||||
# First turn
|
||||
messages = [ChatMessage.from_user("What's quantum computing?")]
|
||||
response = client.run(messages)
|
||||
response_id = response["replies"][0].meta.get("id")
|
||||
|
||||
# Second turn - reference previous response
|
||||
messages = [ChatMessage.from_user("Can you explain that in simpler terms?")]
|
||||
response = client.run(messages, generation_kwargs={"previous_response_id": response_id})
|
||||
```
|
||||
|
||||
### Structured Output
|
||||
|
||||
`OpenAIResponsesChatGenerator` supports structured output generation through the `text_format` and `text` parameters in `generation_kwargs`:
|
||||
|
||||
- **`text_format`**: Pass a Pydantic model to define the structure
|
||||
- **`text`**: Pass a JSON schema directly
|
||||
|
||||
**Using a Pydantic model**:
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel
|
||||
from haystack.components.generators.chat import OpenAIResponsesChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
|
||||
class BookInfo(BaseModel):
|
||||
title: str
|
||||
author: str
|
||||
year: int
|
||||
genre: str
|
||||
|
||||
|
||||
client = OpenAIResponsesChatGenerator(
|
||||
model="gpt-4o",
|
||||
generation_kwargs={"text_format": BookInfo},
|
||||
)
|
||||
|
||||
response = client.run(
|
||||
messages=[
|
||||
ChatMessage.from_user(
|
||||
"Extract book information: '1984 by George Orwell, published in 1949, is a dystopian novel.'",
|
||||
),
|
||||
],
|
||||
)
|
||||
print(response["replies"][0].text)
|
||||
```
|
||||
|
||||
**Using a JSON schema**:
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import OpenAIResponsesChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
json_schema = {
|
||||
"format": {
|
||||
"type": "json_schema",
|
||||
"name": "BookInfo",
|
||||
"strict": True,
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"title": {"type": "string"},
|
||||
"author": {"type": "string"},
|
||||
"year": {"type": "integer"},
|
||||
"genre": {"type": "string"},
|
||||
},
|
||||
"required": ["title", "author", "year", "genre"],
|
||||
"additionalProperties": False,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
client = OpenAIResponsesChatGenerator(
|
||||
model="gpt-4o",
|
||||
generation_kwargs={"text": json_schema},
|
||||
)
|
||||
|
||||
response = client.run(
|
||||
messages=[
|
||||
ChatMessage.from_user(
|
||||
"Extract book information: '1984 by George Orwell, published in 1949, is a dystopian novel.'",
|
||||
),
|
||||
],
|
||||
)
|
||||
print(response["replies"][0].text)
|
||||
```
|
||||
|
||||
:::info[Model Compatibility and Limitations]
|
||||
- Both Pydantic models and JSON schemas are supported for latest models starting from GPT-4o.
|
||||
- If both `text_format` and `text` are provided, `text_format` takes precedence and the JSON schema passed to `text` is ignored.
|
||||
- Streaming is not supported when using structured outputs.
|
||||
- Older models only support basic JSON mode through `{"type": "json_object"}`. For details, see [OpenAI JSON mode documentation](https://platform.openai.com/docs/guides/structured-outputs#json-mode).
|
||||
- For complete information, check the [OpenAI Structured Outputs documentation](https://platform.openai.com/docs/guides/structured-outputs).
|
||||
:::
|
||||
|
||||
### Tool Support
|
||||
|
||||
`OpenAIResponsesChatGenerator` supports function calling through the `tools` parameter. It accepts flexible tool configurations:
|
||||
|
||||
- **Haystack Tool objects and Toolsets**: Pass Haystack `Tool` objects or `Toolset` objects, including mixed lists of both
|
||||
- **OpenAI/MCP tool definitions**: Pass pre-defined OpenAI or MCP tool definitions as dictionaries
|
||||
|
||||
Note that you cannot mix Haystack tools and OpenAI/MCP tools in the same call - choose one format or the other.
|
||||
|
||||
```python
|
||||
from haystack.tools import Tool
|
||||
from haystack.components.generators.chat import OpenAIResponsesChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
|
||||
def get_weather(city: str) -> str:
|
||||
"""Get weather information for a city."""
|
||||
return f"Weather in {city}: Sunny, 22°C"
|
||||
|
||||
|
||||
weather_tool = Tool(
|
||||
name="get_weather",
|
||||
description="Get current weather for a city",
|
||||
function=get_weather,
|
||||
parameters={"type": "object", "properties": {"city": {"type": "string"}}},
|
||||
)
|
||||
|
||||
generator = OpenAIResponsesChatGenerator(tools=[weather_tool])
|
||||
messages = [ChatMessage.from_user("What's the weather in Paris?")]
|
||||
response = generator.run(messages)
|
||||
```
|
||||
|
||||
You can control strict schema adherence with the `tools_strict` parameter. When set to `True` (default is `False`), the model will follow the tool schema exactly. Note that the Responses API has its own strictness enforcement mechanisms independent of this parameter.
|
||||
|
||||
For more details on working with tools, see the [Tool](../../tools/tool.mdx) and [Toolset](../../tools/toolset.mdx) documentation.
|
||||
|
||||
### Streaming
|
||||
|
||||
You can stream output as it's generated. Pass a callback to `streaming_callback`. Use the built-in `print_streaming_chunk` to print text tokens and tool events (tool calls and tool results).
|
||||
|
||||
```python
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
|
||||
## Configure any `Generator` or `ChatGenerator` with a streaming callback
|
||||
component = SomeGeneratorOrChatGenerator(streaming_callback=print_streaming_chunk)
|
||||
|
||||
## If this is a `ChatGenerator`, pass a list of messages:
|
||||
## from haystack.dataclasses import ChatMessage
|
||||
## component.run([ChatMessage.from_user("Your question here")])
|
||||
|
||||
## If this is a (non-chat) `Generator`, pass a prompt:
|
||||
## component.run({"prompt": "Your prompt here"})
|
||||
```
|
||||
|
||||
:::info
|
||||
Streaming works only with a single response. If a provider supports multiple candidates, set `n=1`.
|
||||
:::
|
||||
|
||||
See our [Streaming Support](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) docs to learn more how `StreamingChunk` works and how to write a custom callback.
|
||||
|
||||
Give preference to `print_streaming_chunk` by default. Write a custom callback only if you need a specific transport (for example, SSE/WebSocket) or custom UI formatting.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
Here is an example of using `OpenAIResponsesChatGenerator` independently with reasoning and streaming:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.components.generators.chat import OpenAIResponsesChatGenerator
|
||||
from haystack.components.generators.utils import print_streaming_chunk
|
||||
|
||||
client = OpenAIResponsesChatGenerator(
|
||||
streaming_callback=print_streaming_chunk,
|
||||
generation_kwargs={"reasoning": {"effort": "high", "summary": "auto"}},
|
||||
)
|
||||
response = client.run(
|
||||
[
|
||||
ChatMessage.from_user(
|
||||
"Solve this logic puzzle: If all roses are flowers and some flowers fade quickly, can we conclude that some roses fade quickly?",
|
||||
),
|
||||
],
|
||||
)
|
||||
print(response["replies"][0].reasoning) # Access reasoning summary if available
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
This example shows a pipeline that uses `ChatPromptBuilder` to create dynamic prompts and `OpenAIResponsesChatGenerator` with reasoning enabled to generate explanations of complex topics:
|
||||
|
||||
```python
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIResponsesChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack import Pipeline
|
||||
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
llm = OpenAIResponsesChatGenerator(
|
||||
generation_kwargs={"reasoning": {"effort": "low", "summary": "auto"}},
|
||||
)
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
topic = "quantum computing"
|
||||
messages = [
|
||||
ChatMessage.from_system(
|
||||
"You are a helpful assistant that explains complex topics clearly.",
|
||||
),
|
||||
ChatMessage.from_user("Explain {{topic}} in simple terms"),
|
||||
]
|
||||
result = pipe.run(
|
||||
data={
|
||||
"prompt_builder": {
|
||||
"template_variables": {"topic": topic},
|
||||
"template": messages,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
print(result)
|
||||
```
|
||||
+161
@@ -0,0 +1,161 @@
|
||||
---
|
||||
title: "OpenRouterChatGenerator"
|
||||
id: openrouterchatgenerator
|
||||
slug: "/openrouterchatgenerator"
|
||||
description: "This component enables chat completion with any model hosted on [OpenRouter](https://openrouter.ai/)."
|
||||
---
|
||||
|
||||
# OpenRouterChatGenerator
|
||||
|
||||
This component enables chat completion with any model hosted on [OpenRouter](https://openrouter.ai/).
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: An OpenRouter API key. Can be set with `OPENROUTER_API_KEY` env variable or passed to `init()` method. |
|
||||
| **Mandatory run variables** | `messages`: A list of [ChatMessage](../../concepts/data-classes/chatmessage.mdx) objects |
|
||||
| **Output variables** | `replies`: A list of [ChatMessage](../../concepts/data-classes/chatmessage.mdx) objects |
|
||||
| **API reference** | [OpenRouter](/reference/integrations-openrouter) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/openrouter |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `OpenRouterChatGenerator` enables you to use models from multiple providers (such as `openai/gpt-4o`, `anthropic/claude-3.5-sonnet`, and others) by making chat completion calls to the [OpenRouter API](https://openrouter.ai/docs/quickstart).
|
||||
|
||||
This generator also supports OpenRouter-specific features such as:
|
||||
|
||||
- Provider routing and model fallback that are configurable with the `generation_kwargs` parameter during initialization or runtime.
|
||||
- Custom HTTP headers that can be supplied using the `extra_headers` parameter.
|
||||
|
||||
This component uses the same `ChatMessage` format as other Haystack Chat Generators for structured input and output. For more information, see the [ChatMessage documentation](../../concepts/data-classes/chatmessage.mdx).
|
||||
|
||||
### Tool Support
|
||||
|
||||
`OpenRouterChatGenerator` supports function calling through the `tools` parameter, which accepts flexible tool configurations:
|
||||
|
||||
- **A list of Tool objects**: Pass individual tools as a list
|
||||
- **A single Toolset**: Pass an entire Toolset directly
|
||||
- **Mixed Tools and Toolsets**: Combine multiple Toolsets with standalone tools in a single list
|
||||
|
||||
This allows you to organize related tools into logical groups while also including standalone tools as needed.
|
||||
|
||||
```python
|
||||
from haystack.tools import Tool, Toolset
|
||||
from haystack_integrations.components.generators.openrouter import OpenRouterChatGenerator
|
||||
|
||||
# Create individual tools
|
||||
weather_tool = Tool(name="weather", description="Get weather info", ...)
|
||||
news_tool = Tool(name="news", description="Get latest news", ...)
|
||||
|
||||
# Group related tools into a toolset
|
||||
math_toolset = Toolset([add_tool, subtract_tool, multiply_tool])
|
||||
|
||||
# Pass mixed tools and toolsets to the generator
|
||||
generator = OpenRouterChatGenerator(
|
||||
tools=[math_toolset, weather_tool, news_tool] # Mix of Toolset and Tool objects
|
||||
)
|
||||
```
|
||||
|
||||
For more details on working with tools, see the [Tool](../../tools/tool.mdx) and [Toolset](../../tools/toolset.mdx) documentation.
|
||||
|
||||
### Initialization
|
||||
|
||||
To use this integration, you must have an active OpenRouter subscription with sufficient credits and an API key. You can provide it with the `OPENROUTER_API_KEY` environment variable or by using a [Secret](../../concepts/secret-management.mdx).
|
||||
|
||||
Then, install the `openrouter-haystack` integration:
|
||||
|
||||
```shell
|
||||
pip install openrouter-haystack
|
||||
```
|
||||
|
||||
### Streaming
|
||||
|
||||
`OpenRouterChatGenerator` supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) responses from the LLM, allowing tokens to be emitted as they are generated. To enable streaming, pass a callable to the `streaming_callback` parameter during initialization.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.openrouter import (
|
||||
OpenRouterChatGenerator,
|
||||
)
|
||||
|
||||
client = OpenRouterChatGenerator()
|
||||
response = client.run([ChatMessage.from_user("What are Agentic Pipelines? Be brief.")])
|
||||
print(response["replies"][0].text)
|
||||
```
|
||||
|
||||
With streaming and model routing:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.openrouter import (
|
||||
OpenRouterChatGenerator,
|
||||
)
|
||||
|
||||
client = OpenRouterChatGenerator(
|
||||
model="openrouter/auto",
|
||||
streaming_callback=lambda chunk: print(chunk.content, end="", flush=True),
|
||||
)
|
||||
|
||||
response = client.run([ChatMessage.from_user("What are Agentic Pipelines? Be brief.")])
|
||||
|
||||
## check the model used for the response
|
||||
print("\n\n Model used: ", response["replies"][0].meta["model"])
|
||||
```
|
||||
|
||||
With multimodal inputs:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage, ImageContent
|
||||
from haystack_integrations.components.generators.openrouter import (
|
||||
OpenRouterChatGenerator,
|
||||
)
|
||||
|
||||
llm = OpenRouterChatGenerator(model="anthropic/claude-3-5-sonnet")
|
||||
|
||||
image = ImageContent.from_file_path("apple.jpg")
|
||||
user_message = ChatMessage.from_user(
|
||||
content_parts=["What does the image show? Max 5 words.", image],
|
||||
)
|
||||
|
||||
response = llm.run([user_message])["replies"][0].text
|
||||
print(response)
|
||||
|
||||
# Red apple on straw.
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.openrouter import (
|
||||
OpenRouterChatGenerator,
|
||||
)
|
||||
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
llm = OpenRouterChatGenerator(model="openai/gpt-4o-mini")
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
pipe.connect("builder.prompt", "llm.messages")
|
||||
|
||||
messages = [
|
||||
ChatMessage.from_system("Give brief answers."),
|
||||
ChatMessage.from_user("Tell me about {{city}}"),
|
||||
]
|
||||
|
||||
response = pipe.run(
|
||||
data={"builder": {"template": messages, "template_variables": {"city": "Berlin"}}},
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
+107
@@ -0,0 +1,107 @@
|
||||
---
|
||||
title: "SagemakerGenerator"
|
||||
id: sagemakergenerator
|
||||
slug: "/sagemakergenerator"
|
||||
description: "This component enables text generation using LLMs deployed on Amazon Sagemaker."
|
||||
---
|
||||
|
||||
# SagemakerGenerator
|
||||
|
||||
This component enables text generation using LLMs deployed on Amazon Sagemaker.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`PromptBuilder`](../builders/promptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `model`: The model to use <br /> <br />`aws_access_key_id`: AWS access key ID. Can be set with `AWS_ACCESS_KEY_ID` env var. <br /> <br />`aws_secret_access_key`: AWS secret access key. Can be set with `AWS_SECRET_ACCESS_KEY` env var. |
|
||||
| **Mandatory run variables** | `prompt`: A string containing the prompt for the LLM |
|
||||
| **Output variables** | `replies`: A list of strings with all the replies generated by the LLM <br /> <br />`meta`: A list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and so on |
|
||||
| **API reference** | [Amazon Sagemaker](/reference/integrations-amazon-sagemaker) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/amazon_sagemaker |
|
||||
|
||||
</div>
|
||||
|
||||
`SagemakerGenerator` allows you to make use of models deployed on [AWS SageMaker](https://docs.aws.amazon.com/sagemaker/latest/dg/whatis.html).
|
||||
|
||||
## Parameters Overview
|
||||
|
||||
`SagemakerGenerator` needs AWS credentials to work. Set the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables.
|
||||
|
||||
You also need to specify your Sagemaker endpoint at initialization time for the component to work. Pass the endpoint name to the `model` parameter like this:
|
||||
|
||||
```python
|
||||
generator = SagemakerGenerator(model="jumpstart-dft-hf-llm-falcon-7b-instruct-bf16")
|
||||
```
|
||||
|
||||
Additionally, you can pass any text generation parameters valid for your specific model directly to `SagemakerGenerator` using the `generation_kwargs` parameter, both at initialization and to `run()` method.
|
||||
|
||||
If your model also needs custom attributes, pass those as a dictionary at initialization time by setting the `aws_custom_attributes` parameter.
|
||||
|
||||
One notable family of models that needs these custom parameters is Llama2, which needs to be initialized with `{"accept_eula": True}` :
|
||||
|
||||
```python
|
||||
generator = SagemakerGenerator(
|
||||
model="jumpstart-dft-meta-textgenerationneuron-llama-2-7b",
|
||||
aws_custom_attributes={"accept_eula": True},
|
||||
)
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
You need to install `amazon-sagemaker-haystack` package to use the `SagemakerGenerator`:
|
||||
|
||||
```shell
|
||||
pip install amazon-sagemaker-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.amazon_sagemaker import SagemakerGenerator
|
||||
|
||||
client = SagemakerGenerator(model="jumpstart-dft-hf-llm-falcon-7b-instruct-bf16")
|
||||
response = client.run("Briefly explain what NLP is in one sentence.")
|
||||
print(response)
|
||||
|
||||
>>> {'replies': ["Natural Language Processing (NLP) is a subfield of artificial intelligence and computational linguistics that focuses on the interaction between computers and human languages..."],
|
||||
'metadata': [{}]}
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
In a RAG pipeline:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.amazon_sagemaker import (
|
||||
SagemakerGenerator,
|
||||
)
|
||||
from haystack import Pipeline
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.components.builders import PromptBuilder
|
||||
|
||||
template = """
|
||||
Given the following information, answer the question.
|
||||
|
||||
Context:
|
||||
{% for document in documents %}
|
||||
{{ document.content }}
|
||||
{% endfor %}
|
||||
|
||||
Question: What's the official language of {{ country }}?
|
||||
"""
|
||||
pipe = Pipeline()
|
||||
|
||||
pipe.add_component("retriever", InMemoryBM25Retriever(document_store=docstore))
|
||||
pipe.add_component("prompt_builder", PromptBuilder(template=template))
|
||||
pipe.add_component(
|
||||
"llm",
|
||||
SagemakerGenerator(model="jumpstart-dft-hf-llm-falcon-7b-instruct-bf16"),
|
||||
)
|
||||
pipe.connect("retriever", "prompt_builder.documents")
|
||||
pipe.connect("prompt_builder", "llm")
|
||||
|
||||
pipe.run({"prompt_builder": {"country": "France"}})
|
||||
```
|
||||
+119
@@ -0,0 +1,119 @@
|
||||
---
|
||||
title: "STACKITChatGenerator"
|
||||
id: stackitchatgenerator
|
||||
slug: "/stackitchatgenerator"
|
||||
description: "This component enables chat completions using the STACKIT API."
|
||||
---
|
||||
|
||||
# STACKITChatGenerator
|
||||
|
||||
This component enables chat completions using the STACKIT API.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`ChatPromptBuilder`](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `model`: The model used through the STACKIT API |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects |
|
||||
| **Output variables** | `replies`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects <br /> <br />`meta`: A list of dictionaries with the metadata associated with each reply (such as token count, finish reason, and so on) |
|
||||
| **API reference** | [STACKIT](/reference/integrations-stackit) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/stackit |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`STACKITChatGenerator` enables text generation models served by STACKIT through their API.
|
||||
|
||||
### Parameters
|
||||
|
||||
To use the `STACKITChatGenerator`, ensure you have set a `STACKIT_API_KEY` as an environment variable. Alternatively, provide the API key as another environment variable or a token by setting
|
||||
`api_key` and using Haystack’s [secret management](../../concepts/secret-management.mdx).
|
||||
|
||||
Set your preferred supported model with the `model` parameter when initializing the component. See the full list of all supported models on the [STACKIT website](https://docs.stackit.cloud/stackit/en/models-licenses-319914532.html).
|
||||
|
||||
Optionally, you can change the default `api_base_url`, which is `"https://api.openai-compat.model-serving.eu01.onstackit.cloud/v1"`.
|
||||
|
||||
You can pass any text generation parameters valid for the STACKIT Chat Completion API directly to this component with the `generation_kwargs` parameter in the init or run methods.
|
||||
|
||||
The component needs a list of `ChatMessage` objects to run. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata. Find out more about it [ChatMessage documentation](../../concepts/data-classes/chatmessage.mdx).
|
||||
|
||||
### Streaming
|
||||
|
||||
This ChatGenerator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly into the output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `stackit-haystack` package to use the `STACKITChatGenerator`:
|
||||
|
||||
```shell
|
||||
pip install stackit-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.stackit import STACKITChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
generator = STACKITChatGenerator(model="neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8")
|
||||
|
||||
result = generator.run([ChatMessage.from_user("Tell me a joke.")])
|
||||
print(result)
|
||||
```
|
||||
|
||||
With multimodal inputs:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage, ImageContent
|
||||
from haystack_integrations.components.generators.stackit import STACKITChatGenerator
|
||||
|
||||
llm = STACKITChatGenerator(model="meta-llama/Llama-3.2-11B-Vision-Instruct")
|
||||
|
||||
image = ImageContent.from_file_path("apple.jpg")
|
||||
user_message = ChatMessage.from_user(
|
||||
content_parts=["What does the image show? Max 5 words.", image],
|
||||
)
|
||||
|
||||
response = llm.run([user_message])["replies"][0].text
|
||||
print(response)
|
||||
|
||||
# Red apple on straw.
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
You can also use `STACKITChatGenerator` in your pipeline.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
from haystack_integrations.components.generators.stackit import STACKITChatGenerator
|
||||
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
llm = STACKITChatGenerator(model="neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8")
|
||||
|
||||
messages = [ChatMessage.from_user("Question: {{question}} \\n")]
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("prompt_builder", prompt_builder)
|
||||
pipeline.add_component("llm", llm)
|
||||
|
||||
pipeline.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
result = pipeline.run(
|
||||
{
|
||||
"prompt_builder": {
|
||||
"template_variables": {"question": "Tell me a joke."},
|
||||
"template": messages,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
print(result)
|
||||
```
|
||||
|
||||
For an example of streaming in a pipeline, refer to the examples in the STACKIT integration [repository](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/stackit/examples) and on its dedicated [integration page](https://haystack.deepset.ai/integrations/stackit).
|
||||
+142
@@ -0,0 +1,142 @@
|
||||
---
|
||||
title: "TogetherAIChatGenerator"
|
||||
id: togetheraichatgenerator
|
||||
slug: "/togetheraichatgenerator"
|
||||
description: "This component enables chat completion using models hosted on Together AI."
|
||||
---
|
||||
|
||||
# TogetherAIChatGenerator
|
||||
|
||||
This component enables chat completion using models hosted on Together AI.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: A Together API key. Can be set with `TOGETHER_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects |
|
||||
| **Output variables** | `replies`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects |
|
||||
| **API reference** | [TogetherAI](/reference/integrations-togetherai) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/togetherai |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`TogetherAIChatGenerator` supports models hosted on [Together AI](https://docs.together.ai/intro), such as `meta-llama/Llama-3.3-70B-Instruct-Turbo`. For the full list of supported models, see [Together AI documentation](https://docs.together.ai/docs/chat-models).
|
||||
|
||||
This component needs a list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects to operate. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata.
|
||||
|
||||
You can pass any text generation parameters valid for the Together AI chat completion API directly to this component using the `generation_kwargs` parameter in `__init__` or the `generation_kwargs` parameter in `run` method. For more details on the parameters supported by the Together AI API, see [Together AI API documentation](https://docs.together.ai/reference/chat-completions-1).
|
||||
|
||||
To use this integration, you need to have an active TogetherAI subscription with sufficient credits and an API key. You can provide it with:
|
||||
|
||||
- The `TOGETHER_API_KEY` environment variable (recommended)
|
||||
- The `api_key` init parameter and Haystack [Secret](../../concepts/secret-management.mdx) API: `Secret.from_token("your-api-key-here")`
|
||||
|
||||
By default, the component uses Together AI's OpenAI-compatible base URL `https://api.together.xyz/v1`, which you can override with `api_base_url` if needed.
|
||||
|
||||
### Tool Support
|
||||
|
||||
`TogetherAIChatGenerator` supports function calling through the `tools` parameter, which accepts flexible tool configurations:
|
||||
|
||||
- **A list of Tool objects**: Pass individual tools as a list
|
||||
- **A single Toolset**: Pass an entire Toolset directly
|
||||
- **Mixed Tools and Toolsets**: Combine multiple Toolsets with standalone tools in a single list
|
||||
|
||||
This allows you to organize related tools into logical groups while also including standalone tools as needed.
|
||||
|
||||
```python
|
||||
from haystack.tools import Tool, Toolset
|
||||
from haystack_integrations.components.generators.togetherai import TogetherAIChatGenerator
|
||||
|
||||
# Create individual tools
|
||||
weather_tool = Tool(name="weather", description="Get weather info", ...)
|
||||
news_tool = Tool(name="news", description="Get latest news", ...)
|
||||
|
||||
# Group related tools into a toolset
|
||||
math_toolset = Toolset([add_tool, subtract_tool, multiply_tool])
|
||||
|
||||
# Pass mixed tools and toolsets to the generator
|
||||
generator = TogetherAIChatGenerator(
|
||||
tools=[math_toolset, weather_tool, news_tool] # Mix of Toolset and Tool objects
|
||||
)
|
||||
```
|
||||
|
||||
For more details on working with tools, see the [Tool](../../tools/tool.mdx) and [Toolset](../../tools/toolset.mdx) documentation.
|
||||
|
||||
### Streaming
|
||||
|
||||
`TogetherAIChatGenerator` supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) responses from the LLM, allowing tokens to be emitted as they are generated. To enable streaming, pass a callable to the `streaming_callback` parameter during initialization.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `togetherai-haystack` package to use the `TogetherAIChatGenerator`:
|
||||
|
||||
```shell
|
||||
pip install togetherai-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.togetherai import (
|
||||
TogetherAIChatGenerator,
|
||||
)
|
||||
|
||||
client = TogetherAIChatGenerator()
|
||||
response = client.run([ChatMessage.from_user("What are Agentic Pipelines? Be brief.")])
|
||||
print(response["replies"][0].text)
|
||||
```
|
||||
|
||||
With streaming:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.togetherai import (
|
||||
TogetherAIChatGenerator,
|
||||
)
|
||||
|
||||
client = TogetherAIChatGenerator(
|
||||
model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
|
||||
streaming_callback=lambda chunk: print(chunk.content, end="", flush=True),
|
||||
)
|
||||
|
||||
response = client.run([ChatMessage.from_user("What are Agentic Pipelines? Be brief.")])
|
||||
|
||||
# check the model used for the response
|
||||
print("\n\nModel used:", response["replies"][0].meta.get("model"))
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.togetherai import (
|
||||
TogetherAIChatGenerator,
|
||||
)
|
||||
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
llm = TogetherAIChatGenerator(model="meta-llama/Llama-3.3-70B-Instruct-Turbo")
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
pipe.connect("builder.prompt", "llm.messages")
|
||||
|
||||
messages = [
|
||||
ChatMessage.from_system("Give brief answers."),
|
||||
ChatMessage.from_user("Tell me about {{city}}"),
|
||||
]
|
||||
|
||||
response = pipe.run(
|
||||
data={"builder": {"template": messages, "template_variables": {"city": "Berlin"}}},
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
+148
@@ -0,0 +1,148 @@
|
||||
---
|
||||
title: "TogetherAIGenerator"
|
||||
id: togetheraigenerator
|
||||
slug: "/togetheraigenerator"
|
||||
description: "This component enables text generation using models hosted on Together AI."
|
||||
---
|
||||
|
||||
# TogetherAIGenerator
|
||||
|
||||
This component enables text generation using models hosted on Together AI.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`PromptBuilder`](../builders/promptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: A Together API key. Can be set with `TOGETHER_API_KEY` env var. |
|
||||
| **Mandatory run variables** | `prompt`: A string containing the prompt for the LLM |
|
||||
| **Output variables** | `replies`: A list of strings with all the replies generated by the LLM <br /> <br />`meta`: A list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and so on |
|
||||
| **API reference** | [TogetherAI](/reference/integrations-togetherai) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/togetherai |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`TogetherAIGenerator` supports models hosted on [Together AI](https://docs.together.ai/intro), such as `meta-llama/Llama-3.3-70B-Instruct-Turbo`. For the full list of supported models, see [Together AI documentation](https://docs.together.ai/docs/chat-models).
|
||||
|
||||
This component needs a prompt string to operate. You can pass any text generation parameters valid for the Together AI chat completion API directly to this component using the `generation_kwargs` parameter in `__init__` or the `generation_kwargs` parameter in `run` method. For more details on the parameters supported by the Together AI API, see [Together AI API documentation](https://docs.together.ai/reference/chat-completions-1).
|
||||
|
||||
You can also provide an optional `system_prompt` to set context or instructions for text generation. If not provided, the system prompt is omitted, and the default system prompt of the model is used.
|
||||
|
||||
To use this integration, you need to have an active TogetherAI subscription with sufficient credits and an API key. You can provide it with:
|
||||
|
||||
- The `TOGETHER_API_KEY` environment variable (recommended)
|
||||
- The `api_key` init parameter and Haystack [Secret](../../concepts/secret-management.mdx) API: `Secret.from_token("your-api-key-here")`
|
||||
|
||||
By default, the component uses Together AI's OpenAI-compatible base URL `https://api.together.xyz/v1`, which you can override with `api_base_url` if needed.
|
||||
|
||||
### Streaming
|
||||
|
||||
`TogetherAIGenerator` supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) responses from the LLM, allowing tokens to be emitted as they are generated. To enable streaming, pass a callable to the `streaming_callback` parameter during initialization.
|
||||
|
||||
:::info
|
||||
This component is designed for text generation, not for chat. If you want to use Together AI LLMs for chat, use [`TogetherAIChatGenerator`](togetheraichatgenerator.mdx) instead.
|
||||
:::
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `togetherai-haystack` package to use the `TogetherAIGenerator`:
|
||||
|
||||
```shell
|
||||
pip install togetherai-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.togetherai import TogetherAIGenerator
|
||||
|
||||
client = TogetherAIGenerator(model="meta-llama/Llama-3.3-70B-Instruct-Turbo")
|
||||
response = client.run("What's Natural Language Processing? Be brief.")
|
||||
print(response)
|
||||
|
||||
>> {'replies': ['Natural Language Processing (NLP) is a branch of artificial intelligence
|
||||
>> that focuses on enabling computers to understand, interpret, and generate human language
|
||||
>> in a way that is meaningful and useful.'],
|
||||
>> 'meta': [{'model': 'meta-llama/Llama-3.3-70B-Instruct-Turbo', 'index': 0,
|
||||
>> 'finish_reason': 'stop', 'usage': {'prompt_tokens': 15, 'completion_tokens': 36,
|
||||
>> 'total_tokens': 51}}]}
|
||||
```
|
||||
|
||||
With streaming:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.togetherai import TogetherAIGenerator
|
||||
|
||||
client = TogetherAIGenerator(
|
||||
model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
|
||||
streaming_callback=lambda chunk: print(chunk.content, end="", flush=True),
|
||||
)
|
||||
|
||||
response = client.run("What's Natural Language Processing? Be brief.")
|
||||
print(response)
|
||||
```
|
||||
|
||||
With system prompt:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.togetherai import TogetherAIGenerator
|
||||
|
||||
client = TogetherAIGenerator(
|
||||
model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
|
||||
system_prompt="You are a helpful assistant that provides concise answers.",
|
||||
)
|
||||
|
||||
response = client.run("What's Natural Language Processing?")
|
||||
print(response["replies"][0])
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline, Document
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.components.builders.prompt_builder import PromptBuilder
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.generators.togetherai import TogetherAIGenerator
|
||||
|
||||
docstore = InMemoryDocumentStore()
|
||||
docstore.write_documents([
|
||||
Document(content="Rome is the capital of Italy"),
|
||||
Document(content="Paris is the capital of France")
|
||||
])
|
||||
|
||||
query = "What is the capital of France?"
|
||||
|
||||
template = """
|
||||
Given the following information, answer the question.
|
||||
|
||||
Context:
|
||||
{% for document in documents %}
|
||||
{{ document.content }}
|
||||
{% endfor %}
|
||||
|
||||
Question: {{ query }}?
|
||||
"""
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("retriever", InMemoryBM25Retriever(document_store=docstore))
|
||||
pipe.add_component("prompt_builder", PromptBuilder(template=template))
|
||||
pipe.add_component("llm", TogetherAIGenerator(model="meta-llama/Llama-3.3-70B-Instruct-Turbo"))
|
||||
|
||||
pipe.connect("retriever", "prompt_builder.documents")
|
||||
pipe.connect("prompt_builder", "llm")
|
||||
|
||||
result = pipe.run({
|
||||
"prompt_builder": {"query": query},
|
||||
"retriever": {"query": query}
|
||||
})
|
||||
|
||||
print(result)
|
||||
|
||||
>> {'llm': {'replies': ['The capital of France is Paris.'],
|
||||
>> 'meta': [{'model': 'meta-llama/Llama-3.3-70B-Instruct-Turbo', ...}]}}
|
||||
```
|
||||
+99
@@ -0,0 +1,99 @@
|
||||
---
|
||||
title: "VertexAICodeGenerator"
|
||||
id: vertexaicodegenerator
|
||||
slug: "/vertexaicodegenerator"
|
||||
description: "This component enables code generation using Google Vertex AI generative model."
|
||||
---
|
||||
|
||||
# VertexAICodeGenerator
|
||||
|
||||
This component enables code generation using Google Vertex AI generative model.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Mandatory run variables** | `prefix`: A string of code before the current point <br /> <br />`suffix`: An optional string of code after the current point |
|
||||
| **Output variables** | `replies`: Code generated by the model |
|
||||
| **API reference** | [Google Vertex](/reference/integrations-google-vertex) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/google_vertex |
|
||||
|
||||
</div>
|
||||
|
||||
`VertexAICodeGenerator` supports `code-bison`, `code-bison-32k`, and `code-gecko`.
|
||||
|
||||
### Parameters Overview
|
||||
|
||||
`VertexAICodeGenerator` uses Google Cloud Application Default Credentials (ADCs) for authentication. For more information on how to set up ADCs, see the [official documentation](https://cloud.google.com/docs/authentication/provide-credentials-adc).
|
||||
|
||||
Keep in mind that it’s essential to use an account that has access to a project authorized to use Google Vertex AI endpoints.
|
||||
|
||||
You can find your project ID in the [GCP resource manager](https://console.cloud.google.com/cloud-resource-manager) or locally by running `gcloud projects list` in your terminal. For more info on the gcloud CLI, see its [official documentation](https://cloud.google.com/cli).
|
||||
|
||||
## Usage
|
||||
|
||||
You need to install `google-vertex-haystack` package first to use the `VertexAIImageCaptioner`:
|
||||
|
||||
```shell
|
||||
pip install google-vertex-haystack
|
||||
```
|
||||
|
||||
Basic usage:
|
||||
|
||||
````python
|
||||
from haystack_integrations.components.generators.google_vertex import VertexAICodeGenerator
|
||||
|
||||
generator = VertexAICodeGenerator()
|
||||
|
||||
result = generator.run(prefix="def to_json(data):")
|
||||
|
||||
for answer in result["replies"]:
|
||||
print(answer)
|
||||
|
||||
>>> ```python
|
||||
>>> import json
|
||||
>>>
|
||||
>>> def to_json(data):
|
||||
>>> """Converts a Python object to a JSON string.
|
||||
>>>
|
||||
>>> Args:
|
||||
>>> data: The Python object to convert.
|
||||
>>>
|
||||
>>> Returns:
|
||||
>>> A JSON string representing the Python object.
|
||||
>>> """
|
||||
>>>
|
||||
>>> return json.dumps(data)
|
||||
>>> ```
|
||||
````
|
||||
|
||||
You can also set other parameters like the number of output tokens, temperature, stop sequences, and the number of candidates.
|
||||
|
||||
Let’s try a different model:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.google_vertex import VertexAICodeGenerator
|
||||
|
||||
generator = VertexAICodeGenerator(
|
||||
model="code-gecko",
|
||||
temperature=0.8,
|
||||
candidate_count=3
|
||||
)
|
||||
|
||||
result = generator.run(prefix="def convert_temperature(degrees):")
|
||||
|
||||
for answer in result["replies"]:
|
||||
print(answer)
|
||||
|
||||
>>>
|
||||
>>> return degrees * (9/5) + 32
|
||||
|
||||
>>>
|
||||
>>> return round(degrees * (9.0 / 5.0) + 32, 1)
|
||||
|
||||
>>>
|
||||
>>> return 5 * (degrees - 32) /9
|
||||
>>>
|
||||
>>> def convert_temperature_back(degrees):
|
||||
>>> return 9 * (degrees / 5) + 32
|
||||
```
|
||||
+177
@@ -0,0 +1,177 @@
|
||||
---
|
||||
title: "VertexAIGeminiChatGenerator"
|
||||
id: vertexaigeminichatgenerator
|
||||
slug: "/vertexaigeminichatgenerator"
|
||||
description: "`VertexAIGeminiChatGenerator` enables chat completion using Google Gemini models."
|
||||
---
|
||||
|
||||
# VertexAIGeminiChatGenerator
|
||||
|
||||
`VertexAIGeminiChatGenerator` enables chat completion using Google Gemini models.
|
||||
|
||||
:::warning[Deprecation Notice]
|
||||
|
||||
This integration uses the deprecated google-generativeai SDK, which will lose support after August 2025.
|
||||
|
||||
We recommend switching to the new [GoogleGenAIChatGenerator](googlegenaichatgenerator.mdx) integration instead.
|
||||
:::
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects representing the chat |
|
||||
| **Output variables** | `replies`: A list of alternative replies of the model to the input chat |
|
||||
| **API reference** | [Google Vertex](/reference/integrations-google-vertex) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/google_vertex |
|
||||
|
||||
</div>
|
||||
|
||||
`VertexAIGeminiGenerator` supports `gemini-1.5-pro` and `gemini-1.5-flash`/ `gemini-2.0-flash` models. Note that [Google recommends upgrading](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/model-versions) from `gemini-1.5-pro` to `gemini-2.0-flash`.
|
||||
|
||||
For available models, see https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models.
|
||||
|
||||
:::info
|
||||
To explore the full capabilities of Gemini check out this [article](https://haystack.deepset.ai/blog/gemini-models-with-google-vertex-for-haystack) and the related [🧑🍳 Cookbook](https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/vertexai-gemini-examples.ipynb).
|
||||
:::
|
||||
|
||||
### Parameters Overview
|
||||
|
||||
`VertexAIGeminiChatGenerator` uses Google Cloud Application Default Credentials (ADCs) for authentication. For more information on how to set up ADCs, see the [official documentation](https://cloud.google.com/docs/authentication/provide-credentials-adc).
|
||||
|
||||
Keep in mind that it’s essential to use an account that has access to a project authorized to use Google Vertex AI endpoints.
|
||||
|
||||
You can find your project ID in the [GCP resource manager](https://console.cloud.google.com/cloud-resource-manager) or locally by running `gcloud projects list` in your terminal. For more info on the gcloud CLI, see its [official documentation](https://cloud.google.com/cli).
|
||||
|
||||
### Streaming
|
||||
|
||||
This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
You need to install the `google-vertex-haystack` package to use the `VertexAIGeminiChatGenerator`:
|
||||
|
||||
```shell
|
||||
pip install google-vertex-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.google_vertex import VertexAIGeminiChatGenerator
|
||||
|
||||
gemini_chat = VertexAIGeminiChatGenerator()
|
||||
|
||||
messages = [ChatMessage.from_user("Tell me the name of a movie")]
|
||||
res = gemini_chat.run(messages)
|
||||
|
||||
print(res["replies"][0].text)
|
||||
>>> The Shawshank Redemption
|
||||
|
||||
messages += [res["replies"][0], ChatMessage.from_user("Who's the main actor?")]
|
||||
res = gemini_chat.run(messages)
|
||||
|
||||
print(res["replies"][0].text)
|
||||
>>> Tim Robbins
|
||||
```
|
||||
|
||||
When chatting with Gemini Pro, you can also easily use function calls. First, define the function locally and convert into a [Tool](../../tools/tool.mdx):
|
||||
|
||||
```python
|
||||
from typing import Annotated
|
||||
from haystack.tools import create_tool_from_function
|
||||
|
||||
|
||||
## example function to get the current weather
|
||||
def get_current_weather(
|
||||
location: Annotated[
|
||||
str,
|
||||
"The city for which to get the weather, e.g. 'San Francisco'",
|
||||
] = "Munich",
|
||||
unit: Annotated[str, "The unit for the temperature, e.g. 'celsius'"] = "celsius",
|
||||
) -> str:
|
||||
return f"The weather in {location} is sunny. The temperature is 20 {unit}."
|
||||
|
||||
|
||||
tool = create_tool_from_function(get_current_weather)
|
||||
```
|
||||
|
||||
Create a new instance of `VertexAIGeminiChatGenerator` to set the tools and a [ToolInvoker](../tools/toolinvoker.mdx) to invoke the tools.:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.google_vertex import (
|
||||
VertexAIGeminiChatGenerator,
|
||||
)
|
||||
from haystack.components.tools import ToolInvoker
|
||||
|
||||
gemini_chat = VertexAIGeminiChatGenerator(model="gemini-2.0-flash-exp", tools=[tool])
|
||||
|
||||
tool_invoker = ToolInvoker(tools=[tool])
|
||||
```
|
||||
|
||||
And then ask our question:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
messages = [ChatMessage.from_user("What is the temperature in celsius in Berlin?")]
|
||||
res = gemini_chat.run(messages=messages)
|
||||
|
||||
print(res["replies"][0].tool_calls)
|
||||
>>> [ToolCall(tool_name='get_current_weather',
|
||||
>>> arguments={'unit': 'celsius', 'location': 'Berlin'}, id=None)]
|
||||
|
||||
tool_messages = tool_invoker.run(messages=replies)["tool_messages"]
|
||||
messages = user_message + replies + tool_messages
|
||||
|
||||
messages += res["replies"][0] + [ChatMessage.from_function(content=weather, name="get_current_weather")]
|
||||
|
||||
final_replies = gemini_chat.run(messages=messages)["replies"]
|
||||
print(final_replies[0].text)
|
||||
>>> The temperature in Berlin is 20 degrees Celsius.
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack import Pipeline
|
||||
from haystack_integrations.components.generators.google_vertex import VertexAIGeminiChatGenerator
|
||||
|
||||
## no parameter init, we don't use any runtime template variables
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
gemini_chat = VertexAIGeminiChatGenerator()
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("gemini", gemini)
|
||||
pipe.connect("prompt_builder.prompt", "gemini.messages")
|
||||
|
||||
location = "Rome"
|
||||
messages = [ChatMessage.from_user("Tell me briefly about {{location}} history")]
|
||||
res = pipe.run(data={"prompt_builder": {"template_variables":{"location": location}, "template": messages}})
|
||||
|
||||
print(res)
|
||||
|
||||
>>> - **753 B.C.:** Traditional date of the founding of Rome by Romulus and Remus.
|
||||
>>> - **509 B.C.:** Establishment of the Roman Republic, replacing the Etruscan monarchy.
|
||||
>>> - **492-264 B.C.:** Series of wars against neighboring tribes, resulting in the expansion of the Roman Republic's territory.
|
||||
>>> - **264-146 B.C.:** Three Punic Wars against Carthage, resulting in the destruction of Carthage and the Roman Republic becoming the dominant power in the Mediterranean.
|
||||
>>> - **133-73 B.C.:** Series of civil wars and slave revolts, leading to the rise of Julius Caesar.
|
||||
>>> - **49 B.C.:** Julius Caesar crosses the Rubicon River, starting the Roman Civil War.
|
||||
>>> - **44 B.C.:** Julius Caesar is assassinated, leading to the Second Triumvirate of Octavian, Mark Antony, and Lepidus.
|
||||
>>> - **31 B.C.:** Battle of Actium, where Octavian defeats Mark Antony and Cleopatra, becoming the sole ruler of Rome.
|
||||
>>> - **27 B.C.:** The Roman Republic is transformed into the Roman Empire, with Octavian becoming the first Roman emperor, known as Augustus.
|
||||
>>> - **1st century A.D.:** The Roman Empire reaches its greatest extent, stretching from Britain to Egypt.
|
||||
>>> - **3rd century A.D.:** The Roman Empire begins to decline, facing internal instability, invasions by Germanic tribes, and the rise of Christianity.
|
||||
>>> - **476 A.D.:** The last Western Roman emperor, Romulus Augustulus, is overthrown by the Germanic leader Odoacer, marking the end of the Roman Empire in the West.
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Function Calling and Multimodal QA with Gemini](https://haystack.deepset.ai/cookbook/vertexai-gemini-examples)
|
||||
+159
@@ -0,0 +1,159 @@
|
||||
---
|
||||
title: "VertexAIGeminiGenerator"
|
||||
id: vertexaigeminigenerator
|
||||
slug: "/vertexaigeminigenerator"
|
||||
description: "`VertexAIGeminiGenerator` enables text generation using Google Gemini models."
|
||||
---
|
||||
|
||||
# VertexAIGeminiGenerator
|
||||
|
||||
`VertexAIGeminiGenerator` enables text generation using Google Gemini models.
|
||||
|
||||
:::warning[Deprecation Notice]
|
||||
|
||||
This integration uses the deprecated google-generativeai SDK, which will lose support after August 2025.
|
||||
|
||||
We recommend switching to the new [GoogleGenAIChatGenerator](googlegenaichatgenerator.mdx) integration instead.
|
||||
:::
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`PromptBuilder`](../builders/promptbuilder.mdx) |
|
||||
| **Mandatory run variables** | `parts`: A variadic list containing a mix of images, audio, video, and text to prompt Gemini |
|
||||
| **Output variables** | `replies`: A list of strings or dictionaries with all the replies generated by the model |
|
||||
| **API reference** | [Google Vertex](/reference/integrations-google-vertex) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/google_vertex |
|
||||
|
||||
</div>
|
||||
|
||||
`VertexAIGeminiGenerator` supports `gemini-1.5-pro` and `gemini-1.5-flash`/ `gemini-2.0-flash` models. Note that [Google recommends upgrading](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/model-versions) from `gemini-1.5-pro` to `gemini-2.0-flash`.
|
||||
|
||||
For details on available models, see https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models.
|
||||
|
||||
:::info
|
||||
To explore the full capabilities of Gemini check out this [article](https://haystack.deepset.ai/blog/gemini-models-with-google-vertex-for-haystack) and the related [Colab notebook](https://colab.research.google.com/drive/10SdXvH2ATSzqzA3OOmTM8KzD5ZdH_Q6Z?usp=sharing).
|
||||
:::
|
||||
|
||||
### Parameters Overview
|
||||
|
||||
`VertexAIGeminiGenerator` uses Google Cloud Application Default Credentials (ADCs) for authentication. For more information on how to set up ADCs, see the [official documentation](https://cloud.google.com/docs/authentication/provide-credentials-adc).
|
||||
|
||||
Keep in mind that it’s essential to use an account that has access to a project authorized to use Google Vertex AI endpoints.
|
||||
|
||||
You can find your project ID in the [GCP resource manager](https://console.cloud.google.com/cloud-resource-manager) or locally by running `gcloud projects list` in your terminal. For more info on the gcloud CLI, see its [official documentation](https://cloud.google.com/cli).
|
||||
|
||||
### Streaming
|
||||
|
||||
This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
You should install `google-vertex-haystack` package to use the `VertexAIGeminiGenerator`:
|
||||
|
||||
```shell
|
||||
pip install google-vertex-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.google_vertex import VertexAIGeminiGenerator
|
||||
|
||||
gemini = VertexAIGeminiGenerator()
|
||||
result = gemini.run(parts = ["What is the most interesting thing you know?"])
|
||||
for answer in result["replies"]:
|
||||
print(answer)
|
||||
|
||||
>>> 1. **The Origin of Life:** How and where did life begin? The answers to this question are still shrouded in mystery, but scientists continuously uncover new insights into the remarkable story of our planet's earliest forms of life.
|
||||
>>> 2. **The Unseen Universe:** The vast majority of the universe is comprised of matter and energy that we cannot directly observe. Dark matter and dark energy make up over 95% of the universe, yet we still don't fully understand their properties or how they influence the cosmos.
|
||||
>>> 3. **Quantum Entanglement:** This eerie phenomenon in quantum mechanics allows two particles to become so intertwined that they share the same fate, regardless of how far apart they are. This has mind-bending implications for our understanding of reality and could potentially lead to advancements in communication and computing.
|
||||
>>> 4. **Time Dilation:** Einstein's theory of relativity revealed that time can pass at different rates for different observers. Astronauts traveling at high speeds, for example, experience time dilation relative to people on Earth. This phenomenon could have significant implications for future space travel.
|
||||
>>> 5. **The Fermi Paradox:** Despite the vastness of the universe and the abundance of potential life-supporting planets, we have yet to find any concrete evidence of extraterrestrial life. This contradiction between scientific expectations and observational reality is known as the Fermi Paradox and remains one of the most intriguing mysteries in modern science.
|
||||
>>> 6. **Biological Evolution:** The idea that life evolves over time through natural selection is one of the most profound and transformative scientific discoveries. It explains the diversity of life on Earth and provides insights into our own origins and the interconnectedness of all living things.
|
||||
>>> 7. **Neuroplasticity:** The brain's ability to adapt and change throughout life, known as neuroplasticity, is a remarkable phenomenon that has important implications for learning, memory, and recovery from brain injuries.
|
||||
>>> 8. **The Goldilocks Zone:** The concept of the habitable zone, or the Goldilocks zone, refers to the range of distances from a star within which liquid water can exist on a planet's surface. This zone is critical for the potential existence of life as we know it and has been used to guide the search for exoplanets that could support life.
|
||||
>>> 9. **String Theory:** This theoretical framework in physics aims to unify all the fundamental forces of nature into a single coherent theory. It suggests that the universe has extra dimensions beyond the familiar three spatial dimensions and time.
|
||||
>>> 10. **Consciousness:** The nature of human consciousness and how it arises from the brain's physical processes remain one of the most profound and elusive mysteries in science. Understanding consciousness is crucial for unraveling the complexities of the human mind and our place in the universe.
|
||||
```
|
||||
|
||||
Advanced usage, multi-modal prompting:
|
||||
|
||||
```python
|
||||
import requests
|
||||
from haystack.dataclasses.byte_stream import ByteStream
|
||||
from haystack_integrations.components.generators.google_vertex import VertexAIGeminiGenerator
|
||||
|
||||
URLS = [
|
||||
"https://raw.githubusercontent.com/silvanocerza/robots/main/robot1.jpg",
|
||||
"https://raw.githubusercontent.com/silvanocerza/robots/main/robot2.jpg",
|
||||
"https://raw.githubusercontent.com/silvanocerza/robots/main/robot3.jpg",
|
||||
"https://raw.githubusercontent.com/silvanocerza/robots/main/robot4.jpg"
|
||||
]
|
||||
images = [
|
||||
ByteStream(data=requests.get(url).content, mime_type="image/jpeg")
|
||||
for url in URLS
|
||||
]
|
||||
|
||||
gemini = VertexAIGeminiGenerator()
|
||||
result = gemini.run(parts = ["What can you tell me about this robots?", *images])
|
||||
for answer in result["replies"]:
|
||||
print(answer)
|
||||
>>> The first image is of C-3PO and R2-D2 from the Star Wars franchise. C-3PO is a protocol droid, while R2-D2 is an astromech droid. They are both loyal companions to the heroes of the Star Wars saga.
|
||||
>>> The second image is of Maria from the 1927 film Metropolis. Maria is a robot who is created to be the perfect woman. She is beautiful, intelligent, and obedient. However, she is also soulless and lacks any real emotions.
|
||||
>>> The third image is of Gort from the 1951 film The Day the Earth Stood Still. Gort is a robot who is sent to Earth to warn humanity about the dangers of nuclear war. He is a powerful and intelligent robot, but he is also compassionate and understanding.
|
||||
>>> The fourth image is of Marvin from the 1977 film The Hitchhiker's Guide to the Galaxy. Marvin is a robot who is depressed and pessimistic. He is constantly complaining about everything, but he is also very intelligent and has a dry sense of humor.
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
In a RAG pipeline:
|
||||
|
||||
```python
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.components.builders import PromptBuilder
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.generators.google_vertex import (
|
||||
VertexAIGeminiGenerator,
|
||||
)
|
||||
|
||||
docstore = InMemoryDocumentStore()
|
||||
docstore.write_documents(
|
||||
[
|
||||
Document(content="Rome is the capital of Italy"),
|
||||
Document(content="Paris is the capital of France"),
|
||||
],
|
||||
)
|
||||
|
||||
query = "What is the capital of France?"
|
||||
|
||||
template = """
|
||||
Given the following information, answer the question.
|
||||
|
||||
Context:
|
||||
{% for document in documents %}
|
||||
{{ document.content }}
|
||||
{% endfor %}
|
||||
|
||||
Question: {{ query }}?
|
||||
"""
|
||||
pipe = Pipeline()
|
||||
|
||||
pipe.add_component("retriever", InMemoryBM25Retriever(document_store=docstore))
|
||||
pipe.add_component("prompt_builder", PromptBuilder(template=template))
|
||||
pipe.add_component("gemini", VertexAIGeminiGenerator())
|
||||
pipe.connect("retriever", "prompt_builder.documents")
|
||||
pipe.connect("prompt_builder", "gemini")
|
||||
|
||||
res = pipe.run({"prompt_builder": {"query": query}, "retriever": {"query": query}})
|
||||
|
||||
print(res)
|
||||
```
|
||||
|
||||
## Additional References
|
||||
|
||||
🧑🍳 Cookbook: [Function Calling and Multimodal QA with Gemini](https://haystack.deepset.ai/cookbook/vertexai-gemini-examples)
|
||||
+82
@@ -0,0 +1,82 @@
|
||||
---
|
||||
title: "VertexAIImageCaptioner"
|
||||
id: vertexaiimagecaptioner
|
||||
slug: "/vertexaiimagecaptioner"
|
||||
description: "`VertexAIImageCaptioner` enables text generation using Google Vertex AI `imagetext` generative model."
|
||||
---
|
||||
|
||||
# VertexAIImageCaptioner
|
||||
|
||||
`VertexAIImageCaptioner` enables text generation using Google Vertex AI `imagetext` generative model.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Mandatory run variables** | `image`: A [`ByteStream`](../../concepts/data-classes.mdx#bytestream) object storing an image |
|
||||
| **Output variables** | `captions`: A list of strings generated by the model |
|
||||
| **API reference** | [Google Vertex](/reference/integrations-google-vertex) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/google_vertex |
|
||||
|
||||
</div>
|
||||
|
||||
### Parameters Overview
|
||||
|
||||
`VertexAIImageCaptioner` uses Google Cloud Application Default Credentials (ADCs) for authentication. For more information on how to set up ADCs, see the [official documentation](https://cloud.google.com/docs/authentication/provide-credentials-adc).
|
||||
|
||||
Keep in mind that it’s essential to use an account that has access to a project authorized to use Google Vertex AI endpoints.
|
||||
|
||||
You can find your project ID in the [GCP resource manager](https://console.cloud.google.com/cloud-resource-manager) or locally by running `gcloud projects list` in your terminal. For more info on the gcloud CLI, see its [official documentation](https://cloud.google.com/cli).
|
||||
|
||||
## Usage
|
||||
|
||||
You need to install `google-vertex-haystack` package to use the `VertexAIImageCaptioner`:
|
||||
|
||||
```shell
|
||||
pip install google-vertex-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage:
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
from haystack.dataclasses.byte_stream import ByteStream
|
||||
from haystack_integrations.components.generators.google_vertex import VertexAIImageCaptioner
|
||||
|
||||
captioner = VertexAIImageCaptioner()
|
||||
|
||||
image = ByteStream(data=requests.get("https://raw.githubusercontent.com/silvanocerza/robots/main/robot1.jpg").content)
|
||||
result = captioner.run(image=image)
|
||||
|
||||
for caption in result["captions"]:
|
||||
print(caption)
|
||||
|
||||
>>> two gold robots are standing next to each other in the desert
|
||||
```
|
||||
|
||||
You can also set the caption language and the number of results:
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
from haystack.dataclasses.byte_stream import ByteStream
|
||||
from haystack_integrations.components.generators.google_vertex import VertexAIImageCaptioner
|
||||
|
||||
captioner = VertexAIImageCaptioner(
|
||||
number_of_results=3, # Can't be greater than 3
|
||||
language="it",
|
||||
)
|
||||
|
||||
image = ByteStream(data=requests.get("https://raw.githubusercontent.com/silvanocerza/robots/main/robot1.jpg").content)
|
||||
result = captioner.run(image=image)
|
||||
|
||||
for caption in result["captions"]:
|
||||
print(caption)
|
||||
|
||||
>>> due robot dorati sono in piedi uno accanto all'altro in un deserto
|
||||
>>> un c3p0 e un r2d2 stanno in piedi uno accanto all'altro in un deserto
|
||||
>>> due robot dorati sono in piedi uno accanto all'altro
|
||||
```
|
||||
+80
@@ -0,0 +1,80 @@
|
||||
---
|
||||
title: "VertexAIImageGenerator"
|
||||
id: vertexaiimagegenerator
|
||||
slug: "/vertexaiimagegenerator"
|
||||
description: "This component enables image generation using Google Vertex AI generative model."
|
||||
---
|
||||
|
||||
# VertexAIImageGenerator
|
||||
|
||||
This component enables image generation using Google Vertex AI generative model.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Mandatory run variables** | `prompt`: A string containing the prompt for the model |
|
||||
| **Output variables** | `images`: A list of [`ByteStream`](../../concepts/data-classes.mdx#bytestream) containing images generated by the model |
|
||||
| **API reference** | [Google Vertex](/reference/integrations-google-vertex) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/google_vertex |
|
||||
|
||||
</div>
|
||||
|
||||
`VertexAIImageGenerator` supports the `imagegeneration` model.
|
||||
|
||||
### Parameters Overview
|
||||
|
||||
`VertexAIImageGenerator` uses Google Cloud Application Default Credentials (ADCs) for authentication. For more information on how to set up ADCs, see the [official documentation](https://cloud.google.com/docs/authentication/provide-credentials-adc).
|
||||
|
||||
Keep in mind that it’s essential to use an account that has access to a project authorized to use Google Vertex AI endpoints.
|
||||
|
||||
You can find your project ID in the [GCP resource manager](https://console.cloud.google.com/cloud-resource-manager) or locally by running `gcloud projects list` in your terminal. For more info on the gcloud CLI, see its [official documentation](https://cloud.google.com/cli).
|
||||
|
||||
## Usage
|
||||
|
||||
You need to install `google-vertex-haystack` package to use the `VertexAIImageGenerator`:
|
||||
|
||||
```python
|
||||
pip install google-vertex-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from haystack_integrations.components.generators.google_vertex import (
|
||||
VertexAIImageGenerator,
|
||||
)
|
||||
|
||||
generator = VertexAIImageGenerator()
|
||||
result = generator.run(prompt="Generate an image of a cute cat")
|
||||
result["images"][0].to_file(Path("my_image.png"))
|
||||
```
|
||||
|
||||
You can also set other parameters like the number of images generated and the guidance scale to change the strength of the prompt.
|
||||
|
||||
Let’s also use a negative prompt to omit something from the image:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from haystack_integrations.components.generators.google_vertex import (
|
||||
VertexAIImageGenerator,
|
||||
)
|
||||
|
||||
generator = VertexAIImageGenerator(
|
||||
number_of_images=3,
|
||||
guidance_scale=12,
|
||||
)
|
||||
|
||||
result = generator.run(
|
||||
prompt="Generate an image of a cute cat",
|
||||
negative_prompt="window, chair",
|
||||
)
|
||||
|
||||
for i, image in enumerate(result["images"]):
|
||||
images.to_file(Path(f"image_{i}.png"))
|
||||
```
|
||||
+79
@@ -0,0 +1,79 @@
|
||||
---
|
||||
title: "VertexAIImageQA"
|
||||
id: vertexaiimageqa
|
||||
slug: "/vertexaiimageqa"
|
||||
description: "This component enables text generation (image captioning) using Google Vertex AI generative models."
|
||||
---
|
||||
|
||||
# VertexAIImageQA
|
||||
|
||||
This component enables text generation (image captioning) using Google Vertex AI generative models.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Mandatory run variables** | `image`: A [`ByteStream`](../../concepts/data-classes.mdx#bytestream) containing an image data <br /> <br />`question`: A string of a question about the image |
|
||||
| **Output variables** | `replies`: A list of strings containing answers generated by the model |
|
||||
| **API reference** | [Google Vertex](/reference/integrations-google-vertex) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/google_vertex |
|
||||
|
||||
</div>
|
||||
|
||||
`VertexAIImageQA` supports the `imagetext` model.
|
||||
|
||||
### Parameters Overview
|
||||
|
||||
`VertexAIImageQA` uses Google Cloud Application Default Credentials (ADCs) for authentication. For more information on how to set up ADCs, see the [official documentation](https://cloud.google.com/docs/authentication/provide-credentials-adc).
|
||||
|
||||
Keep in mind that it’s essential to use an account that has access to a project authorized to use Google Vertex AI endpoints.
|
||||
|
||||
You can find your project ID in the [GCP resource manager](https://console.cloud.google.com/cloud-resource-manager) or locally by running `gcloud projects list` in your terminal. For more info on the gcloud CLI, see its [official documentation](https://cloud.google.com/cli).
|
||||
|
||||
## Usage
|
||||
|
||||
You need to install `google-vertex-haystack` package to use the `VertexAIImageQA`:
|
||||
|
||||
```python
|
||||
pip install google-vertex-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses.byte_stream import ByteStream
|
||||
from haystack_integrations.components.generators.google_vertex import VertexAIImageQA
|
||||
|
||||
qa = VertexAIImageQA()
|
||||
|
||||
image = ByteStream.from_file_path("dog.jpg")
|
||||
|
||||
res = qa.run(image=image, question="What color is this dog")
|
||||
|
||||
print(res["replies"][0])
|
||||
|
||||
>>> white
|
||||
```
|
||||
|
||||
You can also set the number of answers generated:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses.byte_stream import ByteStream
|
||||
from haystack_integrations.components.generators.google_vertex import VertexAIImageQA
|
||||
|
||||
qa = VertexAIImageQA(
|
||||
number_of_results=3,
|
||||
)
|
||||
image = ByteStream.from_file_path("dog.jpg")
|
||||
|
||||
res = qa.run(image=image, question="Tell me something about this dog")
|
||||
|
||||
for answer in res["replies"]:
|
||||
print(answer)
|
||||
|
||||
>>> pomeranian
|
||||
>>> white
|
||||
>>> pomeranian puppy
|
||||
```
|
||||
+87
@@ -0,0 +1,87 @@
|
||||
---
|
||||
title: "VertexAITextGenerator"
|
||||
id: vertexaitextgenerator
|
||||
slug: "/vertexaitextgenerator"
|
||||
description: "This component enables text generation using Google Vertex AI generative models."
|
||||
---
|
||||
|
||||
# VertexAITextGenerator
|
||||
|
||||
This component enables text generation using Google Vertex AI generative models.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Mandatory run variables** | `prompt`: A string containing the prompt for the model |
|
||||
| **Output variables** | `replies`: A list of strings containing answers generated by the model <br /> <br />`safety_attributes`: A dictionary containing scores for safety attributes <br /> <br />`citations`: A list of dictionaries containing grounding citations |
|
||||
| **API reference** | [Google Vertex](/reference/integrations-google-vertex) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/google_vertex |
|
||||
|
||||
</div>
|
||||
|
||||
`VertexAITextGenerator` supports `text-bison`, `text-unicorn` and `text-bison-32k` models.
|
||||
|
||||
### Parameters Overview
|
||||
|
||||
`VertexAITextGenerator` uses Google Cloud Application Default Credentials (ADCs) for authentication. For more information on how to set up ADCs, see the [official documentation](https://cloud.google.com/docs/authentication/provide-credentials-adc).
|
||||
|
||||
Keep in mind that it’s essential to use an account that has access to a project authorized to use Google Vertex AI endpoints.
|
||||
|
||||
You can find your project ID in the [GCP resource manager](https://console.cloud.google.com/cloud-resource-manager) or locally by running `gcloud projects list` in your terminal. For more info on the gcloud CLI, see its [official documentation](https://cloud.google.com/cli).
|
||||
|
||||
## Usage
|
||||
|
||||
You need to install `google-vertex-haystack` package to use the `VertexAITextGenerator`:
|
||||
|
||||
```python
|
||||
pip install google-vertex-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage:
|
||||
|
||||
````python
|
||||
from haystack_integrations.components.generators.google_vertex import VertexAITextGenerator
|
||||
|
||||
generator = VertexAITextGenerator()
|
||||
res = generator.run("Tell me a good interview question for a software engineer.")
|
||||
|
||||
print(res["replies"][0])
|
||||
|
||||
>>> **Question:** You are given a list of integers and a target sum. Find all unique combinations of numbers in the list that add up to the target sum.
|
||||
>>>
|
||||
>>> **Example:**
|
||||
>>>
|
||||
>>> ```
|
||||
>>> Input: [1, 2, 3, 4, 5], target = 7
|
||||
>>> Output: [[1, 2, 4], [3, 4]]
|
||||
>>> ```
|
||||
>>>
|
||||
>>> **Follow-up:** What if the list contains duplicate numbers?
|
||||
````
|
||||
|
||||
You can also set other parameters like the number of answers generated, temperature to control the randomness, and stop sequences to stop generation. For a full list of possible parameters, see the documentation of [`TextGenerationModel.predict()`](https://cloud.google.com/python/docs/reference/aiplatform/latest/vertexai.language_models.TextGenerationModel#vertexai_language_models_TextGenerationModel_predict).
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.google_vertex import VertexAITextGenerator
|
||||
|
||||
generator = VertexAITextGenerator(
|
||||
candidate_count=3,
|
||||
temperature=0.2,
|
||||
stop_sequences=["example", "Example"],
|
||||
)
|
||||
res = generator.run("Tell me a good interview question for a software engineer.")
|
||||
|
||||
for answer in res["replies"]:
|
||||
print(answer)
|
||||
print("-----")
|
||||
|
||||
>>> **Question:** You are given a list of integers, and you need to find the longest increasing subsequence. What is the most efficient algorithm to solve this problem?
|
||||
>>> -----
|
||||
>>> **Question:** You are given a list of integers and a target sum. Find all unique combinations in the list that sum up to the target sum. The same number can be used multiple times in a combination.
|
||||
>>> -----
|
||||
>>> **Question:** You are given a list of integers and a target sum. Find all unique combinations of numbers in the list that add up to the target sum.
|
||||
>>> -----
|
||||
```
|
||||
+196
@@ -0,0 +1,196 @@
|
||||
---
|
||||
title: "VLLMChatGenerator"
|
||||
id: vllmchatgenerator
|
||||
slug: "/vllmchatgenerator"
|
||||
description: "This component enables chat completion using models served with vLLM."
|
||||
---
|
||||
|
||||
# VLLMChatGenerator
|
||||
|
||||
This component enables chat completion using models served with [vLLM](https://docs.vllm.ai/).
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [`ChatPromptBuilder`](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `model`: The name of the model served by vLLM |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects |
|
||||
| **Output variables** | `replies`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects |
|
||||
| **API reference** | [vLLM](/reference/integrations-vllm) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/vllm |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
[vLLM](https://docs.vllm.ai/) is a high-throughput and memory-efficient inference and serving engine for LLMs. It exposes an OpenAI-compatible HTTP server, which `VLLMChatGenerator` uses to run chat completions.
|
||||
|
||||
`VLLMChatGenerator` expects a vLLM server to be running and accessible at the `api_base_url` parameter (by default, `http://localhost:8000/v1`). The component needs a list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects to operate. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata.
|
||||
|
||||
You can pass any text generation parameters valid for the vLLM OpenAI-compatible Chat Completion API directly to this component using the `generation_kwargs` parameter in `__init__` or in the `run` method. vLLM-specific parameters not part of the standard OpenAI API (such as `top_k`, `min_tokens`, `repetition_penalty`) can be passed through `generation_kwargs["extra_body"]`. For more details, see the [vLLM documentation](https://docs.vllm.ai/en/stable/serving/openai_compatible_server/).
|
||||
|
||||
If the vLLM server was started with `--api-key`, provide the API key through the `VLLM_API_KEY` environment variable or the `api_key` init parameter using Haystack's [Secret](../../concepts/secret-management.mdx) API.
|
||||
|
||||
### Tool Support
|
||||
|
||||
`VLLMChatGenerator` supports function calling through the `tools` parameter, which accepts flexible tool configurations:
|
||||
|
||||
- **A list of Tool objects**: Pass individual tools as a list
|
||||
- **A single Toolset**: Pass an entire Toolset directly
|
||||
- **Mixed Tools and Toolsets**: Combine multiple Toolsets with standalone tools in a single list
|
||||
|
||||
This allows you to organize related tools into logical groups while also including standalone tools as needed.
|
||||
|
||||
For tool calling to work, the vLLM server must be started with `--enable-auto-tool-choice` and `--tool-call-parser`. The available tool call parsers depend on the model. See the [vLLM tool calling docs](https://docs.vllm.ai/en/stable/features/tool_calling/) for the full list.
|
||||
|
||||
For more details on working with tools, see the [Tool](../../tools/tool.mdx) and [Toolset](../../tools/toolset.mdx) documentation.
|
||||
|
||||
### Streaming
|
||||
|
||||
`VLLMChatGenerator` supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) responses from the LLM, allowing tokens to be emitted as they are generated. To enable streaming, pass a callable to the `streaming_callback` parameter during initialization.
|
||||
|
||||
### Reasoning models
|
||||
|
||||
`VLLMChatGenerator` supports reasoning models. To use them, start the vLLM server with the appropriate `--reasoning-parser`. The reasoning content produced by the model is exposed in the `reasoning` field of the returned `ChatMessage`.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `vllm-haystack` package to use the `VLLMChatGenerator`:
|
||||
|
||||
```shell
|
||||
pip install vllm-haystack
|
||||
```
|
||||
|
||||
### Starting the vLLM server
|
||||
|
||||
Before using this component, start a vLLM server:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-4B-Instruct-2507
|
||||
```
|
||||
|
||||
For reasoning models, start the server with the appropriate reasoning parser:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-0.6B --reasoning-parser qwen3
|
||||
```
|
||||
|
||||
For tool calling, start the server with `--enable-auto-tool-choice` and `--tool-call-parser`:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-0.6B --enable-auto-tool-choice --tool-call-parser hermes
|
||||
```
|
||||
|
||||
For details on server options, see the [vLLM CLI docs](https://docs.vllm.ai/en/stable/cli/serve/).
|
||||
|
||||
### On its own
|
||||
|
||||
Basic usage:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.vllm import VLLMChatGenerator
|
||||
|
||||
generator = VLLMChatGenerator(
|
||||
model="Qwen/Qwen3-4B-Instruct-2507",
|
||||
generation_kwargs={"max_tokens": 512, "temperature": 0.7},
|
||||
)
|
||||
|
||||
messages = [ChatMessage.from_user("What's Natural Language Processing?")]
|
||||
response = generator.run(messages=messages)
|
||||
print(response["replies"][0].text)
|
||||
```
|
||||
|
||||
### With vLLM-specific parameters
|
||||
|
||||
Pass vLLM-specific parameters through the `generation_kwargs["extra_body"]` dictionary:
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.vllm import VLLMChatGenerator
|
||||
|
||||
generator = VLLMChatGenerator(
|
||||
model="Qwen/Qwen3-4B-Instruct-2507",
|
||||
generation_kwargs={
|
||||
"max_tokens": 512,
|
||||
"extra_body": {
|
||||
"top_k": 50,
|
||||
"min_tokens": 10,
|
||||
"repetition_penalty": 1.1,
|
||||
},
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
### With tool calling
|
||||
|
||||
Start the vLLM server with `--enable-auto-tool-choice` and `--tool-call-parser`, then:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.tools import tool
|
||||
from haystack_integrations.components.generators.vllm import VLLMChatGenerator
|
||||
|
||||
|
||||
@tool
|
||||
def weather(city: str) -> str:
|
||||
"""Get the weather in a given city."""
|
||||
return f"The weather in {city} is sunny"
|
||||
|
||||
|
||||
generator = VLLMChatGenerator(model="Qwen/Qwen3-0.6B", tools=[weather])
|
||||
|
||||
messages = [ChatMessage.from_user("What is the weather in Paris?")]
|
||||
response = generator.run(messages=messages)
|
||||
print(response["replies"][0].tool_calls)
|
||||
```
|
||||
|
||||
### With reasoning models
|
||||
|
||||
Start the vLLM server with `--reasoning-parser`, then:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.vllm import VLLMChatGenerator
|
||||
|
||||
generator = VLLMChatGenerator(model="Qwen/Qwen3-0.6B")
|
||||
|
||||
messages = [ChatMessage.from_user("Solve step by step: what is 15 * 37?")]
|
||||
response = generator.run(messages=messages)
|
||||
reply = response["replies"][0]
|
||||
if reply.reasoning:
|
||||
print("Reasoning:", reply.reasoning.reasoning_text)
|
||||
print("Answer:", reply.text)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.vllm import VLLMChatGenerator
|
||||
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
llm = VLLMChatGenerator(model="Qwen/Qwen3-4B-Instruct-2507")
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
|
||||
messages = [
|
||||
ChatMessage.from_system("Give brief answers."),
|
||||
ChatMessage.from_user("Tell me about {{city}}"),
|
||||
]
|
||||
|
||||
response = pipe.run(
|
||||
data={
|
||||
"prompt_builder": {
|
||||
"template": messages,
|
||||
"template_variables": {"city": "Berlin"},
|
||||
},
|
||||
},
|
||||
)
|
||||
print(response)
|
||||
```
|
||||
+134
@@ -0,0 +1,134 @@
|
||||
---
|
||||
title: "WatsonxChatGenerator"
|
||||
id: watsonxchatgenerator
|
||||
slug: "/watsonxchatgenerator"
|
||||
description: "Use this component with IBM watsonx models like `granite-3-2b-instruct` for chat generation."
|
||||
---
|
||||
|
||||
# WatsonxChatGenerator
|
||||
|
||||
Use this component with IBM watsonx models like `granite-3-2b-instruct` for chat generation.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: The IBM Cloud API key. Can be set with `WATSONX_API_KEY` env var. <br /> <br />`project_id`: The IBM Cloud project ID. Can be set with `WATSONX_PROJECT_ID` env var. |
|
||||
| **Mandatory run variables** | `messages` A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects |
|
||||
| **Output variables** | `replies`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects |
|
||||
| **API reference** | [Watsonx](/reference/integrations-watsonx) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/watsonx |
|
||||
|
||||
</div>
|
||||
|
||||
This integration supports IBM watsonx.ai foundation models such as `ibm/granite-13b-chat-v2`, `ibm/llama-2-70b-chat`, `ibm/llama-3-70b-instruct`, and similar. These models provide high-quality chat completion capabilities through IBM's cloud platform. Check out the most recent full list in the [IBM watsonx.ai documentation](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models-ibm.html?context=wx).
|
||||
|
||||
## Overview
|
||||
|
||||
`WatsonxChatGenerator` needs IBM Cloud credentials to work. You can set these in:
|
||||
|
||||
- The `api_key` and `project_id` init parameters using [Secret API](../../concepts/secret-management.mdx)
|
||||
- The `WATSONX_API_KEY` and `WATSONX_PROJECT_ID` environment variables (recommended)
|
||||
|
||||
Then, the component needs a prompt to operate, but you can pass any text generation parameters valid for the IBM watsonx.ai API directly to this component using the `generation_kwargs` parameter, both at initialization and to `run()` method. For more details on the parameters supported by the IBM watsonx.ai API, refer to the [IBM watsonx.ai documentation](https://cloud.ibm.com/apidocs/watsonx-ai).
|
||||
|
||||
Finally, the component needs a list of `ChatMessage` objects to operate. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata.
|
||||
|
||||
### Streaming
|
||||
|
||||
This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
You need to install `watsonx-haystack` package to use the `WatsonxChatGenerator`:
|
||||
|
||||
```shell
|
||||
pip install watsonx-haystack
|
||||
```
|
||||
|
||||
#### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.watsonx.chat.chat_generator import (
|
||||
WatsonxChatGenerator,
|
||||
)
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.utils import Secret
|
||||
|
||||
generator = WatsonxChatGenerator(
|
||||
api_key=Secret.from_env_var("WATSONX_API_KEY"),
|
||||
project_id=Secret.from_env_var("WATSONX_PROJECT_ID"),
|
||||
model="ibm/granite-13b-instruct-v2",
|
||||
)
|
||||
|
||||
message = ChatMessage.from_user("What's Natural Language Processing? Be brief.")
|
||||
print(generator.run([message]))
|
||||
```
|
||||
|
||||
With multimodal inputs:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import ChatMessage, ImageContent
|
||||
from haystack_integrations.components.generators.watsonx.chat.chat_generator import (
|
||||
WatsonxChatGenerator,
|
||||
)
|
||||
|
||||
# Use a multimodal model
|
||||
llm = WatsonxChatGenerator(model="meta-llama/llama-3-2-11b-vision-instruct")
|
||||
|
||||
image = ImageContent.from_file_path("apple.jpg")
|
||||
user_message = ChatMessage.from_user(
|
||||
content_parts=["What does the image show? Max 5 words.", image],
|
||||
)
|
||||
|
||||
response = llm.run([user_message])["replies"][0].text
|
||||
print(response)
|
||||
|
||||
# Red apple on straw.
|
||||
```
|
||||
|
||||
#### In a Pipeline
|
||||
|
||||
You can also use `WatsonxChatGenerator` to use IBM watsonx.ai chat models in your pipeline.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.generators.watsonx.chat.chat_generator import (
|
||||
WatsonxChatGenerator,
|
||||
)
|
||||
from haystack.utils import Secret
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", ChatPromptBuilder())
|
||||
pipe.add_component(
|
||||
"llm",
|
||||
WatsonxChatGenerator(
|
||||
api_key=Secret.from_env_var("WATSONX_API_KEY"),
|
||||
project_id=Secret.from_env_var("WATSONX_PROJECT_ID"),
|
||||
model="ibm/granite-13b-instruct-v2",
|
||||
),
|
||||
)
|
||||
pipe.connect("prompt_builder", "llm")
|
||||
|
||||
country = "Germany"
|
||||
system_message = ChatMessage.from_system(
|
||||
"You are an assistant giving out valuable information to language learners.",
|
||||
)
|
||||
messages = [
|
||||
system_message,
|
||||
ChatMessage.from_user("What's the official language of {{ country }}?"),
|
||||
]
|
||||
|
||||
res = pipe.run(
|
||||
data={
|
||||
"prompt_builder": {
|
||||
"template_variables": {"country": country},
|
||||
"template": messages,
|
||||
},
|
||||
},
|
||||
)
|
||||
print(res)
|
||||
```
|
||||
+107
@@ -0,0 +1,107 @@
|
||||
---
|
||||
title: "WatsonxGenerator"
|
||||
id: watsonxgenerator
|
||||
slug: "/watsonxgenerator"
|
||||
description: "Use this component with IBM watsonx models like `granite-3-2b-instruct` for simple text generation tasks."
|
||||
---
|
||||
|
||||
# WatsonxGenerator
|
||||
|
||||
Use this component with IBM watsonx models like `granite-3-2b-instruct` for simple text generation tasks.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [PromptBuilder](../builders/promptbuilder.mdx) |
|
||||
| **Mandatory init variables** | `api_key`: An IBM Cloud API key. Can be set with `WATSONX_API_KEY` env var. <br /> <br />`project_id`: An IBM Cloud project ID. Can be set with `WATSONX_PROJECT_ID` env var. |
|
||||
| **Mandatory run variables** | `prompt`: A string containing the prompt for the LLM |
|
||||
| **Output variables** | `replies`: A list of strings with all the replies generated by the LLM <br /> <br />`meta`: A list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and so on |
|
||||
| **API reference** | [Watsonx](/reference/integrations-watsonx) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/watsonx |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
This integration supports IBM watsonx.ai foundation models such as `ibm/granite-13b-chat-v2`, `ibm/llama-2-70b-chat`, `ibm/llama-3-70b-instruct`, and similar. These models provide high-quality text generation capabilities through IBM's cloud platform. Check out the most recent full list in the [IBM watsonx.ai documentation](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models-ibm.html?context=wx).
|
||||
|
||||
### Parameters
|
||||
|
||||
`WatsonxGenerator` needs IBM Cloud credentials to work. You can provide these in:
|
||||
|
||||
- The `WATSONX_API_KEY` environment variable (recommended)
|
||||
- The `WATSONX_PROJECT_ID` environment variable (recommended)
|
||||
- The `api_key` and `project_id` init parameters using Haystack [Secret](../../concepts/secret-management.mdx) API: `Secret.from_token("your-api-key-here")`
|
||||
|
||||
Set your preferred IBM watsonx.ai model in the `model` parameter when initializing the component. The default model is `ibm/granite-3-2b-instruct`.
|
||||
|
||||
`WatsonxGenerator` requires a prompt to generate text, but you can pass any text generation parameters available in the IBM watsonx.ai API directly to this component using the `generation_kwargs` parameter, both at initialization and to `run()` method. For more details on the parameters supported by the IBM watsonx.ai API, see [IBM watsonx.ai documentation](https://cloud.ibm.com/apidocs/watsonx-ai).
|
||||
|
||||
The component also supports system prompts that can be set at initialization or passed during runtime to provide context or instructions for the generation.
|
||||
|
||||
Finally, the component run method requires a single string prompt to generate text.
|
||||
|
||||
### Streaming
|
||||
|
||||
This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `watsonx-haystack` package to use the `WatsonxGenerator`:
|
||||
|
||||
```shell
|
||||
pip install watsonx-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.watsonx.generator import (
|
||||
WatsonxGenerator,
|
||||
)
|
||||
from haystack.utils import Secret
|
||||
|
||||
generator = WatsonxGenerator(
|
||||
api_key=Secret.from_env_var("WATSONX_API_KEY"),
|
||||
project_id=Secret.from_env_var("WATSONX_PROJECT_ID"),
|
||||
)
|
||||
|
||||
print(generator.run("What's Natural Language Processing? Be brief."))
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
You can also use `WatsonxGenerator` with the IBM watsonx.ai models in your pipeline.
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders import PromptBuilder
|
||||
from haystack_integrations.components.generators.watsonx.generator import (
|
||||
WatsonxGenerator,
|
||||
)
|
||||
from haystack.utils import Secret
|
||||
|
||||
template = """
|
||||
You are an assistant giving out valuable information to language learners.
|
||||
Answer this question, be brief.
|
||||
|
||||
Question: {{ query }}?
|
||||
"""
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", PromptBuilder(template))
|
||||
pipe.add_component(
|
||||
"llm",
|
||||
WatsonxGenerator(
|
||||
api_key=Secret.from_env_var("WATSONX_API_KEY"),
|
||||
project_id=Secret.from_env_var("WATSONX_PROJECT_ID"),
|
||||
),
|
||||
)
|
||||
pipe.connect("prompt_builder", "llm")
|
||||
|
||||
query = "What language is spoken in Germany?"
|
||||
res = pipe.run(data={"prompt_builder": {"query": query}})
|
||||
|
||||
print(res)
|
||||
```
|
||||
Reference in New Issue
Block a user