chore: import upstream snapshot with attribution
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,101 @@
|
||||
---
|
||||
title: "TransformersChatGenerator"
|
||||
id: transformerschatgenerator
|
||||
slug: "/transformerschatgenerator"
|
||||
description: "Provides an interface for chat completion using a Hugging Face model that runs locally."
|
||||
---
|
||||
|
||||
# TransformersChatGenerator
|
||||
|
||||
Provides an interface for chat completion using a Hugging Face model that runs locally.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | None |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects representing the chat or a plain string |
|
||||
| **Output variables** | `replies`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects generated by the LLM |
|
||||
| **API reference** | [Transformers](/reference/integrations-transformers) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/transformers |
|
||||
| **Package name** | `transformers-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
Keep in mind that if LLMs run locally, you may need a powerful machine to run them. This depends strongly on the model you select and its parameter count.
|
||||
|
||||
If a string is passed to `messages`, it is converted into a list containing a single `ChatMessage` with the `user` role.
|
||||
|
||||
Authentication with a Hugging Face API token is only required to access private or gated models. You can pass the token at initialization with `token`, or set the `HF_API_TOKEN` or `HF_TOKEN` environment variable:
|
||||
|
||||
```python
|
||||
generator = TransformersChatGenerator(
|
||||
token=Secret.from_token("<your-api-key>"),
|
||||
)
|
||||
```
|
||||
|
||||
### Streaming
|
||||
|
||||
This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `transformers-haystack` package to use the `TransformersChatGenerator`:
|
||||
|
||||
```shell
|
||||
pip install transformers-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.transformers import (
|
||||
TransformersChatGenerator,
|
||||
)
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
generator = TransformersChatGenerator(model="Qwen/Qwen3-0.6B")
|
||||
messages = [ChatMessage.from_user("What's Natural Language Processing? Be brief.")]
|
||||
print(generator.run(messages))
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders.prompt_builder import ChatPromptBuilder
|
||||
from haystack_integrations.components.generators.transformers import (
|
||||
TransformersChatGenerator,
|
||||
)
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.utils import Secret
|
||||
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
llm = TransformersChatGenerator(
|
||||
model="Qwen/Qwen3-0.6B",
|
||||
token=Secret.from_env_var("HF_API_TOKEN"),
|
||||
)
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
location = "Berlin"
|
||||
messages = [
|
||||
ChatMessage.from_system(
|
||||
"Always respond in German even if some input data is in other languages.",
|
||||
),
|
||||
ChatMessage.from_user("Tell me about {{location}}"),
|
||||
]
|
||||
pipe.run(
|
||||
data={
|
||||
"prompt_builder": {
|
||||
"template_variables": {"location": location},
|
||||
"template": messages,
|
||||
},
|
||||
},
|
||||
)
|
||||
```
|
||||
Reference in New Issue
Block a user