---
title: "HuggingFaceLocalChatGenerator"
id: huggingfacelocalchatgenerator
slug: "/huggingfacelocalchatgenerator"
description: "Provides an interface for chat completion using a Hugging Face model that runs locally."
---

# HuggingFaceLocalChatGenerator

Provides an interface for chat completion using a Hugging Face model that runs locally.

<div className="key-value-table">

|  |  |
| --- | --- |
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx)                                                         |
| **Mandatory init variables**           | `token`: The Hugging Face API token. Can be set with `HF_API_TOKEN` or `HF_TOKEN` env var.                   |
| **Mandatory run variables**            | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx)  objects representing the chat         |
| **Output variables**                   | `replies`: A list of strings with all the replies generated by the LLM                                       |
| **API reference**                      | [Generators](/reference/generators-api)                                                                             |
| **GitHub link**                        | https://github.com/deepset-ai/haystack/blob/main/haystack/components/generators/chat/hugging_face_local.py |

</div>

## Overview

Keep in mind that if LLMs run locally, you may need a powerful machine to run them. This depends strongly on the model you select and its parameter count.

:::info
This component is designed for chat completion, not for text generation. If you want to use Hugging Face LLMs for text generation, use [`HuggingFaceLocalGenerator`](huggingfacelocalgenerator.mdx) instead.
:::

For remote file authorization, this component uses a `HF_API_TOKEN` environment variable by default. Otherwise, you can pass a Hugging Face API token at initialization with `token`:

```python
local_generator = HuggingFaceLocalChatGenerator(
    token=Secret.from_token("<your-api-key>"),
)
```

### Streaming

This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.

## Usage

### On its own

```python
from haystack.components.generators.chat import HuggingFaceLocalChatGenerator
from haystack.dataclasses import ChatMessage

generator = HuggingFaceLocalChatGenerator(model="HuggingFaceH4/zephyr-7b-beta")
generator.warm_up()
messages = [ChatMessage.from_user("What's Natural Language Processing? Be brief.")]
print(generator.run(messages))
```

### In a Pipeline

```python
from haystack import Pipeline
from haystack.components.builders.prompt_builder import ChatPromptBuilder
from haystack.components.generators.chat import HuggingFaceLocalChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret

prompt_builder = ChatPromptBuilder()
llm = HuggingFaceLocalChatGenerator(
    model="HuggingFaceH4/zephyr-7b-beta",
    token=Secret.from_env_var("HF_API_TOKEN"),
)

pipe = Pipeline()
pipe.add_component("prompt_builder", prompt_builder)
pipe.add_component("llm", llm)
pipe.connect("prompt_builder.prompt", "llm.messages")
location = "Berlin"
messages = [
    ChatMessage.from_system(
        "Always respond in German even if some input data is in other languages.",
    ),
    ChatMessage.from_user("Tell me about {{location}}"),
]
pipe.run(
    data={
        "prompt_builder": {
            "template_variables": {"location": location},
            "template": messages,
        },
    },
)
```