---
title: "HuggingFaceAPIChatGenerator"
id: huggingfaceapichatgenerator
slug: "/huggingfaceapichatgenerator"
description: "This generator enables chat completion using various Hugging Face APIs."
---

# HuggingFaceAPIChatGenerator

This generator enables chat completion using various Hugging Face APIs.

<div className="key-value-table">

|  |  |
| --- | --- |
| **Most common position in a pipeline** | After a [`ChatPromptBuilder`](../builders/chatpromptbuilder.mdx) |
| **Mandatory init variables** | `api_type`: The type of Hugging Face API to use  <br /> <br />`api_params`: A dictionary with one of the following keys:  <br /> <br />- `model`: Hugging Face model ID. Required when `api_type` is `SERVERLESS_INFERENCE_API`.**OR** - `url`: URL of the inference endpoint. Required when `api_type` is `INFERENCE_ENDPOINTS` or `TEXT_EMBEDDINGS_INFERENCE`.`token`: The Hugging Face API token. Can be set with `HF_API_TOKEN` or `HF_TOKEN` env var. |
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects representing the chat or a plain string |
| **Output variables** | `replies`: A list of replies of the LLM to the input chat |
| **API reference** | [Hugging Face API](/reference/integrations-huggingface-api) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/huggingface_api |
| **Package name** | `huggingface-api-haystack` |

</div>

## Overview

`HuggingFaceAPIChatGenerator` can be used to generate chat completions using different Hugging Face APIs:

- [Serverless Inference API (Inference Providers)](https://huggingface.co/docs/inference-providers) - free tier available
- [Paid Inference Endpoints](https://huggingface.co/inference-endpoints)
- [Self-hosted Text Generation Inference](https://github.com/huggingface/text-generation-inference)

This component's main input is a list of `ChatMessage` objects. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata. If a string is passed, it is converted into a list containing a single `ChatMessage` with the `user` role. For more information, check out our [`ChatMessage` docs](../../concepts/data-classes/chatmessage.mdx).

The component uses a `HF_API_TOKEN` environment variable by default. Otherwise, you can pass a Hugging Face API token at initialization with `token` – see code examples below.
The token is needed:

- If you use the Serverless Inference API, or
- If you use the Inference Endpoints.

### Streaming

This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.

## Usage

Install the `huggingface-api-haystack` package to use the `HuggingFaceAPIChatGenerator`:

```shell
pip install huggingface-api-haystack
```

### On its own

#### Using Serverless Inference API (Inference Providers) - Free Tier Available

This API allows you to quickly experiment with many models hosted on the Hugging Face Hub, offloading the inference to Hugging Face servers. It's rate-limited and not meant for production.

To use this API, you need a [free Hugging Face token](https://huggingface.co/settings/tokens).
The Generator expects the `model` in `api_params`. It's also recommended to specify a `provider` for better performance and reliability.

```python
from haystack_integrations.components.generators.huggingface_api import (
    HuggingFaceAPIChatGenerator,
)
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret
from haystack_integrations.components.common.huggingface_api.utils import (
    HFGenerationAPIType,
)

messages = [
    ChatMessage.from_system("\\nYou are a helpful, respectful and honest assistant"),
    ChatMessage.from_user("What's Natural Language Processing?"),
]

# the api_type can be expressed using the HFGenerationAPIType enum or as a string
api_type = HFGenerationAPIType.SERVERLESS_INFERENCE_API
api_type = "serverless_inference_api"  # this is equivalent to the above

generator = HuggingFaceAPIChatGenerator(
    api_type=api_type,
    api_params={"model": "Qwen/Qwen2.5-7B-Instruct", "provider": "together"},
    token=Secret.from_env_var("HF_API_TOKEN"),
)

result = generator.run(messages)
print(result)
```

#### Using Paid Inference Endpoints

In this case, a private instance of the model is deployed by Hugging Face, and you typically pay per hour.

To understand how to spin up an Inference Endpoint, visit [Hugging Face documentation](https://huggingface.co/inference-endpoints/dedicated).

Additionally, in this case, you need to provide your Hugging Face token.
The Generator expects the `url` of your endpoint in `api_params`.

```python
from haystack_integrations.components.generators.huggingface_api import (
    HuggingFaceAPIChatGenerator,
)
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret

messages = [
    ChatMessage.from_system("\\nYou are a helpful, respectful and honest assistant"),
    ChatMessage.from_user("What's Natural Language Processing?"),
]

generator = HuggingFaceAPIChatGenerator(
    api_type="inference_endpoints",
    api_params={"url": "<your-inference-endpoint-url>"},
    token=Secret.from_env_var("HF_API_TOKEN"),
)

result = generator.run(messages)
print(result)
```

#### Using Serverless Inference API (Inference Providers) with Text+Image Input

You can also use this component with multimodal models that support both text and image input:

```python
from haystack_integrations.components.generators.huggingface_api import (
    HuggingFaceAPIChatGenerator,
)
from haystack.dataclasses import ChatMessage, ImageContent
from haystack.utils import Secret
from haystack_integrations.components.common.huggingface_api.utils import (
    HFGenerationAPIType,
)

# Create an image from file path, URL, or base64
image = ImageContent.from_file_path("path/to/your/image.jpg")

# Create a multimodal message with both text and image
messages = [
    ChatMessage.from_user(content_parts=["Describe this image in detail", image]),
]

generator = HuggingFaceAPIChatGenerator(
    api_type=HFGenerationAPIType.SERVERLESS_INFERENCE_API,
    api_params={
        "model": "Qwen/Qwen2.5-VL-7B-Instruct",  # Vision Language Model
        "provider": "hyperbolic",
    },
    token=Secret.from_token("<your-api-key>"),
)

result = generator.run(messages)
print(result)
```

#### Using Self-Hosted Text Generation Inference (TGI)

[Hugging Face Text Generation Inference](https://github.com/huggingface/text-generation-inference) is a toolkit for efficiently deploying and serving LLMs.

While it powers the most recent versions of Serverless Inference API and Inference Endpoints, it can be used easily on-premise through Docker.

For example, you can run a TGI container as follows:

```shell
model=HuggingFaceH4/zephyr-7b-beta
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run

docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:1.4 --model-id $model
```

For more information, refer to the [official TGI repository](https://github.com/huggingface/text-generation-inference).

The Generator expects the `url` of your TGI instance in `api_params`.

```python
from haystack_integrations.components.generators.huggingface_api import (
    HuggingFaceAPIChatGenerator,
)
from haystack.dataclasses import ChatMessage

messages = [
    ChatMessage.from_system("\\nYou are a helpful, respectful and honest assistant"),
    ChatMessage.from_user("What's Natural Language Processing?"),
]

generator = HuggingFaceAPIChatGenerator(
    api_type="text_generation_inference",
    api_params={"url": "http://localhost:8080"},
)

result = generator.run(messages)
print(result)
```

### In a pipeline

```python
from haystack.components.builders import ChatPromptBuilder
from haystack_integrations.components.generators.huggingface_api import (
    HuggingFaceAPIChatGenerator,
)
from haystack.dataclasses import ChatMessage
from haystack import Pipeline
from haystack.utils import Secret
from haystack_integrations.components.common.huggingface_api.utils import (
    HFGenerationAPIType,
)

# no parameter init, we don't use any runtime template variables
prompt_builder = ChatPromptBuilder()
llm = HuggingFaceAPIChatGenerator(
    api_type=HFGenerationAPIType.SERVERLESS_INFERENCE_API,
    api_params={"model": "Qwen/Qwen2.5-7B-Instruct", "provider": "together"},
    token=Secret.from_env_var("HF_API_TOKEN"),
)

pipe = Pipeline()
pipe.add_component("prompt_builder", prompt_builder)
pipe.add_component("llm", llm)
pipe.connect("prompt_builder.prompt", "llm.messages")
location = "Berlin"
messages = [
    ChatMessage.from_system(
        "Always respond in German even if some input data is in other languages.",
    ),
    ChatMessage.from_user("Tell me about {{location}}"),
]
result = pipe.run(
    data={
        "prompt_builder": {
            "template_variables": {"location": location},
            "template": messages,
        },
    },
)

print(result)
```

## Additional References

🧑‍🍳 Cookbook: [Build with Google Gemma: chat and RAG](https://haystack.deepset.ai/cookbook/gemma_chat_rag)
