deepset-ai--haystack/docs-website/versioned_docs/version-2.21/pipeline-components/generators/togetheraigenerator.mdx

---
title: "TogetherAIGenerator"
id: togetheraigenerator
slug: "/togetheraigenerator"
description: "This component enables text generation using models hosted on Together AI."
---

# TogetherAIGenerator

This component enables text generation using models hosted on Together AI.

<div className="key-value-table">

|  |  |
| --- | --- |
| **Most common position in a pipeline** | After a [`PromptBuilder`](../builders/promptbuilder.mdx) |
| **Mandatory init variables** | `api_key`: A Together API key. Can be set with `TOGETHER_API_KEY` env var. |
| **Mandatory run variables** | `prompt`: A string containing the prompt for the LLM |
| **Output variables** | `replies`: A list of strings with all the replies generated by the LLM  <br /> <br />`meta`: A list of dictionaries with the metadata associated with each reply, such as token count, finish reason, and so on |
| **API reference** | [TogetherAI](/reference/integrations-togetherai) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/togetherai |

</div>

## Overview

`TogetherAIGenerator` supports models hosted on [Together AI](https://docs.together.ai/intro), such as `meta-llama/Llama-3.3-70B-Instruct-Turbo`. For the full list of supported models, see [Together AI documentation](https://docs.together.ai/docs/chat-models).

This component needs a prompt string to operate. You can pass any text generation parameters valid for the Together AI chat completion API directly to this component using the `generation_kwargs` parameter in `__init__` or the `generation_kwargs` parameter in `run` method. For more details on the parameters supported by the Together AI API, see [Together AI API documentation](https://docs.together.ai/reference/chat-completions-1).

You can also provide an optional `system_prompt` to set context or instructions for text generation. If not provided, the system prompt is omitted, and the default system prompt of the model is used.

To use this integration, you need to have an active TogetherAI subscription with sufficient credits and an API key. You can provide it with:

- The `TOGETHER_API_KEY` environment variable (recommended)
- The `api_key` init parameter and Haystack [Secret](../../concepts/secret-management.mdx) API: `Secret.from_token("your-api-key-here")`

By default, the component uses Together AI's OpenAI-compatible base URL `https://api.together.xyz/v1`, which you can override with `api_base_url` if needed.

### Streaming

`TogetherAIGenerator` supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) responses from the LLM, allowing tokens to be emitted as they are generated. To enable streaming, pass a callable to the `streaming_callback` parameter during initialization.

:::info
This component is designed for text generation, not for chat. If you want to use Together AI LLMs for chat, use [`TogetherAIChatGenerator`](togetheraichatgenerator.mdx) instead.
:::

## Usage

Install the `togetherai-haystack` package to use the `TogetherAIGenerator`:

```shell
pip install togetherai-haystack
```

### On its own

Basic usage:

```python
from haystack_integrations.components.generators.togetherai import TogetherAIGenerator

client = TogetherAIGenerator(model="meta-llama/Llama-3.3-70B-Instruct-Turbo")
response = client.run("What's Natural Language Processing? Be brief.")
print(response)

>> {'replies': ['Natural Language Processing (NLP) is a branch of artificial intelligence
>> that focuses on enabling computers to understand, interpret, and generate human language
>> in a way that is meaningful and useful.'],
>> 'meta': [{'model': 'meta-llama/Llama-3.3-70B-Instruct-Turbo', 'index': 0,
>> 'finish_reason': 'stop', 'usage': {'prompt_tokens': 15, 'completion_tokens': 36,
>> 'total_tokens': 51}}]}
```

With streaming:

```python
from haystack_integrations.components.generators.togetherai import TogetherAIGenerator

client = TogetherAIGenerator(
    model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
    streaming_callback=lambda chunk: print(chunk.content, end="", flush=True),
)

response = client.run("What's Natural Language Processing? Be brief.")
print(response)
```

With system prompt:

```python
from haystack_integrations.components.generators.togetherai import TogetherAIGenerator

client = TogetherAIGenerator(
    model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
    system_prompt="You are a helpful assistant that provides concise answers.",
)

response = client.run("What's Natural Language Processing?")
print(response["replies"][0])
```

### In a Pipeline

```python
from haystack import Pipeline, Document
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.components.builders.prompt_builder import PromptBuilder
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack_integrations.components.generators.togetherai import TogetherAIGenerator

docstore = InMemoryDocumentStore()
docstore.write_documents([
    Document(content="Rome is the capital of Italy"),
    Document(content="Paris is the capital of France")
])

query = "What is the capital of France?"

template = """
Given the following information, answer the question.

Context:
{% for document in documents %}
    {{ document.content }}
{% endfor %}

Question: {{ query }}?
"""

pipe = Pipeline()
pipe.add_component("retriever", InMemoryBM25Retriever(document_store=docstore))
pipe.add_component("prompt_builder", PromptBuilder(template=template))
pipe.add_component("llm", TogetherAIGenerator(model="meta-llama/Llama-3.3-70B-Instruct-Turbo"))

pipe.connect("retriever", "prompt_builder.documents")
pipe.connect("prompt_builder", "llm")

result = pipe.run({
    "prompt_builder": {"query": query},
    "retriever": {"query": query}
})

print(result)

>> {'llm': {'replies': ['The capital of France is Paris.'],
>> 'meta': [{'model': 'meta-llama/Llama-3.3-70B-Instruct-Turbo', ...}]}}
```