deepset-ai--haystack/docs-website/versioned_docs/version-2.29/pipeline-components/preprocessors/chonkietokendocumentsplitter.mdx

---
title: "ChonkieTokenDocumentSplitter"
id: chonkietokendocumentsplitter
slug: "/chonkietokendocumentsplitter"
description: "Use `ChonkieTokenDocumentSplitter` to split documents into token-based chunks using the Chonkie library."
---

# ChonkieTokenDocumentSplitter

`ChonkieTokenDocumentSplitter` splits documents into fixed-size token-based chunks using [Chonkie](https://docs.chonkie.ai/)'s `TokenChunker`.
It supports multiple tokenizers and is well-suited for splitting long documents before indexing.

<div className="key-value-table">

|  |  |
| --- | --- |
| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx) and [`DocumentCleaner`](documentcleaner.mdx), before [Embedders](../embedders.mdx) |
| **Mandatory run variables**            | `documents`: A list of documents |
| **Output variables**                   | `documents`: A list of documents |
| **API reference**                      | [Chonkie](/reference/integrations-chonkie) |
| **GitHub link**                        | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/chonkie |

</div>

## Overview

`ChonkieTokenDocumentSplitter` wraps Chonkie's `TokenChunker` to split each input document into smaller chunks based on token count.
You can configure the tokenizer, chunk size, and overlap between chunks.

Each output document includes the original document's metadata plus:
- `source_id`: ID of the original document
- `page_number`: Page number of the chunk within the original document
- `split_id`: Index of the chunk within the document
- `split_idx_start` / `split_idx_end`: Character offsets of the chunk in the original text
- `token_count`: Number of tokens in the chunk

## Installation

```bash
pip install chonkie-haystack
```

## Configuration

| Parameter | Default | Description |
| --- | --- | --- |
| `tokenizer` | `"character"` | Tokenizer to use. Common options: `"character"`, `"gpt2"`, `"cl100k_base"`. See [Chonkie docs](https://docs.chonkie.ai/) for all options. |
| `chunk_size` | `2048` | Maximum number of tokens per chunk. |
| `chunk_overlap` | `0` | Number of overlapping tokens between consecutive chunks. |
| `skip_empty_documents` | `True` | Whether to skip documents with empty content. |
| `page_break_character` | `"\f"` | Character used to detect page breaks when tracking page numbers. |

## Usage

### On its own

```python
from haystack import Document
from haystack_integrations.components.preprocessors.chonkie import (
    ChonkieTokenDocumentSplitter,
)

chunker = ChonkieTokenDocumentSplitter(
    tokenizer="gpt2",
    chunk_size=512,
    chunk_overlap=50,
)
documents = [
    Document(
        content="Haystack is an open-source framework for building LLM applications.",
    ),
]
result = chunker.run(documents=documents)
print(result["documents"])
```

### In a pipeline

```python
from pathlib import Path

from haystack import Pipeline
from haystack.components.converters import TextFileToDocument
from haystack.components.preprocessors import DocumentCleaner
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack_integrations.components.preprocessors.chonkie import (
    ChonkieTokenDocumentSplitter,
)

document_store = InMemoryDocumentStore()

p = Pipeline()
p.add_component("converter", TextFileToDocument())
p.add_component("cleaner", DocumentCleaner())
p.add_component(
    "splitter",
    ChonkieTokenDocumentSplitter(tokenizer="gpt2", chunk_size=512),
)
p.add_component("writer", DocumentWriter(document_store=document_store))

p.connect("converter.documents", "cleaner.documents")
p.connect("cleaner.documents", "splitter.documents")
p.connect("splitter.documents", "writer.documents")

files = list(Path("path/to/your/files").glob("*.txt"))
p.run({"converter": {"sources": files}})
```