chore: import upstream snapshot with attribution
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
+190
@@ -0,0 +1,190 @@
|
||||
---
|
||||
title: "ChineseDocumentSplitter"
|
||||
id: chinesedocumentsplitter
|
||||
slug: "/chinesedocumentsplitter"
|
||||
description: "`ChineseDocumentSplitter` divides Chinese text documents into smaller chunks using advanced Chinese language processing capabilities. It leverages HanLP for accurate Chinese word segmentation and sentence tokenization, making it ideal for processing Chinese text that requires linguistic awareness."
|
||||
---
|
||||
|
||||
# ChineseDocumentSplitter
|
||||
|
||||
`ChineseDocumentSplitter` divides Chinese text documents into smaller chunks using advanced Chinese language processing capabilities. It leverages HanLP for accurate Chinese word segmentation and sentence tokenization, making it ideal for processing Chinese text that requires linguistic awareness.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx) and [DocumentCleaner](documentcleaner.mdx), before [Classifiers](../classifiers.mdx) |
|
||||
| **Mandatory run variables** | `documents`: A list of documents with Chinese text content |
|
||||
| **Output variables** | `documents`: A list of documents, each containing a chunk of the original Chinese text |
|
||||
| **API reference** | [HanLP](/reference/integrations-hanlp) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/hanlp |
|
||||
| **Package name** | `hanlp-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`ChineseDocumentSplitter` is a specialized document splitter designed specifically for Chinese text processing. Unlike English text where words are separated by spaces, Chinese text is written continuously without spaces between words.
|
||||
|
||||
This component leverages HanLP (Han Language Processing) to provide accurate Chinese word segmentation and sentence tokenization. It supports two granularity levels:
|
||||
|
||||
- **Coarse granularity**: Provides broader word segmentation suitable for most general use cases. Uses `COARSE_ELECTRA_SMALL_ZH` model for general-purpose segmentation.
|
||||
- **Fine granularity**: Offers more detailed word segmentation for specialized applications. Uses `FINE_ELECTRA_SMALL_ZH` model for detailed segmentation.
|
||||
|
||||
The splitter can divide documents by various units:
|
||||
|
||||
- `word`: Splits by Chinese words (multi-character tokens)
|
||||
- `sentence`: Splits by sentences using HanLP sentence tokenizer
|
||||
- `passage`: Splits by double line breaks ("\\n\\n")
|
||||
- `page`: Splits by form feed characters ("\\f")
|
||||
- `line`: Splits by single line breaks ("\\n")
|
||||
- `period`: Splits by periods (".")
|
||||
- `function`: Uses a custom splitting function
|
||||
|
||||
Each extracted chunk retains metadata from the original document and includes additional fields:
|
||||
|
||||
- `source_id`: The ID of the original document
|
||||
- `page_number`: The page number the chunk belongs to
|
||||
- `split_id`: The sequential ID of the split within the document
|
||||
- `split_idx_start`: The starting index of the chunk in the original document
|
||||
|
||||
When `respect_sentence_boundary=True` is set, the component uses HanLP's sentence tokenizer (`UD_CTB_EOS_MUL`) to ensure that splits occur only between complete sentences, preserving the semantic integrity of the text.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
You can use `ChineseDocumentSplitter` outside of a pipeline to process Chinese documents directly:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.preprocessors.hanlp import ChineseDocumentSplitter
|
||||
|
||||
# Initialize the splitter with word-based splitting
|
||||
splitter = ChineseDocumentSplitter(
|
||||
split_by="word",
|
||||
split_length=10,
|
||||
split_overlap=3,
|
||||
granularity="coarse",
|
||||
)
|
||||
|
||||
# Create a Chinese document
|
||||
doc = Document(
|
||||
content="这是第一句话，这是第二句话，这是第三句话。这是第四句话，这是第五句话，这是第六句话！",
|
||||
)
|
||||
|
||||
# Split the document
|
||||
result = splitter.run(documents=[doc])
|
||||
print(result["documents"]) # List of split documents
|
||||
```
|
||||
|
||||
### With sentence boundary respect
|
||||
|
||||
When splitting by words, you can ensure that sentence boundaries are respected:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.preprocessors.hanlp import ChineseDocumentSplitter
|
||||
|
||||
doc = Document(
|
||||
content="这是第一句话，这是第二句话，这是第三句话。"
|
||||
"这是第四句话，这是第五句话，这是第六句话！"
|
||||
"这是第七句话，这是第八句话，这是第九句话？",
|
||||
)
|
||||
|
||||
splitter = ChineseDocumentSplitter(
|
||||
split_by="word",
|
||||
split_length=10,
|
||||
split_overlap=3,
|
||||
respect_sentence_boundary=True,
|
||||
granularity="coarse",
|
||||
)
|
||||
result = splitter.run(documents=[doc])
|
||||
|
||||
# Each chunk will end with a complete sentence
|
||||
for doc in result["documents"]:
|
||||
print(f"Chunk: {doc.content}")
|
||||
print(f"Ends with sentence: {doc.content.endswith(('。', '！', '？'))}")
|
||||
```
|
||||
|
||||
### With fine granularity
|
||||
|
||||
For more detailed word segmentation:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.preprocessors.hanlp import ChineseDocumentSplitter
|
||||
|
||||
doc = Document(content="人工智能技术正在快速发展，改变着我们的生活方式。")
|
||||
|
||||
splitter = ChineseDocumentSplitter(
|
||||
split_by="word",
|
||||
split_length=5,
|
||||
split_overlap=0,
|
||||
granularity="fine", # More detailed segmentation
|
||||
)
|
||||
result = splitter.run(documents=[doc])
|
||||
print(result["documents"])
|
||||
```
|
||||
|
||||
### With custom splitting function
|
||||
|
||||
You can also use a custom function for splitting:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.preprocessors.hanlp import ChineseDocumentSplitter
|
||||
|
||||
|
||||
def custom_split(text: str) -> list[str]:
|
||||
"""Custom splitting function that splits by commas"""
|
||||
return text.split("，")
|
||||
|
||||
|
||||
doc = Document(content="第一段，第二段，第三段，第四段")
|
||||
|
||||
splitter = ChineseDocumentSplitter(split_by="function", splitting_function=custom_split)
|
||||
splitter.warm_up()
|
||||
result = splitter.run(documents=[doc])
|
||||
print(result["documents"])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Here's how you can integrate `ChineseDocumentSplitter` into a Haystack indexing pipeline:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline, Document
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters.txt import TextFileToDocument
|
||||
from haystack_integrations.components.preprocessors.hanlp import ChineseDocumentSplitter
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
# Initialize components
|
||||
document_store = InMemoryDocumentStore()
|
||||
p = Pipeline()
|
||||
p.add_component(instance=TextFileToDocument(), name="text_file_converter")
|
||||
p.add_component(instance=DocumentCleaner(), name="cleaner")
|
||||
p.add_component(
|
||||
instance=ChineseDocumentSplitter(
|
||||
split_by="word",
|
||||
split_length=100,
|
||||
split_overlap=20,
|
||||
respect_sentence_boundary=True,
|
||||
granularity="coarse",
|
||||
),
|
||||
name="chinese_splitter",
|
||||
)
|
||||
p.add_component(instance=DocumentWriter(document_store=document_store), name="writer")
|
||||
|
||||
# Connect components
|
||||
p.connect("text_file_converter.documents", "cleaner.documents")
|
||||
p.connect("cleaner.documents", "chinese_splitter.documents")
|
||||
p.connect("chinese_splitter.documents", "writer.documents")
|
||||
|
||||
# Run pipeline with Chinese text files
|
||||
p.run({"text_file_converter": {"sources": ["path/to/your/chinese/files.txt"]}})
|
||||
```
|
||||
|
||||
This pipeline processes Chinese text files by converting them to documents, cleaning the text, splitting them into linguistically-aware chunks using Chinese word segmentation, and storing the results in the Document Store for further retrieval and processing.
|
||||
+129
@@ -0,0 +1,129 @@
|
||||
---
|
||||
title: "ChonkieRecursiveDocumentSplitter"
|
||||
id: chonkierecursivedocumentsplitter
|
||||
slug: "/chonkierecursivedocumentsplitter"
|
||||
description: "Use `ChonkieRecursiveDocumentSplitter` to split documents recursively using a hierarchy of rules, powered by the Chonkie library."
|
||||
---
|
||||
|
||||
# ChonkieRecursiveDocumentSplitter
|
||||
|
||||
`ChonkieRecursiveDocumentSplitter` splits documents using a hierarchy of splitting rules via [Chonkie](https://docs.chonkie.ai/)'s `RecursiveChunker`.
|
||||
It applies progressively finer-grained splits until all chunks satisfy the configured size constraints, making it effective for structured text like Markdown or code.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx), before [Embedders](../embedders.mdx) |
|
||||
| **Mandatory run variables** | `documents`: A list of documents |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Chonkie](/reference/integrations-chonkie) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/chonkie |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`ChonkieRecursiveDocumentSplitter` wraps Chonkie's `RecursiveChunker` to split documents by applying splitting rules level by level.
|
||||
If a chunk produced at one level still exceeds `chunk_size`, the next level's rules are applied to it.
|
||||
This continues recursively until all chunks are within the size limit.
|
||||
|
||||
You can customize the splitting behavior by providing `RecursiveRules` from Chonkie.
|
||||
See the [Chonkie documentation](https://docs.chonkie.ai/) for details on defining custom rules.
|
||||
|
||||
Each output document includes the original document's metadata plus:
|
||||
- `source_id`: ID of the original document
|
||||
- `page_number`: Page number of the chunk within the original document
|
||||
- `split_id`: Index of the chunk within the document
|
||||
- `split_idx_start` / `split_idx_end`: Character offsets of the chunk in the original text
|
||||
- `token_count`: Number of tokens in the chunk
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install chonkie-haystack
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
| Parameter | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `tokenizer` | `"character"` | Tokenizer to use. Common options: `"character"`, `"gpt2"`, `"cl100k_base"`. See [Chonkie docs](https://docs.chonkie.ai/) for all options. |
|
||||
| `chunk_size` | `2048` | Maximum number of tokens per chunk. |
|
||||
| `min_characters_per_chunk` | `24` | Minimum number of characters a chunk must contain. |
|
||||
| `rules` | `None` | Custom `RecursiveRules` defining the splitting hierarchy. If `None`, Chonkie's default rules are used. |
|
||||
| `skip_empty_documents` | `True` | Whether to skip documents with empty content. |
|
||||
| `page_break_character` | `"\f"` | Character used to detect page breaks when tracking page numbers. |
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.preprocessors.chonkie import (
|
||||
ChonkieRecursiveDocumentSplitter,
|
||||
)
|
||||
|
||||
chunker = ChonkieRecursiveDocumentSplitter(chunk_size=512)
|
||||
documents = [
|
||||
Document(
|
||||
content="# Introduction\n\nHaystack is a framework.\n\n## Features\n\nIt supports RAG pipelines.",
|
||||
),
|
||||
]
|
||||
result = chunker.run(documents=documents)
|
||||
print(result["documents"])
|
||||
```
|
||||
|
||||
### With custom rules
|
||||
|
||||
```python
|
||||
from chonkie.types.recursive import RecursiveLevel, RecursiveRules
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.preprocessors.chonkie import (
|
||||
ChonkieRecursiveDocumentSplitter,
|
||||
)
|
||||
|
||||
rules = RecursiveRules(
|
||||
levels=[
|
||||
RecursiveLevel(delimiters=["\n\n"]),
|
||||
RecursiveLevel(delimiters=["\n"]),
|
||||
RecursiveLevel(delimiters=[". ", "! ", "? "]),
|
||||
],
|
||||
)
|
||||
|
||||
chunker = ChonkieRecursiveDocumentSplitter(chunk_size=256, rules=rules)
|
||||
documents = [Document(content="First paragraph.\n\nSecond paragraph with more detail.")]
|
||||
result = chunker.run(documents=documents)
|
||||
print(result["documents"])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from haystack import Pipeline
|
||||
from haystack.components.converters import TextFileToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.preprocessors.chonkie import (
|
||||
ChonkieRecursiveDocumentSplitter,
|
||||
)
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
p = Pipeline()
|
||||
p.add_component("converter", TextFileToDocument())
|
||||
p.add_component("cleaner", DocumentCleaner())
|
||||
p.add_component("splitter", ChonkieRecursiveDocumentSplitter(chunk_size=512))
|
||||
p.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
|
||||
p.connect("converter.documents", "cleaner.documents")
|
||||
p.connect("cleaner.documents", "splitter.documents")
|
||||
p.connect("splitter.documents", "writer.documents")
|
||||
|
||||
files = list(Path("path/to/your/files").glob("*.md"))
|
||||
p.run({"converter": {"sources": files}})
|
||||
```
|
||||
+119
@@ -0,0 +1,119 @@
|
||||
---
|
||||
title: "ChonkieSemanticDocumentSplitter"
|
||||
id: chonkiesemanticdocumentsplitter
|
||||
slug: "/chonkiesemanticdocumentsplitter"
|
||||
description: "Use `ChonkieSemanticDocumentSplitter` to split documents at semantic topic boundaries using embedding similarity, powered by the Chonkie library."
|
||||
---
|
||||
|
||||
# ChonkieSemanticDocumentSplitter
|
||||
|
||||
`ChonkieSemanticDocumentSplitter` splits documents at semantically meaningful boundaries using [Chonkie](https://docs.chonkie.ai/)'s `SemanticChunker`.
|
||||
Rather than splitting by a fixed token count, it uses an embedding model to detect topic shifts and keeps related sentences together.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx), before [Embedders](../embedders.mdx) |
|
||||
| **Mandatory run variables** | `documents`: A list of documents |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Chonkie](/reference/integrations-chonkie) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/chonkie |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`ChonkieSemanticDocumentSplitter` wraps Chonkie's `SemanticChunker` to produce context-aware chunks by grouping sentences with similar semantic content.
|
||||
It computes embeddings for sentences and uses cosine similarity to find natural topic boundaries.
|
||||
|
||||
The embedding model is loaded lazily — `warm_up()` is called automatically the first time `run()` is invoked, whether inside a pipeline or standalone.
|
||||
|
||||
Each output document includes the original document's metadata plus:
|
||||
- `source_id`: ID of the original document
|
||||
- `page_number`: Page number of the chunk within the original document
|
||||
- `split_id`: Index of the chunk within the document
|
||||
- `split_idx_start` / `split_idx_end`: Character offsets of the chunk in the original text
|
||||
- `token_count`: Number of tokens in the chunk
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install chonkie-haystack
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
| Parameter | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `embedding_model` | `"minishlab/potion-base-32M"` | The embedding model used to compute sentence similarity. See [Chonkie docs](https://docs.chonkie.ai/) for supported models. |
|
||||
| `threshold` | `0.8` | Cosine similarity threshold below which a sentence boundary becomes a split point. |
|
||||
| `chunk_size` | `2048` | Maximum number of tokens per chunk (based on the embedding model's tokenizer). |
|
||||
| `similarity_window` | `3` | Number of surrounding sentences to include when computing similarity. |
|
||||
| `min_sentences_per_chunk` | `1` | Minimum number of sentences that must be included in each chunk. |
|
||||
| `min_characters_per_sentence` | `24` | Minimum number of characters for a sentence to be considered valid. |
|
||||
| `delim` | `None` | Custom sentence delimiters. If `None`, Chonkie's default delimiters are used. |
|
||||
| `include_delim` | `"prev"` | Whether to attach the delimiter to the previous (`"prev"`) or next (`"next"`) chunk. |
|
||||
| `skip_window` | `0` | Number of sentences to skip when computing similarity scores. |
|
||||
| `filter_window` | `5` | Window size for the Savitzky-Golay smoothing filter applied to similarity scores. |
|
||||
| `filter_polyorder` | `3` | Polynomial order for the Savitzky-Golay filter. |
|
||||
| `filter_tolerance` | `0.2` | Tolerance used when filtering similarity scores. |
|
||||
| `skip_empty_documents` | `True` | Whether to skip documents with empty content. |
|
||||
| `page_break_character` | `"\f"` | Character used to detect page breaks when tracking page numbers. |
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.preprocessors.chonkie import (
|
||||
ChonkieSemanticDocumentSplitter,
|
||||
)
|
||||
|
||||
chunker = ChonkieSemanticDocumentSplitter(chunk_size=512, threshold=0.5)
|
||||
|
||||
documents = [
|
||||
Document(
|
||||
content="Haystack is an open-source framework for LLM applications. "
|
||||
"It makes building RAG pipelines easy. "
|
||||
"The Eiffel Tower is located in Paris. "
|
||||
"Paris is the capital of France.",
|
||||
),
|
||||
]
|
||||
result = chunker.run(documents=documents)
|
||||
print(result["documents"])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from haystack import Pipeline
|
||||
from haystack.components.converters import TextFileToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.preprocessors.chonkie import (
|
||||
ChonkieSemanticDocumentSplitter,
|
||||
)
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
p = Pipeline()
|
||||
p.add_component("converter", TextFileToDocument())
|
||||
p.add_component("cleaner", DocumentCleaner())
|
||||
p.add_component(
|
||||
"splitter",
|
||||
ChonkieSemanticDocumentSplitter(chunk_size=512, threshold=0.5),
|
||||
)
|
||||
p.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
|
||||
p.connect("converter.documents", "cleaner.documents")
|
||||
p.connect("cleaner.documents", "splitter.documents")
|
||||
p.connect("splitter.documents", "writer.documents")
|
||||
|
||||
files = list(Path("path/to/your/files").glob("*.txt"))
|
||||
p.run({"converter": {"sources": files}})
|
||||
```
|
||||
+113
@@ -0,0 +1,113 @@
|
||||
---
|
||||
title: "ChonkieSentenceDocumentSplitter"
|
||||
id: chonkiesentencedocumentsplitter
|
||||
slug: "/chonkiesentencedocumentsplitter"
|
||||
description: "Use `ChonkieSentenceDocumentSplitter` to split documents into sentence-aware chunks using the Chonkie library."
|
||||
---
|
||||
|
||||
# ChonkieSentenceDocumentSplitter
|
||||
|
||||
`ChonkieSentenceDocumentSplitter` splits documents into chunks that respect sentence boundaries using [Chonkie](https://docs.chonkie.ai/)'s `SentenceChunker`.
|
||||
Unlike pure token splitting, it avoids cutting mid-sentence, producing more coherent chunks.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx) and [`DocumentCleaner`](documentcleaner.mdx), before [Embedders](../embedders.mdx) |
|
||||
| **Mandatory run variables** | `documents`: A list of documents |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Chonkie](/reference/integrations-chonkie) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/chonkie |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`ChonkieSentenceDocumentSplitter` wraps Chonkie's `SentenceChunker` to split each input document into chunks whose boundaries align with sentence endings.
|
||||
The chunker groups sentences together until the chunk size limit is reached.
|
||||
|
||||
Each output document includes the original document's metadata plus:
|
||||
- `source_id`: ID of the original document
|
||||
- `page_number`: Page number of the chunk within the original document
|
||||
- `split_id`: Index of the chunk within the document
|
||||
- `split_idx_start` / `split_idx_end`: Character offsets of the chunk in the original text
|
||||
- `token_count`: Number of tokens in the chunk
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install chonkie-haystack
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
| Parameter | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `tokenizer` | `"character"` | Tokenizer to use. Common options: `"character"`, `"gpt2"`, `"cl100k_base"`. See [Chonkie docs](https://docs.chonkie.ai/) for all options. |
|
||||
| `chunk_size` | `2048` | Maximum number of tokens per chunk. |
|
||||
| `chunk_overlap` | `0` | Number of overlapping tokens between consecutive chunks. |
|
||||
| `min_sentences_per_chunk` | `1` | Minimum number of sentences that must be included in each chunk. |
|
||||
| `min_characters_per_sentence` | `12` | Minimum number of characters for a sentence to be considered valid. |
|
||||
| `approximate` | `False` | Whether to use approximate chunking for faster processing. |
|
||||
| `delim` | `None` | Custom sentence delimiters. If `None`, Chonkie's default delimiters are used. |
|
||||
| `include_delim` | `"prev"` | Whether to attach the delimiter to the previous (`"prev"`) or next (`"next"`) chunk. |
|
||||
| `skip_empty_documents` | `True` | Whether to skip documents with empty content. |
|
||||
| `page_break_character` | `"\f"` | Character used to detect page breaks when tracking page numbers. |
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.preprocessors.chonkie import (
|
||||
ChonkieSentenceDocumentSplitter,
|
||||
)
|
||||
|
||||
chunker = ChonkieSentenceDocumentSplitter(
|
||||
tokenizer="gpt2",
|
||||
chunk_size=512,
|
||||
chunk_overlap=0,
|
||||
)
|
||||
documents = [
|
||||
Document(
|
||||
content="Haystack is an open-source framework. It helps you build LLM applications.",
|
||||
),
|
||||
]
|
||||
result = chunker.run(documents=documents)
|
||||
print(result["documents"])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from haystack import Pipeline
|
||||
from haystack.components.converters import TextFileToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.preprocessors.chonkie import (
|
||||
ChonkieSentenceDocumentSplitter,
|
||||
)
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
p = Pipeline()
|
||||
p.add_component("converter", TextFileToDocument())
|
||||
p.add_component("cleaner", DocumentCleaner())
|
||||
p.add_component(
|
||||
"splitter",
|
||||
ChonkieSentenceDocumentSplitter(tokenizer="gpt2", chunk_size=512),
|
||||
)
|
||||
p.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
|
||||
p.connect("converter.documents", "cleaner.documents")
|
||||
p.connect("cleaner.documents", "splitter.documents")
|
||||
p.connect("splitter.documents", "writer.documents")
|
||||
|
||||
files = list(Path("path/to/your/files").glob("*.txt"))
|
||||
p.run({"converter": {"sources": files}})
|
||||
```
|
||||
+108
@@ -0,0 +1,108 @@
|
||||
---
|
||||
title: "ChonkieTokenDocumentSplitter"
|
||||
id: chonkietokendocumentsplitter
|
||||
slug: "/chonkietokendocumentsplitter"
|
||||
description: "Use `ChonkieTokenDocumentSplitter` to split documents into token-based chunks using the Chonkie library."
|
||||
---
|
||||
|
||||
# ChonkieTokenDocumentSplitter
|
||||
|
||||
`ChonkieTokenDocumentSplitter` splits documents into fixed-size token-based chunks using [Chonkie](https://docs.chonkie.ai/)'s `TokenChunker`.
|
||||
It supports multiple tokenizers and is well-suited for splitting long documents before indexing.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx) and [`DocumentCleaner`](documentcleaner.mdx), before [Embedders](../embedders.mdx) |
|
||||
| **Mandatory run variables** | `documents`: A list of documents |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Chonkie](/reference/integrations-chonkie) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/chonkie |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`ChonkieTokenDocumentSplitter` wraps Chonkie's `TokenChunker` to split each input document into smaller chunks based on token count.
|
||||
You can configure the tokenizer, chunk size, and overlap between chunks.
|
||||
|
||||
Each output document includes the original document's metadata plus:
|
||||
- `source_id`: ID of the original document
|
||||
- `page_number`: Page number of the chunk within the original document
|
||||
- `split_id`: Index of the chunk within the document
|
||||
- `split_idx_start` / `split_idx_end`: Character offsets of the chunk in the original text
|
||||
- `token_count`: Number of tokens in the chunk
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install chonkie-haystack
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
| Parameter | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `tokenizer` | `"character"` | Tokenizer to use. Common options: `"character"`, `"gpt2"`, `"cl100k_base"`. See [Chonkie docs](https://docs.chonkie.ai/) for all options. |
|
||||
| `chunk_size` | `2048` | Maximum number of tokens per chunk. |
|
||||
| `chunk_overlap` | `0` | Number of overlapping tokens between consecutive chunks. |
|
||||
| `skip_empty_documents` | `True` | Whether to skip documents with empty content. |
|
||||
| `page_break_character` | `"\f"` | Character used to detect page breaks when tracking page numbers. |
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.preprocessors.chonkie import (
|
||||
ChonkieTokenDocumentSplitter,
|
||||
)
|
||||
|
||||
chunker = ChonkieTokenDocumentSplitter(
|
||||
tokenizer="gpt2",
|
||||
chunk_size=512,
|
||||
chunk_overlap=50,
|
||||
)
|
||||
documents = [
|
||||
Document(
|
||||
content="Haystack is an open-source framework for building LLM applications.",
|
||||
),
|
||||
]
|
||||
result = chunker.run(documents=documents)
|
||||
print(result["documents"])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from haystack import Pipeline
|
||||
from haystack.components.converters import TextFileToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.preprocessors.chonkie import (
|
||||
ChonkieTokenDocumentSplitter,
|
||||
)
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
p = Pipeline()
|
||||
p.add_component("converter", TextFileToDocument())
|
||||
p.add_component("cleaner", DocumentCleaner())
|
||||
p.add_component(
|
||||
"splitter",
|
||||
ChonkieTokenDocumentSplitter(tokenizer="gpt2", chunk_size=512),
|
||||
)
|
||||
p.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
|
||||
p.connect("converter.documents", "cleaner.documents")
|
||||
p.connect("cleaner.documents", "splitter.documents")
|
||||
p.connect("splitter.documents", "writer.documents")
|
||||
|
||||
files = list(Path("path/to/your/files").glob("*.txt"))
|
||||
p.run({"converter": {"sources": files}})
|
||||
```
|
||||
+90
@@ -0,0 +1,90 @@
|
||||
---
|
||||
title: "CSVDocumentCleaner"
|
||||
id: csvdocumentcleaner
|
||||
slug: "/csvdocumentcleaner"
|
||||
description: "Use `CSVDocumentCleaner` to clean CSV documents by removing empty rows and columns while preserving specific ignored rows and columns. It processes CSV content stored in documents and helps standardize data for further analysis."
|
||||
---
|
||||
|
||||
# CSVDocumentCleaner
|
||||
|
||||
Use `CSVDocumentCleaner` to clean CSV documents by removing empty rows and columns while preserving specific ignored rows and columns. It processes CSV content stored in documents and helps standardize data for further analysis.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx) , before [Embedders](../embedders.mdx) or [Writers](../writers/documentwriter.mdx) |
|
||||
| **Mandatory run variables** | `documents`: A list of documents containing CSV content |
|
||||
| **Output variables** | `documents`: A list of cleaned CSV documents |
|
||||
| **API reference** | [PreProcessors](/reference/preprocessors-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/csv_document_cleaner.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`CSVDocumentCleaner` expects a list of `Document` objects as input, each containing CSV-formatted content as text. It cleans the data by removing fully empty rows and columns while allowing users to specify the number of rows and columns to be preserved before cleaning.
|
||||
|
||||
### Parameters
|
||||
|
||||
- `ignore_rows`: Number of rows to ignore from the top of the CSV table before processing. If any columns are removed, the same columns will be dropped from the ignored rows.
|
||||
- `ignore_columns`: Number of columns to ignore from the left of the CSV table before processing. If any rows are removed, the same rows will be dropped from the ignored columns.
|
||||
- `remove_empty_rows`: Whether to remove entirely empty rows.
|
||||
- `remove_empty_columns`: Whether to remove entirely empty columns.
|
||||
- `keep_id`: Whether to retain the original document ID in the output document.
|
||||
|
||||
### Cleaning Process
|
||||
|
||||
The `CSVDocumentCleaner` algorithm follows these steps:
|
||||
|
||||
1. Reads each document's content as a CSV table using pandas.
|
||||
2. Retains the specified number of `ignore_rows` from the top and `ignore_columns` from the left.
|
||||
3. Drops any rows and columns that are entirely empty (contain only NaN values).
|
||||
4. If columns are dropped, they are also removed from ignored rows.
|
||||
5. If rows are dropped, they are also removed from ignored columns.
|
||||
6. Reattaches the remaining ignored rows and columns to maintain their original positions.
|
||||
7. Returns the cleaned CSV content as a new `Document` object.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
You can use `CSVDocumentCleaner` independently to clean up CSV documents:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.preprocessors import CSVDocumentCleaner
|
||||
|
||||
cleaner = CSVDocumentCleaner(ignore_rows=1, ignore_columns=0)
|
||||
|
||||
documents = [Document(content="""col1,col2,col3\n,,\na,b,c\n,,""")]
|
||||
cleaned_docs = cleaner.run(documents=documents)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters import XLSXToDocument
|
||||
from haystack.components.preprocessors import CSVDocumentCleaner
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
p = Pipeline()
|
||||
p.add_component(instance=XLSXToDocument(), name="xlsx_file_converter")
|
||||
p.add_component(
|
||||
instance=CSVDocumentCleaner(ignore_rows=1, ignore_columns=1),
|
||||
name="csv_cleaner",
|
||||
)
|
||||
p.add_component(instance=DocumentWriter(document_store=document_store), name="writer")
|
||||
|
||||
p.connect("xlsx_file_converter.documents", "csv_cleaner.documents")
|
||||
p.connect("csv_cleaner.documents", "writer.documents")
|
||||
|
||||
p.run({"xlsx_file_converter": {"sources": [Path("your_xlsx_file.xlsx")]}})
|
||||
```
|
||||
|
||||
This ensures that CSV documents are properly cleaned before further processing or storage.
|
||||
+118
@@ -0,0 +1,118 @@
|
||||
---
|
||||
title: "CSVDocumentSplitter"
|
||||
id: csvdocumentsplitter
|
||||
slug: "/csvdocumentsplitter"
|
||||
description: "`CSVDocumentSplitter` divides CSV documents into smaller sub-tables based on split arguments. This is useful for handling structured data that contains multiple tables, improving data processing efficiency and retrieval."
|
||||
---
|
||||
|
||||
# CSVDocumentSplitter
|
||||
|
||||
`CSVDocumentSplitter` divides CSV documents into smaller sub-tables based on split arguments. This is useful for handling structured data that contains multiple tables, improving data processing efficiency and retrieval.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx) , before [CSVDocumentCleaner](csvdocumentcleaner.mdx) |
|
||||
| **Mandatory run variables** | `documents`: A list of documents with CSV-formatted content |
|
||||
| **Output variables** | `documents`: A list of documents, each containing a sub-table extracted from the original CSV file |
|
||||
| **API reference** | [PreProcessors](/reference/preprocessors-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/csv_document_splitter.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`CSVDocumentSplitter` expects a list of documents containing CSV-formatted content and returns a list of new `Document` objects, each representing a sub-table extracted from the original document.
|
||||
|
||||
There are two modes of operation for the splitter:
|
||||
|
||||
1. `threshold` (Default): Identifies empty rows or columns exceeding a given threshold and splits the document accordingly.
|
||||
2. `row-wise`: Splits each row into a separate document, treating each as an independent sub-table.
|
||||
|
||||
The splitting process follows these rules:
|
||||
|
||||
1. **Row-Based Splitting**: If `row_split_threshold` is set, consecutive empty rows equalling or exceeding this threshold trigger a split.
|
||||
2. **Column-Based Splitting**: If `column_split_threshold` is set, consecutive empty columns equalling or exceeding this threshold trigger a split.
|
||||
3. **Recursive Splitting**: If both thresholds are provided, `CSVDocumentSplitter` first splits by rows and then by columns. If more empty rows are detected, the splitting process is called again. This ensures that sub-tables are fully separated.
|
||||
|
||||
Each extracted sub-table retains metadata from the original document and includes additional fields:
|
||||
|
||||
- `source_id`: The ID of the original document
|
||||
- `row_idx_start`: The starting row index of the sub-table in the original document
|
||||
- `col_idx_start`: The starting column index of the sub-table in the original document
|
||||
- `split_id`: The sequential ID of the split within the document
|
||||
|
||||
This component is especially useful for document processing pipelines that require structured data to be extracted and stored efficiently.
|
||||
|
||||
### Supported Document Stores
|
||||
|
||||
`CSVDocumentSplitter` is compatible with the following Document Stores:
|
||||
|
||||
- [AstraDocumentStore](../../document-stores/astradocumentstore.mdx)
|
||||
- [ChromaDocumentStore](../../document-stores/chromadocumentstore.mdx)
|
||||
- [ElasticsearchDocumentStore](../../document-stores/elasticsearch-document-store.mdx)
|
||||
- [OpenSearchDocumentStore](../../document-stores/opensearch-document-store.mdx)
|
||||
- [PgvectorDocumentStore](../../document-stores/pgvectordocumentstore.mdx)
|
||||
- [PineconeDocumentStore](../../document-stores/pinecone-document-store.mdx)
|
||||
- [QdrantDocumentStore](../../document-stores/qdrant-document-store.mdx)
|
||||
- [WeaviateDocumentStore](../../document-stores/weaviatedocumentstore.mdx)
|
||||
- [MilvusDocumentStore](https://haystack.deepset.ai/integrations/milvus-document-store)
|
||||
- [Neo4jDocumentStore](https://haystack.deepset.ai/integrations/neo4j-document-store)
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
You can use `CSVDocumentSplitter` outside of a pipeline to process CSV documents directly:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.preprocessors import CSVDocumentSplitter
|
||||
|
||||
splitter = CSVDocumentSplitter(row_split_threshold=1, column_split_threshold=1)
|
||||
|
||||
doc = Document(
|
||||
content="""ID,LeftVal,,,RightVal,Extra
|
||||
1,Hello,,,World,Joined
|
||||
2,StillLeft,,,StillRight,Bridge
|
||||
,,,,,
|
||||
A,B,,,C,D
|
||||
E,F,,,G,H
|
||||
""",
|
||||
)
|
||||
split_result = splitter.run([doc])
|
||||
print(split_result["documents"]) # List of split tables as Documents
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Here's how you can integrate `CSVDocumentSplitter` into a Haystack indexing pipeline:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline, Document
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters.csv import CSVToDocument
|
||||
from haystack.components.preprocessors import CSVDocumentSplitter
|
||||
from haystack.components.preprocessors import CSVDocumentCleaner
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
# Initialize components
|
||||
document_store = InMemoryDocumentStore()
|
||||
p = Pipeline()
|
||||
p.add_component(instance=CSVToDocument(), name="csv_file_converter")
|
||||
p.add_component(instance=CSVDocumentSplitter(), name="splitter")
|
||||
p.add_component(instance=CSVDocumentCleaner(), name="cleaner")
|
||||
p.add_component(instance=DocumentWriter(document_store=document_store), name="writer")
|
||||
|
||||
# Connect components
|
||||
p.connect("csv_file_converter.documents", "splitter.documents")
|
||||
p.connect("splitter.documents", "cleaner.documents")
|
||||
p.connect("cleaner.documents", "writer.documents")
|
||||
|
||||
# Run pipeline
|
||||
p.run({"csv_file_converter": {"sources": ["path/to/your/file.csv"]}})
|
||||
```
|
||||
|
||||
This pipeline extracts CSV content, splits it into structured sub-tables, cleans the CSV documents by removing empty rows and columns, and stores the resulting documents in the Document Store for further retrieval and processing.
|
||||
+152
@@ -0,0 +1,152 @@
|
||||
---
|
||||
title: "DocumentCleaner"
|
||||
id: documentcleaner
|
||||
slug: "/documentcleaner"
|
||||
description: "Use `DocumentCleaner` to make text documents more readable. It removes extra whitespaces, empty lines, specified substrings, regexes, page headers, and footers in this particular order. This is useful for preparing the documents for further processing by LLMs."
|
||||
---
|
||||
|
||||
# DocumentCleaner
|
||||
|
||||
Use `DocumentCleaner` to make text documents more readable. It removes extra whitespaces, empty lines, specified substrings, regexes, page headers, and footers in this particular order. This is useful for preparing the documents for further processing by LLMs.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx) , after [`DocumentSplitter`](documentsplitter.mdx) |
|
||||
| **Mandatory run variables** | `documents`: A list of documents |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [PreProcessors](/reference/preprocessors-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/document_cleaner.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`DocumentCleaner` expects a list of documents as input and returns a list of documents with cleaned texts. Selectable cleaning steps for each input document are to `remove_empty_lines`, `remove_extra_whitespaces` and to `remove_repeated_substrings`. These three parameters are booleans that can be set when the component is initialized.
|
||||
|
||||
- `unicode_normalization` normalizes Unicode characters to a standard form. The parameter can be set to NFC, NFKC, NFD, or NFKD.
|
||||
- `ascii_only` removes accents from characters and replaces them with their closest ASCII equivalents.
|
||||
- `remove_empty_lines` removes empty lines from the document.
|
||||
- `remove_extra_whitespaces` removes extra whitespaces from the document.
|
||||
- `remove_repeated_substrings` removes repeated substrings (headers/footers) from pages in the document. Pages in the text need to be separated by form feed character "\\f", which is supported by [`TextFileToDocument`](../converters/textfiletodocument.mdx), [`AzureOCRDocumentConverter`](../converters/azureocrdocumentconverter.mdx), [`MistralOCRDocumentConverter`](../converters/mistralocrdocumentconverter.mdx), and [`PaddleOCRVLDocumentConverter`](../converters/paddleocrvldocumentconverter.mdx).
|
||||
|
||||
:::note
|
||||
`remove_extra_whitespaces` and `remove_empty_lines` work best on plain-text content. If your converter returns Markdown, such as [`AzureDocumentIntelligenceConverter`](../converters/azuredocumentintelligenceconverter.mdx), [`MarkItDownConverter`](../converters/markitdownconverter.mdx), [`MistralOCRDocumentConverter`](../converters/mistralocrdocumentconverter.mdx), or [`PaddleOCRVLDocumentConverter`](../converters/paddleocrvldocumentconverter.mdx), disable those options to preserve headings, tables, lists, and image tags.
|
||||
:::
|
||||
|
||||
In addition, you can specify a list of strings that should be removed from all documents as part of the cleaning with the parameter `remove_substring`. You can also specify a regular expression with the parameter `remove_regex` and any matches will be removed.
|
||||
|
||||
The cleaning steps are executed in the following order:
|
||||
|
||||
1. unicode_normalization
|
||||
2. ascii_only
|
||||
3. remove_extra_whitespaces
|
||||
4. remove_empty_lines
|
||||
5. remove_substrings
|
||||
6. remove_regex
|
||||
7. remove_repeated_substrings
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
You can use it outside of a pipeline to clean up your documents:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
|
||||
doc = Document(content="This is a document to clean\n\n\nsubstring to remove")
|
||||
|
||||
cleaner = DocumentCleaner(remove_substrings=["substring to remove"])
|
||||
result = cleaner.run(documents=[doc])
|
||||
|
||||
assert result["documents"][0].content == "This is a document to clean "
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters import TextFileToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
p = Pipeline()
|
||||
p.add_component(instance=TextFileToDocument(), name="text_file_converter")
|
||||
p.add_component(instance=DocumentCleaner(), name="cleaner")
|
||||
p.add_component(
|
||||
instance=DocumentSplitter(split_by="sentence", split_length=1),
|
||||
name="splitter",
|
||||
)
|
||||
p.add_component(instance=DocumentWriter(document_store=document_store), name="writer")
|
||||
p.connect("text_file_converter.documents", "cleaner.documents")
|
||||
p.connect("cleaner.documents", "splitter.documents")
|
||||
p.connect("splitter.documents", "writer.documents")
|
||||
|
||||
p.run({"text_file_converter": {"sources": your_files}})
|
||||
```
|
||||
|
||||
### In YAML
|
||||
```yaml
|
||||
components:
|
||||
cleaner:
|
||||
init_parameters:
|
||||
ascii_only: false
|
||||
keep_id: false
|
||||
remove_empty_lines: true
|
||||
remove_extra_whitespaces: true
|
||||
remove_regex: null
|
||||
remove_repeated_substrings: false
|
||||
remove_substrings: null
|
||||
replace_regexes: null
|
||||
strip_whitespaces: false
|
||||
unicode_normalization: null
|
||||
type: haystack.components.preprocessors.document_cleaner.DocumentCleaner
|
||||
splitter:
|
||||
init_parameters:
|
||||
extend_abbreviations: true
|
||||
language: en
|
||||
respect_sentence_boundary: false
|
||||
skip_empty_documents: true
|
||||
split_by: sentence
|
||||
split_length: 1
|
||||
split_overlap: 0
|
||||
split_threshold: 0
|
||||
use_split_rules: true
|
||||
type: haystack.components.preprocessors.document_splitter.DocumentSplitter
|
||||
text_file_converter:
|
||||
init_parameters:
|
||||
encoding: utf-8
|
||||
store_full_path: false
|
||||
type: haystack.components.converters.txt.TextFileToDocument
|
||||
writer:
|
||||
init_parameters:
|
||||
document_store:
|
||||
init_parameters:
|
||||
bm25_algorithm: BM25L
|
||||
bm25_parameters: {}
|
||||
bm25_tokenization_regex: (?u)\\b\\w+\\b
|
||||
embedding_similarity_function: dot_product
|
||||
index: 64e4f9ab-87fb-47fd-b390-dabcfda61447
|
||||
return_embedding: true
|
||||
type: haystack.document_stores.in_memory.document_store.InMemoryDocumentStore
|
||||
policy: NONE
|
||||
type: haystack.components.writers.document_writer.DocumentWriter
|
||||
connection_type_validation: true
|
||||
connections:
|
||||
- receiver: cleaner.documents
|
||||
sender: text_file_converter.documents
|
||||
- receiver: splitter.documents
|
||||
sender: cleaner.documents
|
||||
- receiver: writer.documents
|
||||
sender: splitter.documents
|
||||
max_runs_per_component: 100
|
||||
metadata: {}
|
||||
```
|
||||
+80
@@ -0,0 +1,80 @@
|
||||
---
|
||||
title: "DocumentPreprocessor"
|
||||
id: documentpreprocessor
|
||||
slug: "/documentpreprocessor"
|
||||
description: "Divides a list of text documents into a list of shorter text documents and then makes them more readable by cleaning."
|
||||
---
|
||||
|
||||
# DocumentPreprocessor
|
||||
|
||||
Divides a list of text documents into a list of shorter text documents and then makes them more readable by cleaning.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx) |
|
||||
| **Mandatory run variables** | `documents`: A list of documents |
|
||||
| **Output variables** | `documents`: A list of split and cleaned documents |
|
||||
| **API reference** | [PreProcessors](/reference/preprocessors-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/document_preprocessor.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`DocumentPreprocessor` first splits and then cleans documents.
|
||||
|
||||
It is a SuperComponent that combines a `DocumentSplitter` and a `DocumentCleaner` into a single component.
|
||||
|
||||
### Parameters
|
||||
|
||||
The `DocumentPreprocessor` exposes all initialization parameters of the underlying `DocumentSplitter` and `DocumentCleaner`, and they are all optional. A detailed description of their parameters is in the respective documentation pages:
|
||||
|
||||
- [DocumentSplitter](documentsplitter.mdx)
|
||||
- [DocumentCleaner](documentcleaner.mdx)
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.preprocessors import DocumentPreprocessor
|
||||
|
||||
doc = Document(content="I love pizza!")
|
||||
preprocessor = DocumentPreprocessor()
|
||||
|
||||
result = preprocessor.run(documents=[doc])
|
||||
print(result["documents"])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
You can use the `DocumentPreprocessor` in your indexing pipeline. The example below requires installing additional dependencies for the `MultiFileConverter`:
|
||||
|
||||
```shell
|
||||
pip install pypdf markdown-it-py mdit_plain trafilatura python-pptx python-docx jq openpyxl tabulate pandas
|
||||
```
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.converters import MultiFileConverter
|
||||
from haystack.components.preprocessors import DocumentPreprocessor
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("converter", MultiFileConverter())
|
||||
pipeline.add_component("preprocessor", DocumentPreprocessor())
|
||||
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
pipeline.connect("converter", "preprocessor")
|
||||
pipeline.connect("preprocessor", "writer")
|
||||
|
||||
result = pipeline.run(data={"sources": ["test.txt", "test.pdf"]})
|
||||
print(result)
|
||||
# {'writer': {'documents_written': 3}}
|
||||
```
|
||||
+159
@@ -0,0 +1,159 @@
|
||||
---
|
||||
title: "DocumentSplitter"
|
||||
id: documentsplitter
|
||||
slug: "/documentsplitter"
|
||||
description: "`DocumentSplitter` divides a list of text documents into a list of shorter text documents. This is useful for long texts that otherwise wouldn't fit into the maximum text length of language models and can also speed up question answering."
|
||||
---
|
||||
|
||||
# DocumentSplitter
|
||||
|
||||
`DocumentSplitter` divides a list of text documents into a list of shorter text documents. This is useful for long texts that otherwise wouldn't fit into the maximum text length of language models and can also speed up question answering.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx) and [`DocumentCleaner`](documentcleaner.mdx) , before [Classifiers](../classifiers.mdx) |
|
||||
| **Mandatory run variables** | `documents`: A list of documents |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [PreProcessors](/reference/preprocessors-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/document_splitter.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`DocumentSplitter` expects a list of documents as input and returns a list of documents with split texts. It splits each input document by `split_by` after `split_length` units with an overlap of `split_overlap` units. These additional parameters can be set when the component is initialized:
|
||||
|
||||
- `split_by` can be `"word"`, `"sentence"`, `"passage"` (paragraph), `"page"`, `"line"` or `"function"`.
|
||||
- `split_length` is an integer indicating the chunk size, which is the number of words, sentences, or passages.
|
||||
- `split_overlap` is an integer indicating the number of overlapping words, sentences, or passages between chunks.
|
||||
- `split_threshold` is an integer indicating the minimum number of words, sentences, or passages that the document fragment should have. If the fragment is below the threshold, it will be attached to the previous one.
|
||||
|
||||
A field `"source_id"` is added to each document's `meta` data to keep track of the original document that was split. Another meta field `"page_number"` is added to each document to keep track of the page it belonged to in the original document. Other metadata are copied from the original document.
|
||||
|
||||
The DocumentSplitter is compatible with the following DocumentStores:
|
||||
|
||||
- [AstraDocumentStore](../../document-stores/astradocumentstore.mdx)
|
||||
- [ChromaDocumentStore](../../document-stores/chromadocumentstore.mdx) – limited support, overlapping information is not stored.
|
||||
- [ElasticsearchDocumentStore](../../document-stores/elasticsearch-document-store.mdx)
|
||||
- [OpenSearchDocumentStore](../../document-stores/opensearch-document-store.mdx)
|
||||
- [PgvectorDocumentStore](../../document-stores/pgvectordocumentstore.mdx)
|
||||
- [PineconeDocumentStore](../../document-stores/pinecone-document-store.mdx) – limited support, overlapping information is not stored.
|
||||
- [QdrantDocumentStore](../../document-stores/qdrant-document-store.mdx)
|
||||
- [WeaviateDocumentStore](../../document-stores/weaviatedocumentstore.mdx)
|
||||
- [MilvusDocumentStore](https://haystack.deepset.ai/integrations/milvus-document-store)
|
||||
- [Neo4jDocumentStore](https://haystack.deepset.ai/integrations/neo4j-document-store)
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
You can use this component outside of a pipeline to shorten your documents like this:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
|
||||
doc = Document(
|
||||
content="Moonlight shimmered softly, wolves howled nearby, night enveloped everything.",
|
||||
)
|
||||
|
||||
splitter = DocumentSplitter(split_by="word", split_length=3, split_overlap=0)
|
||||
result = splitter.run(documents=[doc])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Here's how you can use `DocumentSplitter` in an indexing pipeline:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters.txt import TextFileToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
p = Pipeline()
|
||||
p.add_component(instance=TextFileToDocument(), name="text_file_converter")
|
||||
p.add_component(instance=DocumentCleaner(), name="cleaner")
|
||||
p.add_component(
|
||||
instance=DocumentSplitter(split_by="sentence", split_length=1),
|
||||
name="splitter",
|
||||
)
|
||||
p.add_component(instance=DocumentWriter(document_store=document_store), name="writer")
|
||||
p.connect("text_file_converter.documents", "cleaner.documents")
|
||||
p.connect("cleaner.documents", "splitter.documents")
|
||||
p.connect("splitter.documents", "writer.documents")
|
||||
|
||||
path = "path/to/your/files"
|
||||
files = list(Path(path).glob("*.md"))
|
||||
p.run({"text_file_converter": {"sources": files}})
|
||||
```
|
||||
|
||||
### In YAML
|
||||
|
||||
This is the YAML representation of the indexing pipeline shown above. It reads text files, cleans the text, splits it into individual sentences, and writes them to an in-memory document store.
|
||||
|
||||
```yaml
|
||||
components:
|
||||
cleaner:
|
||||
init_parameters:
|
||||
ascii_only: false
|
||||
keep_id: false
|
||||
remove_empty_lines: true
|
||||
remove_extra_whitespaces: true
|
||||
remove_regex: null
|
||||
remove_repeated_substrings: false
|
||||
remove_substrings: null
|
||||
replace_regexes: null
|
||||
strip_whitespaces: false
|
||||
unicode_normalization: null
|
||||
type: haystack.components.preprocessors.document_cleaner.DocumentCleaner
|
||||
splitter:
|
||||
init_parameters:
|
||||
extend_abbreviations: true
|
||||
language: en
|
||||
respect_sentence_boundary: false
|
||||
skip_empty_documents: true
|
||||
split_by: sentence
|
||||
split_length: 1
|
||||
split_overlap: 0
|
||||
split_threshold: 0
|
||||
use_split_rules: true
|
||||
type: haystack.components.preprocessors.document_splitter.DocumentSplitter
|
||||
text_file_converter:
|
||||
init_parameters:
|
||||
encoding: utf-8
|
||||
store_full_path: false
|
||||
type: haystack.components.converters.txt.TextFileToDocument
|
||||
writer:
|
||||
init_parameters:
|
||||
document_store:
|
||||
init_parameters:
|
||||
bm25_algorithm: BM25L
|
||||
bm25_parameters: {}
|
||||
bm25_tokenization_regex: (?u)\\b\\w+\\b
|
||||
embedding_similarity_function: dot_product
|
||||
index: 64e4f9ab-87fb-47fd-b390-dabcfda61447
|
||||
return_embedding: true
|
||||
type: haystack.document_stores.in_memory.document_store.InMemoryDocumentStore
|
||||
policy: NONE
|
||||
type: haystack.components.writers.document_writer.DocumentWriter
|
||||
connection_type_validation: true
|
||||
connections:
|
||||
- receiver: cleaner.documents
|
||||
sender: text_file_converter.documents
|
||||
- receiver: splitter.documents
|
||||
sender: cleaner.documents
|
||||
- receiver: writer.documents
|
||||
sender: splitter.documents
|
||||
max_runs_per_component: 100
|
||||
metadata: {}
|
||||
```
|
||||
+98
@@ -0,0 +1,98 @@
|
||||
---
|
||||
title: "EmbeddingBasedDocumentSplitter"
|
||||
id: embeddingbaseddocumentsplitter
|
||||
slug: "/embeddingbaseddocumentsplitter"
|
||||
description: "Use this component to split documents based on embedding similarity using cosine distances between sequential sentence groups."
|
||||
---
|
||||
|
||||
# EmbeddingBasedDocumentSplitter
|
||||
|
||||
Use this component to split documents based on embedding similarity using cosine distances between sequential sentence groups.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx) and [`DocumentCleaner`](documentcleaner.mdx) |
|
||||
| **Mandatory run variables** | `documents`: A list of documents to split each into smaller documents based on embedding similarity. |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [PreProcessors](/reference/preprocessors-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/embedding_based_document_splitter.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
This component splits documents based on embedding similarity using cosine distances between sequential sentence groups.
|
||||
|
||||
It first splits text into sentences, optionally groups them, calculates embeddings for each group, and then uses cosine
|
||||
distance between sequential embeddings to determine split points. Any distance above the specified percentile is treated
|
||||
as a break point. The component also tracks page numbers based on form feed characters (`\f`) in the original document.
|
||||
|
||||
This component is inspired by [5 Levels of Text Splitting](https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/tutorials/LevelsOfTextSplitting/5_Levels_Of_Text_Splitting.ipynb) by Greg Kamradt.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
|
||||
from haystack import Document
|
||||
from haystack.components.embedders import SentenceTransformersDocumentEmbedder
|
||||
from haystack.components.preprocessors import EmbeddingBasedDocumentSplitter
|
||||
|
||||
# Create a document with content that has a clear topic shift
|
||||
doc = Document(
|
||||
content="This is a first sentence. This is a second sentence. This is a third sentence. "
|
||||
"Completely different topic. The same completely different topic.",
|
||||
)
|
||||
|
||||
# Initialize the embedder to calculate semantic similarities
|
||||
embedder = SentenceTransformersDocumentEmbedder()
|
||||
|
||||
# Configure the splitter with parameters that control splitting behavior
|
||||
splitter = EmbeddingBasedDocumentSplitter(
|
||||
document_embedder=embedder,
|
||||
sentences_per_group=2, # Group 2 sentences before calculating embeddings
|
||||
percentile=0.95, # Split when cosine distance exceeds 95th percentile
|
||||
min_length=50, # Merge splits shorter than 50 characters
|
||||
max_length=1000, # Further split chunks longer than 1000 characters
|
||||
)
|
||||
result = splitter.run(documents=[doc])
|
||||
|
||||
# The result contains a list of Document objects, each representing a semantic chunk
|
||||
# Each split document includes metadata: source_id, split_id, and page_number
|
||||
print(f"Original document split into {len(result['documents'])} chunks")
|
||||
for i, split_doc in enumerate(result["documents"]):
|
||||
print(f"Chunk {i}: {split_doc.content[:50]}...")
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters.txt import TextFileToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.preprocessors import EmbeddingBasedDocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
Pipeline = Pipeline()
|
||||
Pipeline.add_component(instance=TextFileToDocument(), name="text_file_converter")
|
||||
Pipeline.add_component(instance=DocumentCleaner(), name="cleaner")
|
||||
Pipeline.add_component(instance=EmbeddingBasedDocumentSplitter(document_embedder=embedder, sentences_per_group=2, percentile=0.95, min_length=50,max_length=1000)
|
||||
Pipeline.add_component(instance=DocumentWriter(document_store=document_store), name="writer")
|
||||
Pipeline.connect("text_file_converter.documents", "cleaner.documents")
|
||||
Pipeline.connect("cleaner.documents", "splitter.documents")
|
||||
Pipeline.connect("splitter.documents", "writer.documents")
|
||||
|
||||
path = "path/to/your/files"
|
||||
files = list(Path(path).glob("*.md"))
|
||||
Pipeline.run({"text_file_converter": {"sources": files}})
|
||||
```
|
||||
+98
@@ -0,0 +1,98 @@
|
||||
---
|
||||
title: "HierarchicalDocumentSplitter"
|
||||
id: hierarchicaldocumentsplitter
|
||||
slug: "/hierarchicaldocumentsplitter"
|
||||
description: "Use this component to create a multi-level document structure based on parent-children relationships between text segments."
|
||||
---
|
||||
|
||||
# HierarchicalDocumentSplitter
|
||||
|
||||
Use this component to create a multi-level document structure based on parent-children relationships between text segments.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx) and [`DocumentCleaner`](documentcleaner.mdx) |
|
||||
| **Mandatory init variables** | `block_sizes`: Set of block sizes to split the document into. The blocks are split in descending order. |
|
||||
| **Mandatory run variables** | `documents`: A list of documents to split into hierarchical blocks |
|
||||
| **Output variables** | `documents`: A list of hierarchical documents |
|
||||
| **API reference** | [PreProcessors](/reference/preprocessors-api) |
|
||||
| **GitHub link** | [https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/hierarchical_document_splitter.py](https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/hierarchical_document_splitter.py#L12) |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `HierarchicalDocumentSplitter` divides documents into blocks of different sizes, creating a tree-like structure.
|
||||
|
||||
A block is one of the chunks of text that the splitter produces. It is similar to cutting a long piece of text into smaller pieces: each piece is a block. Blocks form a tree structure where your full document is the root block, and as you split it into smaller and smaller pieces you get child-blocks and leaf-blocks, down to whatever smallest size specified.
|
||||
|
||||
The [`AutoMergingRetriever`](../retrievers/automergingretriever.mdx) component then leverages this hierarchical structure to improve document retrieval.
|
||||
|
||||
To initialize the component, you need to specify the `block_size`, which is the “maximum length” of each of the blocks, measured in the specific unit (see `split_by` parameter). Pass a set of sizes (for example, `{20, 5}`), and it will:
|
||||
|
||||
- First, split the document into blocks of up to 20 units each (the “parent” blocks).
|
||||
- Then, it will split each of those into blocks of up to 5 units each (the “child” blocks).
|
||||
|
||||
This descending order of sizes builds the hierarchy.
|
||||
|
||||
These additional parameters can be set when the component is initialized:
|
||||
|
||||
- `split_by` can be `"word"` (default), `"sentence"`, `"passage"`, `"page"`.
|
||||
- `split_overlap` is an integer indicating the number of overlapping words, sentences, or passages between chunks, 0 being the default.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.preprocessors import HierarchicalDocumentSplitter
|
||||
|
||||
doc = Document(content="This is a simple test document")
|
||||
splitter = HierarchicalDocumentSplitter(block_sizes={3, 2}, split_overlap=0, split_by="word")
|
||||
splitter.run([doc])
|
||||
|
||||
>> {'documents': [Document(id=3f7..., content: 'This is a simple test document', meta: {'block_size': 0, 'parent_id': None, 'children_ids': ['5ff..', '8dc..'], 'level': 0}),
|
||||
>> Document(id=5ff.., content: 'This is a ', meta: {'block_size': 3, 'parent_id': '3f7..', 'children_ids': ['f19..', '52c..'], 'level': 1, 'source_id': '3f7..', 'page_number': 1, 'split_id': 0, 'split_idx_start': 0}),
|
||||
>> Document(id=8dc.., content: 'simple test document', meta: {'block_size': 3, 'parent_id': '3f7..', 'children_ids': ['39d..', 'e23..'], 'level': 1, 'source_id': '3f7..', 'page_number': 1, 'split_id': 1, 'split_idx_start': 10}),
|
||||
>> Document(id=f19.., content: 'This is ', meta: {'block_size': 2, 'parent_id': '5ff..', 'children_ids': [], 'level': 2, 'source_id': '5ff..', 'page_number': 1, 'split_id': 0, 'split_idx_start': 0}),
|
||||
>> Document(id=52c.., content: 'a ', meta: {'block_size': 2, 'parent_id': '5ff..', 'children_ids': [], 'level': 2, 'source_id': '5ff..', 'page_number': 1, 'split_id': 1, 'split_idx_start': 8}),
|
||||
>> Document(id=39d.., content: 'simple test ', meta: {'block_size': 2, 'parent_id': '8dc..', 'children_ids': [], 'level': 2, 'source_id': '8dc..', 'page_number': 1, 'split_id': 0, 'split_idx_start': 0}),
|
||||
>> Document(id=e23.., content: 'document', meta: {'block_size': 2, 'parent_id': '8dc..', 'children_ids': [], 'level': 2, 'source_id': '8dc..', 'page_number': 1, 'split_id': 1, 'split_idx_start': 12})]}
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
This Haystack pipeline processes `.md` files by converting them to documents, cleaning the text, splitting it into sentence-based chunks, and storing the results in an In-Memory Document Store.
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters.txt import TextFileToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.preprocessors import HierarchicalDocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
Pipeline = Pipeline()
|
||||
Pipeline.add_component(instance=TextFileToDocument(), name="text_file_converter")
|
||||
Pipeline.add_component(instance=DocumentCleaner(), name="cleaner")
|
||||
Pipeline.add_component(instance=HierarchicalDocumentSplitter(
|
||||
block_sizes={10, 6, 3}, split_overlap=0, split_by="sentence", name="splitter"
|
||||
)
|
||||
Pipeline.add_component(instance=DocumentWriter(document_store=document_store), name="writer")
|
||||
Pipeline.connect("text_file_converter.documents", "cleaner.documents")
|
||||
Pipeline.connect("cleaner.documents", "splitter.documents")
|
||||
Pipeline.connect("splitter.documents", "writer.documents")
|
||||
|
||||
path = "path/to/your/files"
|
||||
files = list(Path(path).glob("*.md"))
|
||||
Pipeline.run({"text_file_converter": {"sources": files}})
|
||||
```
|
||||
+123
@@ -0,0 +1,123 @@
|
||||
---
|
||||
title: "MarkdownHeaderSplitter"
|
||||
id: markdownheadersplitter
|
||||
slug: "/markdownheadersplitter"
|
||||
description: "Split documents at ATX-style Markdown headers (#), with optional secondary splitting. Preserves header hierarchy as metadata."
|
||||
---
|
||||
|
||||
# MarkdownHeaderSplitter
|
||||
|
||||
Split documents at ATX-style Markdown headers (`#`, `##`, and so on), with optional secondary splitting. Header hierarchy is preserved as metadata on each chunk.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx) and [`DocumentCleaner`](documentcleaner.mdx) |
|
||||
| **Mandatory run variables** | `documents`: A list of text documents to split. |
|
||||
| **Output variables** | `documents`: A list of documents split at headers (and optionally by secondary split). |
|
||||
| **API reference** | [PreProcessors](/reference/preprocessors-api) |
|
||||
| **GitHub link** | [https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/markdown_header_splitter.py](https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/markdown_header_splitter.py) |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `MarkdownHeaderSplitter` processes text documents by:
|
||||
|
||||
- Splitting them into chunks at ATX-style Markdown headers (`#`, `##`, …, `######`), preserving header hierarchy as metadata.
|
||||
- Optionally applying a secondary split (by word, passage, period, or line) to each chunk using Haystack's [`DocumentSplitter`](documentsplitter.mdx).
|
||||
- Preserving and propagating metadata such as parent headers, page numbers, and split IDs.
|
||||
|
||||
Only ATX-style headers are recognized (e.g. `# Title`). Setext-style headers (`Underline with ===`) aren't supported.
|
||||
|
||||
Parameters you can set when initializing the component:
|
||||
|
||||
- `page_break_character`: Character used to identify page breaks. Defaults to form feed `\f`.
|
||||
- `keep_headers`: If `True`, headers remain in the chunk content. If `False`, headers are moved to metadata only. Defaults to `True`.
|
||||
- `secondary_split`: Optional secondary split after header splitting. Options: `None`, `"word"`, `"passage"`, `"period"`, `"line"`. Defaults to `None`.
|
||||
- `split_length`: Maximum number of units per split when using secondary splitting. Defaults to `200`.
|
||||
- `split_overlap`: Number of overlapping units between splits when using secondary splitting. Defaults to `0`.
|
||||
- `split_threshold`: Minimum number of units per split when using secondary splitting. Defaults to `0`.
|
||||
- `skip_empty_documents`: Whether to skip documents with empty content. Defaults to `True`.
|
||||
|
||||
Each output document's metadata includes:
|
||||
|
||||
- `source_id`: ID of the original document.
|
||||
- `page_number`: Page number. Updated when `page_break_character` is found.
|
||||
- `split_id`: Index of the chunk within its parent.
|
||||
- `header`: The header text for this chunk.
|
||||
- `parent_headers`: List of parent header texts in hierarchy order.
|
||||
|
||||
The component only works with text documents. Documents with `None` or non-string content raise a `ValueError`.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.preprocessors import MarkdownHeaderSplitter
|
||||
|
||||
text = (
|
||||
"# Introduction\n"
|
||||
"This is the intro section.\n"
|
||||
"## Getting Started\n"
|
||||
"Here is how to start.\n"
|
||||
"## Advanced\n"
|
||||
"Advanced content here."
|
||||
)
|
||||
doc = Document(content=text)
|
||||
splitter = MarkdownHeaderSplitter(keep_headers=True)
|
||||
result = splitter.run(documents=[doc])
|
||||
|
||||
# result["documents"] contains one document per header section,
|
||||
# with meta["header"], meta["parent_headers"], meta["source_id"], and so on
|
||||
```
|
||||
|
||||
### With secondary splitting
|
||||
|
||||
When sections are long, you can add a secondary split, for example by word, so each chunk stays within a maximum size:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.preprocessors import MarkdownHeaderSplitter
|
||||
|
||||
text = "# Section\n" + "Some long body text. " * 50
|
||||
doc = Document(content=text)
|
||||
splitter = MarkdownHeaderSplitter(
|
||||
keep_headers=True,
|
||||
secondary_split="word",
|
||||
split_length=20,
|
||||
split_overlap=2,
|
||||
)
|
||||
result = splitter.run(documents=[doc])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
This pipeline converts Markdown files to documents, cleans them, splits by headers, and writes to an in-memory document store:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters.txt import TextFileToDocument
|
||||
from haystack.components.preprocessors import MarkdownHeaderSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
p = Pipeline()
|
||||
p.add_component("text_file_converter", TextFileToDocument())
|
||||
p.add_component("splitter", MarkdownHeaderSplitter(keep_headers=True))
|
||||
p.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
p.connect("text_file_converter.documents", "splitter.documents")
|
||||
p.connect("splitter.documents", "writer.documents")
|
||||
|
||||
path = "path/to/your/files"
|
||||
files = list(Path(path).glob("*.md"))
|
||||
p.run({"text_file_converter": {"sources": files}})
|
||||
```
|
||||
+147
@@ -0,0 +1,147 @@
|
||||
---
|
||||
title: "PresidioDocumentCleaner"
|
||||
id: presidiodocumentcleaner
|
||||
slug: "/presidiodocumentcleaner"
|
||||
description: "Use `PresidioDocumentCleaner` to replace PII in Document text with entity type placeholders, powered by Microsoft Presidio."
|
||||
---
|
||||
|
||||
# PresidioDocumentCleaner
|
||||
|
||||
`PresidioDocumentCleaner` replaces personally identifiable information (PII) in the text content of Documents with entity type placeholders such as `<PERSON>` or `<EMAIL_ADDRESS>`. Original Documents are not mutated. Documents without text content pass through unchanged.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In an indexing pipeline, before writing Documents to a Document Store |
|
||||
| **Mandatory run variables** | `documents`: A list of Document objects |
|
||||
| **Output variables** | `documents`: A list of Document objects with PII replaced |
|
||||
| **API reference** | [Presidio](/reference/integrations-presidio) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/presidio |
|
||||
| **Package name** | `presidio-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
[Microsoft Presidio](https://microsoft.github.io/presidio/) is an open-source framework for PII detection and anonymization. `PresidioDocumentCleaner` uses Presidio's Analyzer and Anonymizer engines to scan document text and replace detected entities with type placeholders such as `<PERSON>` or `<EMAIL_ADDRESS>`.
|
||||
|
||||
This is useful when you want to store sanitized versions of your documents in a Document Store — for example, to prevent sensitive information from being indexed or returned in search results.
|
||||
|
||||
If you want to annotate PII without modifying the text, see [`PresidioEntityExtractor`](../extractors/presidioentityextractor.mdx). For sanitizing plain strings such as user queries, see [`PresidioTextCleaner`](./presidiotextcleaner.mdx).
|
||||
|
||||
## Configuration
|
||||
|
||||
| Parameter | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `language` | `"en"` | ISO 639-1 language code for PII detection. The appropriate spaCy model is selected automatically for [supported languages](#non-english-languages). See [Presidio supported languages](https://microsoft.github.io/presidio/analyzer/languages/). |
|
||||
| `entities` | `None` | List of PII entity types to detect and anonymize (e.g. `["PERSON", "EMAIL_ADDRESS"]`). If `None`, all supported types are detected. See [supported entities](https://microsoft.github.io/presidio/supported_entities/). |
|
||||
| `score_threshold` | `0.35` | Minimum confidence score (0–1) for a detected entity to be anonymized. |
|
||||
| `models` | `None` | Advanced override: explicit list of spaCy model configs, e.g. `[{"lang_code": "fr", "model_name": "fr_core_news_md"}]`. Use this only when you need a specific model variant or a language not in the built-in mapping. If `None`, the model is selected automatically based on `language`. |
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `presidio-haystack` package to use the `PresidioDocumentCleaner`.
|
||||
|
||||
```bash
|
||||
pip install presidio-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.preprocessors.presidio import (
|
||||
PresidioDocumentCleaner,
|
||||
)
|
||||
|
||||
cleaner = PresidioDocumentCleaner()
|
||||
result = cleaner.run(
|
||||
documents=[
|
||||
Document(content="Contact Alice Smith at alice@example.com or 212-555-1234."),
|
||||
],
|
||||
)
|
||||
print(result["documents"][0].content)
|
||||
# Contact <PERSON> at <EMAIL_ADDRESS> or <PHONE_NUMBER>.
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack_integrations.components.preprocessors.presidio import (
|
||||
PresidioDocumentCleaner,
|
||||
)
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
indexing_pipeline = Pipeline()
|
||||
indexing_pipeline.add_component("cleaner", PresidioDocumentCleaner())
|
||||
indexing_pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
indexing_pipeline.connect("cleaner", "writer")
|
||||
|
||||
indexing_pipeline.run(
|
||||
{
|
||||
"cleaner": {
|
||||
"documents": [
|
||||
Document(content="Alice Smith's email is alice@example.com"),
|
||||
Document(content="Call Bob at 212-555-9876"),
|
||||
],
|
||||
},
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
### Using Custom Parameters
|
||||
|
||||
Use `entities` to limit anonymization to the PII types you actually care about. This reduces false positives and improves performance by skipping recognizers you don't need.
|
||||
|
||||
Use `score_threshold` to tune the precision-recall tradeoff. The default `0.35` casts a wide net and may anonymize some false positives. Raise it (e.g. `0.7`) when you need high confidence before replacing text; lower it when missing any PII is the bigger risk.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.preprocessors.presidio import (
|
||||
PresidioDocumentCleaner,
|
||||
)
|
||||
|
||||
cleaner = PresidioDocumentCleaner(
|
||||
language="de",
|
||||
entities=["PERSON", "EMAIL_ADDRESS"], # only anonymize names and emails
|
||||
score_threshold=0.7, # higher precision, fewer false positives
|
||||
)
|
||||
```
|
||||
|
||||
### Non-English languages
|
||||
|
||||
For any language in the built-in mapping, just set `language` — the right spaCy model is selected and loaded automatically at warm-up time.
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.preprocessors.presidio import (
|
||||
PresidioDocumentCleaner,
|
||||
)
|
||||
|
||||
# No `models` parameter needed — de_core_news_lg is selected automatically
|
||||
cleaner = PresidioDocumentCleaner(language="de")
|
||||
result = cleaner.run(
|
||||
documents=[
|
||||
Document(
|
||||
content="Mein Name ist Hans Müller und meine E-Mail ist hans@example.com",
|
||||
),
|
||||
],
|
||||
)
|
||||
print(result["documents"][0].content)
|
||||
# Mein Name ist <PERSON> und meine E-Mail ist <EMAIL_ADDRESS>
|
||||
```
|
||||
|
||||
Supported languages and their default models are listed in `PresidioDocumentCleaner.SPACY_DEFAULT_MODELS`. Using a language not in that mapping without providing `models` raises a `ValueError` at warm-up time with a list of the supported language codes.
|
||||
|
||||
To use a non-default model variant, or a language outside the built-in mapping, pass `models` explicitly:
|
||||
|
||||
```python
|
||||
cleaner = PresidioDocumentCleaner(
|
||||
language="fr",
|
||||
models=[{"lang_code": "fr", "model_name": "fr_core_news_md"}],
|
||||
)
|
||||
```
|
||||
+125
@@ -0,0 +1,125 @@
|
||||
---
|
||||
title: "PresidioTextCleaner"
|
||||
id: presidiotextcleaner
|
||||
slug: "/presidiotextcleaner"
|
||||
description: "Use `PresidioTextCleaner` to replace PII in plain strings, powered by Microsoft Presidio."
|
||||
---
|
||||
|
||||
# PresidioTextCleaner
|
||||
|
||||
`PresidioTextCleaner` replaces personally identifiable information (PII) in plain strings. It takes a `list[str]` as input and returns a `list[str]`, making it easy to sanitize user queries before they are sent to an LLM.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In a query pipeline, before a Generator or Chat Generator |
|
||||
| **Mandatory run variables** | `texts`: A list of strings |
|
||||
| **Output variables** | `texts`: A list of strings with PII replaced |
|
||||
| **API reference** | [Presidio](/reference/integrations-presidio) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/presidio |
|
||||
| **Package name** | `presidio-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
[Microsoft Presidio](https://microsoft.github.io/presidio/) is an open-source framework for PII detection and anonymization. `PresidioTextCleaner` uses Presidio's Analyzer and Anonymizer engines to scan plain text strings and replace detected entities with type placeholders such as `<PERSON>` or `<US_SSN>`.
|
||||
|
||||
This is useful when you want to sanitize user queries before sending them to an LLM, ensuring that no personally identifiable information is passed to the model.
|
||||
|
||||
For sanitizing Haystack `Document` objects rather than plain strings, see [`PresidioDocumentCleaner`](./presidiodocumentcleaner.mdx).
|
||||
|
||||
## Configuration
|
||||
|
||||
| Parameter | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `language` | `"en"` | ISO 639-1 language code for PII detection. The appropriate spaCy model is selected automatically for [supported languages](#non-english-languages). See [Presidio supported languages](https://microsoft.github.io/presidio/analyzer/languages/). |
|
||||
| `entities` | `None` | List of PII entity types to detect and anonymize (e.g. `["PERSON", "EMAIL_ADDRESS"]`). If `None`, all supported types are detected. See [supported entities](https://microsoft.github.io/presidio/supported_entities/). |
|
||||
| `score_threshold` | `0.35` | Minimum confidence score (0–1) for a detected entity to be anonymized. |
|
||||
| `models` | `None` | Advanced override: explicit list of spaCy model configs, e.g. `[{"lang_code": "fr", "model_name": "fr_core_news_md"}]`. Use this only when you need a specific model variant or a language not in the built-in mapping. If `None`, the model is selected automatically based on `language`. |
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `presidio-haystack` package to use the `PresidioTextCleaner`.
|
||||
|
||||
```bash
|
||||
pip install presidio-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.preprocessors.presidio import PresidioTextCleaner
|
||||
|
||||
cleaner = PresidioTextCleaner()
|
||||
result = cleaner.run(texts=["My name is John Doe, my SSN is 123-45-6789"])
|
||||
print(result["texts"][0])
|
||||
# My name is <PERSON>, my SSN is <US_SSN>
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import OpenAIChatGenerator
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack_integrations.components.preprocessors.presidio import PresidioTextCleaner
|
||||
|
||||
template = [ChatMessage.from_user("Answer this question: {{query}}")]
|
||||
|
||||
query_pipeline = Pipeline()
|
||||
query_pipeline.add_component("cleaner", PresidioTextCleaner())
|
||||
query_pipeline.add_component("prompt_builder", ChatPromptBuilder(template=template))
|
||||
query_pipeline.add_component("llm", OpenAIChatGenerator(model="gpt-4o-mini"))
|
||||
query_pipeline.connect("cleaner.texts[0]", "prompt_builder.query")
|
||||
query_pipeline.connect("prompt_builder", "llm")
|
||||
|
||||
query_pipeline.run(
|
||||
{"cleaner": {"texts": ["My name is John Smith. What is the capital of France?"]}},
|
||||
)
|
||||
```
|
||||
|
||||
### Using Custom Parameters
|
||||
|
||||
Use `entities` to limit anonymization to the PII types you actually care about. This reduces false positives and improves performance by skipping recognizers you don't need.
|
||||
|
||||
Use `score_threshold` to tune the precision-recall tradeoff. The default `0.35` casts a wide net and may anonymize some false positives. Raise it (e.g. `0.7`) when you need high confidence before replacing text; lower it when missing any PII is the bigger risk.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.preprocessors.presidio import PresidioTextCleaner
|
||||
|
||||
cleaner = PresidioTextCleaner(
|
||||
language="de",
|
||||
entities=["PERSON", "EMAIL_ADDRESS"], # only anonymize names and emails
|
||||
score_threshold=0.7, # higher precision, fewer false positives
|
||||
)
|
||||
```
|
||||
|
||||
### Non-English languages
|
||||
|
||||
For any language in the built-in mapping, just set `language` — the right spaCy model is selected and loaded automatically at warm-up time.
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.preprocessors.presidio import PresidioTextCleaner
|
||||
|
||||
# No `models` parameter needed — de_core_news_lg is selected automatically
|
||||
cleaner = PresidioTextCleaner(language="de")
|
||||
result = cleaner.run(
|
||||
texts=["Hallo, ich bin Thomas Schmidt und meine E-Mail ist thomas@example.com"],
|
||||
)
|
||||
print(result["texts"][0])
|
||||
# Hallo, ich bin <PERSON> und meine E-Mail ist <EMAIL_ADDRESS>
|
||||
```
|
||||
|
||||
Supported languages and their default models are listed in `PresidioTextCleaner.SPACY_DEFAULT_MODELS`. Using a language not in that mapping without providing `models` raises a `ValueError` at warm-up time with a list of the supported language codes.
|
||||
|
||||
To use a non-default model variant, or a language outside the built-in mapping, pass `models` explicitly:
|
||||
|
||||
```python
|
||||
cleaner = PresidioTextCleaner(
|
||||
language="fr",
|
||||
models=[{"lang_code": "fr", "model_name": "fr_core_news_md"}],
|
||||
)
|
||||
```
|
||||
+179
@@ -0,0 +1,179 @@
|
||||
---
|
||||
title: "PythonCodeSplitter"
|
||||
id: pythoncodesplitter
|
||||
slug: "/pythoncodesplitter"
|
||||
description: "Split Python source documents into syntax-aware chunks using Python's AST, with metadata for line ranges, classes, decorators, and docstrings."
|
||||
---
|
||||
|
||||
# PythonCodeSplitter
|
||||
|
||||
`PythonCodeSplitter` splits Python source code documents into syntax-aware chunks. It is designed for Python files and keeps code units such as imports, functions, classes, and methods together where possible.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx), before [Embedders](../embedders.mdx) or [`DocumentWriter`](../writers/documentwriter.mdx) |
|
||||
| **Mandatory run variables** | `documents`: A list of Python source code documents |
|
||||
| **Output variables** | `documents`: A list of Python source code documents split into syntax-aware chunks |
|
||||
| **API reference** | [PreProcessors](/reference/preprocessors-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/python_code_splitter.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`PythonCodeSplitter` expects each input document's `content` to be valid Python source code. It parses the source with Python's `ast` module and creates ordered split units for:
|
||||
|
||||
- Module docstrings
|
||||
- Consecutive import blocks
|
||||
- Top-level functions
|
||||
- Class headers
|
||||
- Methods and nested classes
|
||||
- Remaining top-level statements
|
||||
|
||||
The splitter merges these units in source order toward `max_effective_lines`. Effective lines are calculated from character length with `ceil(len(source) / expected_chars_per_line)`, so long lines count as more than one line.
|
||||
|
||||
Functions and methods are kept whole by the primary AST split. If one syntactic unit is larger than `oversized_factor * max_effective_lines`, the splitter falls back to a line-based secondary split using [`DocumentSplitter`](documentsplitter.mdx). This oversized fallback is the only case where chunks can overlap; the primary AST split does not add overlap.
|
||||
|
||||
By default, `preserve_class_definition=True`. When a chunk contains class members without the original class header, the splitter prefixes the bare class signature so the chunk still carries the class context.
|
||||
|
||||
If `strip_docstrings=True`, function, method, and class docstrings are removed from chunk content and stored in `meta["docstrings"]`. Module docstrings stay in the chunk content because they are their own top-level unit.
|
||||
|
||||
### Per-chunk metadata
|
||||
|
||||
Each output document carries the metadata below. All fields from the parent document's `meta` (except `split_id`) are also propagated.
|
||||
|
||||
| Field | Description |
|
||||
| --- | --- |
|
||||
| `source_id` | ID of the originating document |
|
||||
| `split_id` | Sequential index of this chunk within its source document |
|
||||
| `start_line` | First line of the chunk in the original source (1-indexed). Oversized secondary chunks keep the originating unit's range. |
|
||||
| `end_line` | Last line of the chunk in the original source (1-indexed). Oversized secondary chunks keep the originating unit's range. |
|
||||
| `unit_kinds` | List of syntactic unit kinds included in this chunk, such as `imports`, `function`, `class_header`, or `method` |
|
||||
| `include_classes` | *(when applicable)* Ordered list of class names whose members appear in this chunk |
|
||||
| `decorators` | *(when applicable)* Ordered list of decorator strings found on included functions, methods, or classes |
|
||||
| `docstrings` | *(when `strip_docstrings=True`)* List of stripped docstring strings in source order |
|
||||
| `secondary_split` | `True` if this chunk was produced by the oversized fallback splitter |
|
||||
| `secondary_split_index` | Index of this piece within the secondary split sequence |
|
||||
| `secondary_split_total` | Total number of pieces produced by the secondary split |
|
||||
|
||||
Documents with `None` content raise `ValueError`, documents with non-string content raise `TypeError`, and invalid Python source raises `SyntaxError`. Empty documents are skipped.
|
||||
|
||||
## Configuration
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
| --- | --- | --- | --- |
|
||||
| `min_effective_lines` | `int` | `20` | Minimum effective lines per chunk. While a chunk is below this value, the splitter keeps merging in the next unit. |
|
||||
| `max_effective_lines` | `int` | `100` | Target effective lines per chunk. Units are merged greedily toward this value. |
|
||||
| `expected_chars_per_line` | `int` | `45` | Character count used to estimate effective lines via `ceil(len(source) / expected_chars_per_line)`. |
|
||||
| `oversized_factor` | `int` | `3` | Multiplier that triggers secondary line-based splitting for oversized syntactic units. |
|
||||
| `strip_docstrings` | `bool` | `False` | Moves function, method, and class docstrings from content into `meta["docstrings"]`. |
|
||||
| `preserve_class_definition` | `bool` | `True` | Prefixes class signatures on chunks that contain class members without the class header. |
|
||||
| `secondary_split_overlap` | `int` | `5` | Line overlap used only by the oversized secondary split. |
|
||||
| `secondary_split_length` | `int \| None` | `None` | Line length for the oversized secondary split. Defaults to `max_effective_lines` when `None`. |
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
import textwrap
|
||||
|
||||
from haystack import Document
|
||||
from haystack.components.preprocessors import PythonCodeSplitter
|
||||
|
||||
source = textwrap.dedent(
|
||||
'''
|
||||
"""Math utilities."""
|
||||
from math import pi
|
||||
|
||||
|
||||
class Circle:
|
||||
"""A circle."""
|
||||
|
||||
def __init__(self, radius: float) -> None:
|
||||
self.radius = radius
|
||||
|
||||
def area(self) -> float:
|
||||
return pi * self.radius * self.radius
|
||||
'''
|
||||
).lstrip()
|
||||
|
||||
splitter = PythonCodeSplitter(
|
||||
min_effective_lines=4,
|
||||
max_effective_lines=12,
|
||||
strip_docstrings=True,
|
||||
)
|
||||
|
||||
result = splitter.run(
|
||||
documents=[Document(content=source, meta={"file_name": "geometry.py"})],
|
||||
)
|
||||
|
||||
for chunk in result["documents"]:
|
||||
print(chunk.meta["start_line"], chunk.meta["end_line"], chunk.meta.get("include_classes"))
|
||||
```
|
||||
|
||||
### With docstring stripping for RAG
|
||||
|
||||
Set `strip_docstrings=True` when docstrings are verbose. The docstring text is moved out of the chunk content into `meta["docstrings"]`, keeping the stored chunk compact. Pass `meta_fields_to_embed=["docstrings"]` to your embedder so the docstring text still influences retrieval even though it is no longer in the chunk content.
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.preprocessors import PythonCodeSplitter
|
||||
|
||||
source = '''
|
||||
"""Example module."""
|
||||
from math import pi
|
||||
|
||||
|
||||
class Circle:
|
||||
"""A circle defined by its radius."""
|
||||
|
||||
def __init__(self, r: float) -> None:
|
||||
"""Store the radius."""
|
||||
self.r = r
|
||||
|
||||
def area(self) -> float:
|
||||
"""Return the area of the circle."""
|
||||
return pi * self.r * self.r
|
||||
'''
|
||||
|
||||
splitter = PythonCodeSplitter(
|
||||
min_effective_lines=20,
|
||||
max_effective_lines=100,
|
||||
strip_docstrings=True,
|
||||
)
|
||||
result = splitter.run(documents=[Document(content=source, meta={"file_name": "my_module.py"})])
|
||||
for chunk in result["documents"]:
|
||||
print(chunk.content)
|
||||
print(chunk.meta.get("docstrings"))
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
This pipeline converts Python files to documents, splits them with `PythonCodeSplitter`, and writes the chunks to an in-memory document store.
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from haystack import Pipeline
|
||||
from haystack.components.converters.txt import TextFileToDocument
|
||||
from haystack.components.preprocessors import PythonCodeSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
p = Pipeline()
|
||||
p.add_component("converter", TextFileToDocument())
|
||||
p.add_component("splitter", PythonCodeSplitter(max_effective_lines=80))
|
||||
p.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
|
||||
p.connect("converter.documents", "splitter.documents")
|
||||
p.connect("splitter.documents", "writer.documents")
|
||||
|
||||
files = list(Path("path/to/your/project").glob("**/*.py"))
|
||||
p.run({"converter": {"sources": files}})
|
||||
```
|
||||
+99
@@ -0,0 +1,99 @@
|
||||
---
|
||||
title: "RecursiveDocumentSplitter"
|
||||
id: recursivesplitter
|
||||
slug: "/recursivesplitter"
|
||||
description: "This component recursively breaks down text into smaller chunks by applying a given list of separators to the text."
|
||||
---
|
||||
|
||||
# RecursiveDocumentSplitter
|
||||
|
||||
This component recursively breaks down text into smaller chunks by applying a given list of separators to the text.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| Most common position in a pipeline | In indexing pipelines after [Converters](../converters.mdx) and [`DocumentCleaner`](documentcleaner.mdx) , before [Classifiers](../classifiers.mdx) |
|
||||
| Mandatory run variables | `documents`: A list of documents |
|
||||
| Output variables | `documents`: A list of documents |
|
||||
| API reference | [PreProcessors](/reference/preprocessors-api) |
|
||||
| Github link | https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/recursive_splitter.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `RecursiveDocumentSplitter` expects a list of documents as input and returns a list of documents with split texts. You can set the following parameters when initializing the component:
|
||||
|
||||
- `split_length`: The maximum length of each chunk, in words, by default. See the `split_units` parameter to change the the unit.
|
||||
- `split_overlap`: The number of characters or words that overlap between consecutive chunks.
|
||||
- `split_unit`: The unit of the `split_length` parameter. Can be either `"word"`, `"char"`, or `"token"`.
|
||||
- `separators`: An optional list of separator strings to use for splitting the text. If you don’t provide any separators, the default ones are `["\n\n", "sentence", "\n", " "]`. The string separators will be treated as regular expressions. If the separator is `"sentence"`, the text will be split into sentences using a custom sentence tokenizer based on NLTK. See [SentenceSplitter](https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/sentence_tokenizer.py#L116) code for more information.
|
||||
- `sentence_splitter_params`: Optional parameters to pass to the [SentenceSplitter](https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/sentence_tokenizer.py#L116).
|
||||
|
||||
The separators are applied in the same order as they are defined in the list. The first separator is used on the text; any resulting chunk that is within the specified `chunk_size` is retained. For chunks that exceed the defined `chunk_size`, the next separator in the list is applied. If all separators are used and the chunk still exceeds the `chunk_size`, a hard split occurs based on the `chunk_size`, taking into account whether words or characters are used as counting units. This process is repeated until all chunks are within the limits of the specified `chunk_size`.
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.preprocessors import RecursiveDocumentSplitter
|
||||
|
||||
chunker = RecursiveDocumentSplitter(split_length=260, split_overlap=0, separators=["\n\n", "\n", ".", " "])
|
||||
text = ('''Artificial intelligence (AI) - Introduction
|
||||
|
||||
AI, in its broadest sense, is intelligence exhibited by machines, particularly computer systems.
|
||||
AI technology is widely used throughout industry, government, and science. Some high-profile applications include advanced web search engines; recommendation systems; interacting via human speech; autonomous vehicles; generative and creative tools; and superhuman play and analysis in strategy games.''')
|
||||
doc = Document(content=text)
|
||||
doc_chunks = chunker.run([doc])
|
||||
print(doc_chunks["documents"])
|
||||
>[
|
||||
>Document(id=..., content: 'Artificial intelligence (AI) - Introduction\n\n', meta: {'original_id': '...', 'split_id': 0, 'split_idx_start': 0, '_split_overlap': []})
|
||||
>Document(id=..., content: 'AI, in its broadest sense, is intelligence exhibited by machines, particularly computer systems.\n', meta: {'original_id': '...', 'split_id': 1, 'split_idx_start': 45, '_split_overlap': []})
|
||||
>Document(id=..., content: 'AI technology is widely used throughout industry, government, and science.', meta: {'original_id': '...', 'split_id': 2, 'split_idx_start': 142, '_split_overlap': []})
|
||||
>Document(id=..., content: ' Some high-profile applications include advanced web search engines; recommendation systems; interac...', meta: {'original_id': '...', 'split_id': 3, 'split_idx_start': 216, '_split_overlap': []})
|
||||
>]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Here's how you can use `RecursiveSplitter` in an indexing pipeline:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters.txt import TextFileToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.preprocessors import RecursiveDocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
p = Pipeline()
|
||||
p.add_component(instance=TextFileToDocument(), name="text_file_converter")
|
||||
p.add_component(instance=DocumentCleaner(), name="cleaner")
|
||||
p.add_component(
|
||||
instance=RecursiveDocumentSplitter(
|
||||
split_length=400,
|
||||
split_overlap=0,
|
||||
split_unit="char",
|
||||
separators=["\n\n", "\n", "sentence", " "],
|
||||
sentence_splitter_params={
|
||||
"language": "en",
|
||||
"use_split_rules": True,
|
||||
"keep_white_spaces": False,
|
||||
},
|
||||
),
|
||||
name="recursive_splitter",
|
||||
)
|
||||
p.add_component(instance=DocumentWriter(document_store=document_store), name="writer")
|
||||
p.connect("text_file_converter.documents", "cleaner.documents")
|
||||
p.connect("cleaner.documents", "splitter.documents")
|
||||
p.connect("splitter.documents", "writer.documents")
|
||||
|
||||
path = "path/to/your/files"
|
||||
files = list(Path(path).glob("*.md"))
|
||||
p.run({"text_file_converter": {"sources": files}})
|
||||
```
|
||||
+120
@@ -0,0 +1,120 @@
|
||||
---
|
||||
title: "TextCleaner"
|
||||
id: textcleaner
|
||||
slug: "/textcleaner"
|
||||
description: "Use `TextCleaner` to make text data more readable. It removes regexes, punctuation, and numbers, as well as converts text to lowercase. This is especially useful to clean up text data before evaluation."
|
||||
---
|
||||
|
||||
# TextCleaner
|
||||
|
||||
Use `TextCleaner` to make text data more readable. It removes regexes, punctuation, and numbers, as well as converts text to lowercase. This is especially useful to clean up text data before evaluation.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Between a [Generator](../generators.mdx) and an [Evaluator](../evaluators.mdx) |
|
||||
| **Mandatory run variables** | `texts`: A list of strings to be cleaned |
|
||||
| **Output variables** | `texts`: A list of cleaned texts |
|
||||
| **API reference** | [PreProcessors](/reference/preprocessors-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/text_cleaner.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`TextCleaner` expects a list of strings as input and returns a list of strings with cleaned texts. Selectable cleaning steps are to `convert_to_lowercase`, `remove_punctuation`, and to `remove_numbers`. These three parameters are booleans that need to be set when the component is initialized.
|
||||
|
||||
- `convert_to_lowercase` converts all characters in texts to lowercase.
|
||||
- `remove_punctuation` removes all punctuation from the text.
|
||||
- `remove_numbers` removes all numerical digits from the text.
|
||||
|
||||
In addition, you can specify a regular expression with the parameter `remove_regexps`, and any matches will be removed.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
You can use it outside of a pipeline to clean up any texts:
|
||||
|
||||
```python
|
||||
from haystack.components.preprocessors import TextCleaner
|
||||
|
||||
text_to_clean = (
|
||||
"1Moonlight shimmered softly, 300 Wolves howled nearby, Night enveloped everything."
|
||||
)
|
||||
|
||||
cleaner = TextCleaner(
|
||||
convert_to_lowercase=True,
|
||||
remove_punctuation=False,
|
||||
remove_numbers=True,
|
||||
)
|
||||
result = cleaner.run(texts=[text_to_clean])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
In this example, we are using `TextCleaner` after an `ExtractiveReader` and an `OutputAdapter` to remove the punctuation in texts. Then, our custom-made `ExactMatchEvaluator` component compares the retrieved answer to the ground truth answer.
|
||||
|
||||
```python
|
||||
from typing import List
|
||||
from haystack import component, Document, Pipeline
|
||||
from haystack.components.converters import OutputAdapter
|
||||
from haystack.components.preprocessors import TextCleaner
|
||||
from haystack.components.readers import ExtractiveReader
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
document_store.write_documents(documents=documents)
|
||||
|
||||
|
||||
@component
|
||||
class ExactMatchEvaluator:
|
||||
@component.output_types(score=int)
|
||||
def run(self, expected: str, provided: List[str]):
|
||||
return {"score": int(expected in provided)}
|
||||
|
||||
|
||||
adapter = OutputAdapter(
|
||||
template="{{answers | extract_data}}",
|
||||
output_type=List[str],
|
||||
custom_filters={
|
||||
"extract_data": lambda data: [answer.data for answer in data if answer.data],
|
||||
},
|
||||
)
|
||||
|
||||
p = Pipeline()
|
||||
p.add_component("retriever", InMemoryBM25Retriever(document_store=document_store))
|
||||
p.add_component("reader", ExtractiveReader())
|
||||
p.add_component("adapter", adapter)
|
||||
p.add_component("cleaner", TextCleaner(remove_punctuation=True))
|
||||
p.add_component("evaluator", ExactMatchEvaluator())
|
||||
|
||||
p.connect("retriever", "reader")
|
||||
p.connect("reader", "adapter")
|
||||
p.connect("adapter", "cleaner.texts")
|
||||
p.connect("cleaner", "evaluator.provided")
|
||||
|
||||
question = "What behavior indicates a high level of self-awareness of elephants?"
|
||||
ground_truth_answer = "recognizing themselves in mirrors"
|
||||
|
||||
result = p.run(
|
||||
{
|
||||
"retriever": {"query": question},
|
||||
"reader": {"query": question},
|
||||
"evaluator": {"expected": ground_truth_answer},
|
||||
},
|
||||
)
|
||||
print(result)
|
||||
```
|
||||
Reference in New Issue
Block a user