chore: import upstream snapshot with attribution
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
+194
@@ -0,0 +1,194 @@
|
||||
---
|
||||
title: "ChineseDocumentSplitter"
|
||||
id: chinesedocumentsplitter
|
||||
slug: "/chinesedocumentsplitter"
|
||||
description: "`ChineseDocumentSplitter` divides Chinese text documents into smaller chunks using advanced Chinese language processing capabilities. It leverages HanLP for accurate Chinese word segmentation and sentence tokenization, making it ideal for processing Chinese text that requires linguistic awareness."
|
||||
---
|
||||
|
||||
# ChineseDocumentSplitter
|
||||
|
||||
`ChineseDocumentSplitter` divides Chinese text documents into smaller chunks using advanced Chinese language processing capabilities. It leverages HanLP for accurate Chinese word segmentation and sentence tokenization, making it ideal for processing Chinese text that requires linguistic awareness.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx) and [DocumentCleaner](documentcleaner.mdx), before [Classifiers](../classifiers.mdx) |
|
||||
| **Mandatory run variables** | `documents`: A list of documents with Chinese text content |
|
||||
| **Output variables** | `documents`: A list of documents, each containing a chunk of the original Chinese text |
|
||||
| **API reference** | [HanLP](/reference/integrations-hanlp) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/hanlp |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`ChineseDocumentSplitter` is a specialized document splitter designed specifically for Chinese text processing. Unlike English text where words are separated by spaces, Chinese text is written continuously without spaces between words.
|
||||
|
||||
This component leverages HanLP (Han Language Processing) to provide accurate Chinese word segmentation and sentence tokenization. It supports two granularity levels:
|
||||
|
||||
- **Coarse granularity**: Provides broader word segmentation suitable for most general use cases. Uses `COARSE_ELECTRA_SMALL_ZH` model for general-purpose segmentation.
|
||||
- **Fine granularity**: Offers more detailed word segmentation for specialized applications. Uses `FINE_ELECTRA_SMALL_ZH` model for detailed segmentation.
|
||||
|
||||
The splitter can divide documents by various units:
|
||||
|
||||
- `word`: Splits by Chinese words (multi-character tokens)
|
||||
- `sentence`: Splits by sentences using HanLP sentence tokenizer
|
||||
- `passage`: Splits by double line breaks ("\\n\\n")
|
||||
- `page`: Splits by form feed characters ("\\f")
|
||||
- `line`: Splits by single line breaks ("\\n")
|
||||
- `period`: Splits by periods (".")
|
||||
- `function`: Uses a custom splitting function
|
||||
|
||||
Each extracted chunk retains metadata from the original document and includes additional fields:
|
||||
|
||||
- `source_id`: The ID of the original document
|
||||
- `page_number`: The page number the chunk belongs to
|
||||
- `split_id`: The sequential ID of the split within the document
|
||||
- `split_idx_start`: The starting index of the chunk in the original document
|
||||
|
||||
When `respect_sentence_boundary=True` is set, the component uses HanLP's sentence tokenizer (`UD_CTB_EOS_MUL`) to ensure that splits occur only between complete sentences, preserving the semantic integrity of the text.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
You can use `ChineseDocumentSplitter` outside of a pipeline to process Chinese documents directly:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.preprocessors.hanlp import ChineseDocumentSplitter
|
||||
|
||||
## Initialize the splitter with word-based splitting
|
||||
splitter = ChineseDocumentSplitter(
|
||||
split_by="word",
|
||||
split_length=10,
|
||||
split_overlap=3,
|
||||
granularity="coarse",
|
||||
)
|
||||
|
||||
## Create a Chinese document
|
||||
doc = Document(
|
||||
content="这是第一句话，这是第二句话，这是第三句话。这是第四句话，这是第五句话，这是第六句话！",
|
||||
)
|
||||
|
||||
## Warm up the component (loads the necessary models)
|
||||
splitter.warm_up()
|
||||
|
||||
## Split the document
|
||||
result = splitter.run(documents=[doc])
|
||||
print(result["documents"]) # List of split documents
|
||||
```
|
||||
|
||||
### With sentence boundary respect
|
||||
|
||||
When splitting by words, you can ensure that sentence boundaries are respected:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.preprocessors.hanlp import ChineseDocumentSplitter
|
||||
|
||||
doc = Document(
|
||||
content="这是第一句话，这是第二句话，这是第三句话。"
|
||||
"这是第四句话，这是第五句话，这是第六句话！"
|
||||
"这是第七句话，这是第八句话，这是第九句话？",
|
||||
)
|
||||
|
||||
splitter = ChineseDocumentSplitter(
|
||||
split_by="word",
|
||||
split_length=10,
|
||||
split_overlap=3,
|
||||
respect_sentence_boundary=True,
|
||||
granularity="coarse",
|
||||
)
|
||||
splitter.warm_up()
|
||||
result = splitter.run(documents=[doc])
|
||||
|
||||
## Each chunk will end with a complete sentence
|
||||
for doc in result["documents"]:
|
||||
print(f"Chunk: {doc.content}")
|
||||
print(f"Ends with sentence: {doc.content.endswith(('。', '！', '？'))}")
|
||||
```
|
||||
|
||||
### With fine granularity
|
||||
|
||||
For more detailed word segmentation:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.preprocessors.hanlp import ChineseDocumentSplitter
|
||||
|
||||
doc = Document(content="人工智能技术正在快速发展，改变着我们的生活方式。")
|
||||
|
||||
splitter = ChineseDocumentSplitter(
|
||||
split_by="word",
|
||||
split_length=5,
|
||||
split_overlap=0,
|
||||
granularity="fine", # More detailed segmentation
|
||||
)
|
||||
splitter.warm_up()
|
||||
result = splitter.run(documents=[doc])
|
||||
print(result["documents"])
|
||||
```
|
||||
|
||||
### With custom splitting function
|
||||
|
||||
You can also use a custom function for splitting:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.preprocessors.hanlp import ChineseDocumentSplitter
|
||||
|
||||
|
||||
def custom_split(text: str) -> list[str]:
|
||||
"""Custom splitting function that splits by commas"""
|
||||
return text.split("，")
|
||||
|
||||
|
||||
doc = Document(content="第一段，第二段，第三段，第四段")
|
||||
|
||||
splitter = ChineseDocumentSplitter(split_by="function", splitting_function=custom_split)
|
||||
splitter.warm_up()
|
||||
result = splitter.run(documents=[doc])
|
||||
print(result["documents"])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Here's how you can integrate `ChineseDocumentSplitter` into a Haystack indexing pipeline:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline, Document
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters.txt import TextFileToDocument
|
||||
from haystack_integrations.components.preprocessors.hanlp import ChineseDocumentSplitter
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
## Initialize components
|
||||
document_store = InMemoryDocumentStore()
|
||||
p = Pipeline()
|
||||
p.add_component(instance=TextFileToDocument(), name="text_file_converter")
|
||||
p.add_component(instance=DocumentCleaner(), name="cleaner")
|
||||
p.add_component(
|
||||
instance=ChineseDocumentSplitter(
|
||||
split_by="word",
|
||||
split_length=100,
|
||||
split_overlap=20,
|
||||
respect_sentence_boundary=True,
|
||||
granularity="coarse",
|
||||
),
|
||||
name="chinese_splitter",
|
||||
)
|
||||
p.add_component(instance=DocumentWriter(document_store=document_store), name="writer")
|
||||
|
||||
## Connect components
|
||||
p.connect("text_file_converter.documents", "cleaner.documents")
|
||||
p.connect("cleaner.documents", "chinese_splitter.documents")
|
||||
p.connect("chinese_splitter.documents", "writer.documents")
|
||||
|
||||
## Run pipeline with Chinese text files
|
||||
p.run({"text_file_converter": {"sources": ["path/to/your/chinese/files.txt"]}})
|
||||
```
|
||||
|
||||
This pipeline processes Chinese text files by converting them to documents, cleaning the text, splitting them into linguistically-aware chunks using Chinese word segmentation, and storing the results in the Document Store for further retrieval and processing.
|
||||
+89
@@ -0,0 +1,89 @@
|
||||
---
|
||||
title: "CSVDocumentCleaner"
|
||||
id: csvdocumentcleaner
|
||||
slug: "/csvdocumentcleaner"
|
||||
description: "Use `CSVDocumentCleaner` to clean CSV documents by removing empty rows and columns while preserving specific ignored rows and columns. It processes CSV content stored in documents and helps standardize data for further analysis."
|
||||
---
|
||||
|
||||
# CSVDocumentCleaner
|
||||
|
||||
Use `CSVDocumentCleaner` to clean CSV documents by removing empty rows and columns while preserving specific ignored rows and columns. It processes CSV content stored in documents and helps standardize data for further analysis.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx) , before [Embedders](../embedders.mdx) or [Writers](../writers/documentwriter.mdx) |
|
||||
| **Mandatory run variables** | `documents`: A list of documents containing CSV content |
|
||||
| **Output variables** | `documents`: A list of cleaned CSV documents |
|
||||
| **API reference** | [PreProcessors](/reference/preprocessors-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/csv_document_cleaner.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`CSVDocumentCleaner` expects a list of `Document` objects as input, each containing CSV-formatted content as text. It cleans the data by removing fully empty rows and columns while allowing users to specify the number of rows and columns to be preserved before cleaning.
|
||||
|
||||
### Parameters
|
||||
|
||||
- `ignore_rows`: Number of rows to ignore from the top of the CSV table before processing. If any columns are removed, the same columns will be dropped from the ignored rows.
|
||||
- `ignore_columns`: Number of columns to ignore from the left of the CSV table before processing. If any rows are removed, the same rows will be dropped from the ignored columns.
|
||||
- `remove_empty_rows`: Whether to remove entirely empty rows.
|
||||
- `remove_empty_columns`: Whether to remove entirely empty columns.
|
||||
- `keep_id`: Whether to retain the original document ID in the output document.
|
||||
|
||||
### Cleaning Process
|
||||
|
||||
The `CSVDocumentCleaner` algorithm follows these steps:
|
||||
|
||||
1. Reads each document's content as a CSV table using pandas.
|
||||
2. Retains the specified number of `ignore_rows` from the top and `ignore_columns` from the left.
|
||||
3. Drops any rows and columns that are entirely empty (contain only NaN values).
|
||||
4. If columns are dropped, they are also removed from ignored rows.
|
||||
5. If rows are dropped, they are also removed from ignored columns.
|
||||
6. Reattaches the remaining ignored rows and columns to maintain their original positions.
|
||||
7. Returns the cleaned CSV content as a new `Document` object.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
You can use `CSVDocumentCleaner` independently to clean up CSV documents:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.preprocessors import CSVDocumentCleaner
|
||||
|
||||
cleaner = CSVDocumentCleaner(ignore_rows=1, ignore_columns=0)
|
||||
|
||||
documents = [Document(content="""col1,col2,col3\n,,\na,b,c\n,,""")]
|
||||
cleaned_docs = cleaner.run(documents=documents)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters import XLSXToDocument
|
||||
from haystack.components.preprocessors import CSVDocumentCleaner
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
p = Pipeline()
|
||||
p.add_component(instance=XLSXToDocument(), name="xlsx_file_converter")
|
||||
p.add_component(
|
||||
instance=CSVDocumentCleaner(ignore_rows=1, ignore_columns=1),
|
||||
name="csv_cleaner",
|
||||
)
|
||||
p.add_component(instance=DocumentWriter(document_store=document_store), name="writer")
|
||||
|
||||
p.connect("xlsx_file_converter.documents", "csv_cleaner.documents")
|
||||
p.connect("csv_cleaner.documents", "writer.documents")
|
||||
|
||||
p.run({"xlsx_file_converter": {"sources": [Path("your_xlsx_file.xlsx")]}})
|
||||
```
|
||||
|
||||
This ensures that CSV documents are properly cleaned before further processing or storage.
|
||||
+117
@@ -0,0 +1,117 @@
|
||||
---
|
||||
title: "CSVDocumentSplitter"
|
||||
id: csvdocumentsplitter
|
||||
slug: "/csvdocumentsplitter"
|
||||
description: "`CSVDocumentSplitter` divides CSV documents into smaller sub-tables based on split arguments. This is useful for handling structured data that contains multiple tables, improving data processing efficiency and retrieval."
|
||||
---
|
||||
|
||||
# CSVDocumentSplitter
|
||||
|
||||
`CSVDocumentSplitter` divides CSV documents into smaller sub-tables based on split arguments. This is useful for handling structured data that contains multiple tables, improving data processing efficiency and retrieval.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx) , before [CSVDocumentCleaner](csvdocumentcleaner.mdx) |
|
||||
| **Mandatory run variables** | `documents`: A list of documents with CSV-formatted content |
|
||||
| **Output variables** | `documents`: A list of documents, each containing a sub-table extracted from the original CSV file |
|
||||
| **API reference** | [PreProcessors](/reference/preprocessors-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/csv_document_splitter.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`CSVDocumentSplitter` expects a list of documents containing CSV-formatted content and returns a list of new `Document` objects, each representing a sub-table extracted from the original document.
|
||||
|
||||
There are two modes of operation for the splitter:
|
||||
|
||||
1. `threshold` (Default): Identifies empty rows or columns exceeding a given threshold and splits the document accordingly.
|
||||
2. `row-wise`: Splits each row into a separate document, treating each as an independent sub-table.
|
||||
|
||||
The splitting process follows these rules:
|
||||
|
||||
1. **Row-Based Splitting**: If `row_split_threshold` is set, consecutive empty rows equalling or exceeding this threshold trigger a split.
|
||||
2. **Column-Based Splitting**: If `column_split_threshold` is set, consecutive empty columns equalling or exceeding this threshold trigger a split.
|
||||
3. **Recursive Splitting**: If both thresholds are provided, `CSVDocumentSplitter` first splits by rows and then by columns. If more empty rows are detected, the splitting process is called again. This ensures that sub-tables are fully separated.
|
||||
|
||||
Each extracted sub-table retains metadata from the original document and includes additional fields:
|
||||
|
||||
- `source_id`: The ID of the original document
|
||||
- `row_idx_start`: The starting row index of the sub-table in the original document
|
||||
- `col_idx_start`: The starting column index of the sub-table in the original document
|
||||
- `split_id`: The sequential ID of the split within the document
|
||||
|
||||
This component is especially useful for document processing pipelines that require structured data to be extracted and stored efficiently.
|
||||
|
||||
### Supported Document Stores
|
||||
|
||||
`CSVDocumentSplitter` is compatible with the following Document Stores:
|
||||
|
||||
- [AstraDocumentStore](../../document-stores/astradocumentstore.mdx)
|
||||
- [ChromaDocumentStore](../../document-stores/chromadocumentstore.mdx)
|
||||
- [ElasticsearchDocumentStore](../../document-stores/elasticsearch-document-store.mdx)
|
||||
- [OpenSearchDocumentStore](../../document-stores/opensearch-document-store.mdx)
|
||||
- [PgvectorDocumentStore](../../document-stores/pgvectordocumentstore.mdx)
|
||||
- [PineconeDocumentStore](../../document-stores/pinecone-document-store.mdx)
|
||||
- [QdrantDocumentStore](../../document-stores/qdrant-document-store.mdx)
|
||||
- [WeaviateDocumentStore](../../document-stores/weaviatedocumentstore.mdx)
|
||||
- [MilvusDocumentStore](https://haystack.deepset.ai/integrations/milvus-document-store)
|
||||
- [Neo4jDocumentStore](https://haystack.deepset.ai/integrations/neo4j-document-store)
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
You can use `CSVDocumentSplitter` outside of a pipeline to process CSV documents directly:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.preprocessors import CSVDocumentSplitter
|
||||
|
||||
splitter = CSVDocumentSplitter(row_split_threshold=1, column_split_threshold=1)
|
||||
|
||||
doc = Document(
|
||||
content="""ID,LeftVal,,,RightVal,Extra
|
||||
1,Hello,,,World,Joined
|
||||
2,StillLeft,,,StillRight,Bridge
|
||||
,,,,,
|
||||
A,B,,,C,D
|
||||
E,F,,,G,H
|
||||
""",
|
||||
)
|
||||
split_result = splitter.run([doc])
|
||||
print(split_result["documents"]) # List of split tables as Documents
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Here's how you can integrate `CSVDocumentSplitter` into a Haystack indexing pipeline:
|
||||
|
||||
```python
|
||||
from haystack import Pipeline, Document
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters.csv import CSVToDocument
|
||||
from haystack.components.preprocessors import CSVDocumentSplitter
|
||||
from haystack.components.preprocessors import CSVDocumentCleaner
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
## Initialize components
|
||||
document_store = InMemoryDocumentStore()
|
||||
p = Pipeline()
|
||||
p.add_component(instance=CSVToDocument(), name="csv_file_converter")
|
||||
p.add_component(instance=CSVDocumentSplitter(), name="splitter")
|
||||
p.add_component(instance=CSVDocumentCleaner(), name="cleaner")
|
||||
p.add_component(instance=DocumentWriter(document_store=document_store), name="writer")
|
||||
|
||||
## Connect components
|
||||
p.connect("csv_file_converter.documents", "splitter.documents")
|
||||
p.connect("splitter.documents", "cleaner.documents")
|
||||
p.connect("cleaner.documents", "writer.documents")
|
||||
|
||||
## Run pipeline
|
||||
p.run({"csv_file_converter": {"sources": ["path/to/your/file.csv"]}})
|
||||
```
|
||||
|
||||
This pipeline extracts CSV content, splits it into structured sub-tables, cleans the CSV documents by removing empty rows and columns, and stores the resulting documents in the Document Store for further retrieval and processing.
|
||||
+89
@@ -0,0 +1,89 @@
|
||||
---
|
||||
title: "DocumentCleaner"
|
||||
id: documentcleaner
|
||||
slug: "/documentcleaner"
|
||||
description: "Use `DocumentCleaner` to make text documents more readable. It removes extra whitespaces, empty lines, specified substrings, regexes, page headers, and footers in this particular order. This is useful for preparing the documents for further processing by LLMs."
|
||||
---
|
||||
|
||||
# DocumentCleaner
|
||||
|
||||
Use `DocumentCleaner` to make text documents more readable. It removes extra whitespaces, empty lines, specified substrings, regexes, page headers, and footers in this particular order. This is useful for preparing the documents for further processing by LLMs.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx) , after [`DocumentSplitter`](documentsplitter.mdx) |
|
||||
| **Mandatory run variables** | `documents`: A list of documents |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [PreProcessors](/reference/preprocessors-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/document_cleaner.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`DocumentCleaner` expects a list of documents as input and returns a list of documents with cleaned texts. Selectable cleaning steps for each input document are to `remove_empty_lines`, `remove_extra_whitespaces` and to `remove_repeated_substrings`. These three parameters are booleans that can be set when the component is initialized.
|
||||
|
||||
- `unicode_normalization` normalizes Unicode characters to a standard form. The parameter can be set to NFC, NFKC, NFD, or NFKD.
|
||||
- `ascii_only` removes accents from characters and replaces them with their closest ASCII equivalents.
|
||||
- `remove_empty_lines` removes empty lines from the document.
|
||||
- `remove_extra_whitespaces` removes extra whitespaces from the document.
|
||||
- `remove_repeated_substrings` removes repeated substrings (headers/footers) from pages in the document. Pages in the text need to be separated by form feed character "\\f", which is supported by [`TextFileToDocument`](../converters/textfiletodocument.mdx) and [`AzureOCRDocumentConverter`](../converters/azureocrdocumentconverter.mdx).
|
||||
|
||||
In addition, you can specify a list of strings that should be removed from all documents as part of the cleaning with the parameter `remove_substring`. You can also specify a regular expression with the parameter `remove_regex` and any matches will be removed.
|
||||
|
||||
The cleaning steps are executed in the following order:
|
||||
|
||||
1. unicode_normalization
|
||||
2. ascii_only
|
||||
3. remove_extra_whitespaces
|
||||
4. remove_empty_lines
|
||||
5. remove_substrings
|
||||
6. remove_regex
|
||||
7. remove_repeated_substrings
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
You can use it outside of a pipeline to clean up your documents:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
|
||||
doc = Document(content="This is a document to clean\n\n\nsubstring to remove")
|
||||
|
||||
cleaner = DocumentCleaner(remove_substrings=["substring to remove"])
|
||||
result = cleaner.run(documents=[doc])
|
||||
|
||||
assert result["documents"][0].content == "This is a document to clean "
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters import TextFileToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
p = Pipeline()
|
||||
p.add_component(instance=TextFileToDocument(), name="text_file_converter")
|
||||
p.add_component(instance=DocumentCleaner(), name="cleaner")
|
||||
p.add_component(
|
||||
instance=DocumentSplitter(split_by="sentence", split_length=1),
|
||||
name="splitter",
|
||||
)
|
||||
p.add_component(instance=DocumentWriter(document_store=document_store), name="writer")
|
||||
p.connect("text_file_converter.documents", "cleaner.documents")
|
||||
p.connect("cleaner.documents", "splitter.documents")
|
||||
p.connect("splitter.documents", "writer.documents")
|
||||
|
||||
p.run({"text_file_converter": {"sources": your_files}})
|
||||
```
|
||||
+79
@@ -0,0 +1,79 @@
|
||||
---
|
||||
title: "DocumentPreprocessor"
|
||||
id: documentpreprocessor
|
||||
slug: "/documentpreprocessor"
|
||||
description: "Divides a list of text documents into a list of shorter text documents and then makes them more readable by cleaning."
|
||||
---
|
||||
|
||||
# DocumentPreprocessor
|
||||
|
||||
Divides a list of text documents into a list of shorter text documents and then makes them more readable by cleaning.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx) |
|
||||
| **Mandatory run variables** | `documents`: A list of documents |
|
||||
| **Output variables** | `documents`: A list of split and cleaned documents |
|
||||
| **API reference** | [PreProcessors](/reference/preprocessors-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/document_preprocessor.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`DocumentPreprocessor` first splits and then cleans documents.
|
||||
|
||||
It is a SuperComponent that combines a `DocumentSplitter` and a `DocumentCleaner` into a single component.
|
||||
|
||||
### Parameters
|
||||
|
||||
The `DocumentPreprocessor` exposes all initialization parameters of the underlying `DocumentSplitter` and `DocumentCleaner`, and they are all optional. A detailed description of their parameters is in the respective documentation pages:
|
||||
|
||||
- [DocumentSplitter](documentsplitter.mdx)
|
||||
- [DocumentCleaner](documentcleaner.mdx)
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.preprocessors import DocumentPreprocessor
|
||||
|
||||
doc = Document(content="I love pizza!")
|
||||
preprocessor = DocumentPreprocessor()
|
||||
|
||||
result = preprocessor.run(documents=[doc])
|
||||
print(result["documents"])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
You can use the `DocumentPreprocessor` in your indexing pipeline. The example below requires installing additional dependencies for the `MultiFileConverter`:
|
||||
|
||||
```shell
|
||||
pip install pypdf markdown-it-py mdit_plain trafilatura python-pptx python-docx jq openpyxl tabulate pandas
|
||||
```
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.converters import MultiFileConverter
|
||||
from haystack.components.preprocessors import DocumentPreprocessor
|
||||
from haystack.components.writers import DocumentWriter
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component("converter", MultiFileConverter())
|
||||
pipeline.add_component("preprocessor", DocumentPreprocessor())
|
||||
pipeline.add_component("writer", DocumentWriter(document_store=document_store))
|
||||
pipeline.connect("converter", "preprocessor")
|
||||
pipeline.connect("preprocessor", "writer")
|
||||
|
||||
result = pipeline.run(data={"sources": ["test.txt", "test.pdf"]})
|
||||
print(result)
|
||||
## {'writer': {'documents_written': 3}}
|
||||
```
|
||||
+97
@@ -0,0 +1,97 @@
|
||||
---
|
||||
title: "DocumentSplitter"
|
||||
id: documentsplitter
|
||||
slug: "/documentsplitter"
|
||||
description: "`DocumentSplitter` divides a list of text documents into a list of shorter text documents. This is useful for long texts that otherwise wouldn't fit into the maximum text length of language models and can also speed up question answering."
|
||||
---
|
||||
|
||||
# DocumentSplitter
|
||||
|
||||
`DocumentSplitter` divides a list of text documents into a list of shorter text documents. This is useful for long texts that otherwise wouldn't fit into the maximum text length of language models and can also speed up question answering.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx) and [`DocumentCleaner`](documentcleaner.mdx) , before [Classifiers](../classifiers.mdx) |
|
||||
| **Mandatory run variables** | `documents`: A list of documents |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [PreProcessors](/reference/preprocessors-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/document_splitter.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`DocumentSplitter` expects a list of documents as input and returns a list of documents with split texts. It splits each input document by `split_by` after `split_length` units with an overlap of `split_overlap` units. These additional parameters can be set when the component is initialized:
|
||||
|
||||
- `split_by` can be `"word"`, `"sentence"`, `"passage"` (paragraph), `"page"`, `"line"` or `"function"`.
|
||||
- `split_length` is an integer indicating the chunk size, which is the number of words, sentences, or passages.
|
||||
- `split_overlap` is an integer indicating the number of overlapping words, sentences, or passages between chunks.
|
||||
- `split_threshold` is an integer indicating the minimum number of words, sentences, or passages that the document fragment should have. If the fragment is below the threshold, it will be attached to the previous one.
|
||||
|
||||
A field `"source_id"` is added to each document's `meta` data to keep track of the original document that was split. Another meta field `"page_number"` is added to each document to keep track of the page it belonged to in the original document. Other metadata are copied from the original document.
|
||||
|
||||
The DocumentSplitter is compatible with the following DocumentStores:
|
||||
|
||||
- [AstraDocumentStore](../../document-stores/astradocumentstore.mdx)
|
||||
- [ChromaDocumentStore](../../document-stores/chromadocumentstore.mdx) – limited support, overlapping information is not stored.
|
||||
- [ElasticsearchDocumentStore](../../document-stores/elasticsearch-document-store.mdx)
|
||||
- [OpenSearchDocumentStore](../../document-stores/opensearch-document-store.mdx)
|
||||
- [PgvectorDocumentStore](../../document-stores/pgvectordocumentstore.mdx)
|
||||
- [PineconeDocumentStore](../../document-stores/pinecone-document-store.mdx) – limited support, overlapping information is not stored.
|
||||
- [QdrantDocumentStore](../../document-stores/qdrant-document-store.mdx)
|
||||
- [WeaviateDocumentStore](../../document-stores/weaviatedocumentstore.mdx)
|
||||
- [MilvusDocumentStore](https://haystack.deepset.ai/integrations/milvus-document-store)
|
||||
- [Neo4jDocumentStore](https://haystack.deepset.ai/integrations/neo4j-document-store)
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
You can use this component outside of a pipeline to shorten your documents like this:
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
|
||||
doc = Document(
|
||||
content="Moonlight shimmered softly, wolves howled nearby, night enveloped everything.",
|
||||
)
|
||||
|
||||
splitter = DocumentSplitter(split_by="word", split_length=3, split_overlap=0)
|
||||
result = splitter.run(documents=[doc])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Here's how you can use `DocumentSplitter` in an indexing pipeline:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters.txt import TextFileToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.preprocessors import DocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
p = Pipeline()
|
||||
p.add_component(instance=TextFileToDocument(), name="text_file_converter")
|
||||
p.add_component(instance=DocumentCleaner(), name="cleaner")
|
||||
p.add_component(
|
||||
instance=DocumentSplitter(split_by="sentence", split_length=1),
|
||||
name="splitter",
|
||||
)
|
||||
p.add_component(instance=DocumentWriter(document_store=document_store), name="writer")
|
||||
p.connect("text_file_converter.documents", "cleaner.documents")
|
||||
p.connect("cleaner.documents", "splitter.documents")
|
||||
p.connect("splitter.documents", "writer.documents")
|
||||
|
||||
path = "path/to/your/files"
|
||||
files = list(Path(path).glob("*.md"))
|
||||
p.run({"text_file_converter": {"sources": files}})
|
||||
```
|
||||
+98
@@ -0,0 +1,98 @@
|
||||
---
|
||||
title: "EmbeddingBasedDocumentSplitter"
|
||||
id: embeddingbaseddocumentsplitter
|
||||
slug: "/embeddingbaseddocumentsplitter"
|
||||
description: "Use this component to split documents based on embedding similarity using cosine distances between sequential sentence groups."
|
||||
---
|
||||
|
||||
# EmbeddingBasedDocumentSplitter
|
||||
|
||||
Use this component to split documents based on embedding similarity using cosine distances between sequential sentence groups.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx) and [`DocumentCleaner`](documentcleaner.mdx) |
|
||||
| **Mandatory run variables** | `documents`: A list of documents to split each into smaller documents based on embedding similarity. |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [PreProcessors](/reference/preprocessors-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/embedding_based_document_splitter.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
This component splits documents based on embedding similarity using cosine distances between sequential sentence groups.
|
||||
|
||||
It first splits text into sentences, optionally groups them, calculates embeddings for each group, and then uses cosine
|
||||
distance between sequential embeddings to determine split points. Any distance above the specified percentile is treated
|
||||
as a break point. The component also tracks page numbers based on form feed characters (`\f`) in the original document.
|
||||
|
||||
This component is inspired by [5 Levels of Text Splitting](https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/tutorials/LevelsOfTextSplitting/5_Levels_Of_Text_Splitting.ipynb) by Greg Kamradt.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
|
||||
from haystack import Document
|
||||
from haystack.components.embedders import SentenceTransformersDocumentEmbedder
|
||||
from haystack.components.preprocessors import EmbeddingBasedDocumentSplitter
|
||||
|
||||
# Create a document with content that has a clear topic shift
|
||||
doc = Document(
|
||||
content="This is a first sentence. This is a second sentence. This is a third sentence. "
|
||||
"Completely different topic. The same completely different topic.",
|
||||
)
|
||||
|
||||
# Initialize the embedder to calculate semantic similarities
|
||||
embedder = SentenceTransformersDocumentEmbedder()
|
||||
|
||||
# Configure the splitter with parameters that control splitting behavior
|
||||
splitter = EmbeddingBasedDocumentSplitter(
|
||||
document_embedder=embedder,
|
||||
sentences_per_group=2, # Group 2 sentences before calculating embeddings
|
||||
percentile=0.95, # Split when cosine distance exceeds 95th percentile
|
||||
min_length=50, # Merge splits shorter than 50 characters
|
||||
max_length=1000, # Further split chunks longer than 1000 characters
|
||||
)
|
||||
splitter.warm_up()
|
||||
result = splitter.run(documents=[doc])
|
||||
|
||||
# The result contains a list of Document objects, each representing a semantic chunk
|
||||
# Each split document includes metadata: source_id, split_id, and page_number
|
||||
print(f"Original document split into {len(result['documents'])} chunks")
|
||||
for i, split_doc in enumerate(result["documents"]):
|
||||
print(f"Chunk {i}: {split_doc.content[:50]}...")
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters.txt import TextFileToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.preprocessors import EmbeddingBasedDocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
Pipeline = Pipeline()
|
||||
Pipeline.add_component(instance=TextFileToDocument(), name="text_file_converter")
|
||||
Pipeline.add_component(instance=DocumentCleaner(), name="cleaner")
|
||||
Pipeline.add_component(instance=EmbeddingBasedDocumentSplitter(document_embedder=embedder, sentences_per_group=2, percentile=0.95, min_length=50,max_length=1000)
|
||||
Pipeline.add_component(instance=DocumentWriter(document_store=document_store), name="writer")
|
||||
Pipeline.connect("text_file_converter.documents", "cleaner.documents")
|
||||
Pipeline.connect("cleaner.documents", "splitter.documents")
|
||||
Pipeline.connect("splitter.documents", "writer.documents")
|
||||
|
||||
path = "path/to/your/files"
|
||||
files = list(Path(path).glob("*.md"))
|
||||
Pipeline.run({"text_file_converter": {"sources": files}})
|
||||
```
|
||||
+97
@@ -0,0 +1,97 @@
|
||||
---
|
||||
title: "HierarchicalDocumentSplitter"
|
||||
id: hierarchicaldocumentsplitter
|
||||
slug: "/hierarchicaldocumentsplitter"
|
||||
description: "Use this component to create a multi-level document structure based on parent-children relationships between text segments."
|
||||
---
|
||||
|
||||
# HierarchicalDocumentSplitter
|
||||
|
||||
Use this component to create a multi-level document structure based on parent-children relationships between text segments.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In indexing pipelines after [Converters](../converters.mdx) and [`DocumentCleaner`](documentcleaner.mdx) |
|
||||
| **Mandatory init variables** | `block_sizes`: Set of block sizes to split the document into. The blocks are split in descending order. |
|
||||
| **Mandatory run variables** | `documents`: A list of documents to split into hierarchical blocks |
|
||||
| **Output variables** | `documents`: A list of hierarchical documents |
|
||||
| **API reference** | [PreProcessors](/reference/preprocessors-api) |
|
||||
| **GitHub link** | [https://github.com/deepset-ai/haystack/blob/dae8c7babaf28d2ffab4f2a8dedecd63e2394fb4/haystack/components/preprocessors/hierarchical_document_splitter.py](https://github.com/deepset-ai/haystack/blob/dae8c7babaf28d2ffab4f2a8dedecd63e2394fb4/haystack/components/preprocessors/hierarchical_document_splitter.py#L12) |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `HierarchicalDocumentSplitter` divides documents into blocks of different sizes, creating a tree-like structure.
|
||||
|
||||
A block is one of the chunks of text that the splitter produces. It is similar to cutting a long piece of text into smaller pieces: each piece is a block. Blocks form a tree structure where your full document is the root block, and as you split it into smaller and smaller pieces you get child-blocks and leaf-blocks, down to whatever smallest size specified.
|
||||
|
||||
The [`AutoMergingRetriever`](../retrievers/automergingretriever.mdx) component then leverages this hierarchical structure to improve document retrieval.
|
||||
|
||||
To initialize the component, you need to specify the `block_size`, which is the “maximum length” of each of the blocks, measured in the specific unit (see `split_by` parameter). Pass a set of sizes (for example, `{20, 5}`), and it will:
|
||||
|
||||
- First, split the document into blocks of up to 20 units each (the “parent” blocks).
|
||||
- Then, it will split each of those into blocks of up to 5 units each (the “child” blocks).
|
||||
|
||||
This descending order of sizes builds the hierarchy.
|
||||
|
||||
These additional parameters can be set when the component is initialized:
|
||||
|
||||
- `split_by` can be `"word"` (default), `"sentence"`, `"passage"`, `"page"`.
|
||||
- `split_overlap` is an integer indicating the number of overlapping words, sentences, or passages between chunks, 0 being the default.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.preprocessors import HierarchicalDocumentSplitter
|
||||
|
||||
doc = Document(content="This is a simple test document")
|
||||
splitter = HierarchicalDocumentSplitter(block_sizes={3, 2}, split_overlap=0, split_by="word")
|
||||
splitter.run([doc])
|
||||
|
||||
>> {'documents': [Document(id=3f7..., content: 'This is a simple test document', meta: {'block_size': 0, 'parent_id': None, 'children_ids': ['5ff..', '8dc..'], 'level': 0}),
|
||||
>> Document(id=5ff.., content: 'This is a ', meta: {'block_size': 3, 'parent_id': '3f7..', 'children_ids': ['f19..', '52c..'], 'level': 1, 'source_id': '3f7..', 'page_number': 1, 'split_id': 0, 'split_idx_start': 0}),
|
||||
>> Document(id=8dc.., content: 'simple test document', meta: {'block_size': 3, 'parent_id': '3f7..', 'children_ids': ['39d..', 'e23..'], 'level': 1, 'source_id': '3f7..', 'page_number': 1, 'split_id': 1, 'split_idx_start': 10}),
|
||||
>> Document(id=f19.., content: 'This is ', meta: {'block_size': 2, 'parent_id': '5ff..', 'children_ids': [], 'level': 2, 'source_id': '5ff..', 'page_number': 1, 'split_id': 0, 'split_idx_start': 0}),
|
||||
>> Document(id=52c.., content: 'a ', meta: {'block_size': 2, 'parent_id': '5ff..', 'children_ids': [], 'level': 2, 'source_id': '5ff..', 'page_number': 1, 'split_id': 1, 'split_idx_start': 8}),
|
||||
>> Document(id=39d.., content: 'simple test ', meta: {'block_size': 2, 'parent_id': '8dc..', 'children_ids': [], 'level': 2, 'source_id': '8dc..', 'page_number': 1, 'split_id': 0, 'split_idx_start': 0}),
|
||||
>> Document(id=e23.., content: 'document', meta: {'block_size': 2, 'parent_id': '8dc..', 'children_ids': [], 'level': 2, 'source_id': '8dc..', 'page_number': 1, 'split_id': 1, 'split_idx_start': 12})]}
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
This Haystack pipeline processes `.md` files by converting them to documents, cleaning the text, splitting it into sentence-based chunks, and storing the results in an In-Memory Document Store.
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters.txt import TextFileToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.preprocessors import HierarchicalDocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
|
||||
Pipeline = Pipeline()
|
||||
Pipeline.add_component(instance=TextFileToDocument(), name="text_file_converter")
|
||||
Pipeline.add_component(instance=DocumentCleaner(), name="cleaner")
|
||||
Pipeline.add_component(instance=HierarchicalDocumentSplitter(
|
||||
block_sizes={10, 6, 3}, split_overlap=0, split_by="sentence", name="splitter"
|
||||
)
|
||||
Pipeline.add_component(instance=DocumentWriter(document_store=document_store), name="writer")
|
||||
Pipeline.connect("text_file_converter.documents", "cleaner.documents")
|
||||
Pipeline.connect("cleaner.documents", "splitter.documents")
|
||||
Pipeline.connect("splitter.documents", "writer.documents")
|
||||
|
||||
path = "path/to/your/files"
|
||||
files = list(Path(path).glob("*.md"))
|
||||
Pipeline.run({"text_file_converter": {"sources": files}})
|
||||
```
|
||||
+100
@@ -0,0 +1,100 @@
|
||||
---
|
||||
title: "RecursiveDocumentSplitter"
|
||||
id: recursivesplitter
|
||||
slug: "/recursivesplitter"
|
||||
description: "This component recursively breaks down text into smaller chunks by applying a given list of separators to the text."
|
||||
---
|
||||
|
||||
# RecursiveDocumentSplitter
|
||||
|
||||
This component recursively breaks down text into smaller chunks by applying a given list of separators to the text.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| Most common position in a pipeline | In indexing pipelines after [Converters](../converters.mdx) and [`DocumentCleaner`](documentcleaner.mdx) , before [Classifiers](../classifiers.mdx) |
|
||||
| Mandatory run variables | `documents`: A list of documents |
|
||||
| Output variables | `documents`: A list of documents |
|
||||
| API reference | [PreProcessors](/reference/preprocessors-api) |
|
||||
| Github link | https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/recursive_splitter.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
The `RecursiveDocumentSplitter` expects a list of documents as input and returns a list of documents with split texts. You can set the following parameters when initializing the component:
|
||||
|
||||
- `split_length`: The maximum length of each chunk, in words, by default. See the `split_units` parameter to change the the unit.
|
||||
- `split_overlap`: The number of characters or words that overlap between consecutive chunks.
|
||||
- `split_unit`: The unit of the `split_length` parameter. Can be either `"word"`, `"char"`, or `"token"`.
|
||||
- `separators`: An optional list of separator strings to use for splitting the text. If you don’t provide any separators, the default ones are `["\n\n", "sentence", "\n", " "]`. The string separators will be treated as regular expressions. If the separator is `"sentence"`, the text will be split into sentences using a custom sentence tokenizer based on NLTK. See [SentenceSplitter](https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/sentence_tokenizer.py#L116) code for more information.
|
||||
- `sentence_splitter_params`: Optional parameters to pass to the [SentenceSplitter](https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/sentence_tokenizer.py#L116).
|
||||
|
||||
The separators are applied in the same order as they are defined in the list. The first separator is used on the text; any resulting chunk that is within the specified `chunk_size` is retained. For chunks that exceed the defined `chunk_size`, the next separator in the list is applied. If all separators are used and the chunk still exceeds the `chunk_size`, a hard split occurs based on the `chunk_size`, taking into account whether words or characters are used as counting units. This process is repeated until all chunks are within the limits of the specified `chunk_size`.
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.preprocessors import RecursiveDocumentSplitter
|
||||
|
||||
chunker = RecursiveDocumentSplitter(split_length=260, split_overlap=0, separators=["\n\n", "\n", ".", " "])
|
||||
text = ('''Artificial intelligence (AI) - Introduction
|
||||
|
||||
AI, in its broadest sense, is intelligence exhibited by machines, particularly computer systems.
|
||||
AI technology is widely used throughout industry, government, and science. Some high-profile applications include advanced web search engines; recommendation systems; interacting via human speech; autonomous vehicles; generative and creative tools; and superhuman play and analysis in strategy games.''')
|
||||
chunker.warm_up()
|
||||
doc = Document(content=text)
|
||||
doc_chunks = chunker.run([doc])
|
||||
print(doc_chunks["documents"])
|
||||
>[
|
||||
>Document(id=..., content: 'Artificial intelligence (AI) - Introduction\n\n', meta: {'original_id': '...', 'split_id': 0, 'split_idx_start': 0, '_split_overlap': []})
|
||||
>Document(id=..., content: 'AI, in its broadest sense, is intelligence exhibited by machines, particularly computer systems.\n', meta: {'original_id': '...', 'split_id': 1, 'split_idx_start': 45, '_split_overlap': []})
|
||||
>Document(id=..., content: 'AI technology is widely used throughout industry, government, and science.', meta: {'original_id': '...', 'split_id': 2, 'split_idx_start': 142, '_split_overlap': []})
|
||||
>Document(id=..., content: ' Some high-profile applications include advanced web search engines; recommendation systems; interac...', meta: {'original_id': '...', 'split_id': 3, 'split_idx_start': 216, '_split_overlap': []})
|
||||
>]
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Here's how you can use `RecursiveSplitter` in an indexing pipeline:
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from haystack import Document
|
||||
from haystack import Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.converters.txt import TextFileToDocument
|
||||
from haystack.components.preprocessors import DocumentCleaner
|
||||
from haystack.components.preprocessors import RecursiveDocumentSplitter
|
||||
from haystack.components.writers import DocumentWriter
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
p = Pipeline()
|
||||
p.add_component(instance=TextFileToDocument(), name="text_file_converter")
|
||||
p.add_component(instance=DocumentCleaner(), name="cleaner")
|
||||
p.add_component(
|
||||
instance=RecursiveDocumentSplitter(
|
||||
split_length=400,
|
||||
split_overlap=0,
|
||||
split_unit="char",
|
||||
separators=["\n\n", "\n", "sentence", " "],
|
||||
sentence_splitter_params={
|
||||
"language": "en",
|
||||
"use_split_rules": True,
|
||||
"keep_white_spaces": False,
|
||||
},
|
||||
),
|
||||
name="recursive_splitter",
|
||||
)
|
||||
p.add_component(instance=DocumentWriter(document_store=document_store), name="writer")
|
||||
p.connect("text_file_converter.documents", "cleaner.documents")
|
||||
p.connect("cleaner.documents", "splitter.documents")
|
||||
p.connect("splitter.documents", "writer.documents")
|
||||
|
||||
path = "path/to/your/files"
|
||||
files = list(Path(path).glob("*.md"))
|
||||
p.run({"text_file_converter": {"sources": files}})
|
||||
```
|
||||
+119
@@ -0,0 +1,119 @@
|
||||
---
|
||||
title: "TextCleaner"
|
||||
id: textcleaner
|
||||
slug: "/textcleaner"
|
||||
description: "Use `TextCleaner` to make text data more readable. It removes regexes, punctuation, and numbers, as well as converts text to lowercase. This is especially useful to clean up text data before evaluation."
|
||||
---
|
||||
|
||||
# TextCleaner
|
||||
|
||||
Use `TextCleaner` to make text data more readable. It removes regexes, punctuation, and numbers, as well as converts text to lowercase. This is especially useful to clean up text data before evaluation.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | Between a [Generator](../generators.mdx) and an [Evaluator](../evaluators.mdx) |
|
||||
| **Mandatory run variables** | `texts`: A list of strings to be cleaned |
|
||||
| **Output variables** | `texts`: A list of cleaned texts |
|
||||
| **API reference** | [PreProcessors](/reference/preprocessors-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/preprocessors/text_cleaner.py |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`TextCleaner` expects a list of strings as input and returns a list of strings with cleaned texts. Selectable cleaning steps are to `convert_to_lowercase`, `remove_punctuation`, and to `remove_numbers`. These three parameters are booleans that need to be set when the component is initialized.
|
||||
|
||||
- `convert_to_lowercase` converts all characters in texts to lowercase.
|
||||
- `remove_punctuation` removes all punctuation from the text.
|
||||
- `remove_numbers` removes all numerical digits from the text.
|
||||
|
||||
In addition, you can specify a regular expression with the parameter `remove_regexps`, and any matches will be removed.
|
||||
|
||||
## Usage
|
||||
|
||||
### On its own
|
||||
|
||||
You can use it outside of a pipeline to clean up any texts:
|
||||
|
||||
```python
|
||||
from haystack.components.preprocessors import TextCleaner
|
||||
|
||||
text_to_clean = (
|
||||
"1Moonlight shimmered softly, 300 Wolves howled nearby, Night enveloped everything."
|
||||
)
|
||||
|
||||
cleaner = TextCleaner(
|
||||
convert_to_lowercase=True,
|
||||
remove_punctuation=False,
|
||||
remove_numbers=True,
|
||||
)
|
||||
result = cleaner.run(texts=[text_to_clean])
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
In this example, we are using `TextCleaner` after an `ExtractiveReader` and an `OutputAdapter` to remove the punctuation in texts. Then, our custom-made `ExactMatchEvaluator` component compares the retrieved answer to the ground truth answer.
|
||||
|
||||
```python
|
||||
from typing import List
|
||||
from haystack import component, Document, Pipeline
|
||||
from haystack.components.converters import OutputAdapter
|
||||
from haystack.components.preprocessors import TextCleaner
|
||||
from haystack.components.readers import ExtractiveReader
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
|
||||
document_store = InMemoryDocumentStore()
|
||||
documents = [
|
||||
Document(content="There are over 7,000 languages spoken around the world today."),
|
||||
Document(
|
||||
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
|
||||
),
|
||||
Document(
|
||||
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
|
||||
),
|
||||
]
|
||||
document_store.write_documents(documents=documents)
|
||||
|
||||
|
||||
@component
|
||||
class ExactMatchEvaluator:
|
||||
@component.output_types(score=int)
|
||||
def run(self, expected: str, provided: List[str]):
|
||||
return {"score": int(expected in provided)}
|
||||
|
||||
|
||||
adapter = OutputAdapter(
|
||||
template="{{answers | extract_data}}",
|
||||
output_type=List[str],
|
||||
custom_filters={
|
||||
"extract_data": lambda data: [answer.data for answer in data if answer.data],
|
||||
},
|
||||
)
|
||||
|
||||
p = Pipeline()
|
||||
p.add_component("retriever", InMemoryBM25Retriever(document_store=document_store))
|
||||
p.add_component("reader", ExtractiveReader())
|
||||
p.add_component("adapter", adapter)
|
||||
p.add_component("cleaner", TextCleaner(remove_punctuation=True))
|
||||
p.add_component("evaluator", ExactMatchEvaluator())
|
||||
|
||||
p.connect("retriever", "reader")
|
||||
p.connect("reader", "adapter")
|
||||
p.connect("adapter", "cleaner.texts")
|
||||
p.connect("cleaner", "evaluator.provided")
|
||||
|
||||
question = "What behavior indicates a high level of self-awareness of elephants?"
|
||||
ground_truth_answer = "recognizing themselves in mirrors"
|
||||
|
||||
result = p.run(
|
||||
{
|
||||
"retriever": {"query": question},
|
||||
"reader": {"query": question},
|
||||
"evaluator": {"expected": ground_truth_answer},
|
||||
},
|
||||
)
|
||||
print(result)
|
||||
```
|
||||
Reference in New Issue
Block a user