chore: import upstream snapshot with attribution

docs-website/reference_versioned_docs/version-2.30/integrations-api/chonkie.md

@@ -0,0 +1,394 @@
+---
+title: "Chonkie"
+id: integrations-chonkie
+description: "Chonkie integration for Haystack"
+slug: "/integrations-chonkie"
+---
+
+
+## haystack_integrations.components.preprocessors.chonkie.recursive_splitter
+
+### ChonkieRecursiveDocumentSplitter
+
+A Document Splitter that uses Chonkie's RecursiveChunker to split documents.
+
+### Usage example
+
+```python
+from haystack import Document
+from haystack_integrations.components.preprocessors.chonkie import ChonkieRecursiveDocumentSplitter
+
+chunker = ChonkieRecursiveDocumentSplitter(chunk_size=512)
+documents = [Document(content="Hello world. This is a test.")]
+result = chunker.run(documents=documents)
+print(result["documents"])
+```
+
+#### __init__
+
+```python
+__init__(
+    *,
+    tokenizer: str = "character",
+    chunk_size: int = 2048,
+    min_characters_per_chunk: int = 24,
+    rules: RecursiveRules | dict[str, Any] | None = None,
+    skip_empty_documents: bool = True,
+    page_break_character: str = "\x0c"
+) -> None
+```
+
+Initializes the ChonkieRecursiveDocumentSplitter.
+
+**Parameters:**
+
+- **tokenizer** (<code>str</code>) – The tokenizer to use for chunking. Defaults to "character".
+  Common options include "character", "gpt2", and "cl100k_base".
+  See the [Chonkie documentation](https://docs.chonkie.ai/) for more information on available tokenizers.
+- **chunk_size** (<code>int</code>) – The maximum number of tokens per chunk. The actual length depends on the chosen tokenizer.
+- **min_characters_per_chunk** (<code>int</code>) – The minimum number of characters per chunk.
+- **rules** (<code>RecursiveRules | dict\[str, Any\] | None</code>) – Custom rules for recursive chunking. If None, default rules are used.
+  See the [Chonkie documentation](https://docs.chonkie.ai/) for more information.
+- **skip_empty_documents** (<code>bool</code>) – Whether to skip empty documents.
+- **page_break_character** (<code>str</code>) – The character to use for page breaks.
+
+#### run
+
+```python
+run(documents: list[Document]) -> dict[str, list[Document]]
+```
+
+Splits a list of documents into smaller chunks.
+
+**Parameters:**
+
+- **documents** (<code>list\[Document\]</code>) – The list of documents to split.
+
+**Returns:**
+
+- <code>dict\[str, list\[Document\]\]</code> – A dictionary with the "documents" key containing the list of chunks.
+
+#### to_dict
+
+```python
+to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+**Returns:**
+
+- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
+
+#### from_dict
+
+```python
+from_dict(data: dict[str, Any]) -> ChonkieRecursiveDocumentSplitter
+```
+
+Deserializes the component from a dictionary.
+
+**Parameters:**
+
+- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
+
+**Returns:**
+
+- <code>ChonkieRecursiveDocumentSplitter</code> – Deserialized component.
+
+## haystack_integrations.components.preprocessors.chonkie.semantic_splitter
+
+### ChonkieSemanticDocumentSplitter
+
+A Document Splitter that uses Chonkie's SemanticChunker to split documents.
+
+### Usage example
+
+```python
+from haystack import Document
+from haystack_integrations.components.preprocessors.chonkie import ChonkieSemanticDocumentSplitter
+
+chunker = ChonkieSemanticDocumentSplitter(chunk_size=512)
+documents = [Document(content="Hello world. This is a test.")]
+result = chunker.run(documents=documents)
+print(result["documents"])
+```
+
+#### __init__
+
+```python
+__init__(
+    *,
+    embedding_model: Any = "minishlab/potion-base-32M",
+    threshold: float = 0.8,
+    chunk_size: int = 2048,
+    similarity_window: int = 3,
+    min_sentences_per_chunk: int = 1,
+    min_characters_per_sentence: int = 24,
+    delim: Any = None,
+    include_delim: str = "prev",
+    skip_window: int = 0,
+    filter_window: int = 5,
+    filter_polyorder: int = 3,
+    filter_tolerance: float = 0.2,
+    skip_empty_documents: bool = True,
+    page_break_character: str = "\x0c"
+) -> None
+```
+
+Initializes the ChonkieSemanticDocumentSplitter.
+
+**Parameters:**
+
+- **embedding_model** (<code>Any</code>) – The embedding model to use for semantic similarity.
+  See the [Chonkie documentation](https://docs.chonkie.ai/) for more information on supported models.
+- **threshold** (<code>float</code>) – The semantic similarity threshold.
+- **chunk_size** (<code>int</code>) – The maximum number of tokens per chunk. The actual length depends on the
+  embedding model's tokenizer.
+- **similarity_window** (<code>int</code>) – The window size for similarity calculations.
+- **min_sentences_per_chunk** (<code>int</code>) – The minimum number of sentences per chunk.
+- **min_characters_per_sentence** (<code>int</code>) – The minimum number of characters per sentence.
+- **delim** (<code>Any</code>) – Delimiters to use for splitting. If None, default delimiters are used.
+- **include_delim** (<code>str</code>) – Whether to include the delimiter in the chunks.
+- **skip_window** (<code>int</code>) – The skip window for similarity calculations.
+- **filter_window** (<code>int</code>) – The filter window for similarity calculations.
+- **filter_polyorder** (<code>int</code>) – The polynomial order for similarity filtering.
+- **filter_tolerance** (<code>float</code>) – The tolerance for similarity filtering.
+- **skip_empty_documents** (<code>bool</code>) – Whether to skip empty documents.
+- **page_break_character** (<code>str</code>) – The character to use for page breaks.
+
+#### warm_up
+
+```python
+warm_up() -> None
+```
+
+Initializes the component by loading the embedding model.
+
+#### run
+
+```python
+run(documents: list[Document]) -> dict[str, list[Document]]
+```
+
+Splits a list of documents into smaller semantic chunks.
+
+**Parameters:**
+
+- **documents** (<code>list\[Document\]</code>) – The list of documents to split.
+
+**Returns:**
+
+- <code>dict\[str, list\[Document\]\]</code> – A dictionary with the "documents" key containing the list of chunks.
+
+#### to_dict
+
+```python
+to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+**Returns:**
+
+- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
+
+#### from_dict
+
+```python
+from_dict(data: dict[str, Any]) -> ChonkieSemanticDocumentSplitter
+```
+
+Deserializes the component from a dictionary.
+
+**Parameters:**
+
+- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
+
+**Returns:**
+
+- <code>ChonkieSemanticDocumentSplitter</code> – Deserialized component.
+
+## haystack_integrations.components.preprocessors.chonkie.sentence_splitter
+
+### ChonkieSentenceDocumentSplitter
+
+A Document Splitter that uses Chonkie's SentenceChunker to split documents.
+
+### Usage example
+
+```python
+from haystack import Document
+from haystack_integrations.components.preprocessors.chonkie import ChonkieSentenceDocumentSplitter
+
+chunker = ChonkieSentenceDocumentSplitter(chunk_size=512)
+documents = [Document(content="Hello world. This is a test.")]
+result = chunker.run(documents=documents)
+print(result["documents"])
+```
+
+#### __init__
+
+```python
+__init__(
+    *,
+    tokenizer: str = "character",
+    chunk_size: int = 2048,
+    chunk_overlap: int = 0,
+    min_sentences_per_chunk: int = 1,
+    min_characters_per_sentence: int = 12,
+    approximate: bool = False,
+    delim: Any = None,
+    include_delim: str = "prev",
+    skip_empty_documents: bool = True,
+    page_break_character: str = "\x0c"
+) -> None
+```
+
+Initializes the ChonkieSentenceDocumentSplitter.
+
+**Parameters:**
+
+- **tokenizer** (<code>str</code>) – The tokenizer to use for chunking. Defaults to "character".
+  Common options include "character", "gpt2", and "cl100k_base".
+  See the [Chonkie documentation](https://docs.chonkie.ai/) for more information on available tokenizers.
+- **chunk_size** (<code>int</code>) – The maximum number of tokens per chunk. The actual length depends on the chosen tokenizer.
+- **chunk_overlap** (<code>int</code>) – The overlap between consecutive chunks.
+- **min_sentences_per_chunk** (<code>int</code>) – The minimum number of sentences per chunk.
+- **min_characters_per_sentence** (<code>int</code>) – The minimum number of characters per sentence.
+- **approximate** (<code>bool</code>) – Whether to use approximate chunking.
+- **delim** (<code>Any</code>) – Delimiters to use for splitting. If None, default delimiters are used.
+- **include_delim** (<code>str</code>) – Whether to include the delimiter in the chunks ("prev" or "next").
+- **skip_empty_documents** (<code>bool</code>) – Whether to skip empty documents.
+- **page_break_character** (<code>str</code>) – The character to use for page breaks.
+
+#### run
+
+```python
+run(documents: list[Document]) -> dict[str, list[Document]]
+```
+
+Splits a list of documents into smaller sentence-based chunks.
+
+**Parameters:**
+
+- **documents** (<code>list\[Document\]</code>) – The list of documents to split.
+
+**Returns:**
+
+- <code>dict\[str, list\[Document\]\]</code> – A dictionary with the "documents" key containing the list of chunks.
+
+#### to_dict
+
+```python
+to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+**Returns:**
+
+- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
+
+#### from_dict
+
+```python
+from_dict(data: dict[str, Any]) -> ChonkieSentenceDocumentSplitter
+```
+
+Deserializes the component from a dictionary.
+
+**Parameters:**
+
+- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
+
+**Returns:**
+
+- <code>ChonkieSentenceDocumentSplitter</code> – Deserialized component.
+
+## haystack_integrations.components.preprocessors.chonkie.token_splitter
+
+### ChonkieTokenDocumentSplitter
+
+A Document Splitter that uses Chonkie's TokenChunker to split documents.
+
+### Usage example
+
+```python
+from haystack import Document
+from haystack_integrations.components.preprocessors.chonkie import ChonkieTokenDocumentSplitter
+
+chunker = ChonkieTokenDocumentSplitter(chunk_size=512, chunk_overlap=50)
+documents = [Document(content="Hello world. This is a test.")]
+result = chunker.run(documents=documents)
+print(result["documents"])
+```
+
+#### __init__
+
+```python
+__init__(
+    *,
+    tokenizer: str = "character",
+    chunk_size: int = 2048,
+    chunk_overlap: int = 0,
+    skip_empty_documents: bool = True,
+    page_break_character: str = "\x0c"
+) -> None
+```
+
+Initializes the ChonkieTokenDocumentSplitter.
+
+**Parameters:**
+
+- **tokenizer** (<code>str</code>) – The tokenizer to use for chunking. Defaults to "character".
+  Common options include "character", "gpt2", and "cl100k_base".
+  See the [Chonkie documentation](https://docs.chonkie.ai/) for more information on available tokenizers.
+- **chunk_size** (<code>int</code>) – The maximum number of tokens per chunk. The actual length depends on the chosen tokenizer.
+- **chunk_overlap** (<code>int</code>) – The overlap between consecutive chunks.
+- **skip_empty_documents** (<code>bool</code>) – Whether to skip empty documents.
+- **page_break_character** (<code>str</code>) – The character to use for page breaks.
+
+#### run
+
+```python
+run(documents: list[Document]) -> dict[str, list[Document]]
+```
+
+Splits a list of documents into smaller token-based chunks.
+
+**Parameters:**
+
+- **documents** (<code>list\[Document\]</code>) – The list of documents to split.
+
+**Returns:**
+
+- <code>dict\[str, list\[Document\]\]</code> – A dictionary with the "documents" key containing the list of chunks.
+
+#### to_dict
+
+```python
+to_dict() -> dict[str, Any]
+```
+
+Serializes the component to a dictionary.
+
+**Returns:**
+
+- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
+
+#### from_dict
+
+```python
+from_dict(data: dict[str, Any]) -> ChonkieTokenDocumentSplitter
+```
+
+Deserializes the component from a dictionary.
+
+**Parameters:**
+
+- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
+
+**Returns:**
+
+- <code>ChonkieTokenDocumentSplitter</code> – Deserialized component.