c56bef871b
Sync docs with Docusaurus / sync (push) Waiting to run
Tests / Check if changed (push) Waiting to run
Tests / format (push) Blocked by required conditions
Tests / check-imports (push) Blocked by required conditions
Tests / Unit / macos-latest (push) Blocked by required conditions
Tests / Unit / ubuntu-latest (push) Blocked by required conditions
Tests / Unit / windows-latest (push) Blocked by required conditions
Tests / mypy (push) Blocked by required conditions
Tests / Integration / ubuntu-latest (push) Blocked by required conditions
Tests / Integration / macos-latest (push) Blocked by required conditions
Tests / Integration / windows-latest (push) Blocked by required conditions
Tests / notify-slack-on-failure (push) Blocked by required conditions
Tests / Mark tests as completed (push) Blocked by required conditions
Docker image release / Build base image (push) Waiting to run
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
77 lines
2.1 KiB
Markdown
77 lines
2.1 KiB
Markdown
---
|
|
title: "Preprocessors"
|
|
id: experimental-preprocessors-api
|
|
description: "Pipelines wrapped as components."
|
|
slug: "/experimental-preprocessors-api"
|
|
---
|
|
|
|
<a id="haystack_experimental.components.preprocessors.md_header_level_inferrer"></a>
|
|
|
|
## Module haystack\_experimental.components.preprocessors.md\_header\_level\_inferrer
|
|
|
|
<a id="haystack_experimental.components.preprocessors.md_header_level_inferrer.MarkdownHeaderLevelInferrer"></a>
|
|
|
|
### MarkdownHeaderLevelInferrer
|
|
|
|
Infers and rewrites header levels in Markdown text to normalize hierarchy.
|
|
|
|
First header → Always becomes level 1 (#)
|
|
Subsequent headers → Level increases if no content between headers, stays same if content exists
|
|
Maximum level → Capped at 6 (######)
|
|
|
|
### Usage example
|
|
```python
|
|
from haystack import Document
|
|
from haystack_experimental.components.preprocessors import MarkdownHeaderLevelInferrer
|
|
|
|
# Create a document with uniform header levels
|
|
text = "## Title
|
|
## Subheader
|
|
Section
|
|
## Subheader
|
|
More Content"
|
|
doc = Document(content=text)
|
|
|
|
# Initialize the inferrer and process the document
|
|
inferrer = MarkdownHeaderLevelInferrer()
|
|
result = inferrer.run([doc])
|
|
|
|
# The headers are now normalized with proper hierarchy
|
|
print(result["documents"][0].content)
|
|
> # Title
|
|
## Subheader
|
|
Section
|
|
## Subheader
|
|
More Content
|
|
```
|
|
|
|
<a id="haystack_experimental.components.preprocessors.md_header_level_inferrer.MarkdownHeaderLevelInferrer.__init__"></a>
|
|
|
|
#### MarkdownHeaderLevelInferrer.\_\_init\_\_
|
|
|
|
```python
|
|
def __init__()
|
|
```
|
|
|
|
Initializes the MarkdownHeaderLevelInferrer.
|
|
|
|
<a id="haystack_experimental.components.preprocessors.md_header_level_inferrer.MarkdownHeaderLevelInferrer.run"></a>
|
|
|
|
#### MarkdownHeaderLevelInferrer.run
|
|
|
|
```python
|
|
@component.output_types(documents=list[Document])
|
|
def run(documents: list[Document]) -> dict
|
|
```
|
|
|
|
Infers and rewrites the header levels in the content for documents that use uniform header levels.
|
|
|
|
**Arguments**:
|
|
|
|
- `documents`: list of Document objects to process.
|
|
|
|
**Returns**:
|
|
|
|
dict: a dictionary with the key 'documents' containing the processed Document objects.
|
|
|