wehub-resource-sync
c56bef871b
CodeQL / Analyze (python) (push) Has been cancelled
Update Platform Components Table / update (push) Has been cancelled
Docker image release / Build base image (push) Has been cancelled
Sync docs with Docusaurus / sync (push) Has been cancelled
Tests / Check if changed (push) Has been cancelled
Tests / format (push) Has been cancelled
Tests / check-imports (push) Has been cancelled
Tests / Unit / macos-latest (push) Has been cancelled
Tests / Unit / ubuntu-latest (push) Has been cancelled
Tests / Unit / windows-latest (push) Has been cancelled
Tests / mypy (push) Has been cancelled
Tests / Integration / ubuntu-latest (push) Has been cancelled
Tests / Integration / macos-latest (push) Has been cancelled
Tests / Integration / windows-latest (push) Has been cancelled
Tests / notify-slack-on-failure (push) Has been cancelled
Tests / Mark tests as completed (push) Has been cancelled
9 lines
987 B
YAML
9 lines
987 B
YAML
---
|
|
features:
|
|
- |
|
|
Introduced the ``PythonCodeSplitter`` component, a syntax-aware splitter for Python source files:
|
|
- Parses sources with the ``ast`` module and merges units (module docstring, import blocks, top-level functions, class headers, methods, nested classes, and remaining statements) greedily into chunks of roughly ``max_effective_lines``.
|
|
- Keeps whole functions and methods together; falls back to a line-based secondary split (using ``DocumentSplitter``) with overlap only for functions whose effective length exceeds ``oversized_factor * max_effective_lines``.
|
|
- Optionally strips docstrings into chunk metadata via ``strip_docstrings=True``, and prepends the enclosing class signature to chunks whose members live in a later chunk via ``preserve_class_definition=True``.
|
|
- Emits per-chunk metadata including ``start_line``, ``end_line``, ``unit_kinds``, ``include_classes``, ``decorators``, ``docstrings``, ``source_id``, and ``split_id``.
|