yvgude--lean-ctx/docs/contracts/extractors-v1.md

Format Extractors & Chunkers — extractors-v1

Status: stable · EPIC 12.13 · Code: rust/src/core/extractors/

The front-door that turns a non-code document/data file into clean LLM text plus structure-aware chunks. It complements ingestion-v1 (which decides whether to index a path) by deciding how to read a given format. This is what lets the Context OS index a corpus of emails, spreadsheets, web pages, and reports — not just source code.

Dispatch

extractors::extract(path, bytes) -> Extracted { kind, text, chunks } selects an extractor by file extension:

Extension(s)	kind	Extractor	Chunking strategy
.json .jsonl .ndjson	json	json	one chunk per top-level array element / object entry
.csv .tsv	csv	csv	header-prefixed row groups (RFC-4180 quoting)
.eml	eml	eml	salient-header summary + body paragraphs; text/plain parts of multipart
.html .htm .xhtml	html	web::html_to_text	paragraphs of rendered Markdown
.pdf	pdf	web::pdf	paragraphs of extracted text
anything else	text	verbatim	paragraphs (blank-line split)

Invariants

Every extractor is total and graceful — required because input is arbitrary (agent-supplied, possibly malformed):

1. Never panics on any byte input (PDF parsing is catch_unwind-guarded).
2. Empty input ⇒ no chunks.
3. Non-empty input ⇒ ≥1 non-empty chunk (invalid JSON/CSV/EML degrades to a single text chunk rather than dropping content).
4. Deterministic — identical input yields identical output.

These are the same invariants the conformance suite (conformance-v1) enforces.

Registry integration

The text-based chunkers (csv, json, eml, html) register into the extension-registry-v1 through the same public API extensions use, so they:

• appear under extensions.chunkers in GET /v1/capabilities, and
• are exercised by lean-ctx conformance on every build.

PDF is byte-only (binary input) and is reached via extract(), not the text→chunks registry.

Versioning

extractors-v1 is additive — new formats / extensions may be added in a minor revision. Changing an existing format's kind tag or chunking contract is a breaking change requiring -v2.