mempalace--mempalace/docs/format-coverage.md

Format coverage — mempalace mine --mode extract

Shipped in mempalace 3.3.6 (PR #1555). Module: mempalace/format_miner.py CLI entry: mempalace/cli.py — cmd_mine dispatcher routes --mode extract here. Tests: tests/test_format_miner.py — covers all 14 fringe cases plus orchestrator behavior. See the file for the current test inventory (the count drifts as polish PRs land).

---

What it does

Adds a third miner alongside the existing two:

mempalace mine <dir>                  → miner.py        (project files: code, docs, notes)
mempalace mine <dir> --mode convos    → convo_miner.py  (chat exports)
mempalace mine <dir> --mode extract   → format_miner.py (binary office documents)   ← NEW

Supported formats and the transformer each routes through:

Extension	Transformer	Python	License
.pdf	MarkItDown	≥ 3.10	MIT
.docx	MarkItDown	≥ 3.10	MIT
.pptx	MarkItDown	≥ 3.10	MIT
.xlsx	MarkItDown	≥ 3.10	MIT
.epub	MarkItDown	≥ 3.10	MIT
.rtf	striprtf	≥ 3.6	MIT

Per-format routing — why .rtf doesn't go through MarkItDown. Verified live on 2026-05-19 against a local mixed-format test directory: MarkItDown 0.1.5 does NOT convert .rtf. It returns the raw RTF control-code source unchanged ({\rtf1\ansi\ansicpg1252...). For RTF input, format_miner routes to striprtf (pure-Python, MIT, ~150 lines, purpose-built for RTF → plain text). The principle stays intact: transform at read time, never modify the source — only the choice of transformer is per-format.

Both libraries are optional runtime dependencies. Users who only mine projects or convos don't pay either install footprint. Users on Python 3.9 can still install striprtf to extract RTF; MarkItDown requires 3.10+ and silently skips if not present. Per-extension install messages route to the right library:

• Missing MarkItDown (any of pdf/docx/pptx/xlsx/epub) → SKIP_NO_MARKITDOWN → log: pip install markitdown
• Missing striprtf (any .rtf) → SKIP_NO_STRIPRTF → log: pip install striprtf

---

Design — read-time conversion, never modifies source files

Same architectural principle as virtual line numbering (the other 3.3.6 feature in this package): transform at read time, leave storage untouched.

PDF on disk → (read bytes into memory) → MarkItDown → Markdown text → existing chunker → drawers in ChromaDB

• The PDF on the user's disk is read but never modified.
• No intermediate .md files are written to the user's filesystem.
• The conversion lives entirely in memory; once the text reaches the existing chunker, it's handled identically to any other ingestion.
• Closets / drawers / halls all work the same — the format miner produces the same drawer-shape as the other miners.

---

Why a new miner, not edits to miner.py

Three reasons, in order of importance:

1. Matches the existing miner-per-content-type pattern. Mempalace already has miner.py and convo_miner.py. A third miner is a third instance of the same shape — immediately recognizable to anyone reading the codebase.
2. Smaller review surface. format_miner.py is a pure addition. Reviewer reads one new file in isolation; the stable code paths in miner.py and convo_miner.py are untouched.
3. Cleaner failure isolation. If the format miner has a bug, the existing miners are physically incapable of being affected.

The trade-off vs. integrating into miner.py was decided during the 2026-05-19 design pass: the new-miner pattern wins on architecture and on review surface.

---

Fringe cases handled — 13 (original 12 + striprtf parity)

Each case maps to a specific ExtractionStatus enum value so callers (and reviewers) can audit exactly which path fired. All thirteen are covered by dedicated tests in tests/test_format_miner.py.

#	Case	Status code	Behavior
1	MarkItDown not installed (any non-RTF format)	SKIP_NO_MARKITDOWN	Logs pip install markitdown instruction; skips file
2	File too large (> 500 MB default; caller can override)	SKIP_TOO_LARGE	Logs size and threshold; skips
3	iCloud cloud-only file (.icloud suffix OR st_flags dataless bit)	SKIP_CLOUD_ONLY	Detected before any I/O that would trigger materialization; skips
4	Encrypted / password-protected PDF	SKIP_ENCRYPTED	Catches exceptions matching encrypt/decrypt/password/protected; skips
5	Empty file (zero bytes)	SKIP_EMPTY	Silent skip
6	Permission denied	SKIP_PERMISSION	Catches PermissionError; logs; skips
7	Broken symlink (target missing)	SKIP_BROKEN_SYMLINK	Detected via is_symlink() + not exists(); skips
8	Dirty encoding in extracted text	(recovered)	decode_robust() falls back UTF-8 → CP1252 → UTF-8-with-replace; never raises
9	Windows path semantics	(no special status)	pathlib.Path throughout; case-insensitive suffix matching; accepts str or Path input
10	Transformer internal crash on malformed file	SKIP_EXTRACTION_ERROR	Catches generic Exception at file boundary; logs filename + exception type + message tail; skips
11	Network / sync-drive timeout	SKIP_NETWORK_TIMEOUT	Catches TimeoutError; skips
12	Unrecognized extension	SKIP_UNRECOGNIZED	Cheap suffix-set check; skips
13	striprtf not installed (any .rtf)	SKIP_NO_STRIPRTF	Logs pip install striprtf instruction; skips file. Added 2026-05-19 after a live integration test surfaced MarkItDown 0.1.5's lack of RTF support.

Plus SKIP_UNREADABLE for catch-all OSError during stat (not in the original list, added for defensive completeness — see file-stat block in extract_text).

---

Cases NOT solved by this miner (deferred)

Per the spec, these are documented limits, not bugs:

• Custom PDF parsers for specific document types. MarkItDown does its best; we accept its limits.
• OCR on scanned image PDFs. Separate concern, opt-in feature for a later release.
• DRM-locked files. We can't bypass DRM and shouldn't try.
• Pathological corrupt files. They're corrupt — SKIP_EXTRACTION_ERROR with a clear log line is the responsible outcome.

These limitations are reported per-file via the skip-code log, so the user always knows what didn't make it in and why.

---

API reference

All public symbols are exported via __all__. Two functions and one enum form the contract:

extract_text(path, max_file_size=DEFAULT_MAX_FILE_SIZE) -> tuple[Optional[str], ExtractionStatus]

Convert one file to text. Returns (text, status) where text is the extracted Markdown for OK cases and None for every skip. Pure function — source file at path is never modified.

Accepts both Path and str for path.

scan_formats(directory) -> list[Path]

Walk directory recursively, return supported files sorted by path. Skips hidden / build directories (.git, .venv, __pycache__, etc.) and OS metadata files (.DS_Store, Thumbs.db, desktop.ini).

ExtractionStatus (Enum)

Twelve documented values plus SKIP_UNREADABLE. Each is paired with a string value like "skip:no_markitdown" for log readability; tests assert on .name for stability.

decode_robust(raw: bytes) -> str

Identical to the helper in the terminal-session normalizer. UTF-8 first, CP1252 fallback, UTF-8-with-replace as final safety net. Never raises.

is_icloud_dataless(path: Path) -> bool

Two signals:

1. Literal .icloud suffix (iCloud's offloaded-file placeholder convention)
2. macOS st_flags dataless bit (0x40000000) — best-effort, gracefully degrades on non-macOS

Returning True causes extract_text to short-circuit to SKIP_CLOUD_ONLY before any I/O that would trigger iCloud materialization.

---

CLI integration

One-line addition to mempalace/cli.py in cmd_mine:

def cmd_mine(args):
    ...
    try:
        if args.mode == "convos":
            from .convo_miner import mine_convos
            mine_convos(...)
        elif args.mode == "extract":                       # ← NEW
            from .format_miner import mine_formats         # ← NEW
            mine_formats(                                  # ← NEW
                format_dir=args.dir,                       # ← NEW
                palace_path=palace_path,                   # ← NEW
                wing=args.wing,                            # ← NEW
                agent=args.agent,                          # ← NEW
                limit=args.limit,                          # ← NEW
                dry_run=args.dry_run,                      # ← NEW
            )                                              # ← NEW
        else:
            from .miner import mine
            mine(...)

And add "extract" to the --mode choices in the argparse setup:

p_mine.add_argument(
    "--mode",
    choices=["projects", "convos", "extract"],  # ← extract added
    default="projects",
    help="Ingest mode: 'projects' (default), 'convos' for chat exports, 'extract' for office documents",
)

That's the entire CLI change. Two argparse edits + one elif branch.

---

Empirical smoke test — local mixed-format directory

Run on a local test directory containing 52 .rtf files + 11 .pdf files (a mix of long-form correspondence and research/architecture documents, no synthetic fixtures):

Supported formats: ['.docx', '.epub', '.pdf', '.pptx', '.rtf', '.xlsx']
scan_formats: found 63 supported files

Live end-to-end extraction (2026-05-19, Python 3.13, MarkItDown 0.1.5 + striprtf 0.0.27):

Format	Files	Result	Chars extracted
RTF (via striprtf)	52	52/52 OK	1,350,679
PDF (via MarkItDown)	11	10 OK, 1 SKIP_ENCRYPTED	1,340,869

The single PDF skip is a real password-protected file — Fringe Case 4 (SKIP_ENCRYPTED) fired correctly on real data via the exception-message matcher. Designed behavior validated empirically.

Anti-regression assertion: zero raw \rtf1 / \ansi control codes leak into extracted text. Without the per-format routing fix, 52 of 52 RTFs would have ingested as unreadable control-code source. With the fix, every RTF yields clean plain text. This is the integration bug that mocked unit tests cannot catch and that motivated the SKIP_NO_STRIPRTF status + per-format dispatch.

---

Backwards compatibility

• No changes to existing miners. miner.py and convo_miner.py are not touched.
• No new required dependencies. MarkItDown and striprtf are optional extras (pip install markitdown striprtf or pip install mempalace[extract] if declared as such in pyproject.toml). MarkItDown requires Python ≥ 3.10; striprtf works on 3.6+. Recommended pyproject entries use environment markers so 3.9 users automatically get only striprtf:

  [project.optional-dependencies]
  extract = [
      "striprtf>=0.0.27",
      "markitdown>=0.1.5; python_version >= '3.10'",
  ]
  

• No on-disk format changes. Drawers / closets / halls have the exact same shape regardless of which miner produced them.
• No migration step. Users who already have a palace mined from the other modes can run mempalace mine --mode extract ~/docs/ and the new drawers land in the same palace, same wing, same room conventions.
• Optional --max-file-size flag can be added to mempalace mine so users with legitimate large files (e.g. scanned books) can raise the cap from the 500 MB default.

---

Out of scope for 3.3.6

• Streaming extraction for huge PDFs (>500 MB). MarkItDown's default behavior is whole-file in memory. For users with multi-GB PDFs, a streaming path via pypdf page-by-page is the natural extension. Deferred to 3.3.7 or 3.4.0.
• Audio / video transcription. Whisper integration is its own scope.
• OCR for scanned PDFs. Tesseract / cloud OCR integration is its own scope.
• Migration importers (Notion / Obsidian / Mem0 export formats). Each is a separate adapter; not bundled here.

These are real future features; they don't belong in a 3.3.6 release whose scope is "additive format coverage via MarkItDown."