yvgude--lean-ctx/docs/guides/context-infrastructure.md

lean-ctx as Context Infrastructure

lean-ctx is more than a file-read compressor: it is the infrastructure layer between your agents and everything they need to know. This page is the map of that layer — which sources flow in, what one shared pipeline does with them, how retrieval gets them back out, and where the escape hatches are. Every capability below exists today; each section links to its reference.

 SOURCES                    ONE PIPELINE                     RETRIEVAL
 ─────────                  ────────────                     ─────────
 repo files      ┐                                     ┌─ ctx_search (BM25)
 linked repos    │   chunk → index → consolidate       ├─ ctx_search --semantic
 docs/artifacts  ├──▶ BM25 + SPLADE + dense vectors ──▶├─   (hybrid: RRF + rerank)
 GitHub/GitLab   │   + code graph + knowledge facts    ├─ ctx_multi_repo
 Jira/Postgres   │                                     ├─ ctx_graph / ctx_callgraph
 any MCP server  ┘                                     └─ ctx_knowledge
                                                            │
                          PORTABILITY: OKF (Markdown) · signed .ctxpkg
                          EXTENSION:   addons behind the gateway

Sources — what flows in

Source	How it enters	Reference
The repo	tree-sitter AST chunking (20+ languages), incremental BM25 + embedding index	How retrieval works
More repos	ctx_multi_repo roots, or linked workspace projects fused into one result list	Monorepo guide
Docs & artifacts	project doc folders declared in .lean-ctx-artifacts.json, indexed separately from code	ctx_search action=semantic artifacts=true
Issue trackers & DBs	first-class providers: GitHub, GitLab, Jira, Postgres schema	lean-ctx provider …, ctx_provider
Any MCP server	the MCP bridge provider + gateway addons	Addons

Provider results don't just pass through: with providers.auto_index = true they are consolidated — chunked into the BM25 index, linked into the code graph, distilled into knowledge facts. An issue that references src/auth.rs becomes findable from the file, and vice versa.

Retrieval — one hybrid engine, not one signal

Every retrieval surface runs the same engine: lexical BM25 + learned-sparse SPLADE + dense vectors from a local ONNX model, fused with Reciprocal Rank Fusion, sharpened by code-aware reranking and graph spreading activation. No external embedding API, no index-build marathon — and each piece degrades gracefully (cold dense index → BM25 floor, never a failed query).

Two dials matter in practice:

• The embedding model is swappable — any HuggingFace ONNX export via [embedding].model = "hf:org/repo@rev", including code-specialized models. See Custom Embedding Models.
• The vector store is swappable — in-process by default, or delegate dense search to a self-hosted Qdrant. See Dense Backends.

The quality floor is measurable, not asserted: the benchmark scorecard (recall@5/10, MRR, determinism digest) ships with the repo.

Memory — what the layer learns

Sessions distill into knowledge: facts, patterns, decisions and typed relations per project (ctx_knowledge, ctx_session). Retrieval consults this store alongside code — and it is never locked in:

• OKF — export/import the knowledge base as vendor-neutral, git-diffable Markdown (one concept per file, relations as links).
• ctxpkg — the same snapshot as a signed, verifiable bundle for distribution.

Extension — when you want more than the core

The core stays local and lean on purpose. Heavier machinery — external RAG stacks, graph databases, specialized doc search — plugs in as addons behind the gateway instead of bloating the binary: one lean-ctx addon add <name>, and the tool's MCP surface joins your agent's toolbox with scrubbed env, pinned versions and typed output adapters.

Examples from the registry relevant to this page: qmd (local Markdown/notes search), memgraph-ingester (structure-aware RAG on a Memgraph code graph), cognee (GraphRAG knowledge graphs).

Design commitments

1. Local first. Embeddings, indexes and knowledge live on your machine unless you explicitly point them elsewhere.
2. Deterministic outputs. Tool output is a pure function of content + mode, which keeps provider prompt caches hot (#498).
3. The right half of the pipeline. Compress what fits into the window; retrieve only what doesn't. See lean-ctx vs naive RAG for when a dedicated vector DB is genuinely the better tool.
4. No lock-in. Everything the layer accumulates leaves as OKF or ctxpkg.