yvgude--lean-ctx/docs/rfcs/sdk-embedding-v1.md

RFC: lean-ctx SDK / Embedding (v1)

Status: accepted, increment 1 implemented Crate: rust/crates/lean-ctx-sdk (lean_ctx_sdk) Related plans: lean-ctx SDK Embedding, lean-ctx Developer Platform (Track A)

Problem

The Addon system lets the engine call your tool (out-of-process, no access to internals). The opposite need — consume lean-ctx as an embedded engine — has no supported surface. Lean-md is the driving case: it calls engine cores in-process with a shared SessionCache so a read → re-read produces a token delta. Going through lean_ctx::core::… directly couples every consumer to internal churn.

Goals

1. A small, stable Rust façade with its own types (Engine, ReadMode, Output, Error) — engine internals can change without breaking embedders.
2. Shared session cache so the in-process read→re-read delta works (the acceptance property).
3. Safe by default: PathJail on, scoped state dir, auto-update off, write/exec behind explicit opt-in, no forced global allocator.
4. No new mechanism: dispatch the real registered tools, exactly as the MCP server does — zero behavioural drift.

Non-goals (v1)

• Async API. v1 is synchronous (owns a multi-thread runtime, dispatches via the blocking pool like the server). Async wrappers can come later.
• Re-exporting global mutations (Config::update_global, install/uninstall).
• A feature-minimal engine build. The engine references proxy/http_server/ ort unconditionally today, so the SDK pins default-minus-jemalloc rather than a hand-cut feature set. tree-sitter stays on (AST read modes).

Design

The Engine

Engine owns: the resolved project root (the PathJail root), a shared Arc<RwLock<SessionCache>>, a shared Arc<RwLock<SessionState>>, the full ToolRegistry (build_registry()), and a multi-threaded Tokio runtime.

Each call builds a ToolContext wired to the shared cache/session and dispatches the tool via spawn_blocking(move || tool.handle(&args, &ctx)) — the exact path LeanCtxServer uses, so ctx_read's Handle::block_on and ctx_search's block_in_place are both legal.

Own types

Façade type	Wraps
Engine / EngineBuilder	registry + shared cache/session + runtime
ReadMode	the engine mode string (auto/full/signatures/lines:N-M/…)
Output	ToolOutput (text + token accounting), derives Debug/Clone
Error	rmcp::ErrorData + jail/permission/init errors

Safe-by-default

EngineBuilder::build() resolves + validates the project root, sets the engine's data/config/state/cache dirs to a scoped temp dir (unless .data_dir(…)), disables the update check, and constructs the runtime. Write tools (ctx_edit, ctx_fill) and exec tools (ctx_shell, ctx_execute, shell) return Error::NotPermitted unless .allow_write(true) / .allow_exec(true).

Surface map (~26 capabilities)

dasTholo's Lean-md uses ~26 engine capabilities. v1 ships ergonomic typed methods for the read-mostly core and an escape hatch (Engine::call) that reaches every registered tool (write/exec gated). The table tracks how each capability is served today.

Capability	Engine tool	v1 surface
read	ctx_read	typed read()
search	ctx_search	typed search()
symbol	ctx_symbol	typed symbol()
outline	ctx_outline	typed outline()
tree / repomap	ctx_tree / ctx_repomap	typed tree() · call()
find	ctx_glob	call("ctx_glob", …)
count	ctx_cost / tokens	tokens::count · call()
graph	ctx_graph	call("ctx_graph", …)
callgraph	ctx_callgraph	call("ctx_callgraph", …)
impact	ctx_impact	call("ctx_impact", …)
architecture	ctx_architecture	call("ctx_architecture", …)
smells	ctx_smells	call("ctx_smells", …)
refactor	ctx_refactor	call("ctx_refactor", …)
review	ctx_review	call("ctx_review", …)
recall / remember	ctx_knowledge	call("ctx_knowledge", …)
query (semantic)	ctx_semantic_search	call("ctx_semantic_search", …)
render / compose	ctx_compose / ctx_overview	call(…)
inspect / list (tools)	ctx_tools	call("ctx_tools", …)
include / addressing	ctx_read (lines:/paths)	read()
reformat / compress	shell pattern engine	compress::shell_output
date / env / routes	ctx_routes etc.	call(…)
edit	ctx_edit	call() (needs allow_write)
shell / exec	ctx_shell / ctx_execute	call() (needs allow_exec)
hash	engine hash	hash::blake3_*
addon authoring/audit	scaffold + audit gate	addon::scaffold/audit

Promoting a call()-served capability to a typed method is additive and semver-safe; the plan is to graduate them as the Lean-md port exercises them.

Acceptance

• In-repo: tests/engine_read.rs proves read→re-read saves ≥ the first read, PathJail rejects escapes, search finds symbols, and the write/exec gate + unknown-tool paths error correctly. examples/embed.rs shows a live ~99% re-read delta on Cargo.toml.
• External: dasTholo ports Lean-md onto the Engine — the real acceptance test that the surface is sufficient (tracked on the GitLab epic).

Distribution & trust

The SDK is a build substrate; distribution stays the Addon system. A binary built with the SDK and shipped as an addon still runs under the gateway's OS sandbox + output redaction + trust/signing — embedding does not weaken distribution security. Two trust contexts: (1) distributed as an addon = sandboxed; (2) run standalone = the embedder owns the boundary, like any binary.

Build reality (honest)

Full parity keeps tree-sitter on (AST modes are intrinsically costly). Embedders needing only read/search/knowledge can later get a lighter path once the engine compiles under a minimal feature set (today it does not). A lean-ctx-crypto/ML split is deferred until measured build numbers justify it.