yvgude--lean-ctx/BENCHMARKS.md

lean-ctx Benchmark: Head-to-Head Comparison

Generated by lean-ctx v3.8.18 on 2026-07-02 07:06:37

Project: .
Files measured: 50
Total raw tokens: 533.2K

Methodology

All lean-ctx measurements are real values measured on the test repository. Competitor numbers use published figures from their official documentation, papers, or README files. Sources are cited in the comparison table.

• Token counting: tiktoken o200k_base (GPT-4o tokenizer)
• Compression: Each lean-ctx read mode is applied to the same files
• Latency: Wall-clock time, median of all measured files
• Quality: Preservation score (structural + semantic fidelity)

Compression Comparison

Tool	Compression	Tokens	Source
Raw file read	0%	533.2K	Baseline — no compression applied
Repomix	70%	160.0K	Repomix docs (Tree-sitter compress mode)
aider /map	85%	80.0K	aider docs (repo-map with ctags/tree-sitter)
codebase-memory-mcp	99%	4.3K	arXiv paper (graph-query extraction only)
TokenForge	70%	160.0K	TokenForge README (tree-sitter code folding 40-70%; full-stack: code/command/conversation/json/mcp-schema)
lean-ctx (map)	98.1%	8.0K	Measured
lean-ctx (signatures)	96.7%	14.0K	Measured
lean-ctx (aggressive)	16.5%	449.7K	Measured
lean-ctx (entropy)	0.4%	531.4K	Measured

lean-ctx Mode Performance

Mode	Compression	Latency	Quality	Use Case
full	0.0%	0μs	100%	Editing files (cached, ~13 tok on re-read)
map	98.1%	8.2ms	78%	Understanding structure, deps, exports
signatures	96.7%	2.6ms	96%	API surface only
aggressive	16.5%	344μs	100%	Maximum compression for large files
entropy	0.4%	8.4ms	100%	Information-theoretic filtering

Search Latency

Query	BM25 Latency	Results
function	456μs	10
error handling	982μs	10
configuration	60μs	10
parse	415μs	10
test	671μs	10
Average	516μs	—

Cold Start Performance

Phase	Duration
File scan	220μs
BM25 index build	672.5ms
First file read + tokenize	240μs
Total cold start	673.0ms

Disk Footprint

Component	Size
BM25 index	6.7 MB
Total .lean-ctx/	6.7 MB

Feature Comparison

Feature	Raw	Repomix	aider	codebase-memory	lean-ctx
Multi-mode compression	—	—	—	—	✅
BM25 code search	—	—	—	✅	✅
Session caching	—	—	✅	✅	✅
Cross-session memory (CCP)	—	—	—	✅	✅
Shell output compression	—	—	—	—	✅
Call graph analysis	—	—	—	—	✅
Repo map generation	—	✅	✅	—	✅
Knowledge base	—	—	—	✅	✅
Tree-sitter AST (26 langs)	—	✅	✅	—	✅
MCP server	—	—	—	✅	✅

lean-ctx feature count: 23 operations across 81 MCP tools

Session Simulation (30-min coding)

Approach	Tokens	Cost	Savings
Raw (no compression)	721.9K	$1.805	—
lean-ctx (no CCP)	90.7K	$0.227	87.4%
lean-ctx + CCP	85.6K	$0.214	88.1%

Compression Visualization

Compression % (higher = better)

Raw file read                 0.0%
lean-ctx (entropy)            0.4%
lean-ctx (aggressive)        ████████ 16.5%
Repomix                      ███████████████████████████████████ 70.0%
TokenForge                   ███████████████████████████████████ 70.0%
aider /map                   ██████████████████████████████████████████ 85.0%
lean-ctx (signatures)        ████████████████████████████████████████████████ 96.7%
lean-ctx (map)               █████████████████████████████████████████████████ 98.1%
codebase-memory-mcp          █████████████████████████████████████████████████ 99.2%

System Information

• OS: macos aarch64
• CPU: Apple M5 (10 cores)
• RAM: 24.0 GB
• lean-ctx: v3.8.18
• Rust: rustc 1.96.1 (31fca3adb 2026-06-26)

Reproducibility

# Install lean-ctx
cargo install lean-ctx

# Run the comparative benchmark on this repo
lean-ctx benchmark compare

# Run on a specific repository
lean-ctx benchmark compare --repo /path/to/repo

# Output to file
lean-ctx benchmark compare --output BENCHMARKS.md

---

Generated by lean-ctx v3.8.18 — Context Runtime for AI Agents

Disclaimer: Competitor numbers are from published sources (docs, papers, READMEs). lean-ctx numbers are measured live. Different test repos will produce different results. Run lean-ctx benchmark compare on your own codebase for project-specific numbers.