yvgude--lean-ctx/BENCHMARKS.md

# lean-ctx Benchmark: Head-to-Head Comparison

> Generated by lean-ctx v3.8.18 on 2026-07-02 07:06:37

**Project:** `.`  
**Files measured:** 50  
**Total raw tokens:** 533.2K  

## Methodology

All lean-ctx measurements are **real values** measured on the test repository. Competitor numbers use **published figures** from their official documentation, papers, or README files. Sources are cited in the comparison table.

- **Token counting**: tiktoken `o200k_base` (GPT-4o tokenizer)
- **Compression**: Each lean-ctx read mode is applied to the same files
- **Latency**: Wall-clock time, median of all measured files
- **Quality**: Preservation score (structural + semantic fidelity)

## Compression Comparison

| Tool | Compression | Tokens | Source |
|------|------------:|-------:|--------|
| Raw file read | 0% | 533.2K | Baseline — no compression applied |
| Repomix | 70% | 160.0K | Repomix docs (Tree-sitter compress mode) |
| aider /map | 85% | 80.0K | aider docs (repo-map with ctags/tree-sitter) |
| codebase-memory-mcp | 99% | 4.3K | arXiv paper (graph-query extraction only) |
| TokenForge | 70% | 160.0K | TokenForge README (tree-sitter code folding 40-70%; full-stack: code/command/conversation/json/mcp-schema) |
| **lean-ctx (map)** | **98.1%** | **8.0K** | Measured |
| **lean-ctx (signatures)** | **96.7%** | **14.0K** | Measured |
| **lean-ctx (aggressive)** | **16.5%** | **449.7K** | Measured |
| **lean-ctx (entropy)** | **0.4%** | **531.4K** | Measured |

## lean-ctx Mode Performance

| Mode | Compression | Latency | Quality | Use Case |
|------|------------:|--------:|--------:|----------|
| full | 0.0% | 0μs | 100% | Editing files (cached, ~13 tok on re-read) |
| map | 98.1% | 8.2ms | 78% | Understanding structure, deps, exports |
| signatures | 96.7% | 2.6ms | 96% | API surface only |
| aggressive | 16.5% | 344μs | 100% | Maximum compression for large files |
| entropy | 0.4% | 8.4ms | 100% | Information-theoretic filtering |

## Search Latency

| Query | BM25 Latency | Results |
|-------|-------------:|--------:|
| `function` | 456μs | 10 |
| `error handling` | 982μs | 10 |
| `configuration` | 60μs | 10 |
| `parse` | 415μs | 10 |
| `test` | 671μs | 10 |
| **Average** | **516μs** | — |

## Cold Start Performance

| Phase | Duration |
|-------|--------:|
| File scan | 220μs |
| BM25 index build | 672.5ms |
| First file read + tokenize | 240μs |
| **Total cold start** | **673.0ms** |

## Disk Footprint

| Component | Size |
|-----------|-----:|
| BM25 index | 6.7 MB |
| Total `.lean-ctx/` | 6.7 MB |

## Feature Comparison

| Feature | Raw | Repomix | aider | codebase-memory | **lean-ctx** |
|---------|:---:|:-------:|:-----:|:---------------:|:------------:|
| Multi-mode compression | — | — | — | — | ✅ |
| BM25 code search | — | — | — | ✅ | ✅ |
| Session caching | — | — | ✅ | ✅ | ✅ |
| Cross-session memory (CCP) | — | — | — | ✅ | ✅ |
| Shell output compression | — | — | — | — | ✅ |
| Call graph analysis | — | — | — | — | ✅ |
| Repo map generation | — | ✅ | ✅ | — | ✅ |
| Knowledge base | — | — | — | ✅ | ✅ |
| Tree-sitter AST (26 langs) | — | ✅ | ✅ | — | ✅ |
| MCP server | — | — | — | ✅ | ✅ |

**lean-ctx feature count:** 23 operations across 81 MCP tools

## Session Simulation (30-min coding)

| Approach | Tokens | Cost | Savings |
|----------|-------:|-----:|--------:|
| Raw (no compression) | 721.9K | $1.805 | — |
| lean-ctx (no CCP) | 90.7K | $0.227 | 87.4% |
| **lean-ctx + CCP** | **85.6K** | **$0.214** | **88.1%** |

## Compression Visualization

```
Compression % (higher = better)

Raw file read                 0.0%
lean-ctx (entropy)            0.4%
lean-ctx (aggressive)        ████████ 16.5%
Repomix                      ███████████████████████████████████ 70.0%
TokenForge                   ███████████████████████████████████ 70.0%
aider /map                   ██████████████████████████████████████████ 85.0%
lean-ctx (signatures)        ████████████████████████████████████████████████ 96.7%
lean-ctx (map)               █████████████████████████████████████████████████ 98.1%
codebase-memory-mcp          █████████████████████████████████████████████████ 99.2%
```

## System Information

- **OS:** macos aarch64
- **CPU:** Apple M5 (10 cores)
- **RAM:** 24.0 GB
- **lean-ctx:** v3.8.18
- **Rust:** rustc 1.96.1 (31fca3adb 2026-06-26)

## Reproducibility

```bash
# Install lean-ctx
cargo install lean-ctx

# Run the comparative benchmark on this repo
lean-ctx benchmark compare

# Run on a specific repository
lean-ctx benchmark compare --repo /path/to/repo

# Output to file
lean-ctx benchmark compare --output BENCHMARKS.md
```

---

*Generated by [lean-ctx](https://leanctx.com) v3.8.18 — Context Runtime for AI Agents*

**Disclaimer:** Competitor numbers are from published sources (docs, papers, READMEs). lean-ctx numbers are measured live. Different test repos will produce different results. Run `lean-ctx benchmark compare` on your own codebase for project-specific numbers.