yvgude--lean-ctx/bench/compress

compress() Benchmark — lean-ctx vs Headroom

A head-to-head compression benchmark for the drop-in compress(messages, model) contract. It runs both libraries over the same real corpus with the same tokenizer and emits JSON (compression ratio + latency).

Numbers are always measured, never fabricated: a tool that is not installed or whose daemon is unreachable is reported as available: false (with an install hint) instead of being estimated.

Prerequisites

For	Install
lean-ctx side	a daemon serving POST /v1/compress — lean-ctx dev-install
Headroom side (optional)	pip install headroom-ai
Accurate token counts (optional)	pip install tiktoken (else character counts)

The lean-ctx Python SDK is imported directly from packages/python-lean-ctx (no pip install required).

Run

# Default corpus = docs/reference/*.md, model = gpt-4o
python bench/compress/benchmark.py

# Custom corpus, write the JSON log to a file
python bench/compress/benchmark.py --corpus docs/ --model gpt-4o --out report.json

# Bound the corpus size
python bench/compress/benchmark.py --max-files 50 --max-bytes 200000

The corpus is built from real on-disk files (.md .rs .py .ts .txt .json .log .yaml .yml) under --corpus; one user message per file. No fixtures, no mock payloads.

Output

{
  "corpus":   { "path": "docs/reference", "messages": 27, "model": "gpt-4o", "tokenizer": "o200k_base" },
  "lean_ctx": { "available": true, "original_tokens": 69594, "compressed_tokens": 57615,
                "tokens_saved": 11979, "ratio": 0.172, "latency_ms": 1014.03 },
  "headroom": { "available": false, "install": "pip install headroom-ai" }
}

ratio = 1 − compressed/original, computed with the shared tokenizer for both tools so the comparison is apples-to-apples.

Daemon-free lean-ctx benchmark

To benchmark the deterministic funnel without a running daemon (calls the Rust compress_messages directly, o200k_base):

cargo test -p lean-ctx --lib \
  proxy::compress_api::tests::bench_real_corpus_o200k -- --ignored --nocapture

Extractive vs truncation (prose quality)

Prose is the one place a rule-based funnel can only truncate. This benchmark runs the extractive ranker (centrality, reusing the shipped all-MiniLM model) head-to-head against FIFO truncation at an identical 50% char budget over the real docs/reference corpus, and reports both token savings AND a coverage quality per method:

cargo test -p lean-ctx --lib --features embeddings \
  core::extractive::tests::bench_extractive_vs_truncation -- --ignored --nocapture

The report has two honest signals:

• avg_coverage (query-free): mean, over every full-document sentence, of its cosine to the nearest kept sentence — 1.0 means the kept set has a close match for every original sentence. Coverage is the fair extractive-quality proxy: a centroid cosine would reward redundancy (a contiguous prefix tracks the whole-doc centroid while MMR de-duplication deliberately diversifies away from it), whereas coverage rewards spreading the budget across the whole document.
• rag_query_recall (query-aware): a real sentence from each document's back half is used as the query; recall is the cosine of that query to its nearest kept sentence. 1.0 means the answer survived compression. This is the documented highest-value path (research / tool-result prose), and the place prefix truncation fails structurally — it drops the back half wholesale.

The model must already be present (available: false is reported honestly when it is not).

Honest caveat: docs/reference is structured reference material whose first half is already representative, so on the query-free avg_coverage signal, prefix truncation is a strong baseline (often ahead) — we report that as measured rather than spin it. Extractive's structural win shows up in rag_query_recall: when the answer is not in the prefix, truncation cannot recover it and query-aware extraction can. Point --corpus at long, non-front-loaded prose / RAG dumps to widen the coverage gap too.

Notes

• Prose/markdown is a conservative corpus for the rule-based funnel; tool output, logs and RAG dumps (the common agent case) compress far more.
• lean-ctx output is deterministic and prompt-cache safe (#498); the benchmark re-running with identical input yields identical lean_ctx figures.

See the full positioning in docs/comparisons/vs-headroom.md and the recipes in docs/guides/compress-sdk.md.