This directory contains runnable examples organized by use case. The table below describes what each example does, its hardware requirements, and a recommended learning order for infrastructure engineers getting started with LMCache.

Single-node vs. multi-node: Examples marked with NIXL or UCX require a high-bandwidth interconnect (NVLink or PCIe Gen4/5). Running them on a single machine with two GPUs is simpler than a true multi-node setup, but cross-node deployments add network configuration complexity — don't start there if you're still learning the basics.

Before You Start

Example	What it does	Hardware
`kv_cache_calculator/`	Web UI for calculating KV cache size (GB) given model architecture, dtype, and token count. Start here to size GPU memory and cache tiers before running anything.	None (browser)

Tier 1 — Core Concepts (Single-Node)

Focus: local caching within a single machine. No cross-node networking required.

Example	What it does	Hardware
`kv_cache_reuse/local_backends/`	Offload KV cache from GPU to CPU memory or local disk. Repeated requests with identical prefixes skip prefill entirely. Includes a Rust-based NVMe backend with `io_uring` support (Linux kernel 5.10+ required for `io_uring`).	1 GPU; optional NVMe for disk path
`online_session/`	Measure TTFT (time-to-first-token) for cold vs. cache-hit requests. Outputs JSONL for plotting. Includes a sweep script across context lengths.	1 GPU

Focus: separating prefill from decode, and sharing KV cache across vLLM instances.

Example	What it does	Hardware
`disagg_prefill_mp/`	PD disaggregation via the LMCache multiprocess server. P and D exchange KV through the LMCache MP service — no direct NIXL connection required. Recommended starting point for PD disaggregation.	2 GPUs
`disagg_prefill/1p1d/`	PD disaggregation with direct NIXL transfer: 1 prefill server + 1 decode server + a FastAPI proxy. Includes a benchmark script with expected latency numbers. High-bandwidth interconnect (NVLink or PCIe Gen4/5) is strongly recommended — without it, KV transfer overhead may negate the gains.	2 GPUs + NIXL
`kv_cache_reuse/share_across_instances/centralized_sharing/`	Two vLLM instances share one LMCache server. A prefix computed by instance A is reused by instance B. ⚠️ Requires `PYTHONHASHSEED=0` in all processes — without this, hashes differ across processes and cache lookups silently miss.	2 GPUs (same node)
`kv_cache_reuse/share_across_instances/p2p_sharing/`	Two vLLM instances transfer KV directly peer-to-peer using NIXL and UCX RDMA (`UCX_TLS=rc`).	2 GPUs + NIXL + UCX
`p2p/`	P2P KV cache sharing in multiprocess mode: each node runs an LMCache server, and a server reads a prefix it lacks directly from the peer that holds it over RDMA. A coordinator handles peer discovery. Includes single-node (debug) and multi-node setups and the logs to expect.	2+ GPUs (single- or multi-node) + RDMA fabric

Tier 3 — Production Operations and Observability

Focus: deploying, monitoring, and operating LMCache in production.

Example	What it does	Hardware
`observability/`	Full observability stack: OTel → Prometheus + Tempo → Grafana. Pre-provisioned dashboard shows cache hit rate, read/write throughput, and per-request trace waterfalls. Started with `docker compose up`.	1 GPU + Docker
`multi_process/`	Kubernetes DaemonSet YAML for deploying the LMCache server as a per-node sidecar (60 GB L1, 4 workers). Includes resource `requests`/`limits` calibrated to the L1 size.	Kubernetes cluster with GPU nodes
`kubernetes/`	`health_probe.py`: readiness/liveness probe script for LMCache in Kubernetes. Complements the DaemonSet config in `multi_process/`.	None
`chunk_statistics/`	Track chunk reuse rate using a memory Bloom filter or an on-disk hash log. Query via REST (`/chunk_statistics/status`). Use this to answer "does my workload benefit from caching?" before committing to a full deployment.	1+ GPUs
`cache_controller/`	REST APIs for cache orchestration: `lookup` (which instance holds a key), `move` (migrate a hot context between instances), `pin` (prevent eviction), `clear`, and `compress`.	1 GPU
`cache_with_configs/` and `cache_interface/`	Per-request control: attach tags (`lmcache.tag.*`), set TTL (`lmcache.ttl`), or skip caching entirely (`lmcache.skip_save: true`). Essential for multi-tenant deployments.	1 GPU
`remote_config_server/`	Flask reference server for centralised LMCache configuration. Workers POST their current config on startup and receive overrides — useful for managing settings across a large fleet.	None

Tier 4 — Advanced Features and Ecosystem

Framework Integrations

Example	What it does	Hardware
`sgl_integration/`	Using LMCache with SGLang instead of vLLM.	1 GPU
`frontend/`	Streamlit chat UI demo backed by vLLM + LMCache. Uses the ffmpeg man page as a long shared context to demonstrate TTFT reduction across turns.	1 GPU
`disagg_prefill/xpyd/`	xP + yD topology: multiple prefill servers feeding multiple decode servers.	3+ GPUs + NIXL

Storage Backends and Serialization

Example	What it does	Hardware
`kv_cache_reuse/remote_backends/`	Remote storage backends: InfiniStore, Mooncake, S3, Valkey, Redis. Each subdirectory has its own README with backend-specific setup.	1 GPU + respective backend
`serde/fp8/`	Quantize KV cache to fp8 before writing to disk (L2 adapter), then dequantize on prefetch. Halves disk storage requirements.	Hopper / Ada GPU (H100, RTX 40-series)
`redis_lookup/`	Shows the Redis key schema used by LMCache (`model@world_size@worker_id@chunk_hash`) and `redis-cli` commands for inspecting live cache entries.	Redis + 1 GPU

Non-Prefix KV Reuse (CacheBlend)

Example	What it does	Hardware
`blend_kv_v1/`	CacheBlend v1: reuse KV cache even when the new prompt is not a prefix of the cached one (e.g., RAG with swapped documents). Requires a small patch to vLLM source.	1 GPU (experimental)
`blend_kv/`	CacheBlend v0 (legacy): same concept using the old `lmcache_vllm` integration. Kept for reference.	1+ GPUs (legacy)

Developer Extensibility

Example	What it does	Hardware
`lmc_external_l2_adapter/` and `lmc_external_native_connector/`	Templates for writing a custom L2 storage adapter (Python) or a native C++ GPU connector plugin.	Depends on implementation
`runtime_plugins/` and `mp_runtime_plugins/`	Sidecar scripts (Python or shell) that run alongside LMCache workers: heartbeats, metric reporters, alert hooks. Filename prefix controls which role (`scheduler`, `worker_0`, `all`) runs the script.	None
`basic_check/`	CLI tool for verifying storage backend health and generating test keys. Useful in CI and for on-call diagnostics.	Optional GPU
`agents/`	Script for analyzing prefix-hash distribution of a prompt dataset. Useful for estimating cache efficiency before deployment.	None

README.md

LMCache Examples