# LMCache CLI Design **Status:** Proposal | **Date:** 2026-03-11 ## Why Today users must remember `python3 -m lmcache.v1.multiprocess.http_server ...` and similar module paths. We need a single `lmcache` command as the front door to all LMCache functionality. ## Command Overview ``` lmcache ├── server # Launch LMCache server (ZMQ + HTTP) ├── coordinator # Launch the mp coordinator (HTTP) ├── describe {kvcache,engine} # Rich status view of a running endpoint ├── ping {kvcache,engine} # Pure liveness check (OK/FAIL) ├── query {kvcache,engine} # Single-shot query with metrics ├── bench {engine,server,l2} # Sustained performance benchmarking └── kvcache {clear,end-session} # KV cache management actions ``` | Verb | Question it answers | Weight | |------|-------------------|--------| | `ping` | Is it alive? | Single-shot, instant (OK/FAIL) | | `query` | What happens when I send one request? | Single-shot, with metrics | | `describe` | What is this thing? | Rich status dashboard | | `bench` | How fast is it? | Multi-iteration, metrics-heavy | | `kvcache` | Mutate cache state | Clear, end-session, evict (future) | All client commands use a `--url` flag pointing to the **LMCache HTTP server** (e.g. `--url http://localhost:8000`). --- ## Commands in Detail ### `lmcache server` Replaces `python3 -m lmcache.v1.multiprocess.http_server`. Runs in foreground, Ctrl-C to stop. HTTP frontend is enabled by default; use `--no-http` to run ZMQ-only. ```bash lmcache server \ --engine-type blend --host 0.0.0.0 --port 5555 \ --max-gpu-workers 2 \ --l1-size-gb 60 --eviction-policy LRU \ --no-http # opt out of HTTP frontend ``` Server args are composed from existing helpers: `add_mp_server_args()`, `add_storage_manager_args()`, `add_prometheus_args()`, `add_telemetry_args()`, `add_http_frontend_args()`. ### `lmcache coordinator` Replaces `python3 -m lmcache.v1.mp_coordinator`. Runs the mp coordinator's FastAPI/HTTP app in the foreground (Ctrl-C to stop). The coordinator tracks mp server instances in a registry and evicts those whose heartbeats lapse. ```bash lmcache coordinator \ --host 0.0.0.0 --port 9300 \ --instance-timeout 30 \ --health-check-interval 10 ``` Config resolves from `MPCoordinatorConfig.from_env()` (the `LMCACHE_MP_COORDINATOR_*` environment variables); any CLI flag that is supplied overrides the corresponding field. Each flag defaults to unset so env-only deployments keep working. See [../v1/mp_coordinator/README.md](../v1/mp_coordinator/README.md). ### `lmcache describe` ```bash $ lmcache describe kvcache --url localhost:5555 ============ LMCache KV Cache Service ============ Health: OK ZMQ endpoint: tcp://localhost:5555 HTTP endpoint: http://localhost:8000 Engine type: blend Chunk size: 256 L1 capacity (GB): 60.0 L1 used (GB): 42.3 (70.5%) Eviction policy: LRU Cached objects: 1024 Uptime: 2h 14m 32s ================================================== $ lmcache describe engine --url http://localhost:8000 ================ Inference Engine ================ Model: meta-llama/Llama-3.1-70B-Instruct Max context (tokens): 131072 Status: healthy Running requests: 3 ================================================== ``` `describe kvcache` gathers data from multiple ZMQ request types (`NOOP` for debug info, `GET_CHUNK_SIZE` for chunk size) and `/status` (HTTP) to build a consolidated view. ### `lmcache ping` Pure liveness check for both targets. Returns OK/FAIL with round-trip time, measuring only the network round-trip excluding local Python overhead. **`ping kvcache`** -- pings the LMCache server process via HTTP `/healthcheck`: ```bash $ lmcache ping kvcache --url http://localhost:8080 ======= Ping KV Cache ======= Status: OK Round trip time (ms): 0.42 ============================== ``` **`ping engine`** -- pings the vLLM server process via HTTP `/health`: ```bash $ lmcache ping engine --url http://localhost:8000 ======== Ping Engine ========= Status: OK Round trip time (ms): 12.3 ============================== ``` ### `lmcache query` Single-shot query with detailed metrics. Use this to test a specific request and see what happened. **`query engine`** -- single inference request with TTFT/TPOT. Supports `{corpus}` templates for realistic long-context prompts: ```bash $ lmcache query engine --url http://localhost:8000 \ --prompt "{ffmpeg} What is the example usage of ffmpeg?" --max-tokens 128 ========== Query Engine Result ========== Prompt tokens: 8192 Corpus 'ffmpeg': 8186 Query: 6 Output tokens: 128 -----------Latency Metrics--------------- TTFT (ms): 892.3 TPOT (ms/token): 11.8 Total latency (ms): 2403.7 Throughput (tokens/s): 53.2 ========================================= ``` **`query kvcache`** -- query KV cache state for specific keys or tokens: ```bash # Check if a specific token sequence is cached (lookup) $ lmcache query kvcache --url localhost:5555 \ --prompt "{ffmpeg} What is the example usage of ffmpeg?" \ --model meta-llama/Llama-3.1-8B-Instruct ======== Query KV Cache Result ========== Prompt tokens: 8192 Cached chunks: 30/32 (93.8%) Cached tokens: 7680/8192 Cache status: HIT (partial) ========================================= # Store-retrieve round-trip with latency and correctness $ lmcache query kvcache --url localhost:5555 --round-trip ==== Query KV Cache Result (round-trip) ==== Store latency (ms): 1.23 Retrieve latency (ms): 0.87 Checksum: OK ============================================ ``` ### `lmcache bench` **`bench server`** -- end-to-end sanity test for a running LMCache MP cache server (ZMQ + HTTP). For each sequence in ``[--start, --end)`` the tool runs a cold pass (``LOOKUP`` miss → ``STORE``) and a warm pass (``LOOKUP`` hit → ``RETRIEVE``), then cross-checks per-chunk checksums against the server's HTTP API. Exercises the full RPC path (``REGISTER_KV_CACHE → GET_CHUNK_SIZE → LOOKUP → QUERY_PREFETCH_STATUS → RETRIEVE → STORE → END_SESSION``). Supports two run modes via ``--mode``: - **``gpu``** (default) -- allocates real CUDA tensors and uses CUDA IPC (LMCache-driven handle transfer path). - **``cpu``** -- allocates POSIX-SHM-backed tensors; the server maps the same physical pages for zero-copy STORE/RETRIEVE (engine-driven transfer path by default). To use the zero-copy SHM handle path, add ``--transfer-mode lmcache_driven``. The transfer path can be overridden explicitly with ``--transfer-mode {auto,engine_driven,lmcache_driven}``. ``auto`` keeps the historical mapping: gpu→lmcache_driven, cpu→engine_driven. ```bash $ lmcache bench server \ --rpc-url tcp://localhost:5555 \ --url http://localhost:8080 \ --start 0 --end 2 Connecting to LMCache MP Server at tcp://localhost:5555 (mode=gpu, transfer=auto) ... Server chunk_size = 256 Resolved KV shape spec: (2,1024,16,8,128):float16:32 [seq=0] LOOKUP cold: 0/2 chunks hit (1.82 ms) [seq=0] STORE: 2 chunks stored (1.74 ms) [seq=0] LOOKUP warm: 2/2 chunks hit (1.31 ms) [seq=0] RETRIEVE: 2 chunks retrieved (1.48 ms) [seq=0] CHECKSUM MATCH OK [seq=1] ... ``` With ``--end`` unset, the loop runs forever; stop with ``Ctrl-C``. The KV tensor layout is controlled by ``--kvcache-shape-spec`` (see ``lmcache/v1/kv_layer_groups.py``); see :doc:`bench_server` in the user guide for the full flag list. **`bench l2`** -- store / lookup / load throughput benchmark against an ``L2AdapterInterface`` implementation (no MP server required). Implemented at ``lmcache/cli/commands/bench/l2_adapter_bench/``; see the ``docs/source/cli/bench_l2.rst`` user guide for full options. **`bench engine`** -- **superset of `vllm bench serve`**. Same CLI args, same output format, plus an extra LMCache KV cache metrics section: ```bash # vllm bench serve compatible -- just swap the command name $ lmcache bench engine \ --url http://localhost:8000 \ --model meta-llama/Llama-3.1-8B-Instruct \ --dataset-name random --random-input-len 7500 --random-output-len 200 \ --num-prompts 30 --request-rate 1 --ignore-eos ============ Serving Benchmark Result ============ Successful requests: 30 Benchmark duration (s): 31.34 Total input tokens: 224970 Total generated tokens: 6000 Request throughput (req/s): 0.96 Output token throughput (tok/s): 191.44 Total Token throughput (tok/s): 7369.36 ---------------Time to First Token---------------- Mean TTFT (ms): 313.41 Median TTFT (ms): 272.83 P99 TTFT (ms): 837.32 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 8.84 Median TPOT (ms): 8.72 P99 TPOT (ms): 11.35 ----------LMCache KV Cache Performance------------ KV cache hit rate (L1): 92.3% KV cache hit rate (L2): 67.8% L1 read bandwidth: 12.4 GB/s L1 write bandwidth: 8.7 GB/s Avg tokens saved by cache (per req): 6420 Cache-assisted TTFT savings (est.): 58.2% ================================================== ``` LMCache-specific additions on top of vLLM args: `--url` (replaces `--port`), `--prompt` with `{corpus}` templates, `--corpus name=path` for custom corpora. ### `lmcache kvcache` ```bash $ lmcache kvcache clear --url localhost:5555 ========== KV Cache Clear ========== Status: OK Objects removed: 1024 ==================================== $ lmcache kvcache end-session --url localhost:5555 ======== KV Cache End Session ======== Status: OK Request ID: ====================================== ``` --- ## Prompt Corpora `query engine`, `bench engine`, and `query kvcache` support `{name}` in `--prompt` to expand built-in text corpora (e.g., `{paul_graham}` ~12k tokens, `{ffmpeg}` ~8k tokens). Custom corpora: `--corpus my_doc=./file.txt`. Built-in corpora ship in `lmcache/cli/corpora/`. ## Implementation Notes ### Architecture - **Auto-discovery (N-level):** Commands at all levels are discovered automatically via `discover_subclasses()` (in `lmcache/v1/utils/subclass_discovery.py`). No manual registration is needed — adding a new command at any depth is a single-file change. - **Leaf commands:** Inherit from `BaseCommand` directly. - **Command groups:** Inherit from `CompositeCommand(BaseCommand)`. Its `register()` scans the package where the concrete subclass is defined for nested `BaseCommand` subclasses and registers each one automatically. - **Recursive nesting:** A discovered subcommand can itself be a `CompositeCommand`, enabling arbitrary depth (e.g. `tool → cache-simulator → simulate`). - **Class hierarchy:** - `BaseCommand` — abstract base class for all CLI commands (leaf or composite). - `CompositeCommand(BaseCommand)` — base class for commands that contain auto-discovered sub-subcommands (e.g. `query`, `bench`, `quota`, `trace`, `tool`). Subclasses only need to implement `name()` and `help()`. - **Adding a new command:** - *Top-level:* Create a new `.py` file (or sub-package with `__init__.py`) under `commands/` with a concrete `BaseCommand` subclass. Done. - *Second-level:* Create a new `.py` file (or sub-package with `__init__.py`) under the parent command's package with a concrete `BaseCommand` subclass. Done. No edits to the parent's `__init__.py` required. - **`send_request()` helper:** Creates a temporary `MessageQueueClient`, submits a ZMQ request, waits with timeout (default 5s), tears down. All ZMQ commands use this. Extended to handle HTTP targets alongside ZMQ. - **Framework:** `argparse` with subparsers (no new deps). Reuses existing `add_*_args()` helpers. - **`--url` flag:** Configured per-subcommand (ZMQ vs HTTP semantics vary). ### File layout ``` lmcache/cli/ ├── __init__.py ├── main.py # main() entry point ├── metrics/ # Metrics system (see framework-and-metrics.md) ├── commands/ │ ├── __init__.py # Auto-discovers ALL_COMMANDS (no manual edits) │ ├── base.py # BaseCommand ABC + CompositeCommand │ ├── mock.py # lmcache mock (example/test command) │ ├── server.py # lmcache server │ ├── coordinator.py # lmcache coordinator │ ├── describe.py # lmcache describe {kvcache} │ ├── ping.py # lmcache ping {kvcache,engine} │ ├── kvcache.py # lmcache kvcache {clear,end-session} │ ├── query/ # lmcache query (CompositeCommand) │ │ ├── __init__.py # QueryCommand(CompositeCommand) │ │ ├── engine_command.py # Auto-discovered: lmcache query engine │ │ └── kvcache_command.py # Auto-discovered: lmcache query kvcache │ ├── bench/ # lmcache bench (CompositeCommand) │ │ ├── __init__.py # BenchCommand(CompositeCommand) │ │ ├── engine_bench/ # Auto-discovered: lmcache bench engine │ │ ├── server_bench/ # Auto-discovered: lmcache bench server │ │ └── l2_adapter_bench/ # Auto-discovered: lmcache bench l2 │ ├── quota/ # lmcache quota (CompositeCommand) │ │ ├── __init__.py # QuotaCommand(CompositeCommand) │ │ ├── set_command.py # Auto-discovered: lmcache quota set │ │ ├── get_command.py # Auto-discovered: lmcache quota get │ │ ├── list_command.py # Auto-discovered: lmcache quota list │ │ └── delete_command.py # Auto-discovered: lmcache quota delete │ ├── trace/ # lmcache trace (CompositeCommand) │ │ ├── __init__.py # TraceCommand(CompositeCommand) │ │ ├── info_command.py # Auto-discovered: lmcache trace info │ │ └── replay_command.py # Auto-discovered: lmcache trace replay │ └── tool/ # lmcache tool (CompositeCommand) │ ├── __init__.py # ToolCommand(CompositeCommand) │ └── cache_simulator/ # Auto-discovered: lmcache tool cache-simulator │ ├── __init__.py # CacheSimulatorCommand(CompositeCommand) │ ├── simulate_command.py # Auto-discovered: simulate │ ├── sweep_command.py # Auto-discovered: sweep │ └── gen_dataset_command.py # Auto-discovered: gen-dataset ├── config.py # CLIConfig (centralized config system) └── corpora/ # Built-in prompt corpora ``` ### Other notes - **Entry point:** `lmcache = "lmcache.cli.main:main"` in `pyproject.toml`. - **Auto-discovery mechanism:** Powered by `discover_subclasses()` in `lmcache/v1/utils/subclass_discovery.py`. Uses `pkgutil.iter_modules` to scan direct submodules, then `inspect.getmembers` to find concrete `BaseCommand` subclasses. Each subclass is yielded at most once. - **`CompositeCommand` pattern:** A `CompositeCommand` scans its own package for `BaseCommand` subclasses (excluding itself and abstract classes). Sub-packages with `__init__.py` defining a `BaseCommand` are also discovered, enabling nested command groups (e.g. `tool cache-simulator simulate`). - **`bench engine`:** Wraps `vllm.benchmarks`, then queries `/status` for cache metrics. - **`query kvcache`:** Tokenizes `--prompt` using the model's tokenizer, then performs a lookup over ZMQ to check which chunks are cached. ## Phasing | Phase | Scope | |-------|-------| | **0** | CLI framework (explicit registration, `Metrics`), `mock` example command, entry point — see [framework-and-metrics.md](framework-and-metrics.md) | | **1** | **`server`** (done), `ping kvcache`, `kvcache clear`, `kvcache end-session`, `describe kvcache` | | **2** | `ping engine`, `query engine`, `query kvcache`, `bench engine`, `bench server`, `bench l2`, `describe engine`, corpora | | **3** | `kvcache evict` (future) | Existing `lmcache_server` entry point kept as a deprecated alias for 2 minor releases.