lmcache--lmcache/docs/design/cli/commands/kvcache-command.md

lmcache kvcache CLI Command Design

Status: Proposal | Date: 2026-03-19

Why

Users need a way to manage KV cache state for specific requests from the command line — pin a request's cache to prevent eviction, compress it, or clear it. Part of Phase 1 of the CLI design.

---

Design Principles

lmcache kvcache is a per-request management tool. Every sub-command operates on a specific request's KV cache, identified by request ID or token sequence.

All management goes through HTTP

ZMQ is reserved for performance-critical data-path communication between the inference engine and LMCache (store, retrieve, prefetch). Every lmcache kvcache CLI operation goes through the MP HTTP server (lmcache/v1/multiprocess/http_server.py).

Today some operations (e.g. pin) only have ZMQ implementations. These need new HTTP endpoints on the MP HTTP server before the CLI can use them.

Indexing KV caches

Per-request sub-commands (pin, compress, info) take a positional request ID and optional range flags:

• <request_id> (positional, required) — identifies the request whose KV cache to operate on.
• --start <st> --end <ed> (optional) — narrow the operation to a token range [st, ed) within the request. Defaults to the full sequence.

clear is a bulk operation — it clears all L1 cache and does not take a request ID.

Pipe- and script-friendly output

• Exit codes: 0 = success, 1 = error, 2 = rejected (e.g. pin rejected due to memory pressure). Scripts branch on $? without parsing output.
• --format json: Structured output for piping into jq (already exists).
• --format terminal: Human-readable ASCII table (default, already exists).
• --quiet / -q: Suppress all stdout. Exit code only.
• Stdout vs stderr: Metrics to stdout (pipeable). Errors to stderr.

---

Command Overview

lmcache kvcache
├── info           # Per-request cache state (locations, pinned status)
├── clear          # Clear all cached KV data in L1 (CPU)
├── pin            # Pin a request's KV cache to L1/CPU (may be rejected)
└── compress       # Compress a request's KV cache in-place

Sub-command	Target	Description
info	instance	Show per-request cache state: which chunks, where stored, pinned status
clear	instance	Clear all cached KV data in L1 (CPU)
pin	instance	Pin a request's KV cache to L1/CPU; may be rejected if memory pressure is too high
compress	instance	Compress a request's KV cache to reduce memory footprint

$ lmcache kvcache -h
usage: lmcache kvcache [-h] {clear} ...

Manage KV cache state.

subcommands:
  clear         Clear all cached KV data in L1 (CPU)

# info, pin, compress will appear here once implemented

---

Commands in Detail

info

Status: needs further design — will not be implemented yet. The output format, filtering options, and server-side endpoint are TBD. The sketch below is a placeholder to illustrate intent.

Show the cache state for a specific request: which chunks exist, which storage backend holds each one, and whether they are pinned.

# By request ID
$ lmcache kvcache info --url http://localhost:8000 req-abc-123

===== KV Cache Info (req-abc-123) =====
Total chunks:                         32
Pinned:                                8
---------------- Chunks ---------------
[0:256]:                  L1, pinned
[256:512]:                L1, pinned
[512:768]:                L1, L2
[768:1024]:               L2
...
========================================

# Narrowed to a token range
$ lmcache kvcache info --url http://localhost:8000 \
    req-abc-123 --start 0 --end 512

# JSON for scripting
$ lmcache kvcache info --url http://localhost:8000 \
    req-abc-123 --format json
{
  "title": "KV Cache Info (req-abc-123)",
  "metrics": {
    "total_chunks": 32,
    "pinned": 8,
    "chunks": {
      "0:256": "L1, pinned",
      "256:512": "L1, pinned",
      "512:768": "L1, L2",
      "768:1024": "L2"
    }
  }
}

# Find chunks on L2
$ lmcache kvcache info --url http://localhost:8000 \
    req-abc-123 --format json \
    | jq '.metrics.chunks | to_entries[] | select(.value | contains("L2"))'

clear

Clear all cached KV data in L1 (CPU memory) on the target instance.

$ lmcache kvcache clear --url http://localhost:8000

========== KV Cache Clear ==================
Status:                                   OK
=============================================

pin

Pin a request's KV cache chunks to L1 (CPU memory) to prevent eviction. The server may reject the request if CPU memory pressure is too high.

Exit codes: 0 = pinned, 2 = rejected, 1 = error.

$ lmcache kvcache pin --url http://localhost:8000 req-abc-123

======== KV Cache Pin (req-abc-123) ========
Status:                                   OK
Chunks pinned:                            32
=============================================
$ echo $?
0

# Quiet mode for scripts
if lmcache kvcache pin -q --url http://localhost:8000 req-abc-123; then
    echo "pinned"
else
    echo "rejected or error"
fi

# Narrowed to a token range
$ lmcache kvcache pin --url http://localhost:8000 \
    req-abc-123 --start 0 --end 512

# Rejected case (exit code 2)
$ lmcache kvcache pin --url http://localhost:8000 req-xyz

======== KV Cache Pin (req-xyz) =============
Status:                             REJECTED
Reason:              L1 memory pressure (91%)
=============================================
$ echo $?
2

compress

Compress a request's KV cache chunks in-place to reduce memory footprint.

$ lmcache kvcache compress --url http://localhost:8000 \
    req-abc-123 --method zstd

===== KV Cache Compress (req-abc-123) ======
Status:                                   OK
Method:                                 zstd
Chunks compressed:                        32
=============================================

$ lmcache kvcache compress --url http://localhost:8000 \
    req-abc-123 --start 0 --end 512 --method zstd

Flag	Required	Description
<request_id>	yes (positional)	Target request
--method	yes	Compression method (e.g. zstd)
--start, --end	no	Narrow to token range [st, ed)

---

Existing API Surface & Gaps

All CLI operations target the MP HTTP server (lmcache/v1/multiprocess/http_server.py).

Usable today (no new endpoints needed)

CLI sub-command	Existing MP HTTP endpoint	Notes
clear	POST /cache/clear	Clears all L1 cache. Works as-is.

Needs new MP HTTP endpoints

CLI sub-command	What exists today	New endpoint needed on MP HTTP server
info	No per-request HTTP endpoint	GET /kvcache-info?request_id=... returning chunk ranges, locations, pinned status
pin	ZMQ only (no HTTP)	POST /pin accepting request-id, returning OK or REJECTED with reason
compress	ZMQ only (no HTTP)	POST /compress accepting request-id + method

---

Implementation

• Single KVCacheCommand (BaseCommand subclass) with second-level argparse subparsers. File: lmcache/cli/commands/kvcache.py.
• MP HTTP only: _http_request() wraps urllib.request (no new deps). All requests target the MP HTTP server.
• Indexing args for future per-request sub-commands: positional request_id + optional --start/--end, added via a shared helper.
• Output: self.create_metrics() — use --format json | jq for scripting.
• New --quiet / -q flag on BaseCommand: skips StreamHandler.
• Exit codes: 0 success, 1 error, 2 rejected. Errors to stderr.

Phasing

Phase	Work
1a	clear (HTTP exists)
1b	pin (needs per-instance endpoint), compress (needs per-instance endpoint)
future	info (needs further design — deferred)