.. _lmcache-bench: lmcache bench ============= The ``lmcache bench`` command runs sustained performance benchmarks. It has three sub-commands, each targeting a different layer of the stack: .. list-table:: :header-rows: 1 :widths: 20 80 * - Sub-command - Description * - ``engine`` - Benchmark an inference engine (e.g. vLLM) with workloads that exercise different KV-cache reuse patterns. * - ``server`` - End-to-end sanity test against a running LMCache MP cache server (ZMQ + HTTP). Requires the full ``lmcache`` install and a GPU. * - ``l2`` - Throughput / latency benchmark against an L2 cache adapter (store / lookup / load). .. code-block:: bash lmcache bench {engine,server,l2} [options] .. _lmcache-bench-engine: engine ------ The ``lmcache bench engine`` command runs sustained performance benchmarks against an inference engine (e.g., vLLM). It supports multiple workload types that exercise different caching patterns and reports TTFT, decoding speed, and throughput metrics. .. code-block:: bash lmcache bench engine [options] There are three ways to configure the benchmark: 1. **CLI arguments** -- pass all options on the command line. 2. **Interactive mode** -- run ``lmcache bench engine`` without required args and follow the step-by-step prompts. 3. **Config file** -- save a configuration to JSON and replay it with ``--config``. Quick Start ~~~~~~~~~~~ **Minimal (with all required arguments):** .. code-block:: bash lmcache bench engine \ --engine-url http://localhost:8000 \ --workload long-doc-qa \ --lmcache-url http://localhost:8080 **Interactive mode (guided setup):** .. code-block:: bash lmcache bench engine The interactive mode walks you through each required setting, then asks whether you want to configure general and workload-specific options or use defaults. **From a saved config file:** .. code-block:: bash lmcache bench engine --engine-url http://localhost:8000 \ --config my_bench.json Config files contain benchmark parameters (workload, KV cache settings, etc.) but not the engine URL, so you can reuse the same config against different engines. **Export a config without running the benchmark:** .. code-block:: bash lmcache bench engine \ --engine-url http://localhost:8000 \ --workload long-doc-qa \ --lmcache-url http://localhost:8080 \ --export-config my_bench.json This resolves all auto-detected values (model name, tokens per GB) and saves them to a portable JSON file that works without an LMCache server. **Non-interactive mode (for scripts and CI):** .. code-block:: bash lmcache bench engine \ --engine-url http://localhost:8000 \ --workload long-doc-qa \ --lmcache-url http://localhost:8080 \ --no-interactive Errors immediately if any required argument is missing, instead of entering interactive mode. Useful in automated pipelines. If you don't have an LMCache server, you can pass ``--tokens-per-gb-kvcache`` directly instead of ``--lmcache-url`` (see :ref:`bench-tokens-per-gb` for how to find this value). General Options ~~~~~~~~~~~~~~~ .. list-table:: :header-rows: 1 :widths: 30 10 60 * - Flag - Required - Description * - ``--config FILE`` - No - Load configuration from a JSON file. Skips interactive mode. CLI flags override values in the file. The engine URL is not stored in config files and must be provided separately. * - ``--export-config FILE`` - No - Export resolved configuration to a JSON file and exit. Does not run the benchmark. Auto-detected values (model, tokens per GB) are resolved and saved so the config is portable. Environment- specific values (engine URL, LMCache URL) are excluded. * - ``--no-interactive`` - No - Disable interactive mode. Errors if required arguments are missing instead of prompting. Useful for scripts and CI. * - ``--engine-url URL`` - Yes - Inference engine URL (e.g., ``http://localhost:8000``). Set ``OPENAI_API_KEY`` env var if authentication is needed. * - ``--workload TYPE`` - Yes - Workload type: ``long-doc-qa``, ``multi-round-chat``, ``long-doc-permutator``, ``prefix-suffix-tuner``, or ``random-prefill``. * - ``--tokens-per-gb-kvcache N`` - \* - Tokens per GB of KV cache. Required unless ``--lmcache-url`` is set. See :ref:`bench-tokens-per-gb` for how to find this value. * - ``--lmcache-url URL`` - No - LMCache HTTP server URL. When provided, ``--tokens-per-gb-kvcache`` is auto-detected from the server. * - ``--model NAME`` - No - Model name. Auto-detected from the engine if omitted. * - ``--kv-cache-volume GB`` - No - Target active KV cache volume in GB (default: 100). * - ``--seed N`` - No - Random seed (default: 42). * - ``--output-dir DIR`` - No - Directory for CSV and JSON output files (default: current directory). * - ``--no-csv`` - No - Skip CSV export. * - ``--json`` - No - Export a JSON summary file. * - ``-q`` / ``--quiet`` - No - Suppress the real-time progress display. .. _bench-tokens-per-gb: Finding ``--tokens-per-gb-kvcache`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If you have an LMCache server running, the easiest approach is to pass ``--lmcache-url`` and let the tool auto-detect the value. If you are using **vLLM without LMCache**, look for these lines in vLLM's startup log: .. code-block:: text INFO: Available KV cache memory: 12.34 GiB INFO: GPU KV cache size: 567,890 tokens Then compute:: tokens_per_gb = 567890 / 12.34 = 46,020 Workloads ~~~~~~~~~ long-doc-qa ^^^^^^^^^^^ Simulates repeated Q&A over long documents. Warmup sends each document once to populate the KV cache, then benchmark queries are dispatched with semaphore-controlled concurrency. .. list-table:: :header-rows: 1 :widths: 35 10 55 * - Flag - Default - Description * - ``--ldqa-document-length`` - 10000 - Token length of each synthetic document. * - ``--ldqa-query-per-document`` - 2 - Number of questions asked per document. * - ``--ldqa-shuffle-policy`` - random - Request ordering: ``random`` (shuffled) or ``tile`` (round-by-round). * - ``--ldqa-num-inflight-requests`` - 3 - Maximum concurrent in-flight requests. **Example:** .. code-block:: bash lmcache bench engine \ --engine-url http://localhost:8000 \ --workload long-doc-qa \ --lmcache-url http://localhost:8080 \ --kv-cache-volume 50 \ --ldqa-document-length 8000 \ --ldqa-query-per-document 4 \ --ldqa-shuffle-policy tile multi-round-chat ^^^^^^^^^^^^^^^^ Simulates multi-round chat with stateful sessions. Creates concurrent user sessions, dispatches requests at a fixed QPS rate, and records responses in session history so each subsequent query includes prior context. .. list-table:: :header-rows: 1 :widths: 35 10 55 * - Flag - Default - Description * - ``--mrc-shared-prompt-length`` - 2000 - System prompt token length per session. * - ``--mrc-chat-history-length`` - 10000 - Pre-filled chat history token length. * - ``--mrc-user-input-length`` - 50 - Tokens per user query. * - ``--mrc-output-length`` - 200 - Max tokens to generate per response. * - ``--mrc-qps`` - 1.0 - Target queries per second. * - ``--mrc-duration`` - 60.0 - Benchmark duration in seconds. **Example:** .. code-block:: bash lmcache bench engine \ --engine-url http://localhost:8000 \ --workload multi-round-chat \ --lmcache-url http://localhost:8080 \ --mrc-qps 2.0 \ --mrc-duration 120 long-doc-permutator ^^^^^^^^^^^^^^^^^^^ Stress-tests blended KV cache reuse by sending permutations of a set of context documents. Each request concatenates all context documents in a different order: .. code-block:: text [System Prompt] + [Doc_i1] + [Doc_i2] + ... + [Doc_iN] A single dummy warmup request is sent before the benchmark phase. Requests are dispatched with semaphore-controlled concurrency. .. list-table:: :header-rows: 1 :widths: 35 10 55 * - Flag - Default - Description * - ``--ldp-num-contexts`` - 5 - Number of unique context documents. * - ``--ldp-context-length`` - 5000 - Token length of each context document. * - ``--ldp-system-prompt-length`` - 1000 - Token length of the shared system prompt. Use ``0`` for no system prompt. * - ``--ldp-num-permutations`` - 10 - Number of distinct permutations to send. Capped at N! where N = ``--ldp-num-contexts``. * - ``--ldp-num-inflight-requests`` - 1 - Maximum concurrent in-flight requests. **Example:** .. code-block:: bash lmcache bench engine \ --engine-url http://localhost:8000 \ --workload long-doc-permutator \ --lmcache-url http://localhost:8080 \ --ldp-num-contexts 4 \ --ldp-context-length 8000 \ --ldp-num-permutations 24 \ --ldp-num-inflight-requests 2 prefix-suffix-tuner ^^^^^^^^^^^^^^^^^^^ A two-pass sequential workload designed to be run **unchanged** across three LMCache configurations to demonstrate the value of each cache tier (L0 HBM, L1 DRAM, L2 disk): .. list-table:: :header-rows: 1 :widths: 15 25 30 30 * - Baseline - LMCache config - Targeted overflow - Expected pass-2 hits * - 1 - vanilla vLLM (L0 only) - L0 (HBM) - none -- every request a cold prefill * - 2 - vLLM + LMCache L1 + L2 - L1 (DRAM) - L2 prefix hits (suffix recomputed) * - 3 - vLLM + LMCache L1 + L2 + CacheBlend - L1 (DRAM) - L2 prefix hits + CacheBlend suffix hits Set ``--kv-cache-volume`` to the size in GB of the tier you want to overflow (L0 size for Baseline 1, L1 size for Baselines 2 and 3). The workload itself is identical across baselines. Each request has the layout:: [prefix_i with unique-ID][random breaker][shared suffix] - ``num_prefixes`` distinct prefixes, each starting with ``PREFIX_<8-hex>`` so the prefix's tokenized hash differs across the pool. - A fresh random 32-token breaker per request, defeating ordinary prefix caching past the prefix boundary. - A single shared suffix used by every request -- the only entry CacheBlend can reuse. Pass 1 (warmup) sends each prefix once to populate the cache; its stats are discarded. Pass 2 sends them again in identical order. Because LRU evicts the next-needed prefix on each pass-2 access, even a 1.05x overflow of the targeted tier is enough to make every pass-2 request miss that tier and fall through to the next one. .. list-table:: :header-rows: 1 :widths: 35 10 55 * - Flag - Default - Description * - ``--psf-context-length`` - 8000 - Total tokens per request (prefix + breaker + suffix). * - ``--psf-prefix-ratio`` - 0.8 - Fraction of context-length used by the prefix. Must be in (0.0, 1.0). The remainder (minus a 32-token breaker) is the shared suffix. * - ``--psf-thrash`` - 20.0 - **Size in GB of the KV-cache tier to overflow.** Use the L0 (HBM) size for vanilla vLLM, or the L1 (LMCache DRAM) size for tiered baselines. The workload sizes its prefix pool to slightly more than this (5% overflow internally), enough to drive every pass-2 request to a miss of that tier under sequential dispatch + LRU. The number of pass-2 (measured) requests equals the prefix pool size, computed as ``floor(psf_thrash * 1.05 * tokens_per_gb / prefix_tokens)``. ``--kv-cache-volume`` is unused by this workload — sizing is driven solely by ``--psf-thrash``. **Example:** .. code-block:: bash lmcache bench engine \ --engine-url http://localhost:8000 \ --workload prefix-suffix-tuner \ --lmcache-url http://localhost:8080 \ --psf-context-length 8000 \ --psf-prefix-ratio 0.8 \ --psf-thrash 100 .. note:: For the analytical-model claim "thrash ≈ L1 size → ~0% LMCache hit rate" to hold empirically, the LMCache server must be started with ``--eviction-ratio 0.99`` (default ``0.20`` only clears 20% per cycle, leaving ~60% of pass-1 content in cache through pass 2): .. code-block:: bash lmcache server --l1-size-gb --eviction-policy LRU \ --eviction-trigger-watermark 0.80 \ --eviction-ratio 0.99 The workload itself sleeps 5 seconds between pass 1 (warmup) and pass 2 (measured), so LMCache's 1Hz batched-eviction polling thread has time to actually run. Without that sleep, fast benchmarks complete before any eviction fires. random-prefill ^^^^^^^^^^^^^^ Fires all requests simultaneously with ``max_tokens=1`` to measure pure prefill performance. No warmup phase. .. list-table:: :header-rows: 1 :widths: 35 10 55 * - Flag - Default - Description * - ``--rp-request-length`` - 10000 - Token length per prefill request. * - ``--rp-num-requests`` - 50 - Number of requests to fire. **Example:** .. code-block:: bash lmcache bench engine \ --engine-url http://localhost:8000 \ --workload random-prefill \ --lmcache-url http://localhost:8080 \ --rp-request-length 15000 \ --rp-num-requests 100 Interactive Mode ~~~~~~~~~~~~~~~~ .. image:: /_static/bench_interactive_demo.gif :alt: Interactive mode demo :width: 100% When ``--engine-url`` or ``--workload`` is not provided (and ``--no-interactive`` is not set), the tool enters interactive mode. It guides you through four phases: 1. **Required settings** -- engine URL, workload type, LMCache server (or tokens per GB). 2. **General settings** (optional gate) -- model name, KV cache volume. 3. **Workload settings** (optional gate) -- workload-specific parameters. 4. **Summary and action** -- review configuration, then start the benchmark or export to a JSON file. Each prompt focuses on a single setting. Selection prompts use arrow keys; text and number prompts accept typed input with defaults shown in brackets. .. code-block:: text ══════════════════════════════════════════════════ lmcache bench engine -- Interactive Setup ══════════════════════════════════════════════════ Engine URL URL of the inference engine. [default: http://localhost:8000] > Workload The type of benchmark workload to run. Use up/down to navigate, Enter to select. * long-doc-qa Repeated Q&A over long documents multi-round-chat Multi-turn chat with stateful sessions long-doc-permutator Permutations of context documents prefix-suffix-tuner Two-pass tiered KV-cache demonstrator random-prefill Prefill-only requests fired simultaneously LMCache Server Do you have a running LMCache server? It can auto-detect KV cache size information. [default: Y] (Y/n) > ... ────────────────────────────────────────────────── Configuration Summary ────────────────────────────────────────────────── Workload: long-doc-qa Model: Qwen/Qwen3-14B Tokens per GB: 6553 ... ────────────────────────────────────────────────── What would you like to do? * Start benchmark Export configuration for later use and exit When you choose "Export configuration", all auto-detected values (model name, tokens per GB) are resolved and saved to a portable JSON file. Config File ~~~~~~~~~~~ Config files store benchmark parameters but **not** environment-specific values like engine URL or LMCache URL. This lets you reuse the same config across different environments. You can create a config file in three ways: 1. **Interactive mode** -- choose "Export configuration" at the summary step. 2. **``--export-config``** -- resolve and export from CLI without running. 3. **Manually** -- write JSON with keys matching CLI arg names (dashes replaced by underscores). Example config file: .. code-block:: json { "model": "Qwen/Qwen3-14B", "workload": "long-doc-qa", "tokens_per_gb_kvcache": 6553, "kv_cache_volume": 100.0, "ldqa_document_length": 10000, "ldqa_query_per_document": 2, "ldqa_shuffle_policy": "random", "ldqa_num_inflight_requests": 3 } Load it with ``--config`` (engine URL must be provided separately): .. code-block:: bash lmcache bench engine --engine-url http://localhost:8000 \ --config my_bench.json CLI arguments override config file values, so you can use a base config and tweak individual settings: .. code-block:: bash # Use saved config but override KV cache volume lmcache bench engine --engine-url http://localhost:8000 \ --config my_bench.json --kv-cache-volume 200 Output ~~~~~~ Terminal (real-time progress) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ During the benchmark, a live progress display shows in-flight requests, average TTFT, decode speed, and throughput. Suppress it with ``-q``. Terminal (final summary) ^^^^^^^^^^^^^^^^^^^^^^^^ After completion, a summary table is printed: .. code-block:: text ======= Engine Benchmark Result (long-doc-qa) ======== ---------------------- Configuration ------------------ Engine URL: http://localhost:8000 Model: Qwen/Qwen3-14B Workload: long-doc-qa ------------------------- Results --------------------- Successful requests: 20 Failed requests: 0 Benchmark duration (s): 31.34 Total input tokens: 200000 Total output tokens: 2560 Input throughput (tok/s): 6381.62 Output throughput (tok/s): 81.69 --------------- Time to First Token ------------------- Mean TTFT (ms): 313.41 P50 TTFT (ms): 272.83 P90 TTFT (ms): 587.21 P99 TTFT (ms): 837.32 ------------------ Decoding Speed --------------------- Mean decode (tok/s): 48.23 P99 decode (tok/s): 38.55 ====================================================== CSV and JSON ^^^^^^^^^^^^ - ``bench_results.csv`` -- per-request metrics (TTFT, latency, decode speed, token counts). Written by default; skip with ``--no-csv``. - ``bench_summary.json`` -- aggregate statistics with percentiles and config metadata. Opt-in with ``--json``. Both files are written to ``--output-dir`` (default: current directory). Exit Codes ~~~~~~~~~~ .. list-table:: :header-rows: 1 :widths: 15 85 * - Code - Meaning * - ``0`` - All requests succeeded. * - ``1`` - One or more requests failed. .. _lmcache-bench-server: server ------ The ``lmcache bench server`` command is an end-to-end sanity test for the LMCache Multi-Process (MP) cache server. It connects to a running server over ZMQ and exercises the full KV-cache data path for a sequence of synthetic requests, then optionally verifies per-chunk checksums through the HTTP API. .. code-block:: bash lmcache bench server [options] Unlike :ref:`lmcache bench engine `, this command does **not** require an inference engine. It only needs a running LMCache MP server (ZMQ + HTTP). GPU mode additionally requires a CUDA-capable device. It also requires the full ``lmcache`` install (not the lightweight ``lmcache-cli`` package). What it does ~~~~~~~~~~~~ For each sequence in ``[--start, --end)``, the tool runs two passes: 1. **Cold pass** -- ``LOOKUP`` is expected to miss, so the generated KV tensors are ``STORE``\ d on the server. 2. **Warm pass** -- ``LOOKUP`` is expected to hit; the tool issues ``RETRIEVE`` and compares the retrieved KV chunks' checksums to the originals. The full RPC path exercised is:: REGISTER_KV_CACHE → GET_CHUNK_SIZE → LOOKUP → QUERY_PREFETCH_STATUS → RETRIEVE → STORE → END_SESSION When ``--url`` points to the server's HTTP endpoint, per-chunk checksums are additionally cross-checked against the server-side computation, so a mismatch between producer and consumer surfaces as a loud ``CHECKSUM MISMATCH`` log line. Quick start ~~~~~~~~~~~ Start the MP server in one terminal: .. code-block:: bash lmcache server \ --host localhost --port 15556 \ --chunk-size 256 --l1-size-gb 5 \ --eviction-policy LRU --max-workers 1 Then in another terminal: .. code-block:: bash lmcache bench server \ --rpc-url tcp://localhost:15556 \ --url http://localhost:8080 By default the tool runs forever (``--end`` unset); stop it with ``Ctrl-C`` at any time. Pass ``--end N`` for a bounded run. Options ~~~~~~~ .. list-table:: :header-rows: 1 :widths: 30 15 55 * - Flag - Default - Description * - ``--rpc-url URL`` - ``tcp://localhost:5555`` - ZMQ endpoint of the MP cache server. * - ``--url URL`` - ``http://localhost:8080`` - HTTP base URL of the server's checksum API. Used to verify per-chunk checksums end-to-end. * - ``--mode {gpu,cpu}`` - ``gpu`` - Run mode. ``gpu`` allocates real CUDA tensors and uses CUDA IPC (lmcache-driven handle path). ``cpu`` allocates POSIX-SHM-backed tensors and uses the engine-driven (worker-side gather/scatter) path by default. * - ``--transfer-mode {auto,engine_driven,lmcache_driven}`` - ``auto`` - Transport routing for STORE/RETRIEVE. ``lmcache_driven`` forces the single-shot handle path (``REGISTER_KV_CACHE`` + ``STORE``/``RETRIEVE``), which supports both CUDA IPC and CPU SHM zero-copy transfers. ``engine_driven`` forces the worker-side gather/scatter path (``REGISTER_KV_CACHE_ENGINE_DRIVEN_CONTEXT`` + ``PREPARE``/``COMMIT``). ``auto`` maps gpu→lmcache_driven and cpu→engine_driven. * - ``--num-tokens N`` - ``512`` - Tokens per synthetic request. * - ``--num-blocks N`` - ``1024`` - Number of paged blocks allocated on the GPU. * - ``--block-size N`` - ``16`` - Tokens per paged block. * - ``--start N`` - ``0`` - First sequence number to run. * - ``--end N`` - *(unset)* - Exclusive upper bound on sequence numbers. When omitted the loop runs forever. * - ``--interval SECS`` - ``0.5`` - Delay between successive sub-passes. * - ``--kvcache-shape-spec SPEC`` - ``(2,1024,16,8,128):float16:32`` - KV cache shape spec (see below). * - ``--format FORMAT`` - ``terminal`` - Stdout output format for the final metrics summary. Available: ``terminal``, ``json``. * - ``--output PATH`` - *(unset)* - Save the final metrics summary to a file at PATH (format chosen by ``--format``). * - ``-q`` / ``--quiet`` - *(unset)* - Suppress all progress messages during the run. Only the final structured metrics summary is emitted (unless also redirected via ``--output``). CPU mode (no GPU) ~~~~~~~~~~~~~~~~~ ``--mode cpu`` runs the same end-to-end path without a GPU. The server runs on a CPU-only host (``StubCPUDevice``); the bench tool allocates POSIX-SHM-backed KV tensors and exercises the full RPC path. By default ``--mode cpu`` uses the engine-driven gather/scatter path (``auto`` → ``cpu→engine_driven``). To use the zero-copy SHM handle path instead, pass ``--transfer-mode lmcache_driven``: .. code-block:: bash # Terminal 1 -- start the LMCache server (no GPU required) lmcache server \ --host localhost --port 5555 \ --l1-size-gb 2 --eviction-policy LRU # Terminal 2 -- run bench in CPU + lmcache_driven mode lmcache bench server \ --rpc-url tcp://localhost:5555 \ --url http://localhost:8080 \ --mode cpu --transfer-mode lmcache_driven \ --start 0 --end 2 KV cache shape spec ~~~~~~~~~~~~~~~~~~~ The ``--kvcache-shape-spec`` flag describes how KV tensors are laid out on the GPU. A spec is one or more groups separated by ``;``: .. code-block:: text (kv_size,NB,BS,NH,HS):dtype:layers[;(...):dtype:layers...] Fields: * ``kv_size`` -- 2 for classical attention (separate K/V), 1 for MLA. * ``NB`` -- number of paged blocks. * ``BS`` -- block size (tokens per block). * ``NH`` -- number of attention heads per layer. * ``HS`` -- head size (in elements). * ``dtype`` -- element dtype (e.g. ``float16``, ``bfloat16``, ``float32``, ``uint8``). The full set matches the keys of ``DTYPE_MAP`` in ``lmcache/v1/kv_layer_groups.py``. * ``layers`` -- number of layers in this group. Multi-group specs let you model heterogeneous layers (for example, MLA layers + classical attention layers in the same model): .. code-block:: bash lmcache bench server \ --rpc-url tcp://localhost:15556 \ --kvcache-shape-spec "(1,1024,16,1,128):float16:4;(2,1024,16,8,128):float16:28" All groups must share the same ``NB`` and ``BS`` (this is a physical constraint of paged KV). Layer counts across groups sum to the total layer count registered with the server. See ``parse_kvcache_shape_spec`` in ``lmcache/v1/kv_layer_groups.py`` for the authoritative parsing rules and validation errors. Output ~~~~~~ After the run completes (or is interrupted with ``Ctrl-C``), a structured metrics summary is printed. The summary includes: * **Configuration** -- RPC URL, mode, transfer mode, tokens per request, interval. * **Results** -- total requests, checksum OK / FAIL counts, pass rate. * **Latency sections** -- per-operation latency statistics (count, mean, min, max, p50, p99) for cold lookup, cold store, warm lookup, and warm retrieve. Use ``--format json`` to get machine-readable output, or ``--output FILE`` to save the summary to a file. .. code-block:: text ================ Server Bench Result ================= ---------------------- Configuration ----------------- RPC URL: tcp://localhost:15556 Mode: gpu Transfer mode: auto Tokens / request: 512 Interval (s): 0.5 ------------------------- Results -------------------- Total requests: 3 Checksum OK: 3 Checksum FAIL: 0 Pass rate (%): 100.0 -------------------- Cold Lookup (ms) --------------- count: 3 mean: 1.647 min: 1.312 max: 1.823 p50: 1.647 p99: 1.823 --------------------- Cold Store (ms) --------------- count: 3 mean: 1.740 min: 1.521 max: 1.982 p50: 1.740 p99: 1.982 -------------------- Warm Lookup (ms) --------------- count: 3 mean: 1.310 min: 1.102 max: 1.512 p50: 1.310 p99: 1.512 ------------------- Warm Retrieve (ms) -------------- count: 3 mean: 1.480 min: 1.321 max: 1.612 p50: 1.480 p99: 1.612 ===================================================== Example output (progress) ~~~~~~~~~~~~~~~~~~~~~~~~~ During the run, progress messages are printed to stdout (suppressed by ``-q`` / ``--quiet``): .. code-block:: text Connecting to LMCache MP Server at tcp://localhost:15556 (mode=gpu) ... Server chunk_size = 256 Resolved KV shape spec: (2,1024,16,8,128):float16:32 [seq=0] LOOKUP cold: 0/2 chunks hit (1.82 ms) [seq=0] STORE: 2 chunks stored (1.74 ms) [seq=0] LOOKUP warm: 2/2 chunks hit (1.31 ms) [seq=0] RETRIEVE: 2 chunks retrieved (1.48 ms) [seq=0] CHECKSUM MATCH OK [seq=1] ... Any ``CHECKSUM MISMATCH``, ``ERROR``, or Python traceback in the log indicates a real problem worth investigating. Exit codes ~~~~~~~~~~ .. list-table:: :header-rows: 1 :widths: 15 85 * - Code - Meaning * - ``0`` - Test loop completed (or was interrupted cleanly with Ctrl-C) with no checksum mismatches. * - ``1`` - Fatal error (for example, CUDA unavailable in ``--mode gpu``, server unreachable, or a checksum mismatch). .. _lmcache-bench-l2: l2 -- The ``lmcache bench l2`` command benchmarks an L2 cache adapter (e.g. the local-filesystem adapter) end-to-end through the same ``parse_args_to_l2_adapters_config`` + ``create_l2_adapter`` pipeline that LMCache uses in production. Any registered adapter type can be tested without code changes: you describe the adapter with a single JSON spec and pick the operations to exercise. .. code-block:: bash lmcache bench l2 [options] Unlike :ref:`lmcache bench engine `, this command does **not** require an inference engine or an LMCache MP server. It only needs the adapter's own backing storage to be reachable (for the ``fs`` adapter, that simply means a writable directory). What it does ~~~~~~~~~~~~ For each measured operation the tool drives the adapter directly via its public submit/wait API: * ``Store`` -- ``submit_store_task`` writes ``num_keys`` MemoryObjs per submit and waits for the store eventfd. * ``Lookup`` -- ``submit_lookup_and_lock_task`` checks key existence (no payload transfer) and waits for the lookup eventfd. * ``Load`` -- ``submit_load_task`` reads ``num_keys`` MemoryObjs per submit and waits for the load eventfd. Each measured **round** issues ``--in-flight`` submits sequentially from a single producer thread and then waits for all of them to complete; the round duration is the wall-clock time from the first submit until the last completion. Warmup rounds run before measurement and their results are discarded from the final summary. All three operations share the same key idx universe, so running ``--only store`` followed by ``--only load`` (or ``--only lookup``) with identical other flags hits exactly the same keys. This makes the benchmark useful as a quick regression test for adapters that should support a clean store -> load round-trip. .. note:: When ``--only`` is not given, the three operations are run **in a single process in the order** ``store -> lookup -> load``. For adapters whose backing storage sits behind an OS-level cache -- most notably the local-filesystem (``fs``) adapter, which is subject to the Linux **page cache** -- this means ``lookup`` and ``load`` will almost always observe the data that ``store`` just wrote still hot in RAM, and the reported numbers reflect page-cache throughput rather than the underlying device. To benchmark each operation against a cold cache, run them separately with ``--only`` and drop the OS caches in between, for example:: lmcache bench l2 --l2-adapter '...' --only store sync && echo 3 | sudo tee /proc/sys/vm/drop_caches lmcache bench l2 --l2-adapter '...' --only lookup sync && echo 3 | sudo tee /proc/sys/vm/drop_caches lmcache bench l2 --l2-adapter '...' --only load For adapters that bypass the page cache (e.g. ``fs`` with ``"use_odirect": true``) or that talk to a remote service without a local cache, the default combined run is usually fine. O_DIRECT adapters may also require the benchmark L1 buffer to satisfy the adapter's block alignment. Use ``--l1-align-bytes`` to set that alignment, commonly ``4096`` for local block devices. The payload size (``--data-size-kb * 1024``) must be a multiple of the selected alignment. Quick start ~~~~~~~~~~~ Benchmark the local filesystem adapter with default parameters: .. code-block:: bash lmcache bench l2 \ --l2-adapter '{"type":"fs","base_path":"/tmp/lmcache-bench"}' This runs all three operations (store, lookup, load) with one warmup round and one measurement round. Stress the adapter with more in-flight submits and larger payloads: .. code-block:: bash lmcache bench l2 \ --l2-adapter '{"type":"fs","base_path":"/data/lmcache-bench","relative_tmp_dir":"tmp"}' \ --num-keys 32 --in-flight 4 \ --data-size-kb 512 \ --rounds 5 --warmup-rounds 1 Benchmark an O_DIRECT adapter with aligned L1 buffers: .. code-block:: bash lmcache bench l2 \ --l2-adapter '{"type":"raw_block","device_path":"/dev/nvme0n1","slot_bytes":4194304,"use_odirect":true,"block_align":4096}' \ --data-size-kb 1024 \ --l1-align-bytes 4096 Run only one operation (useful to isolate store vs. load throughput): .. code-block:: bash lmcache bench l2 \ --l2-adapter '{"type":"fs","base_path":"/tmp/lmcache-bench"}' \ --only store Lookup with a controlled hit rate (the benchmark splits the lookup keys between a potentially-existing range and a guaranteed-non-existent range): .. code-block:: bash lmcache bench l2 \ --l2-adapter '{"type":"fs","base_path":"/tmp/lmcache-bench"}' \ --only lookup --lookup-max-hit-rate 0.5 Enable a store -> load round-trip data integrity check on the last measured round: .. code-block:: bash lmcache bench l2 \ --l2-adapter '{"type":"fs","base_path":"/tmp/lmcache-bench"}' \ --no-skip-verify If you prefer to keep the JSON spec out of the command line, set the ``L2_ADAPTER_JSON`` environment variable instead of passing ``--l2-adapter``: .. code-block:: bash export L2_ADAPTER_JSON='{"type":"fs","base_path":"/tmp/lmcache-bench"}' lmcache bench l2 --num-keys 32 --in-flight 2 Options ~~~~~~~ .. list-table:: :header-rows: 1 :widths: 30 15 55 * - Flag - Default - Description * - ``--l2-adapter JSON`` - *(unset)* - L2 adapter spec as JSON with a ``"type"`` field plus adapter-specific configs, e.g. ``'{"type":"fs","base_path":"/tmp/bench"}'``. May be passed multiple times; only the first spec is benchmarked. If not provided, falls back to the ``L2_ADAPTER_JSON`` environment variable. Either the flag or the env var is **required**. * - ``--num-keys N`` - ``32`` - Number of keys per submit. * - ``--in-flight N`` - ``1`` - In-flight submits per round. Each round issues this many submits sequentially from a single producer thread, then waits for all of them. * - ``--data-size-kb N`` - ``256`` - Data size per key, in KiB. * - ``--l1-align-bytes N`` - ``1`` - Alignment in bytes for benchmark L1 buffers. Use a value at least as large as the adapter's block alignment when benchmarking O_DIRECT backends, for example ``4096`` for local block devices. ``--data-size-kb * 1024`` must be a multiple of this value. * - ``--rounds N`` - ``1`` - Measurement rounds per operation. * - ``--warmup-rounds N`` - ``1`` - Warmup rounds run before measurement; their results are discarded. * - ``--lookup-max-hit-rate F`` - ``0.0`` - Upper bound on the lookup hit rate, in ``[0, 1]``. The benchmark requests ``floor(N * rate)`` keys from the potentially-existing range and ``N - hit`` keys from a guaranteed-non-existent range, where ``N`` is the total number of lookup keys. The actual hit rate may be lower if those keys were never stored in this run. * - ``--skip-verify`` / ``--no-skip-verify`` - ``--skip-verify`` - Skip the store -> load round-trip data integrity check (the default). Pass ``--no-skip-verify`` to enable verification on the last measured round; this requires both ``store`` and ``load`` to be exercised. * - ``--only {lookup,store,load}`` - *(unset)* - Run only the specified operation. When omitted, all three operations are run in the order ``store -> lookup -> load``. * - ``--flamegraph {on,off}`` - ``off`` - Capture a flame graph of the measured phases (``on``) or run the benchmark normally (``off``). When ``on``, the benchmark profiles itself and renders an SVG. Default ``off`` leaves benchmark behavior unchanged. See :ref:`Profiling / flame charts `. * - ``--flamegraph-mode {on-cpu,off-cpu}`` - ``on-cpu`` - Flame-graph mode for ``--flamegraph on``. ``on-cpu`` shows where CPU time goes; ``off-cpu`` shows time blocked on I/O / locks (best for I/O-bound adapters). * - ``--flamegraph-output PATH`` - *(auto)* - SVG output path. Default: ``/tmp/lmcache_bench_flames/..svg``. * - ``--flamegraph-dir DIR`` - *($FLAMEGRAPH_DIR or ~/FlameGraph)* - Directory with the FlameGraph scripts (``flamegraph.pl``, ``stackcollapse-perf.pl``). Adapter JSON spec ~~~~~~~~~~~~~~~~~ The ``--l2-adapter`` JSON is parsed by ``lmcache.v1.distributed.l2_adapters.config.parse_args_to_l2_adapters_config``, the same entry point LMCache uses everywhere else. The minimum required field is ``type``; all remaining fields are forwarded to the adapter implementation as keyword arguments. Example for the local-filesystem adapter: .. code-block:: json { "type": "fs", "base_path": "/data/lmcache-bench", "relative_tmp_dir": "tmp", "read_ahead_size": null, "use_odirect": false } See the source under ``lmcache/v1/distributed/l2_adapters/`` for the full list of adapter types and their accepted fields. Example output ~~~~~~~~~~~~~~ Per-round progress (suppressed by ``-q``): .. code-block:: text ============================================================ L2 Adapter Benchmark ============================================================ Adapter config : FSL2AdapterConfig L2 adapter JSON : {"type":"fs","base_path":"/data/lmcache-bench","relative_tmp_dir":"tmp"} Keys / submit : 32 In-flight / round : 3 Keys / round : 96 Data size / key : 256 KB Data / round : 24.00 MB Rounds : 1 (+ 1 warmup) Lookup max hit rate : 0.00% ============================================================ [Init] Creating adapter... [Init] Adapter created successfully (FSL2Adapter). [Store] Running 1 warmup + 1 measurement rounds... [Store] Round 1: 47.83 ms, success_keys=96/96 [Store] Round 2: 46.19 ms, success_keys=96/96 [Lookup] Running 1 warmup + 1 measurement rounds... [Lookup] Round 1: 5.36 ms, found=96/96 [Lookup] Round 2: 5.03 ms, found=96/96 [Load] Running 1 warmup + 1 measurement rounds... [Load] Round 1: 18.15 ms, loaded=96/96 [Load] Round 2: 17.63 ms, loaded=96/96 Final summary (one section per exercised operation): .. code-block:: text ====== L2 Adapter Benchmark Result (FSL2Adapter) ======= ----------------------- Configuration ------------------- Adapter: FSL2Adapter Keys / submit: 32 In-flight / round: 3 Data size / key (KB): 256 Measurement rounds: 1 Warmup rounds: 1 Lookup max hit rate: 0.0 --------------------------- Store ----------------------- Operation: Store Rounds: 1 Keys / round: 96 Total keys: 96 Total success: 96 Duration avg (ms): 46.19 ... Throughput avg (MB/s): 519.62 Avg ops/s: 2078.50 Avg latency / key (ms): 0.481 --------------------------- Lookup ---------------------- ... ---------------------------- Load ----------------------- ... ========================================================= Each operation section reports per-round duration statistics (avg / min / max / p50 / p99 / std), aggregate throughput (``avg_throughput_mbps`` -- 0 for ``Lookup`` since it has no payload), average key-rate (``avg_ops_per_sec``), and a per-key latency. For ``Lookup``, three additional fields are reported when ``--lookup-max-hit-rate`` is non-zero or some keys were found: * ``Expected max hit rate`` -- the configured upper bound. * ``Expected hit keys`` -- ``floor(total_keys * rate)``, scaled for the measured rounds only. * ``Actual hit rate`` -- the measured hit rate over the kept rounds. Round-trip verification ~~~~~~~~~~~~~~~~~~~~~~~~ When ``--no-skip-verify`` is passed and both ``store`` and ``load`` were run, the benchmark compares the load buffers from the last measured round against the byte pattern that ``store`` wrote (see ``make_memory_objects`` in ``lmcache/cli/commands/bench/l2_adapter_bench/data.py``): .. code-block:: text [Verify] Checking store -> load data integrity for last measured round... [Verify] OK Verification is **off** by default because the stricter byte pattern requires both the store and load object batches to stay resident so the loaded data can be compared against the original store pattern. .. _lmcache-bench-l2-profiling: Profiling / flame charts ~~~~~~~~~~~~~~~~~~~~~~~~~~ When ``--flamegraph on`` is passed, the benchmark profiles the L2 adapter's performance and renders a flame graph of the measured phases: .. code-block:: bash lmcache bench l2 \ --l2-adapter '{"type":"fs","base_path":"/data/lmcache-bench"}' \ --rounds 300 --flamegraph on --flamegraph-mode on-cpu # [Profile] on-cpu recording started (pid=12345) -> .../FSL2Adapter.oncpu.svg # [Profile] wrote /tmp/lmcache_bench_flames/FSL2Adapter.oncpu.svg The recorder samples every thread of the process (including native worker threads). Two modes are available via ``--flamegraph-mode``: * **on-cpu** (default) -- ``perf record``; where CPU cycles go (serialization, copies, hashing). * **off-cpu** -- ``offcputime-bpfcc`` (bcc); time spent blocked (waiting on I/O, locks, eventfds). Often the more informative view for I/O-bound adapters such as ``fs`` and ``s3``. Recording covers only the measured store/lookup/load work, so make the run long enough to collect samples (a few seconds is plenty -- use a large ``--rounds``). The SVG is written to ``--flamegraph-output`` (default ``/tmp/lmcache_bench_flames/..svg``). **Requirements.** Rendering needs Brendan Gregg's `FlameGraph `__ scripts (``--flamegraph-dir`` or ``FLAMEGRAPH_DIR``, default ``~/FlameGraph``; auto-cloned to a temp directory when absent). ``on-cpu`` needs ``perf`` (and ``kernel.perf_event_paranoid`` low enough to profile non-root, or run as root); ``off-cpu`` needs ``bcc`` (``offcputime-bpfcc``) and ``sudo``. The required tools are checked up front: when ``--flamegraph on`` is requested but a tool is missing, the benchmark exits with a non-zero status and an actionable message naming the missing tool instead of running unprofiled. The default ``--flamegraph off`` skips all of this and leaves benchmark behavior unchanged. .. note:: When comparing a change, profile each variant with the same ``--rounds`` and ``--flamegraph-mode`` and distinct ``--flamegraph-output`` paths (e.g. ``baseline.svg`` vs ``after.svg``) so the two flame graphs are directly comparable. Exit codes ~~~~~~~~~~ .. list-table:: :header-rows: 1 :widths: 15 85 * - Code - Meaning * - ``0`` - All requested operations completed and (when enabled) the round-trip verification passed. * - ``1`` - Adapter creation failed, round-trip verification failed, or an operation hit a fatal error (e.g. all rounds timed out). * - ``2`` - Invalid invocation: the ``--l2-adapter`` JSON / ``L2_ADAPTER_JSON`` env var was missing or could not be parsed, an option value was invalid, or ``--flamegraph on`` was requested but the profiling toolchain is unavailable.