Configuration Reference ======================= This page documents every CLI argument accepted by the LMCache multiprocess server. Arguments are grouped by the config module that defines them. .. contents:: :local: :depth: 2 MP Server --------- Source: ``lmcache/v1/multiprocess/config.py`` .. list-table:: :header-rows: 1 :widths: 30 15 55 * - Argument - Default - Description * - ``--instance-id`` - *(unset, default UUID v4)* - Stable identity of this MP server. Used as the coordinator membership key and projected onto the OTel ``service.instance.id`` resource attribute on every metric and span (so telemetry and coordinator membership share one id). When the flag is not passed, defaults to a random UUID v4 minted at startup. * - ``--host`` - ``localhost`` - Host address to bind the ZMQ server. * - ``--port`` - ``5555`` - Port to bind the ZMQ server. * - ``--chunk-size`` - ``256`` - Chunk size for KV cache operations (in tokens). * - ``--max-workers`` - ``1`` - Base number of worker threads. Sets the default for both the GPU (affinity) pool and the CPU (normal) pool. Can be overridden per-pool with ``--max-gpu-workers`` and ``--max-cpu-workers``. * - ``--max-gpu-workers`` - (inherits ``--max-workers``) - Worker threads for the GPU affinity pool (STORE/RETRIEVE). Requests from the same vLLM instance are always dispatched to the same thread, eliminating GPU transfer lock contention. * - ``--max-cpu-workers`` - (inherits ``--max-workers``) - Worker threads for the normal CPU pool (LOOKUP, etc.). * - ``--hash-algorithm`` - ``blake3`` - Hash algorithm for token-based operations. Choices: ``builtin``, ``sha256_cbor``, ``blake3``. * - ``--engine-type`` - ``default`` - Cache engine backend type. ``default`` uses standard prefix caching; ``blend`` selects the current CacheBlend V3 implementation (composes a ``BlendV3Module`` into the engine); ``blend_legacy`` selects the original CacheBlend (composes a ``BlendModule``). Both blend variants require ``--supported-transfer-mode`` to be ``lmcache_driven`` or ``auto``. Choices: ``default``, ``blend``, ``blend_legacy``. * - ``--supported-transfer-mode`` - ``auto`` - Which worker → server transfer paths the server loads. ``lmcache_driven`` enables only the server-driven transfer path (STORE/RETRIEVE, supports both CUDA IPC and CPU SHM); ``engine_driven`` enables only the non-GPU (PREPARE/COMMIT) transfer path; ``auto`` (default) loads both so workers of either device type can connect without manual configuration. Choices: ``lmcache_driven``, ``engine_driven``, ``auto``. * - ``--runtime-plugin-locations`` - ``[]`` - Zero or more paths to runtime plugin scripts or directories to launch alongside the server. Plugins are spawned by ``MPRuntimePluginLauncher`` and receive the full server config via the ``LMCACHE_RUNTIME_PLUGIN_CONFIG`` environment variable. * - ``--runtime-plugin-config`` - ``"{}"`` - JSON string of extra key-value config forwarded to runtime plugins via ``LMCACHE_RUNTIME_PLUGIN_EXTRA_CONFIG``. Example: ``'{"plugin.frontend.heartbeat_url": "http://localhost:5000/heartbeat"}'``. * - ``--script-allowed-imports`` - ``[]`` - Space-separated list of Python module names that scripts posted to the HTTP ``/run_script`` endpoint are allowed to import. Example: ``--script-allowed-imports numpy pandas``. * - ``--shm-name`` - *(not set)* - SHM segment name for non-GPU KV transfer (only used when the non-GPU path is loaded, i.e. ``--supported-transfer-mode`` is ``auto`` or ``engine_driven``). Not set (default): auto-allocate a shared-memory pool. ``""`` (empty string): disable SHM and force the pickle transfer path. Any other value: use that exact name for the SHM pool segment. * - ``--worker-reap-timeout-seconds`` - ``120.0`` - Silence budget (seconds) after which a worker that has sent at least one heartbeat PING but then gone quiet has its KV cache registration reaped, freeing the leaked GPU context and CUDA IPC handles. ``0`` disables reaping. Keep this at least 3x the engine adapter's ``lmcache.mp.heartbeat_interval`` (default 10s) so a few missed pings never reap a live worker; the adapter warns at startup if its interval is raised without raising this. * - ``--worker-registration-grace-seconds`` - ``3600.0`` - Silence budget (seconds) for a worker that registered but has never sent a PING (still warming up, or died before its first request). Must be >= ``--worker-reap-timeout-seconds``. Generous by default so slow model warmup is never mistaken for a dead worker. * - ``--enable-segmented-prefix`` - ``False`` - CacheBlend (``--engine-type blend``) only: on a mid-prefix L2 retrieve failure, retain the gapped prefix so the post-gap chunks stay L1-resident and only the dropped gap is recomputed, instead of truncating the prefix at the gap. No effect for other engines. See :doc:`/mp/l2_storage/fault_inject` for a way to exercise it. * - ``--separate-object-groups`` / ``--no-separate-object-groups`` - ``True`` - Split a hybrid model's kernel groups into one object group per cross-chunk attention window (full attention, each sliding-window size, mamba/GDN) at KV-cache registration. On by default; pass ``--no-separate-object-groups`` to keep all layers in a single full-attention object group. Transparent to correctness; a non-hybrid model always resolves to one object group. See :doc:`/mp/hybrid_models`. Lookup Hash Logging ------------------- Source: ``lmcache/v1/mp_observability/subscribers/logging/lookup_hash.py`` When enabled, the server publishes chunk hashes computed during ``lookup()`` as ``MP_LOOKUP`` events on the EventBus. The ``LookupHashLoggingSubscriber`` writes these to rotating JSONL files for offline analysis. Disabled by default. These arguments are part of the Observability group. .. list-table:: :header-rows: 1 :widths: 30 15 55 * - Argument - Default - Description * - ``--lookup-hash-log-dir`` - ``""`` (disabled) - Directory to write lookup hash JSONL files. An empty string disables logging. * - ``--lookup-hash-log-rotation-interval`` - ``21600`` (6 h) - Time interval in seconds before rotating to a new log file. * - ``--lookup-hash-log-rotation-max-size`` - ``104857600`` (100 MB) - Max file size in bytes before rotating even if the time interval has not elapsed. * - ``--lookup-hash-log-max-files`` - ``100`` - Max number of log files to keep. Oldest files are deleted when this limit is exceeded. HTTP Frontend ------------- Source: ``lmcache/v1/multiprocess/config.py`` The HTTP frontend is included when running ``lmcache server``. .. list-table:: :header-rows: 1 :widths: 30 15 55 * - Argument - Default - Description * - ``--http-host`` - ``0.0.0.0`` - Host to bind the HTTP (FastAPI/uvicorn) server. * - ``--http-port`` - ``8080`` - Port to bind the HTTP server. P2P --- Source: ``lmcache/v1/multiprocess/config.py`` These flags configure peer-to-peer KV cache sharing between MP servers (see :doc:`p2p`). They are registered by ``add_p2p_args()`` on the ``lmcache server`` parser. P2P is enabled when ``--p2p-advertise-url`` is set, which additionally requires a coordinator URL via ``--coordinator-url`` (or ``LMCACHE_COORDINATOR_URL``). .. list-table:: :header-rows: 1 :widths: 30 15 55 * - Argument - Default - Description * - ``--p2p-advertise-url`` - ``""`` (P2P disabled) - Transfer-channel server ``host:port`` this instance advertises to peers. Setting it enables P2P (also requires ``--coordinator-url``). * - ``--p2p-listen-url`` - ``""`` - Transfer-channel server ``host:port`` to bind. Defaults to ``--p2p-advertise-url``. * - ``--p2p-lookup-timeout`` - ``30.0`` - Seconds before a peer lookup result counts as a miss. * - ``--p2p-load-timeout`` - ``30.0`` - Seconds before a peer load counts as a failure. * - ``--p2p-transfer-engine`` - ``nixl`` - Transfer-channel implementation to use. L1 Memory Manager ------------------ Source: ``lmcache/v1/distributed/config.py`` .. list-table:: :header-rows: 1 :widths: 30 15 55 * - Argument - Default - Description * - ``--l1-size-gb`` - *required* - Size of the L1 tier in GB. Sizes the pinned-DRAM L1 by default, or the GDS slab file when ``--gds-l1-path`` is set (see *GDS L1 Tier* below). * - ``--l1-use-lazy`` / ``--no-l1-use-lazy`` - ``True`` - Enable or disable lazy allocation for L1 memory. Pass ``--l1-use-lazy`` to enable (default) or ``--no-l1-use-lazy`` to explicitly disable. * - ``--l1-init-size-gb`` - ``20`` - Initial allocation size (GB) when using lazy allocation. * - ``--l1-align-bytes`` - ``4096`` - Alignment size in bytes (default 4 KB). * - ``--l1-devdax-path`` - *(not set)* - Optional ``/dev/dax*`` device or mmap-able file to use as the L1 backing arena. When set, disable lazy allocation with ``--no-l1-use-lazy`` and disable SHM transfer advertising with ``--shm-name ""`` because the L1 bytes live in the DAX mapping. If a DAX L2 adapter with the same ``device_path`` is registered, that adapter's ``max_dax_size_gb`` is used as the L1 Device-DAX overflow size. GDS L1 Tier ----------- Source: ``lmcache/v1/distributed/config.py`` Opt-in. Setting ``--gds-l1-path`` switches the L1 medium from pinned DRAM to an NVMe slab file accessed via GPUDirect Storage DMA. The CPU pinned-DRAM tier is then disabled, and ``--l1-size-gb`` sizes the slab. Disable byte-array L2 adapters when this is on (the GDS tier exposes no L1 memory buffer for them to register). The DMA path is selected automatically by platform: **cuFile** (``libcufile.so``) on NVIDIA and **hipFile** (``libhipfile.so``, `ROCm/hipFile `_) on AMD ROCm. The same flags apply to both; no configuration change is needed to switch vendors. .. note:: AMD hipFile requires ROCm >= 7.2.0. The zero-copy GPUDirect fast path additionally needs a kernel built with ``CONFIG_PCI_P2PDMA``, ``amdgpu-dkms >= 30.20.1``, and the slab on a local NVMe ext4/xfs filesystem; where those are unavailable hipFile transparently falls back to a host-bounce compatibility path (correct, but not zero-copy). .. list-table:: :header-rows: 1 :widths: 30 15 55 * - Argument - Default - Description * - ``--gds-l1-path`` - Not set - NVMe directory for the GDS L1 slab. Setting this enables the GDS L1 tier; one shared slab per process lives at ``/lmcache_gds_slab.bin``. * - ``--gds-l1-use-direct-io`` / ``--no-gds-l1-use-direct-io`` - ``True`` - Open the slab with ``O_DIRECT`` (required for the GDS DMA fast path on ext4). L1 Manager TTLs ---------------- Source: ``lmcache/v1/distributed/config.py`` .. list-table:: :header-rows: 1 :widths: 30 15 55 * - Argument - Default - Description * - ``--l1-write-ttl-seconds`` - ``600`` - Time-to-live for each object's write lock (seconds). * - ``--l1-read-ttl-seconds`` - ``300`` - Time-to-live for each object's read lock (seconds). Eviction Policy --------------- Source: ``lmcache/v1/distributed/config.py`` .. list-table:: :header-rows: 1 :widths: 30 15 55 * - Argument - Default - Description * - ``--eviction-policy`` - *required* - Eviction policy. Choices: ``LRU``, ``IsolatedLRU``, ``noop``. Use ``noop`` for buffer-only mode where L1 acts as a pure write buffer (data is deleted from L1 after L2 store). ``IsolatedLRU`` maintains one LRU list per ``cache_salt`` and requires per-``cache_salt`` quotas to be configured at runtime via the ``/quota`` HTTP endpoints (see :ref:`mp-http-quota-api`); a ``cache_salt`` with no registered quota has an effective limit of ``0`` bytes, so its data is evicted at the next eviction cycle (allowlist semantics). * - ``--eviction-trigger-watermark`` - ``0.8`` - Memory usage ratio (0.0--1.0) that triggers eviction. * - ``--eviction-ratio`` - ``0.2`` - Fraction of allocated memory to evict when triggered (0.0--1.0). L2 Policies ----------- Source: ``lmcache/v1/distributed/config.py`` .. list-table:: :header-rows: 1 :widths: 30 15 55 * - Argument - Default - Description * - ``--l2-store-policy`` - ``default`` - L2 store policy. Determines which adapters receive each key and whether keys are deleted from L1 after L2 store. The ``default`` policy stores all keys to all adapters and keeps L1. The ``skip_l1`` policy stores all keys to all adapters and then deletes them from L1 (buffer-only mode). Choices: ``default``, ``skip_l1``. * - ``--l2-prefetch-policy`` - ``default`` - L2 prefetch policy. Determines which adapter loads each key when multiple adapters have it. The ``default`` policy picks the first adapter (lowest index). Prefetched keys are temporary (deleted after the reader finishes). The ``retain`` policy uses the same load plan but keeps prefetched keys permanently in L1. Choices: ``default``, ``retain``. * - ``--l2-prefetch-max-in-flight`` - ``8`` - Maximum number of concurrent prefetch (L2 load) requests. Limits how many in-flight loads the PrefetchController may issue at once, preventing excessive L1 memory pressure. * - ``--periodic-notifier-interval-ms`` - ``5`` - Interval in milliseconds for the periodic event notifier heartbeat. A native C++ background thread writes to all registered file descriptors at this interval, waking controller poll loops for L2 adapters that lack native async completion callbacks. L2 Adapters ----------- Source: ``lmcache/v1/distributed/l2_adapters/config.py`` L2 adapters are configured via repeatable ``--l2-adapter `` arguments. Each JSON object must include a ``"type"`` field that selects the adapter type. The order of ``--l2-adapter`` arguments determines the adapter order (cascade). Registered adapter types: ``nixl_store``, ``nixl_store_dynamic``, ``fs``, ``fs_native``, ``mock``, ``mooncake_store``, ``aerospike``, ``s3``, ``resp``, ``plugin``, ``native_plugin``, ``raw_block``, ``dax``. Each adapter type's required and optional fields, plus per-backend examples, are documented on its own page under :doc:`Secondary KV Storage ` -- including the adapters not detailed inline here (``fs_native``, ``raw_block``, ``dax``, ``mooncake_store``, ``aerospike``, ``hfbucket``, ``resp``). Multiple adapters (cascade) ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Pass ``--l2-adapter`` multiple times. Adapters are used in the order given: .. code-block:: bash --l2-adapter '{"type": "nixl_store", "backend": "POSIX", "backend_params": {"file_path": "/data/ssd/l2", "use_direct_io": "false"}, "pool_size": 64}' \ --l2-adapter '{"type": "nixl_store", "backend": "GDS", "backend_params": {"file_path": "/data/nvme/l2", "use_direct_io": "true"}, "pool_size": 128}' Observability ------------- Source: ``lmcache/v1/mp_observability/config.py`` See :doc:`observability/index` for full details on the three modes (metrics, logging, tracing). .. list-table:: :header-rows: 1 :widths: 30 15 55 * - Argument - Default - Description * - ``--disable-observability`` - off - Master switch: disable the EventBus entirely. * - ``--disable-metrics`` - off - Skip metrics subscribers (no Prometheus endpoint). * - ``--disable-logging`` - off - Skip logging subscribers. * - ``--enable-tracing`` - off - Register tracing subscribers. Requires ``--otlp-endpoint``. * - ``--event-bus-queue-size`` - ``10000`` - Max events in the EventBus queue before tail-drop. * - ``--otlp-endpoint`` - *(none)* - OTLP gRPC endpoint for exporting metrics and traces. * - ``--prometheus-port`` - ``9090`` - Port for the Prometheus ``/metrics`` endpoint. vLLM Client Configuration -------------------------- On the vLLM side, specify the LMCache server host and port via the ``kv_connector_extra_config`` parameter: .. code-block:: bash vllm serve Qwen/Qwen3-14B \ --kv-transfer-config \ '{"kv_connector":"LMCacheMPConnector", "kv_role":"kv_both", "kv_connector_extra_config": {"lmcache.mp.host": "tcp://127.0.0.1", "lmcache.mp.port": 6000}}' To target multiple LMCache servers from a single vLLM deployment, pass a list (or comma-separated string) of server URLs via ``lmcache.mp.server_urls``. When set, ``server_urls`` takes precedence over the single-server ``host`` / ``port`` keys; vLLM's world size must be divisible by the number of servers, and each worker connects only to its locally-assigned server (global ranks are sliced into contiguous blocks, one block per server). Multi-server mode currently supports tensor parallelism only -- pipeline parallelism (``pp_size > 1``) and data parallelism (``dp_size > 1``) are rejected with a clear error. .. code-block:: bash vllm serve Qwen/Qwen3-14B \ --tensor-parallel-size 4 \ --kv-transfer-config \ '{"kv_connector":"LMCacheMPConnector", "kv_role":"kv_both", "kv_connector_extra_config": {"lmcache.mp.server_urls": "tcp://host1:6667,tcp://host2:6667"}}' ``LMCacheMPConnector`` reads the following keys from ``kv_connector_extra_config``: Connector ``extra_config`` Keys ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ All connector-level options are passed through ``kv_connector_extra_config`` and use the ``lmcache.mp.`` prefix. .. list-table:: :header-rows: 1 :widths: 30 15 55 * - Key - Default - Description * - ``lmcache.mp.server_urls`` - *(unset)* - Multi-server deployment: list (or comma-separated string) of ``://:`` URLs, e.g. ``"tcp://host1:6667,tcp://host2:6667"``. When set, takes precedence over ``lmcache.mp.host`` / ``lmcache.mp.port``; the vLLM world size must be divisible by the number of servers, and each worker connects to its locally-assigned server. * - ``lmcache.mp.host`` - ``tcp://localhost`` - Single-server deployment: host (with ZMQ transport prefix) of the LMCache MP server. Ignored when ``lmcache.mp.server_urls`` is set. * - ``lmcache.mp.port`` - ``5555`` - Single-server deployment: port of the LMCache MP server. Must match the server's ``--port``. Ignored when ``lmcache.mp.server_urls`` is set. * - ``lmcache.mp.mq_timeout`` - ``300.0`` - Timeout (seconds) for blocking message-queue requests, including the initial chunk-size query and KV cache registration/unregistration. If the server does not respond within this window, the connector raises ``ConnectionError`` on startup. * - ``lmcache.mp.heartbeat_interval`` - ``10.0`` - Interval (seconds) between periodic heartbeat pings sent from the connector to the server. * - ``lmcache.mp.mp_transfer_mode`` - ``auto`` - Routing mode for the worker -> server transfer context. One of ``auto`` (CUDA -> lmcache_driven, others -> engine_driven), ``lmcache_driven`` (force the IPC / SHM zero-copy handle path — LMCache server pulls data via device handles), or ``engine_driven`` (force the worker-side gather/scatter copy path). Overrides the ``LMCACHE_MP_TRANSFER_MODE`` env var when set. Environment Variables --------------------- .. list-table:: :header-rows: 1 :widths: 30 70 * - Variable - Description * - ``LMCACHE_LOG_LEVEL`` - Log level for LMCache (``DEBUG``, ``INFO``, ``WARNING``, ``ERROR``). Set to ``DEBUG`` to see L2 store activity, prefetch results, etc. * - ``PYTHONHASHSEED`` - Set to a fixed value for reproducible hashing across processes (relevant when using ``--hash-algorithm builtin``). * - ``LMCACHE_TRACK_USAGE`` - Set to ``false`` to disable anonymous usage statistics (see below). * - ``DO_NOT_TRACK`` - Set to ``1`` to disable anonymous usage statistics (cross-tool convention). Anonymous Usage Statistics -------------------------- At startup the MP server sends a one-time anonymous usage report to the LMCache stats server: an environment snapshot (cloud provider, CPU/GPU, memory) and a configuration snapshot (chunk size, transfer mode, L1 size/medium, L2 adapter types, policies). No prompts, keys, KV-cache data, model names, or the server's ``--instance-id`` are ever sent; reports are correlated only through random UUIDs. Reporting runs on a background thread, never delays startup, and a failure anywhere in reporting never affects serving. To opt out, set ``LMCACHE_TRACK_USAGE=false`` or ``DO_NOT_TRACK=1``. See :ref:`usage-stats-collection` for the full list of collected fields and configuration options. Full Example ------------ .. code-block:: bash lmcache server \ --host 0.0.0.0 \ --port 6555 \ --chunk-size 512 \ --max-workers 4 \ --max-gpu-workers 2 \ --hash-algorithm blake3 \ --engine-type default \ --lookup-hash-log-dir /data/lmcache/lookup_hashes \ --lookup-hash-log-rotation-interval 21600 \ --lookup-hash-log-rotation-max-size 104857600 \ --lookup-hash-log-max-files 100 \ --l1-size-gb 100 \ --l1-use-lazy \ --l1-init-size-gb 20 \ --l1-align-bytes 4096 \ --l1-write-ttl-seconds 600 \ --l1-read-ttl-seconds 300 \ --eviction-policy noop \ --l2-store-policy skip_l1 \ --eviction-trigger-watermark 0.9 \ --eviction-ratio 0.1 \ --l2-prefetch-policy default \ --l2-prefetch-max-in-flight 8 \ --periodic-notifier-interval-ms 5 \ --l2-adapter '{"type": "nixl_store", "backend": "POSIX", "backend_params": {"file_path": "/data/lmcache/l2", "use_direct_io": "false"}, "pool_size": 64}' \ --prometheus-port 9090 \ --metrics-sample-rate 0.01 \ --enable-tracing \ --otlp-endpoint http://localhost:4317