wehub/lmcache--lmcache
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:24:33 +08:00
commit f213ec8976
2101 changed files with 494002 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/kv_cache/async_loading.rst
+132
View File
@@ -0,0 +1,132 @@
Async Loading
=============
.. warning::
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance.
.. contents:: Table of Contents
:local:
:depth: 2
Overview
--------
This document explains the principle, benefits, differences from vLLM PR `19330 <https://github.com/vllm-project/vllm/pull/19330>`_, and limitations of the LMCache ``async_loading`` feature.
It focuses on LMCache v1 integration with vLLM and the internal storage pipeline.
Key change of components in this feature include:
- LMCache async lookup client/server (ZMQ-based)
- Storage manager orchestrating backends and concurrency
- Cache engine async API entrypoints
- vLLM adapter integration points
Principle and Theory
--------------------
At a high level, ``async_loading`` decouples scheduler-side lookup from worker-side prefetch/retrieval, allowing overlap between I/O and computation while preserving prefix-based correctness.
- The scheduler sends lookup requests with token chunk hashes and offsets.
- Worker-side servers perform tiered ``batched_async_contains`` over available backends and eagerly launch non-blocking batched get operations for hit prefixes.
- Completion is tracked via an ``EventManager`` to safely deliver loaded memory objects back to the requesting path.
- A weighted semaphore with an ``AsyncSerializer`` prevents allocator deadlocks by shaping concurrency according to chunk budget.
The following Mermaid sequence diagram illustrates the end-to-end flow:
.. mermaid::
sequenceDiagram
autonumber
participant S as Scheduler (vLLM)
participant LC as LMCacheAsyncLookupClient
participant WS as LMCacheAsyncLookupServer (Worker)
participant SM as StorageManager
participant BE as Backends (LocalCPU/LocalDisk/FSConnector)
participant EM as EventManager
S->>LC: lookup(token_ids, lookup_id, request_configs)
note right of LC: Hashes + offsets via TokenDatabase
LC->>WS: ZMQ PUSH multipart [lookup_id, hashes, offsets, configs]
WS->>SM: async_lookup_and_prefetch(lookup_id, keys, cum_chunk_lengths)
SM->>BE: batched_async_contains(lookup_id, keys, pin=True)
alt prefix hit across tiers
BE-->>SM: num_hit_chunks (per tier)
SM->>BE: batched_get_non_blocking(lookup_id, hit_prefix)
BE-->>SM: Future[List[MemoryObj]]
SM->>EM: add_event(EventType.LOADING, lookup_id, gather_all)
SM-->>WS: send_response_to_scheduler(lookup_id, retrieved_length)
WS-->>LC: ZMQ PUSH [lookup_id, num_hit_tokens]
else cache miss
SM-->>WS: send_response_to_scheduler(lookup_id, 0)
WS-->>LC: ZMQ PUSH [lookup_id, 0]
end
Architecture (Worker Side)
--------------------------
.. mermaid::
:align: center
flowchart LR
subgraph Worker
direction TB
A["LMCacheAsyncLookupServer<br/>ZMQ PULL/PUSH"]
B["StorageManager<br/>Async loop (thread)"]
C["AsyncSerializer<br/>WeightedSemaphore"]
D["EventManager<br/>EventType.LOADING"]
end
subgraph Backends
E["LocalCPUBackend<br/>contains/get"]
F["LocalDiskBackend<br/>async contains/get"]
G["FSConnector<br/>remote FS"]
end
A --> B
B --> C
B --> D
B -.contains/get.-> E
B -.contains/get.-> F
B -.contains/get.-> G
style E fill:#dff,stroke:#333,stroke-width:1px
style F fill:#ffd,stroke:#333,stroke-width:1px
style G fill:#dfd,stroke:#333,stroke-width:1px
Benefits
--------
- Performance overlap
- **I/O–Compute Overlap**: Decoupling lookup/prefetch from loading enables fetching KV chunks while vLLM continues scheduling/computation.
- Robustness and error handling
- **Event-driven Synchronization**: ``EventManager`` ensures safe hand-off of futures and avoids race conditions between threads and the async loop.
- **Backpressure & Deadlock Avoidance**: ``AsyncSerializer`` with a weighted semaphore caps concurrent chunk retrievals based on allocator budget, preventing starvation or allocator lockups.
- **Graceful Miss Path**: Immediate response with ``None`` hit tokens when nothing is retrievable; worker returns quickly without stalling the scheduler.
Comparison with vLLM Load Failure Recovery feature
---------------------------------------------------
The `VLLM_PR_19330 <https://github.com/vllm-project/vllm/pull/19330>`_ introduces a fault recovery mechanism for vLLM's KV connector infrastructure that enables graceful handling of KV cache load failures by automatically detecting failed block loads and rescheduling only affected requests for recomputation from a valid prefix.
By contrast, LMCache’s ``async_loading`` is an externalized caching layer with its own client/server, storage backends, and concurrency control.
Limitations
-----------
- Only works with vllm merged `VLLM_PR_23620 <https://github.com/vllm-project/vllm/pull/23620>`_
- Backend support constraint: This feature currently requires backends that implement ``batched_async_contains``; limited to a few backends, e.g.:
- ``LocalCpuBackend``
- ``LocalDiskBackend``
- ``S3Connector``
- ``FSConnector``
- ``RedisConnector/RedisClusterConnector``
Future Work
-----------
- Introduce a default ``batched_async_contains`` implementation, so all backends can support ``async_loading``.
- Add metrics and observability to track the number of asynchronous lookup requests and the number of occupied ``MemoryObj`` instances.
- Improve the lookup framework by passing vLLM prefix cache hit tokens so that async lookup can skip loading parts already hit in vLLM.
docs/source/kv_cache/caching_policies.rst
+13
View File
@@ -0,0 +1,13 @@
Using Different Caching Policies
===================================
.. warning::
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance.
LMCache supports multiple caching policies.
For example, to use LRU, you can set the environment variable ``LMCACHE_CACHE_POLICY=LRU`` or set it in the configuration file with ``cache_policy="LRU"``.
Currently, LMCache supports "LRU" (Least Recently Used), "MRU" (Most Recently Used), "LFU" (Least Frequently Used) and "FIFO" (First-In-First-Out) caching policies.
docs/source/kv_cache/multiprocess_mode.rst
+12
View File
@@ -0,0 +1,12 @@
:orphan:
Multiprocess Mode (Redirect)
=============================
.. note::
This page has moved. The multiprocess mode documentation is now at
:doc:`/mp/index`.
Please see the new :doc:`/mp/index` section for complete documentation
including quick start, configuration reference, L2 storage, deployment,
observability, and architecture guides.
docs/source/kv_cache/p2p_sharing.rst
+237
View File
@@ -0,0 +1,237 @@
.. _p2p_sharing:
P2P KV Cache Sharing
====================
.. warning::
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/p2p`.
P2P (Peer-to-Peer) KV cache sharing enables direct cache transfer between multiple serving engine instances without requiring a centralized cache server. This approach provides high-performance cache sharing with reduced latency and improved scalability, especially beneficial in distributed inference scenarios.
LMCache supports P2P sharing through a controller-based architecture using NIXL (NVIDIA Inference Xfer Library) for optimized data transfer between instances.
Prerequisites
-------------
- **Multi-GPU Setup**: Your server should have at least 2 GPUs
- **NIC**: RDMA is recommended for more performance.
- **NIXL**: Install from `NIXL <https://github.com/ai-dynamo/nixl>`_
- **vLLM**: v1 version is required, refer to :ref:`installation_guide` for details.
- **LMCache**: Install from :ref:`installation_guide`
Configuration
-------------
Create two configuration files for the P2P sharing setup.
**Instance 1 Configuration (example1.yaml)**:
.. code-block:: yaml
chunk_size: 256
local_cpu: True
max_local_cpu_size: 100
enable_async_loading: True
# P2P configurations
enable_p2p: True
p2p_host: "localhost"
p2p_init_ports: 8200
p2p_lookup_ports: 8201
transfer_channel: "nixl"
# Controller configurations
enable_controller: True
lmcache_instance_id: "lmcache_instance_1"
controller_pull_url: "localhost:8300"
controller_reply_url: "localhost:8400"
lmcache_worker_ports: 8500
extra_config:
lookup_backoff_time: 0.001
**Instance 2 Configuration (example2.yaml)**:
.. code-block:: yaml
chunk_size: 256
local_cpu: True
max_local_cpu_size: 100
enable_async_loading: True
# P2P configurations
enable_p2p: True
p2p_host: "localhost"
p2p_init_ports: 8202
p2p_lookup_ports: 8203
transfer_channel: "nixl"
# Controller configurations
enable_controller: True
lmcache_instance_id: "lmcache_instance_2"
controller_pull_url: "localhost:8300"
controller_reply_url: "localhost:8400"
lmcache_worker_ports: 8501
extra_config:
lookup_backoff_time: 0.001
Setup and Usage
---------------
**Step 1: Start the LMCache Controller**
.. code-block:: bash
PYTHONHASHSEED=123 lmcache_controller --host localhost --port 9000 --monitor-ports '{"pull": 8300, "reply": 8400, "heartbeat": 8082}'
Make sure that the 8300 and 8400 ports are set up in **controller_pull_url** and **controller_reply_url** in the configuration files.
Port 9000 is the controller main port, which is arbitrary and can be changed.
After starting the controller, access the WebUI at:
http://localhost:9000/
**Step 2: Start vLLM Engines with LMCache Workers**
If the NIC supports RDMA:
.. code-block:: bash
export UCX_TLS=rc
If the NIC does not support RDMA:
.. code-block:: bash
export UCX_TLS=tcp
Start vLLM engine 1 at port 8010:
.. code-block:: bash
PYTHONHASHSEED=123 CUDA_VISIBLE_DEVICES=0 LMCACHE_CONFIG_FILE=/path/to/example1.yaml \
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
--gpu-memory-utilization 0.8 \
--port 8010 \
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
Start vLLM engine 2 at port 8011:
.. code-block:: bash
PYTHONHASHSEED=123 CUDA_VISIBLE_DEVICES=1 LMCACHE_CONFIG_FILE=/path/to/example2.yaml \
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
--gpu-memory-utilization 0.8 \
--port 8011 \
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
**Step 3: Test P2P Cache Sharing**
Send a request to vLLM engine 1 to populate the cache:
.. code-block:: bash
curl -X POST http://localhost:8010/v1/completions \
-H "Content-Type: application/json" \
-d "{
\"model\": \"meta-llama/Meta-Llama-3.1-8B-Instruct\",
\"prompt\": \"$(printf 'Explain the significance of KV cache in language models.%.0s' {1..100})\",
\"max_tokens\": 10
}"
Send the same request to vLLM engine 2 to demonstrate cache retrieval from **engine 1**:
.. code-block:: bash
curl -X POST http://localhost:8011/v1/completions \
-H "Content-Type: application/json" \
-d "{
\"model\": \"meta-llama/Meta-Llama-3.1-8B-Instruct\",
\"prompt\": \"$(printf 'Explain the significance of KV cache in language models.%.0s' {1..100})\",
\"max_tokens\": 10
}"
Expected Output
---------------
When the second request successfully retrieves cache from the first instance, you should see logs similar to:
.. code-block:: bash
(EngineCore_DP0 pid=2577584)[2025-09-21 00:00:11,706] LMCache INFO:[0m Established connection to peer_init_url localhost:8200. The peer_lookup_url: localhost:8201 (p2p_backend.py:278:lmcache.v1.storage_backend.p2p_backend)
(EngineCore_DP0 pid=2577584)[2025-09-21 00:00:11,792] LMCache INFO: Retrieved 1002 out of total 1002 out of total 1002 tokens. size: 0.1223 gb, cost 60.3595 ms, throughput: 2.0264 GB/s; (cache_engine.py:496:lmcache.v1.cache_engine)
These logs indicate successful P2P connection establishment and high-throughput cache retrieval.
**Step 4: Benchmarking P2P Cache Sharing**
Send a request workload to instance 1 to populate the cache:
.. code-block:: bash
python benchmarks/long_doc_qa/long_doc_qa.py \
--model meta-llama/Meta-Llama-3.1-8B-Instruct \
--num-documents 50 \
--document-length 10000 \
--output-len 100 \
--repeat-count 1 \
--repeat-mode tile \
--port 8010 \
--max-inflight-requests 4
Send the same request workload to instance 2 to demonstrate cache retrieval from **instance 1**:
.. code-block:: bash
python benchmarks/long_doc_qa/long_doc_qa.py \
--model meta-llama/Meta-Llama-3.1-8B-Instruct \
--num-documents 50 \
--document-length 10000 \
--output-len 100 \
--repeat-count 1 \
--repeat-mode tile \
--port 8011 \
--max-inflight-requests 4
Benchmark Results
-----------------
First instance metrics:
.. code-block:: text
Warmup round mean TTFT: 2.286s
Warmup round time: 37.957s
Warmup round prompt count: 50
Warmup round successful prompt count: 50
=== BENCHMARK RESULTS ===
Query round mean TTFT: 2.028s
Query round time: 38.323s
Query round prompt count: 50
Query round successful prompt count: 50
Second instance metrics:
.. code-block:: text
Warmup round mean TTFT: 1.036s
Warmup round time: 13.814s
Warmup round prompt count: 50
Warmup round successful prompt count: 50
=== BENCHMARK RESULTS ===
Query round mean TTFT: 0.490s
Query round time: 7.964s
Query round prompt count: 50
Query round successful prompt count: 50
In this example, the warm-up round metric in long_doc_qa is used because no existing KV cache is reused within an instance to benefit solely from P2P sharing. With LMCache P2P sharing enabled, the time to first token (TTFT) is reduced by 54.7%, from 2.286 s to 1.036 s, with a 63.6% reduction in total inference time (37.957 s → 13.814 s).
docs/source/kv_cache/storage_backends/3fs.rst
+174
View File
@@ -0,0 +1,174 @@
3FS
====
.. warning::
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/l2_storage/index`.
.. _3FS-overview:
Overview
--------
3FS (Fire-Flyer File System) is an AI native distributed file-system which provides high performance and
low latency. It is a supported option for KV Cache offloading in LMCache. Even though the FSConnector
backend can work with 3FS storage cluster, but it access data in 3FS storage cluster by FUSE interfaces.
It can't leverage 3FS high performance. This particular backend uses 3FS native USRBIO(User Space Ring
Based IO) interfaces to access 3FS storage cluster which get high performance.
Configure LMCache 3FS Offloading
-----------------------------------------
Passed in through ``LMCACHE_CONFIG_FILE=your-lmcache-config.yaml``
Example ``config.yaml``:
.. code-block:: yaml
# 256 Tokens per KV Chunk
chunk_size: 256
local_cpu: False
# Plugin name mode, obtain base_path in extra_config session
remote_storage_plugins: ["hf3fs.primary"]
# URL mode, obtain base_path from the URL
#remote_url: "hf3fs:///3fs/stage/hello, /3fs/stage/world"
extra_config:
# base_path, for a plugin instance
remote_storage_plugin.hf3fs.primary.base_path: "/3fs/stage/dir1,/3fs/stage/dir2"
# base_path, for all plugin instances
#hf3fs_base_path: "/3fs/stage/dir1,/3fs/stage/dir2"
# Mount point of 3FS
hf3fs_mount_point: "/3fs/stage"
# Shared memory size for Iov in hf3fs client,
# range in [104857600(100MB), 2147483648(2GB)], default:209715200
hf3fs_iov_size: 209715200 #200MB
# Max num of concurrent requests that can be submitted in Ior
# range in [128,1024], default: 256
hf3fs_ior_entries: 256
# Control with I/O depth. 0, no control
# >0, only when io_depth requests are in queue, and issue them in one batch
# <0, wait for at most -io_depth requests are in queue and issue them in one batch
# range in [-128, 128], default: 0
hf3fs_io_depth: 0
# NUMA ID for Ior shared memory, -1 for current process NUMA ID.
hf3fs_numa_id: -1
# Number of io thread
# range in [2,16], default: 4
hf3fs_io_thread_num: 4
There are 2 methods to config hf3fs remote backend:
1. Plugin name mode, uses the parameter remote_storage_plugins(Recommend)
2. URL mode, uses the parameter remote_url(Deprecated, will be removed in a future)
For URL mode, the base_path is contained in the url. For plugin name mode, there are 2 methods to set base_path:
1. remote_storage_plugin.{plugin name}.base_path, it sets the base_path for a plugin instance.
e.g.:remote_storage_plugin.hf3fs.primary.base_path
2. hf3fs_base_path, it set the base_path for all plugin instances
Installation
-------------
.. _3FS-prerequisites:
**Prerequisites:**
- A Machine with at least one GPU. You can adjust the max model length of your vllm instance depending on your GPU memory.
- vllm and lmcache installed
**Step 1. Install 3FS hf3fs_py_usrbio package**
The inference server need install 3FS hf3fs_py_usrbio package, recommend to build the package from source:
.. code-block:: bash
git clone https://github.com/deepseek-ai/3fs
cd 3fs
git submodule update --init --recursive
./patches/apply.sh
Install dependencies
pip install -e .
`3FS Build <https://github.com/deepseek-ai/3FS/blob/main/README.md#build-3fs>`_
**Step 2. Setup 3FS storage cluster**
`3FS Setup <https://github.com/deepseek-ai/3FS/blob/main/deploy/README.md>`_
**Step 3. Deploy 3FS FUSE client in inference server**
The inference server must deploy 3FS FUSE client(a fuse daemon process provided by 3FS),
otherwise, it can't access 3FS storage cluster
`Setup 3FS FUSE Client <https://github.com/deepseek-ai/3FS/blob/main/deploy/README.md#step-8-fuse-client>`_
**Step 4. Start a vLLM server with 3FS offloading enabled**
Create a lmcache configuration file called: ``3fs-offload.yaml``
.. code-block:: yaml
# 256 Tokens per KV Chunk
chunk_size: 256
local_cpu: False
# support multiple paths
remote_storage_plugins: ["hf3fs.primary"]
extra_config:
# base_path
remote_storage_plugin.hf3fs.primary.base_path: "/3fs/stage/dir1,/3fs/stage/dir2"
# Mount point of 3FS
hf3fs_mount_point: "/3fs/stage"
# Shared memory size for Iov in hf3fs client,
# range in [104857600(100MB), 2147483648(2GB)], default:209715200 (200MB)
hf3fs_iov_size: 209715200
# Max num of concurrent requests that can be submitted in Ior
# range in [128,1024], default: 256
hf3fs_ior_entries: 256
# Control with I/O depth. 0, no control
# >0, only when io_depth requests are in queue, and issue them in one batch
# <0, wait for at most -io_depth requests are in queue and issue them in one batch
# range in [-128, 128], default: 0
hf3fs_io_depth: 0
# NUMA ID for Ior shared memory, -1 for current process NUMA ID.
hf3fs_numa_id: -1
# Number of io thread
# range in [2,16], default: 4
hf3fs_io_thread_num: 4
Start vllm:
.. code-block:: bash
export VLLM_USE_V1=0
export LMCACHE_USE_EXPERIMENTAL=True
export LMCACHE_LOG_LEVEL=INFO
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_ENABLE_V1_MULTIPROCESSING=1
LMCACHE_CONFIG_FILE="3fs-offload.yaml" \
vllm serve \
meta-llama/Llama-3.1-8B-Instruct \
--max-model-len 65536 \
--kv-transfer-config \
'{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
docs/source/kv_cache/storage_backends/azure.rst
+78
View File
@@ -0,0 +1,78 @@
Azure Blob Storage Backend
==========================
LMCache can offload KV cache to Azure Blob Storage via an async-native
connector (parity with the S3 backend, no NIXL dependency). The connector is
selected by an ``azure://`` ``remote_url``.
Optional Packages
-----------------
The Azure SDKs are optional dependencies and are imported lazily, so install
them only when you use this backend:
.. code-block:: bash
pip install azure-storage-blob
# Only needed for the DefaultAzureCredential fallback (see Authentication):
pip install azure-identity
URL Format
----------
.. code-block:: text
azure://<container_name>
The container name is the only value carried in the URL. The storage account
and credentials are supplied via ``extra_config``.
Example Configuration
---------------------
.. code-block:: yaml
chunk_size: 256
local_cpu: False
save_unfull_chunk: False
remote_url: "azure://my-container"
remote_serde: "naive"
blocking_timeout_secs: 10
extra_config:
azure_account_url: "https://myaccount.blob.core.windows.net"
azure_account_key: "your-account-key"
Configuration Parameters (in extra_config)
-----------------------------------------
* **azure_account_url**: Storage account URL, e.g.
``https://<account>.blob.core.windows.net``. Required unless
``azure_connection_string`` is provided.
* **azure_connection_string**: Full Azure Storage connection string. When set,
it is used on its own and the other fields are ignored.
* **azure_account_key**: Storage account shared key (paired with
``azure_account_url``).
* **azure_sas_token**: Shared Access Signature token (paired with
``azure_account_url``).
Authentication
--------------
Exactly one credential path is resolved, most-explicit-first:
1. **azure_connection_string** — if set, it wins.
2. **azure_account_url + azure_account_key** — shared-key auth.
3. **azure_account_url + azure_sas_token** — SAS-token auth.
4. **azure_account_url** only — falls back to
``DefaultAzureCredential`` (managed identity, environment variables, Azure
CLI login, etc.), which requires the ``azure-identity`` package.
``azure_account_url`` is required for every path except the connection-string
case.
Notes
-----
* Only full chunks are stored; set ``save_unfull_chunk: False``.
* Keys are flattened into one blob per chunk (same flat layout as the S3
backend).
docs/source/kv_cache/storage_backends/bigtable.rst
+152
View File
@@ -0,0 +1,152 @@
Google Cloud Bigtable
=====================
.. warning::
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/l2_storage/index`.
.. _bigtable-overview:
Overview
--------
Google Cloud Bigtable is a petabyte-scale, fully managed NoSQL database. Integrating Cloud Bigtable as a built-in remote storage connector inside LMCache bridges volatile high-cost in-memory tiers (Redis) and low-cost, high-latency archival object stores (S3).
For more information, see the `Cloud Bigtable Overview <https://cloud.google.com/bigtable/docs/overview>`_ and `Cloud Bigtable Pricing <https://cloud.google.com/bigtable/pricing>`_.
Architecture & Payload Limits
-----------------------------
- **Chunk Size Optimization**: Set LMCache's logical ``chunk_size`` to **256 tokens**. This groups payloads to minimize sequential Point-Read gRPC calls, preventing Python event-loop (GIL) bottlenecks.
- **MutateRow Limit**: Enforces a strict **90.0 MB request limit** for a single ``MutateRow`` gRPC request.
- **Storage Tier Row Limits**: The **SSD Tier** ceiling is **100 MiB per cell/row**. The **Enterprise Plus In-Memory Tier** is limited to **1.0 MiB per row**.
- **TTLCache Shielding**: Embeds a thread-safe ``TTLCache`` (10-second TTL default) to shield Bigtable nodes from concurrent prefetch lookup spikes.
Infrastructure Setup
--------------------
**1. Enable GCP APIs**
.. code-block:: bash
gcloud services enable bigtable.googleapis.com bigtableadmin.googleapis.com --project=your-gcp-project-id
**2. Provision Bigtable Instance**
Refer to the `gcloud beta bigtable Reference <https://cloud.google.com/sdk/gcloud/reference/beta/bigtable>`_ for additional parameter details.
.. code-block:: bash
gcloud beta bigtable instances create your-bigtable-instance-id \
--display-name="LMCache SSD Instance" \
--edition=ENTERPRISE \
--cluster-storage-type=ssd \
--cluster-config=id=your-cluster-id,zone=us-central1-a,nodes=1 \
--project=your-gcp-project-id
**3. Create Database Table & Column Family**
.. code-block:: bash
gcloud bigtable instances tables create lmcache-benchmark-v1 \
--instance=your-bigtable-instance-id \
--column-families=cf \
--project=your-gcp-project-id
**4. Install LMCache & Bigtable SDK**
.. code-block:: bash
export NO_NATIVE_EXT=1
pip install --no-cache-dir lmcache google-cloud-bigtable
Configuration
-------------
**Example A: Standard Bigtable SSD Integration (L2 Only)**
.. code-block:: yaml
chunk_size: 256
local_cpu: true
max_local_cpu_size: 10.0
remote_url: "bigtable://your-gcp-project-id/your-bigtable-instance-id"
remote_serde: "naive"
extra_config:
bigtable_project_id: "your-gcp-project-id"
bigtable_instance_id: "your-bigtable-instance-id"
bigtable_table_name: "lmcache-benchmark-v1"
.. note::
Alternatively, you can set the environment variables ``BT_PROJECT_ID``, ``BT_INSTANCE_ID``, and ``BT_TABLE_NAME`` instead of using ``extra_config``.
**Example B: 3-Tier Multi-Connector Hybrid (Local CPU -> Redis L2 -> Bigtable SSD L3)**
Deploy Redis for hot-cache loopbacks while offloading long-tail persistent storage to Bigtable SSD, using LMCache's dynamic OrderedDict routing.
.. code-block:: yaml
chunk_size: 256
local_cpu: true
max_local_cpu_size: 15.0
remote_storage_plugins:
- "redis"
- "bigtable"
extra_config:
remote_storage_plugin.redis.redis_url: "redis://your-redis-host:6379"
remote_storage_plugin.bigtable.bigtable_project_id: "your-gcp-project-id"
remote_storage_plugin.bigtable.bigtable_instance_id: "your-bigtable-instance-id"
remote_storage_plugin.bigtable.bigtable_table_name: "lmcache-benchmark-v1"
remote_storage_plugin.bigtable.bigtable_family_name: "cf"
remote_storage_plugin.bigtable.bigtable_column_name: "data"
remote_storage_plugin.bigtable.credentials_path: "/etc/gcp/key.json"
remote_storage_plugin.bigtable.bigtable_max_chunk_size_mb: 90.0
remote_storage_plugin.bigtable.exists_cache_ttl_seconds: 10.0
remote_storage_plugin.bigtable.exists_cache_size: 10000
remote_storage_plugin.bigtable.bigtable_write_timeout_ms: 10000.0
remote_storage_plugin.bigtable.bigtable_read_timeout_ms: 5000.0
Authentication
--------------
- **Application Default Credentials (ADC)**: If ``credentials_path`` is omitted or ``null``, the connector natively invokes ADC. Compatible with local development via ``gcloud auth application-default login`` or GKE Workload Identity Federation.
- **Explicit Keys**: Pass the absolute filesystem path containing a mounted GCP Service Account JSON secret to ``credentials_path``.
Verification
------------
Ensure you have installed the required dependencies:
.. code-block:: bash
pip install cachetools google-cloud-bigtable
Run the unit tests:
.. code-block:: bash
pytest tests/v1/storage_backend/test_bigtable_connector.py
Troubleshooting Large Payload Warnings
--------------------------------------
If you see a warning in the logs indicating that a chunk size exceeds the limit and is skipped (e.g. ``Bigtable chunk size ... MB exceeds threshold ... MB. Skipping write to prevent hard failures``), choose one of the following approaches:
1. **Reduce the LMCache Chunk Size (Recommended)**:
The serialized chunk size depends on LMCache's logical ``chunk_size`` (number of tokens per chunk) and model shape. You can reduce ``chunk_size`` (e.g., from ``256`` to ``128``) in your configuration file to shrink individual chunk payloads.
2. **Increase the Max Chunk Size**:
If your Bigtable instance uses the SSD storage tier (which supports up to 100 MB per cell/row), you can raise the maximum allowed write threshold in the configuration up to ``99.0`` MB using the ``bigtable_max_chunk_size_mb`` config key (or the ``BT_MAX_CHUNK_SIZE_MB`` environment variable).
.. warning::
Do not set ``bigtable_max_chunk_size_mb`` higher than ``100.0`` MB. While Cloud Bigtable supports up to ``256.0`` MB for a single row, a single cell value (which LMCache uses to store the chunk payload) has a hard limit of ``100.0`` MB. Exceeding this will trigger hard gRPC exceptions.
docs/source/kv_cache/storage_backends/cpu_ram.rst
+330
View File
@@ -0,0 +1,330 @@
CPU RAM
=======
.. warning::
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/l2_storage/index`.
.. _cpu_ram-overview:
Overview
--------
CPU RAM and Local Storage are the two ways of offloading KV cache onto non-GPU
memory of the same machine that is running inference.
Two ways to configure LMCache CPU Offloading:
---------------------------------------------
**1. Environment Variables:**
.. code-block:: bash
# 256 Tokens per KV Chunk
export LMCACHE_CHUNK_SIZE=256
# Enable CPU memory backend
export LMCACHE_LOCAL_CPU=True # default
# 5GB of Pinned CPU memory
export LMCACHE_MAX_LOCAL_CPU_SIZE=5.0 # default
**2. Configuration File**:
Passed in through ``LMCACHE_CONFIG_FILE=your-lmcache-config.yaml``
Example ``config.yaml``:
.. code-block:: yaml
# 256 Tokens per KV Chunk
chunk_size: 256
# Enable CPU memory backend
local_cpu: true # default
# 5GB of Pinned CPU memory
max_local_cpu_size: 5.0 # default
CPU RAM Explanation:
---------------------
The ``LMCACHE_MAX_LOCAL_CPU_SIZE`` is the amount of page-locked (for fast GPU transfer)
CPU memory that LMCache will reserve and must be set to a number greater than 0 since
local and remote backends also use CPU RAM as an intermediate buffer when transferring KV caches
with the GPU. This means it is possible to set ``LMCACHE_LOCAL_CPU=False`` even
though ``LMCACHE_MAX_LOCAL_CPU_SIZE`` is set to a non-zero number.
However, it is recommended to *always* set ``LMCACHE_LOCAL_CPU=True`` (the default is ``True`` so if you
don't specify, CPU offloading will automatically be enabled) since this allows all currently unused pinned CPU RAM that
LMCache has reserved to hold KV caches. When the pinned CPU RAM is required for any disk or remote transfers, the CPU KV caches will be LRU evicted to make
space so there is no danger of running out of pinned CPU RAM.
When ``LMCACHE_LOCAL_CPU=True`` is used in conjunction with the disk backend or
a remote backend (:doc:`Redis <./redis>`, :doc:`Mooncake <./mooncake>`, :doc:`Valkey <./valkey>`,
or :doc:`Infinistore <./infinistore>`), we can think of the CPU RAM as a "hot cache" that
will contain the "hottest" (most recently accessed)subset of KV caches from Disk and Remote storage.
Thus, the cache engine also has a **prefetch** mechanism to preload the KV caches for specified
tokens into the pinned CPU RAM from the disk or remote storage (*if* the KV caches for these
tokens are already stored there). This can preemptively avoid the latency of the disk and
remote KV transfer if we predict these tokens will be requested soon (e.g. structured or agentic workflows).
.. _cpu_ram-hugepage-support:
Hugepage Support
-----------------
By default LMCache allocates CPU-pinned memory using regular 4 KiB pages.
For large KV cache buffers (multiple gigabytes), enabling **Linux hugepages**
(2 MiB pages) can reduce TLB (Translation Lookaside Buffer) pressure and
improve memory access performance.
**System prerequisite**
Hugepages must be pre-allocated at the OS level before LMCache starts.
TO find the number of pages needed, divide the desired buffer size by 2 MiB and round up.
For example, 5 GB requires at least 2560 pages:
.. code-block:: bash
# Allocate 2560 hugepages (5 GB)
sudo sysctl -w vm.nr_hugepages=2560
# Make persistent across reboots
echo 'vm.nr_hugepages=2560' | sudo tee -a /etc/sysctl.conf
Verify that pages are available:
.. code-block:: bash
grep HugePages /proc/meminfo
# HugePages_Total: 2560
# HugePages_Free: 2560
**Configuration**
.. code-block:: yaml
local_cpu_use_hugepages: true
Or via environment variable:
.. code-block:: bash
export LMCACHE_LOCAL_CPU_USE_HUGEPAGES=true
**Restrictions**
- Hugepages are **not compatible with P2P mode** (``enable_p2p: true``).
- Hugepages are **not compatible with shared memory** (``shm_name`` is set).
- On non-CUDA platforms, hugepages are not supported. Regular allocation will be used as fallback.
.. _cpu_ram-online-inference-example:
Online Inference Example
------------------------
Let's feel the TTFT (time to first token) differential!
.. _cpu_ram-prerequisites:
**Prerequisites:**
- A Machine with at least one GPU. Adjust the max model length of your vllm instance depending on your GPU memory and the long context you want to use.
- vllm and lmcache installed (:doc:`Installation Guide <../../getting_started/installation>`)
- Hugging Face access to ``meta-llama/Meta-Llama-3.1-8B-Instruct``
.. code-block:: bash
export HF_TOKEN=your_hugging_face_token
- A few packages:
.. code-block:: bash
pip install openai transformers
**Step 0. Set up a directory for this example:**
.. code-block:: bash
mkdir lmcache-cpu-ram-example
cd lmcache-cpu-ram-example
**Step 1. Prepare a long context!**
We want a context long enough that vllm's prefix caching will not be able to hold the KV caches in
GPU memory and LMCache is necessary to keep KV caches in non-GPU memory:
.. code-block:: bash
# 382757 bytes
man bash > man-bash.txt
**Step 2. Start a vLLM server with CPU offloading enabled:**
Create a an lmcache configuration file called: ``cpu-offload.yaml``
.. code-block:: yaml
chunk_size: 256
local_cpu: true
max_local_cpu_size: 5.0
If you don't want to use a config file, uncomment the first three environment variables
and then comment out the ``LMCACHE_CONFIG_FILE`` below:
.. code-block:: bash
# LMCACHE_CHUNK_SIZE=256 \
# LMCACHE_LOCAL_CPU=True \
# LMCACHE_MAX_LOCAL_CPU_SIZE=5.0 \
LMCACHE_CONFIG_FILE="cpu-offload.yaml" \
vllm serve \
meta-llama/Llama-3.1-8B-Instruct \
--max-model-len 16384 \
--kv-transfer-config \
'{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
- ``--kv-transfer-config``: This is the parameter that actually tells vLLM to use LMCache for KV cache offloading.
- ``kv_connector``: Specifies the LMCache connector for vLLM V1
- ``kv_role``: Set to "kv_both" for both storing and loading KV cache (important because we will run two queries and the first will produce/store a KV cache while the second will consume/load that KV cache)
**Step 3. Query TTFT improvements with LMCache:**
Once the Open AI compatible server is running on default vllm port 8000, let's query it twice with the same long context!
Create a script called ``query-twice.py`` and paste the following code:
.. code-block:: python
import time
from openai import OpenAI
from transformers import AutoTokenizer
client = OpenAI(
api_key="dummy-key", # required by OpenAI client even for local servers
base_url="http://localhost:8000/v1"
)
models = client.models.list()
model = models.data[0].id
# 119512 characters total
# 26054 tokens total
long_context = ""
with open("man-bash.txt", "r") as f:
long_context = f.read()
# a truncation of the long context for the --max-model-len 16384
# if you increase the --max-model-len, you can decrease the truncation i.e.
# use more of the long context
long_context = long_context[:70000]
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct")
question = "Summarize bash in 2 sentences."
prompt = f"{long_context}\n\n{question}"
print(f"Number of tokens in prompt: {len(tokenizer.encode(prompt))}")
def query_and_measure_ttft():
start = time.perf_counter()
ttft = None
chat_completion = client.chat.completions.create(
messages=[{"role": "user", "content": prompt}],
model=model,
temperature=0.7,
stream=True,
)
for chunk in chat_completion:
chunk_message = chunk.choices[0].delta.content
if chunk_message is not None:
if ttft is None:
ttft = time.perf_counter()
print(chunk_message, end="", flush=True)
print("\n") # New line after streaming
return ttft - start
print("Querying vLLM server with cold LMCache CPU Offload")
cold_ttft = query_and_measure_ttft()
print(f"Cold TTFT: {cold_ttft:.3f} seconds")
print("\nQuerying vLLM server with warm LMCache CPU Offload")
warm_ttft = query_and_measure_ttft()
print(f"Warm TTFT: {warm_ttft:.3f} seconds")
print(f"\nTTFT Improvement: {(cold_ttft - warm_ttft):.3f} seconds \
({(cold_ttft/warm_ttft):.1f}x faster)")
Then run:
.. code-block:: bash
python query-twice.py
Since we're in streaming mode, you'll be able to feel the TTFT differential in
real time!
**Example Output:**
.. code-block:: text
Number of tokens in prompt: 15376
Querying vLLM server with cold LMCache
Bash is a Unix shell and command-line interpreter that executes commands read
from the standard input or from a file, incorporating features from the Korn
and C shells. It is an sh-compatible command language interpreter that can be
configured to be POSIX-conformant by default and is intended to be a conformant
implementation of the Shell and Utilities portion of the IEEE POSIX specification.
Cold TTFT: 6.537 seconds
Querying vLLM server with warm LMCache
Bash is a Unix shell and command-line interpreter that eead from the standard
input or from a file, incorporatinhe Korn and C shells. It is intended to be a
conformant tation of the IEEE POSIX specification and can be configured to be
POSIX-conformant by default, with options for setting the shell's behavior and
interacting with the user.
Warm TTFT: 0.147 seconds
TTFT Improvement: 6.390 seconds (44.5x faster)
If you look at the logs of your vLLM server, you should see (the logs are truncated for cleanliness):
.. code-block:: text
# Cold LMCache Miss and then Store
LMCache INFO: Reqid: chatcmpl-8676f9b9ebf04c79a5d47b9ada7b65fd, Total tokens 15410,
LMCache hit tokens: 0, need to load: 0
# you should see 8 of these storing logs total
# 2048 tokens is a multiple of the chunk size
LMCache INFO: Storing KV cache for 2048 out of 12288 tokens for request
chatcmpl-8676f9b9ebf04c79a5d47b9ada7b65fd
LMCache INFO: Storing KV cache for 2048 out of 14336 tokens for request
chatcmpl-8676f9b9ebf04c79a5d47b9ada7b65fd
LMCache INFO: Storing KV cache for 1074 out of 15410 tokens for request
chatcmpl-8676f9b9ebf04c79a5d47b9ada7b65fd
# Warm LMCache Hit!!
LMCache INFO: Reqid: chatcmpl-136d9dac1ba94bd4b4ae85007e8ad437, Total tokens 15410,
LMCache hit tokens: 15409, need to load: 1
.. _cpu_ram-tips:
Tips:
-----
- If you want to run the ``query-twice.py`` script multiple times, you'll need to either restart the vLLM LMCache server or change the prefix of the context you pass in since you've already warmed LMCache.
- The max model length here was decided by running an L4 with only 23GB of GPU memory. If you have more memory, you can increase the max model length and modify ``query-twice.py`` to use more of the long context. LMCache TTFT improvement becomes more pronounced as the context length increases!
docs/source/kv_cache/storage_backends/custom_backend.rst
+6
View File
@@ -0,0 +1,6 @@
Custom Storage Backends
=======================
LMCache supports integrating custom storage backends through dynamic loading or plug and play capability. This allows extending cache storage capabilities without modifying core code.
See :doc:`Storage Plugins <../../developer_guide/extending_lmcache/storage_plugins>` for more details.
docs/source/kv_cache/storage_backends/dax.rst
+254
View File
@@ -0,0 +1,254 @@
# SPDX-License-Identifier: Apache-2.0
Device-DAX (/dev/dax)
=====================
Overview
--------
The DAX storage plugin maps a ``/dev/dax`` device using ``mmap(MAP_SHARED)``
and uses the mapped region as a fixed-size arena for KV cache chunks.
Typical ``/dev/dax`` devices include persistent memory,
CXL-attached memory, and other byte-addressable memory devices.
Data stored on the DAX device may survive process restarts,
but is not guaranteed to be durable.
KV cache data is stored in the DAX region as part of the backend's storage flow.
Reads copy data back into CPU-backed memory objects.
Configuration
-------------
.. code-block:: yaml
local_cpu: true
max_local_cpu_size: 80
storage_plugins: ["dax"]
extra_config:
storage_plugin.dax.module_path: lmcache.v1.storage_backend.plugins.dax_backend
storage_plugin.dax.class_name: DaxBackend
dax.device_path: "/dev/dax1.0"
dax.max_dax_size: 100
dax.restore_workers: 8
dax.restore_max_regions: 8
dax.retrieve_staging_slab_bytes: 268435456
Multiprocess Mode
-----------------
In LMCache multiprocess mode, Device-DAX is configured as a built-in L2
adapter named ``dax``. The MP adapter uses the normal L2 adapter
``submit -> event fd -> query`` contract; no vLLM connector protocol changes
are required.
.. code-block:: bash
lmcache server \
--l1-size-gb 80 \
--eviction-policy LRU \
--l2-adapter '{
"type": "dax",
"device_path": "/dev/dax1.0",
"max_dax_size_gb": 100,
"slot_bytes": 268435456,
"num_store_workers": 1,
"num_lookup_workers": 1,
"num_load_workers": 4
}'
The legacy single-device ``--l2-adapter`` JSON accepts these fields:
- ``device_path``: required path to a readable and writable DAX device.
- ``max_dax_size_gb``: required mapped size in GiB. The value must fit within
the device capacity when capacity can be determined with ``fstat``.
- ``slot_bytes``: required fixed slot size in bytes. It must be large enough
for one full LMCache chunk.
- ``num_store_workers``: optional store worker count, default ``1``.
- ``num_lookup_workers``: optional lookup worker count, default ``1``.
- ``num_load_workers``: optional load worker count, default
``min(4, os.cpu_count())``.
- ``persist_enabled``: accepted by common MP L2 parsing but ignored by
``dax`` in this release.
Runtime hotplug uses the multi-device form. The ``devices`` list may also be
empty when ``hotplug_enabled`` is ``true``.
.. code-block:: bash
lmcache server \
--l1-size-gb 80 \
--eviction-policy LRU \
--l2-adapter '{
"type": "dax",
"devices": [
{"device_path": "/dev/daxX.X", "max_dax_size_gb": 100},
{"device_path": "/dev/daxY.Y", "max_dax_size_gb": 100}
],
"slot_bytes": 268435456,
"hotplug_enabled": true,
"num_store_workers": 1,
"num_lookup_workers": 1,
"num_load_workers": 4
}'
MP DAX stores opaque ``ObjectKey`` values in memory and is volatile-only in
this release. Closing and reopening the server on the same DAX path starts
with an empty index, so previously written bytes are not discoverable after
restart.
MP DAX uses one stable adapter facade per LMCache server. The facade owns
stable event fds and worker pools, and runtime add/remove/resize only changes
the mapped DAX cores behind that facade. It does not add kernel-level CXL or
DAX reconfiguration, per-TP DAX partitions, on-device metadata, or restart
recovery. Capacity accounting and eviction are slot-based: a stored object
occupies one slot even if its payload is smaller than ``slot_bytes``.
Runtime Hotplug API
-------------------
Runtime hotplug is disabled unless ``hotplug_enabled`` is ``true``. The API
changes only LMCache runtime mappings and metadata; the ``/dev/dax*`` device
must already exist and be readable and writable by the LMCache server process.
The runtime endpoints are implemented through StorageManager's generic L2
adapter reconfiguration interface, which routes backend, operation name, and
adapter-specific payload to the selected adapter. DAX owns the path, mode,
migration, and resize semantics; the generic interface is reusable by other
adapters such as P2P.
Use JSON bodies because DAX paths contain slashes:
.. code-block:: bash
curl http://127.0.0.1:9000/reconfigure/dax/status
curl -X POST http://127.0.0.1:9000/reconfigure/dax/add \
-H 'Content-Type: application/json' \
-d '{"device_path": "/dev/daxX.X", "size": "100GiB"}'
curl -X POST http://127.0.0.1:9000/reconfigure/dax/remove \
-H 'Content-Type: application/json' \
-d '{"device_path": "/dev/daxX.X", "mode": "migrate"}'
curl -X POST http://127.0.0.1:9000/reconfigure/dax/resize \
-H 'Content-Type: application/json' \
-d '{"device_path": "/dev/daxX.X", "size": "200GiB"}'
``size`` is required for add and resize. Use an integer byte count or a string
such as ``"100GiB"``. ``remove`` supports these modes:
- ``migrate``: move DAX-resident KV to other active DAX devices before closing
the source device.
- ``evict``: delete DAX-resident KV on the source device. This is destructive
for the DAX tier.
- ``drain``: stop new writes to the source device and leave existing KV
readable until it is evicted or the server closes.
``resize`` supports ``migrate`` and ``evict`` modes. It does not support
``drain`` because resize completes synchronously.
Hotplug operations are lock-safe by default. A remove or shrink that would
delete externally locked or borrowed slots returns ``409 Conflict`` unless
``force`` is set. A migration that has no active destination capacity returns
``507 Insufficient Storage``. Resize grow preserves the in-memory key index and
does not move KV payloads. Resize shrink never silently drops keys; entries
outside the new slot range must migrate first, or the request fails.
Hardware Validation Flow
------------------------
Use the same Qwen 8B or 14B long-context workload before and after a runtime
capacity change. Without hotplug support, ``/reconfigure/dax/status`` and
``/reconfigure/dax/add`` are not available; changing the DAX device set
requires restarting LMCache with a new ``--l2-adapter`` value, which drops the
volatile DAX key index.
.. code-block:: bash
export MODEL=Qwen/Qwen3-8B # or a local Qwen 8B/14B checkpoint
curl http://127.0.0.1:9000/reconfigure/dax/status
python benchmarks/long_doc_qa/long_doc_qa.py \
--model "$MODEL" --num-documents 1 --document-length 1024 \
--output-len 16 --repeat-count 2 --repeat-mode tile \
--completions --host 127.0.0.1 --port 8000 --json-output
curl -X POST http://127.0.0.1:9000/reconfigure/dax/add \
-H 'Content-Type: application/json' \
-d '{"device_path": "/dev/daxX.X", "size": "100GiB"}'
curl http://127.0.0.1:9000/reconfigure/dax/status
Record these fields for the comparison:
- ``total_capacity_bytes`` before and after ``/reconfigure/dax/add``.
- ``total_used_bytes`` while the Qwen workload is running.
- Whether an LMCache restart was required.
- Whether the same cached prompt remains retrievable after the capacity change.
Using The Batched Restore Path
------------------------------
The current DAX optimization is a staged batched restore path for retrieval.
It is enabled automatically whenever the DAX backend is configured. No extra
feature flag is required.
The retrieve flow is:
1. Reserve a batched set of readable DAX chunks.
2. Allocate CPU restore buffers from ``LocalCPUBackend``.
3. Copy DAX data into a backend-owned pinned staging slab in coalesced regions.
4. Copy from the staging slab into the final CPU ``MemoryObj`` outputs.
5. Upload those CPU outputs through the normal GPU connector path.
The store flow is unchanged: KV data is still staged through CPU memory before
being written into the DAX arena.
The new DAX tuning knobs control the batched restore path:
- ``dax.restore_workers``: number of persistent worker threads used to execute
restore regions in parallel.
- ``dax.restore_max_regions``: maximum number of restore regions in one wave.
Larger values increase parallelism but also increase slab space requirements.
- ``dax.retrieve_staging_slab_bytes``: total size in bytes of the reusable
pinned retrieve slab. This must be large enough to hold one full chunk per
configured restore region.
For a first pass, start with:
- ``dax.restore_workers`` equal to the number of CPU workers you want devoted
to DAX restores
- ``dax.restore_max_regions`` equal to ``dax.restore_workers``
- ``dax.retrieve_staging_slab_bytes`` at least
``dax.restore_max_regions * full_chunk_size``, then scale upward if larger
batched restores are common
If retrieve throughput is low, increase the slab size first, then increase
worker and region counts together. If CPU pressure is high, reduce
``dax.restore_workers`` and ``dax.restore_max_regions``.
Runtime Requirements
--------------------
- ``extra_config['dax.device_path']`` is required and must point to a readable
and writable DAX device.
- The process must have read-write access to the DAX device
(e.g., via appropriate permissions or group membership).
- ``LocalCPUBackend`` must be enabled because DAX reads return CPU-backed
memory objects.
Validation and Current Limits
-----------------------------
- Tensor parallelism is currently limited to TP=1
(``metadata.world_size == 1``).
- Only single-tensor chunk layouts are supported. Multi-tensor put
requests are rejected.
- Batched restore uses a backend-owned retrieve staging slab and persistent
restore executors. The slab and region count can be tuned with
``dax.restore_workers``, ``dax.restore_max_regions``, and
``dax.retrieve_staging_slab_bytes``.
- Blocking batched restore preserves positional output semantics, while
asynchronous batched restore returns only the consecutive hit prefix.
docs/source/kv_cache/storage_backends/eic.rst
+45
View File
@@ -0,0 +1,45 @@
EIC
===
.. warning::
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/l2_storage/index`.
EIC(Elastic Instant Cache) is a distributed database designed for LLM KV Cache. It supports RDMA, GDR and has the capabilities of distributed disaster tolerance and expansion.
You can understand the principles and architecture of EIC through these articles:
* https://mp.weixin.qq.com/s/tasDqXf0Gxr3o_WCJ2IJUQ
* https://mp.weixin.qq.com/s/b_4YhTa96Zeklh23lv8qBw
Deploy EIC
----------
You can visit the official link https://console.volcengine.com/eic and deploy EIC KVCache on your compute cluster with web UI. In addition, we provide particular image in volcano engine, which integrates various optimizations based on the official image.
You may use tests/v1/storage_backend/test_eic.py to detect the connectivity of EIC.
Deploy Model With EIC
---------------------
You can enable EIC KVCache offload with the official interface, such as
.. code-block:: bash
export LMCACHE_CONFIG_FILE=/workspace/config/remote-eic.yaml
export VLLM_USE_V1=1
python3 -m vllm.entrypoints.openai.api_server \
... \
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
Example ``config.yaml``:
.. code-block:: yaml
chunk_size: 256
remote_url: "eic://your-eic-endpoint"
eic_instance_id: "your-eic-instance-id"
eic_flag_file: "your-eic-config-path"
For more details, you can see https://www.volcengine.com/docs/85848/1749188.
docs/source/kv_cache/storage_backends/fs.rst
+194
View File
@@ -0,0 +1,194 @@
Filesystem Backend
==================
.. warning::
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/l2_storage/fs`.
The filesystem backend uses ``FSConnector`` to store LMCache remote chunks as
files under one or more POSIX filesystem directories. It is useful when you want
a simple persistent remote backend, or when multiple inference workers can see
the same mounted directory through local disk, NFS, a parallel filesystem, or a
container volume.
This backend is different from :doc:`local_storage`. Local disk offloading is a
per-process local tier configured through ``local_disk``. ``FSConnector`` is a
remote backend configured through ``remote_storage_plugins`` or the legacy
``remote_url`` field, so it participates in the same remote backend path as
Redis, S3, Mooncake, and other remote connectors.
When to use it
--------------
Use ``FSConnector`` when:
* You need a lightweight persistent remote backend for development, examples,
or benchmark runs.
* Multiple LMCache or vLLM processes share a mounted cache directory.
* Your storage is already exposed as a filesystem and does not need a separate
object-store or key-value service.
* You want to test remote-backend behavior before moving to a production
backend such as Redis, Valkey, S3, Mooncake, or InfiniStore.
Avoid using it when:
* The filesystem is not shared by every process that must read the cache.
* You need object-store semantics, cross-region persistence, or service-level
access control.
* The storage path is on a slow network filesystem and sits on the hot request
path.
Recommended configuration
-------------------------
The recommended form is the built-in remote storage plugin configuration. The
plugin name ``fs`` selects ``FSConnector`` and the base path is configured in
``extra_config``.
.. code-block:: yaml
chunk_size: 256
local_cpu: false
max_local_cpu_size: 1
save_unfull_chunk: false
remote_serde: "naive"
blocking_timeout_secs: 10
remote_storage_plugins: ["fs"]
extra_config:
remote_storage_plugin.fs.base_path: "/tmp/lmcache-fs"
save_chunk_meta: false
``FSConnector`` creates the base directory if it does not already exist. Each
cache chunk is written as a ``.data`` file whose name is derived from the
LMCache cache key.
Multiple filesystem instances
-----------------------------
You can configure multiple named ``fs`` instances by appending an instance name
after the connector type. The part before the first dot is still the connector
type; the full plugin name becomes the ``extra_config`` prefix.
.. code-block:: yaml
remote_storage_plugins: ["fs.primary", "fs.backup"]
extra_config:
remote_storage_plugin.fs.primary.base_path: "/mnt/cache-primary/lmcache"
remote_storage_plugin.fs.backup.base_path: "/mnt/cache-backup/lmcache"
save_chunk_meta: false
This is useful when a deployment wants separate filesystem-backed remote stores
for different cache policies, traffic classes, or experiments.
Multiple base paths
-------------------
``remote_storage_plugin.<name>.base_path`` may contain a comma-separated list of
directories. The connector chooses a directory by hashing the cache chunk key,
which spreads files across the configured paths.
.. code-block:: yaml
remote_storage_plugins: ["fs"]
extra_config:
remote_storage_plugin.fs.base_path: "/mnt/nvme0/lmcache,/mnt/nvme1/lmcache"
save_chunk_meta: false
Use multiple paths when each path maps to an independent storage device or mount
point. For best results, keep every path visible to the LMCache processes that
need to retrieve the same chunks.
Legacy ``remote_url`` configuration
-----------------------------------
The legacy ``remote_url`` form is still supported. The host and port are parsed
for compatibility with other remote URL formats; ``FSConnector`` uses the path.
.. code-block:: yaml
chunk_size: 256
local_cpu: false
max_local_cpu_size: 1
save_unfull_chunk: false
remote_url: "fs://localhost:0/tmp/lmcache-fs"
remote_serde: "naive"
blocking_timeout_secs: 10
extra_config:
save_chunk_meta: false
Prefer ``remote_storage_plugins`` for new deployments because it also supports
named instances and keeps connector-specific settings grouped by plugin name.
Optional settings
-----------------
The connector reads the following optional settings from ``extra_config``.
``fs_connector_relative_tmp_dir``
Relative directory used for temporary files before an atomic rename into the
final chunk path. The value must be relative, not absolute. When omitted,
temporary files are created next to the final file with a ``.tmp`` suffix.
``fs_connector_read_ahead_size``
Number of bytes to read first when loading a chunk. If the read fills that
window, the connector reads the remaining bytes. This can trigger filesystem
readahead on filesystems that support it.
``fs_connector_use_odirect``
Enables ``O_DIRECT`` for aligned reads and writes on platforms that expose
it. The connector falls back to normal I/O when a chunk size is not aligned
to the filesystem block size. ``O_DIRECT`` is disabled automatically when
``save_chunk_meta`` is enabled because the metadata prefix is not block
aligned.
``fs_base_path``
Compatibility fallback for plugin mode. Prefer
``remote_storage_plugin.<name>.base_path`` so the setting remains scoped to a
specific plugin instance.
Example with optional settings:
.. code-block:: yaml
remote_storage_plugins: ["fs"]
extra_config:
remote_storage_plugin.fs.base_path: "/data/lmcache-fs"
fs_connector_relative_tmp_dir: ".tmp"
fs_connector_read_ahead_size: 1048576
fs_connector_use_odirect: true
save_chunk_meta: false
Operational notes
-----------------
* Ensure the LMCache process has permission to create directories and write
files under every configured base path.
* Put the path on durable storage if cache reuse must survive process restarts.
Temporary directories such as ``/tmp`` are convenient for tests but may be
cleaned by the operating system.
* Use the same mounted path for every process that should share cache chunks.
If one process writes to a private container path, other processes will miss
those chunks even if they use the same configuration text.
* Leave ``save_chunk_meta`` enabled when workers may infer different metadata
for the same chunk. Disable it only when you need the lower overhead path and
the workers share compatible cache metadata.
* For MP mode L2 storage, see :doc:`../../mp/l2_storage/index`, which documents the
``fs`` and ``fs_native`` L2 adapters configured through ``--l2-adapter``.
Minimal vLLM usage
------------------
After writing ``fs.yaml`` with one of the configurations above, start vLLM with
LMCache enabled:
.. code-block:: bash
LMCACHE_CONFIG_FILE=fs.yaml vllm serve meta-llama/Llama-3.1-8B-Instruct \
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}' \
--disable-log-requests
Then send the same long-prefix request twice. The first request stores chunks in
the filesystem backend. The second request should report LMCache hit tokens and
load matching chunks from the configured filesystem path.
docs/source/kv_cache/storage_backends/gds.rst
+275
View File
@@ -0,0 +1,275 @@
GDS Backend
==================
.. warning::
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/l2_storage/index`.
.. _gds-overview:
Overview
--------
This backend will work with any file system, whether local, remote, and remote
with GDS-based optimizations. Remote file systems allow for multiple LMCache
instances to share data seamlessly. The GDS (GPU-Direct Storage) optimizations
are used for zero-copy I/O from GPU memory to storage systems. Supports both
NVIDIA cuFile and AMD hipFile for GPU-direct storage.
Ways to configure LMCache GDS Backend
-----------------------------------------
**1. Environment Variables:**
.. code-block:: bash
# 256 Tokens per KV Chunk
export LMCACHE_CHUNK_SIZE=256
# Path to store files
export LMCACHE_GDS_PATH="/mnt/gds/cache"
# GDS Buffer Size in MiB
export LMCACHE_GDS_BUFFER_SIZE="8192"
# Disabling CPU RAM offload is sometimes recommended as the
# CPU can get in the way of GPUDirect operations
export LMCACHE_LOCAL_CPU=False
**2. Configuration File**:
Passed in through ``LMCACHE_CONFIG_FILE=your-lmcache-config.yaml``
Example ``config.yaml``:
.. code-block:: yaml
# 256 Tokens per KV Chunk
chunk_size: 256
# Disable local CPU
local_cpu: false
# Path to file system, local, remote or GDS-enabled mount
gds_path: "/mnt/gds/cache"
# GDS Buffer Size in MiB
gds_buffer_size: 8192
Multi-Path (Multi-Device) Support
---------------------------------
When a system has multiple NVMe drives, you can distribute GDS I/O across them
by specifying a comma-separated list of paths in ``gds_path``. The
``gds_path_sharding`` option controls how each GPU worker selects its path.
Currently only ``"by_gpu"`` is supported (the default), which selects a path
based on the device index (``device_id % num_paths``), so traffic is spread
evenly across the drives without any manual pinning.
**Why this helps:** a single PCIe Gen 4 x4 NVMe tops out at ~7 GB/s. With four
drives the aggregate bandwidth can reach ~28 GB/s, matching what multi-GPU
systems need for KV cache eviction and prefetch.
**Environment variables:**
.. code-block:: bash
export LMCACHE_GDS_PATH="/mnt/nvme0/cache,/mnt/nvme1/cache,/mnt/nvme2/cache,/mnt/nvme3/cache"
export LMCACHE_GDS_PATH_SHARDING="by_gpu"
**YAML config:**
.. code-block:: yaml
gds_path: "/mnt/nvme0/cache,/mnt/nvme1/cache,/mnt/nvme2/cache,/mnt/nvme3/cache"
gds_path_sharding: "by_gpu"
With the above configuration on a 4-GPU node:
- ``cuda:0`` writes to ``/mnt/nvme0/cache``
- ``cuda:1`` writes to ``/mnt/nvme1/cache``
- ``cuda:2`` writes to ``/mnt/nvme2/cache``
- ``cuda:3`` writes to ``/mnt/nvme3/cache``
If there are more GPUs than paths, the assignment wraps around (e.g. ``cuda:4``
maps back to ``/mnt/nvme0/cache``). A single path (no commas) works exactly as
before.
All directories are created automatically at startup. Every path in the list
must reside on a filesystem that the rest of the GDS configuration expects
(e.g., all paths on GDS-capable mounts when using cuFile).
**Read behavior:** on startup the backend scans **all** configured paths for
previously-stored KV cache entries, regardless of GPU affinity. This means a
``cuda:0`` worker whose write affinity is ``/mnt/nvme0/cache`` will still
discover entries that were written to ``/mnt/nvme1/cache`` by ``cuda:1`` in a
prior run. Writes, however, always go to the single affinity-selected path.
.. code-block:: text
Startup scan (read): iterate ALL gds_paths → populate hot_cache
Runtime writes: only the affinity path (device_id % num_paths)
Runtime reads: look up hot_cache first; on miss, check ALL
gds_paths on disk → load from whichever path
the entry lives on
GDS Buffer Size Explanation
---------------------------
The backend currently pre-registers buffer space to speed up GDS operations. This buffer space
is registered in VRAM so options like ``--gpu-memory-utilization`` from ``vllm`` should be considered
when setting it. For example, a good rule of thumb for H100 which generally has 80GiBs of VRAM would
be to start with 8GiB and set ``--gpu-memory-utilization 0.85`` and depending on your workflow fine-tune
it from there.
Using AMD hipFile
-----------------
.. note::
hipFile is alpha software and has been tested on limited hardware.
For full installation details, see the
`hipFile install guide <https://github.com/ROCm/rocm-systems/blob/develop/projects/hipfile/INSTALL.md>`__.
**Prerequisites:**
- **ROCm >= 7.2** with ``amdgpu-dkms >= 30.20.1``
(see the `ROCm quick start installation guide <https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/quick-start.html>`__)
- **Supported storage:** local NVMe drives only
- **Supported filesystems:** ext4 (mounted with ``data=ordered``) and xfs
- **Kernel:** ``CONFIG_PCI_P2PDMA`` must be enabled
**Quick install (Ubuntu 24.04):**
.. code-block:: bash
sudo apt install libmount-dev wget
# Install nightly hipFile packages
wget https://github.com/ROCm/hipFile/releases/download/nightly/hipfile_0.2.0.70200-nightly.9999.24.04_amd64.deb
wget https://github.com/ROCm/hipFile/releases/download/nightly/hipfile-dev_0.2.0.70200-nightly.9999.24.04_amd64.deb
sudo dpkg -i hipfile-dev_0.2.0.70200-nightly.9999.24.04_amd64.deb hipfile_0.2.0.70200-nightly.9999.24.04_amd64.deb
You can verify that the HIP libraries and kernel support AIS (AMD Infinity Storage) by running:
.. code-block:: bash
/opt/rocm/bin/ais-check
Successful output will show ``True`` for ``Kernel P2PDMA support``, ``HIP runtime``, and ``amdgpu``.
**LMCache configuration:**
To use AMD hipFile instead of NVIDIA cuFile, set the GDS backend:
**Environment Variables:**
.. code-block:: bash
export LMCACHE_GDS_BACKEND=hipfile
**Configuration File:**
.. code-block:: yaml
gds_backend: "hipfile"
Note: The ``gds_buffer_size`` configuration is used for both cuFile and hipFile buffers.
Setup Example
-------------
.. _gds-prerequisites:
**Prerequisites:**
- A Machine with at least one GPU. You can adjust the max model length of your vllm instance depending on your GPU memory.
- A mounted file system. A file system supportings GDS will work best.
- vllm and lmcache installed (:doc:`Installation Guide <../../getting_started/installation>`)
- Hugging Face access to ``meta-llama/Llama-3.1-8B-Instruct``
.. code-block:: bash
export HF_TOKEN=your_hugging_face_token
**Step 1. Create cache directory under your file system mount:**
To find all the types of file systems supporting GDS in your system, use `gdscheck` from NVIDIA:
.. code-block:: bash
sudo /usr/local/cuda-*/gds/tools/gdscheck -p
Check with your storage vendor on how to mount the remote file system.
(For example, if you want to use a GDS-enabled NFS driver, try the modified [NFS
stack](https://vastnfs.vastdata.com/), which is an open source driver that
works with any standard [NFS
RDMA](https://datatracker.ietf.org/doc/html/rfc5532) server. More
vendor-specific instructions will be added here in the future).
Create a directory under the file systew mount (the name here is arbitrary):
.. code-block:: bash
mkdir /mnt/gds/cache
**Step 2. Start a vLLM server with file backend enabled:**
Create a an lmcache configuration file called: ``gds-backend.yaml``
.. code-block:: yaml
local_cpu: false
chunk_size: 256
gds_path: "/mnt/gds/cache"
gds_buffer_size: 8192
If you don't want to use a config file, uncomment the first three environment variables
and then comment out the ``LMCACHE_CONFIG_FILE`` below:
.. code-block:: bash
# LMCACHE_LOCAL_CPU=False \
# LMCACHE_CHUNK_SIZE=256 \
# LMCACHE_GDS_PATH="/mnt/gds/cache" \
# LMCACHE_GDS_BUFFER_SIZE=8192 \
LMCACHE_CONFIG_FILE="gds-backend.yaml" \
vllm serve \
meta-llama/Llama-3.1-8B-Instruct \
--max-model-len 65536 \
--kv-transfer-config \
'{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
POSIX fallback
--------------
In some cases, libcufile implements its own internal POSIX fallback without `GdsBackend` being aware.
In others, an error such as `RuntimeError: cuFileHandleRegister failed (cuFile err=5030, cuda_err=0)` may be throwned.
Thus, backend can be configured to fallback to its own POSIX implementation when the usage of the GDS APIs is not successful.
To force `GdsBackend` not use GDS APIs for any reason, you can override its behavior via configuration:
.. code-block:: yaml
use_gds: false
Or via environment variable:
.. code-block:: bash
LMCACHE_USE_GDS=False
The ``gds_backend`` field (default: ``cufile``) selects which GDS library to use. Supported
backends are ``cufile`` (NVIDIA cuFile) and ``hipfile`` (AMD hipFile):
.. code-block:: yaml
use_gds: true
gds_backend: "cufile" # or "hipfile"
Note that under this mode it would still use CUDA APIs to map and do operations the pre-registered GPU memory.
docs/source/kv_cache/storage_backends/hfbucket.rst
+183
View File
@@ -0,0 +1,183 @@
Hugging Face Buckets Backend
============================
The Hugging Face Buckets backend stores LMCache chunks in a Hugging Face Bucket
using LMCache's built-in remote storage plugin framework. This is a persistent
remote backend that fits warm and cold KV cache persistence better than the
hottest local tiers.
When to use it
--------------
Use the HFBucket backend when you want:
* A Hub-native persistent store for KV cache data.
* A remote backend that can be configured through ``remote_storage_plugins``.
* Multiple named bucket instances in one LMCache deployment.
Avoid using it as the primary hot path for the lowest-latency cache lookups.
Local CPU, local disk, and other lower-latency backends are a better fit for
the hottest cache tier.
Requirements and limitations
----------------------------
* LMCache uses ``huggingface_hub`` bucket APIs for uploads, downloads, listing,
and deletes.
* The first built-in release is intentionally conservative:
* Only full chunks are supported.
* Partial chunk uploads are rejected.
* Downloads are rejected when the stored object size does not match the
expected full LMCache chunk size.
* Chunk metadata is not stored in the bucket objects.
Minimal configuration
---------------------
.. code-block:: yaml
chunk_size: 256
local_cpu: false
save_unfull_chunk: false
remote_serde: "naive"
blocking_timeout_secs: 10
remote_storage_plugins: ["hfbucket"]
extra_config:
remote_storage_plugin.hfbucket.bucket_handle: "hf://buckets/my-org/lmcache-kv/prod"
remote_storage_plugin.hfbucket.token_env: "HF_TOKEN"
remote_storage_plugin.hfbucket.create_bucket_if_missing: false
remote_storage_plugin.hfbucket.download_tmp_dir: "/tmp/lmcache-hfbucket"
remote_storage_plugin.hfbucket.metadata_cache_ttl_secs: 30
Multiple instances
------------------
Use instance-qualified plugin names to configure more than one bucket-backed
remote store in the same LMCache config.
.. code-block:: yaml
remote_storage_plugins: ["hfbucket.us", "hfbucket.eu"]
extra_config:
remote_storage_plugin.hfbucket.us.bucket_handle: "hf://buckets/my-org/lmcache-kv/us"
remote_storage_plugin.hfbucket.us.token_env: "HF_US_TOKEN"
remote_storage_plugin.hfbucket.eu.bucket_handle: "hf://buckets/my-org/lmcache-kv/eu"
remote_storage_plugin.hfbucket.eu.token_env: "HF_EU_TOKEN"
Configuration reference
-----------------------
All configuration keys live under
``extra_config.remote_storage_plugin.<plugin_name>.*`` where ``plugin_name`` is
either ``hfbucket`` or an instance-qualified name such as ``hfbucket.prod``.
* ``bucket_handle`` (required): Hugging Face Bucket handle in
``hf://buckets/<namespace>/<bucket>[/<prefix>]`` format.
* ``token_env`` (optional, default ``HF_TOKEN``): Environment variable used to
resolve the Hugging Face access token.
* ``token`` (optional): Direct token override. ``token_env`` takes precedence
when both are set.
* ``create_bucket_if_missing`` (optional, default ``false``): Lazily create the
bucket on the first write path.
* ``download_tmp_dir`` (optional): Root directory for connector-local download
scratch space. On Linux, pointing this at a tmpfs mount such as
``/dev/shm/lmcache-hfbucket`` avoids the disk write on the download path.
* ``metadata_cache_ttl_secs`` (optional, default ``30``): TTL for cached exact
existence and size metadata.
MP Mode Configuration
---------------------
In multi-process (MP) mode, Hugging Face Buckets are configured as an L2
adapter through a JSON spec passed to the LMCache server. This is separate from
the non-MP ``remote_storage_plugins`` configuration above. Each
``--l2-adapter`` argument takes a JSON object whose ``"type": "hfbucket"``
field selects the HFBucket adapter.
.. code-block:: json
{
"type": "hfbucket",
"bucket_handle": "hf://buckets/my-org/lmcache-kv/prod",
"token_env": "HF_TOKEN",
"create_bucket_if_missing": false,
"download_tmp_dir": "/tmp/lmcache-hfbucket-mp",
"metadata_cache_ttl_secs": 30,
"num_workers": 4,
"max_capacity_gb": 500,
"eviction": {
"eviction_policy": "LRU",
"trigger_watermark": 0.85,
"eviction_ratio": 0.2
}
}
HFBucket L2 Adapter Fields
~~~~~~~~~~~~~~~~~~~~~~~~~~
* **type** (required): must be ``"hfbucket"``.
* **bucket_handle** (required): Hugging Face Bucket handle in
``hf://buckets/<namespace>/<bucket>[/<prefix>]`` format.
* **token_env**: environment variable used to resolve the Hugging Face access
token (default ``"HF_TOKEN"``).
* **token**: optional direct token fallback. ``token_env`` takes precedence
when the environment variable is set. Prefer ``token_env`` for production
deployments so secrets do not live in adapter JSON.
* **create_bucket_if_missing**: lazily create the bucket on the first store
operation (default ``false``). This only helps when the bucket is missing and
the token has permission to create it; it does not fix invalid credentials,
invalid handles, or network failures.
* **download_tmp_dir**: root directory for temporary load downloads (default
``/tmp/lmcache-hfbucket-mp``). The MP adapter downloads bucket files into
per-task temporary files and then copies their bytes into the destination
``MemoryObj`` buffers supplied by the MP controller.
* **metadata_cache_ttl_secs**: TTL for cached exact path-size metadata (default
``30``). Set this lower when another process may modify the same bucket
prefix outside LMCache and fresher metadata is more important than reducing
Hugging Face metadata calls.
* **num_workers**: number of worker threads used for blocking Hugging Face Hub
bucket API calls (default ``4``). The HFBucket Python APIs are synchronous,
so MP mode runs upload, lookup, load, and delete work on a bounded thread
pool behind the adapter's eventfd-based completion interface.
* **max_capacity_gb**: capacity used by ``get_usage()`` for watermark-based L2
eviction. Set to ``0`` (default) to disable aggregate capacity tracking;
``get_usage()`` then reports the adapter as not providing an eviction signal.
* **eviction**: optional sub-dict enabling the L2 eviction controller for this
adapter. When present, keys that are currently being loaded are protected by
the lookup-and-lock path and skipped by ``delete()`` until they are unlocked.
Differences vs Non-MP HFBucket
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* Hugging Face bucket operations are synchronous but the adapter makes submission
non-blocking by running the blocking calls on worker threads.
* MP loads do not allocate and return new memory. The MP controller provides
destination ``MemoryObj`` buffers, and the adapter copies downloaded bytes
into those buffers.
* Keys are identified by ``ObjectKey`` (``model_name`` + ``kv_rank`` +
``chunk_hash`` + optional ``cache_salt``) rather than ``CacheEngineKey``.
The serialized MP object name is
``<model>@<kv_rank_hex>@<chunk_hash_hex>[@<cache_salt>]`` and is then
encoded for the bucket path. This naming is not compatible with the non-MP
HFBucket connector's ``CacheEngineKey`` object names, so a bucket prefix
populated by non-MP LMCache cannot be read directly by MP LMCache and vice
versa.
* Full object writes are batch based. Hugging Face batch writes are not
transactional, so a failed store task may still leave some objects in the
bucket. The MP adapter reconciles backend metadata after such failures so
any objects that actually landed are counted for usage and later deletion
(submitted store task is still reported as failed).
Notes
-----
* The backend stores objects under the configured bucket prefix using a
reversible encoding of LMCache keys, so ``list()`` returns LMCache key strings
instead of raw bucket object paths.
docs/source/kv_cache/storage_backends/index.rst
+41
View File
@@ -0,0 +1,41 @@
Using Different Storage Backends
================================
.. warning::
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/l2_storage/index`.
LMCache supports various storage backends to offload and share KV cache data.
Supported Backends
-------------------------
.. toctree::
:maxdepth: 1
azure
cpu_ram
custom_backend
dax
eic
fs
gds
hfbucket
infinistore
local_storage
maru
mock
mooncake
nixl
redis
bigtable
resp
s3
sagemaker_hyperpod
valkey
weka
3fs
docs/source/kv_cache/storage_backends/infinistore.rst
+208
View File
@@ -0,0 +1,208 @@
InfiniStore
===========
.. warning::
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/l2_storage/index`.
.. _infinistore-overview:
Overview
--------
`InfiniStore <https://github.com/bytedance/InfiniStore>`_ is an open-source high-performance KV store. It's designed to support LLM Inference clusters, whether the cluster is in prefill-decoding disaggregation mode or not. InfiniStore provides high-performance and low-latency KV cache transfer and KV cache reuse among inference nodes in the cluster.
There are two major scenarios how InfiniStore supports:
* Prefill-Decoding disaggregation clusters: in such mode inference workloads are separated into two node pools: prefill nodes and decoding nodes. InfiniStore enables KV cache transfer among these two types of nodes, and also KV cache reuse.
* Non-disaggregated clusters: in such mode prefill and decoding workloads are mixed on every node. InfiniStore serves as an extra large KV cache pool in addition to GPU cache and local CPU cache, and also enables cross-node KV cache reuse.
.. image:: ../../assets/InfiniStore-usage.png
:alt: InfiniStore Usage Diagram
For more details, please refer to the `InfiniStore Documentation <https://bytedance.github.io/InfiniStore/index.html>`_.
InfiniStore supports both RDMA and TCP for transport. LMCache’s InfiniStore connector only uses the RDMA transport.
Quick Start
-----------
Install InfiniStore via pip:
.. code-block:: bash
pip install infinistore
This package includes the InfiniStore server and the Python bindings.
To build InfiniStore from source, follow the instructions in the `GitHub repository <https://github.com/bytedance/InfiniStore>`_.
Setup and Deployment
~~~~~~~~~~~~~~~~~~~~
**Prerequisites:**
- Machine with at least one GPU for vLLM inference
- RDMA-capable network hardware and drivers
- Python 3.8+ with pip
- vLLM and LMCache installed
**Step 1: Start InfiniStore Server**
For InfiniBand based RDMA:
.. code-block:: bash
infinistore --service-port 12345 --dev-name mlx5_0 --link-type IB
For RoCE based RDMA:
.. code-block:: bash
infinistore --service-port 12345 --dev-name mlx5_0 --link-type Ethernet
You can also specify the ``--hint-gid-index`` option to set the GID index for the InfiniStore server. This is useful when you are in a k8s managed environment.
**Step 2: Create Configuration File**
Create your ``infinistore-config.yaml``:
.. code-block:: yaml
chunk_size: 256
remote_url: "infinistore://127.0.0.1:12345/?device=mlx5_1"
remote_serde: "naive"
local_cpu: False
max_local_cpu_size: 5
**Step 3: Start vLLM with InfiniStore**
.. code-block:: bash
LMCACHE_CONFIG_FILE="infinistore-config.yaml" \
vllm serve \
Qwen/Qwen2.5-7B-Instruct \
--seed 42 \
--max-model-len 16384 \
--gpu-memory-utilization 0.8 \
--kv-transfer-config \
'{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
**Step 4: Verify the Setup**
Test the integration with a sample request:
.. code-block:: bash
curl -X POST "http://localhost:8000/v1/completions" \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-7B-Instruct",
"prompt": "The future of AI is",
"max_tokens": 100,
"temperature": 0.7
}'
**Debugging Tips:**
1. **Enable verbose logging:**
.. code-block:: bash
infinistore --log-level=debug
2. **Check server status:**
.. code-block:: bash
# Check if the server is running
ps aux | grep infinistore
netstat -tlnp | grep -E "12345"
Query TTFT Improvement
~~~~~~~~~~~~~~~~~~~~~~~
Once the OpenAI compatible server is running, let's query it twice and see the TTFT improvement.
Run vLLM's serving benchmark twice with the following parameters:
.. code-block:: bash
vllm bench serve \
--backend vllm \
--model Qwen/Qwen2.5-7B-Instruct \
--num-prompts 50 \
--port 8000 \
--host 127.0.0.1 \
--dataset-name random \
--random-input-len 8192 \
--random-output-len 128 \
--seed 42
**Example Output:**
For the first run, you might see:
.. code-block:: text
============ Serving Benchmark Result ============
Successful requests: 50
Benchmark duration (s): 80.97
Total input tokens: 409544
Total generated tokens: 6273
Request throughput (req/s): 0.62
Output token throughput (tok/s): 77.48
Total Token throughput (tok/s): 5135.74
---------------Time to First Token----------------
Mean TTFT (ms): 36203.54
Median TTFT (ms): 34598.91
P99 TTFT (ms): 76010.91
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 290.30
Median TPOT (ms): 346.25
P99 TPOT (ms): 412.24
---------------Inter-token Latency----------------
Mean ITL (ms): 290.30
Median ITL (ms): 386.78
P99 ITL (ms): 449.83
For the second run, you should see a significant reduction in TTFT:
.. code-block:: text
============ Serving Benchmark Result ============
Successful requests: 50
Benchmark duration (s): 15.14
Total input tokens: 409544
Total generated tokens: 6273
Request throughput (req/s): 3.30
Output token throughput (tok/s): 414.22
Total Token throughput (tok/s): 27457.55
---------------Time to First Token----------------
Mean TTFT (ms): 2880.53
Median TTFT (ms): 3118.50
P99 TTFT (ms): 12027.24
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 73.81
Median TPOT (ms): 71.12
P99 TPOT (ms): 91.24
---------------Inter-token Latency----------------
Mean ITL (ms): 73.81
Median ITL (ms): 63.86
P99 ITL (ms): 565.44
TTFT Improvement: 33.323 seconds (12.6x faster).
**Tips:**
- If you want to run vLLM's serving benchmark multiple times, you'll need to either restart the vLLM LMCache server and the InfiniStore server, or change the ``--seed`` parameter to a different value each time, since you've already warmed up LMCache.
- The benchmark result here was produced by running an L40 with 48GB of GPU memory with ``--gpu-memory-utilization 0.8``. You can adjust the GPU memory utilization and increase the max model length to use more of the long context. LMCache TTFT improvement becomes more pronounced as the context length increases!
Additional Resources
--------------------
- `InfiniStore Documentation <https://bytedance.github.io/InfiniStore/index.html>`_
- `GitHub Repository <https://github.com/bytedance/InfiniStore>`_
docs/source/kv_cache/storage_backends/local_storage.rst
+464
View File
@@ -0,0 +1,464 @@
Local storage
=============
.. warning::
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/l2_storage/index`.
.. _local-storage-overview:
Overview
--------
CPU RAM and Local Storage are the two ways of offloading KV cache onto non-GPU
memory of the same machine that is running inference.
Two ways to configure LMCache Disk Offloading:
----------------------------------------------
**1. Environment Variables:**
.. code-block:: bash
# 256 Tokens per KV Chunk
export LMCACHE_CHUNK_SIZE=256
# None if disabled
# Otherwise, enable by setting the directory where LMCache will
# create files for each KV cache chunks
# (this directory does NOT need to exist beforehand)
export LMCACHE_LOCAL_DISK="file://$HOME/local/disk_test/local_disk/"
# 5GB of Disk
export LMCACHE_MAX_LOCAL_DISK_SIZE=5.0
# Disable page cache
# This should be turned on for better performance if most local CPU memory is used
# Optionally tune the number of I/O worker threads (default: 4)
export LMCACHE_EXTRA_CONFIG='{"use_odirect": true, "disk_io_threads": 8}'
**2. Configuration File**:
Passed in through ``LMCACHE_CONFIG_FILE=your-lmcache-config.yaml``
.. code-block:: yaml
# 256 Tokens per KV Chunk
chunk_size: 256
# Enable Disk backend
local_disk: "file:///local/disk_test/local_disk/"
# 5GB of Disk memory
max_local_disk_size: 5.0
# Disable page cache
# This should be turned on for better performance if most local CPU memory is used
# Optionally tune the number of I/O worker threads (default: 4)
extra_config:
use_odirect: true
disk_io_threads: 8
Multi-Path (Multi-Device) Disk Offloading
-----------------------------------------
If you have **multiple NVMe devices** (or any independent mount points), you can
assign each GPU its own disk path so that each device writes to a dedicated drive.
Specify a **comma-separated list** of paths in ``local_disk``.
Each path can optionally use the ``file://`` prefix. The
``local_disk_path_sharding`` option controls how each GPU worker selects its
path. Currently only ``"by_gpu"`` is supported (the default), which selects a
path based on the device index (``device_id % num_paths``), so all KV cache
files from a given GPU land on the same NVMe. This is especially useful when
GPUs and NVMe devices share a PCIe switch or NUMA node.
For example, with two GPUs and two paths:
- ``cuda:0`` → ``/mnt/nvme0/kvcache/``
- ``cuda:1`` → ``/mnt/nvme1/kvcache/``
``max_local_disk_size`` is the **total budget** shared across all paths.
**Environment variable example:**
.. code-block:: bash
export LMCACHE_LOCAL_DISK="file:///mnt/nvme0/kvcache/,file:///mnt/nvme1/kvcache/"
export LMCACHE_LOCAL_DISK_PATH_SHARDING="by_gpu"
export LMCACHE_MAX_LOCAL_DISK_SIZE=20.0 # combined budget (GB)
**YAML example:**
.. code-block:: yaml
local_disk: "/mnt/nvme0/kvcache/,/mnt/nvme1/kvcache/"
local_disk_path_sharding: "by_gpu"
max_local_disk_size: 20.0
.. note::
Each GPU worker uses only its assigned path, so O_DIRECT alignment
is determined by that path's filesystem block size. Different
devices may have different block sizes without issue.
.. tip::
If you are able to use kernel-level RAID 0 (e.g. ``mdadm --level=0``)
you will get true block-level striping (even a single large file can
use bandwidth from both devices simultaneously). The multi-path
feature is most useful when you cannot or do not want to reconfigure
the block devices — for example, when they already have other data.
Local Storage Explanation:
--------------------------
Unlike CPU RAM offloading, disk offloading is *disabled* by default (``local_disk`` is set to ``None``) and the
max local disk size is set to 0GB instead of 5GB like the default max local cpu size
since the disk space is not strictly necessary for LMCache to function.
Furthermore, instead of greedily allocating the max space up front like the pinned CPU RAM, the disk backend will
create one file per KV cache chunk as they are stored, evicting if capacity is exceeded (LRU currently).
The disk and remote (see :doc:`Redis <./redis>`, :doc:`Mooncake <./mooncake>`, :doc:`Valkey <./valkey>`, :doc:`InfiniStore <./infinistore>`)
backends have asynchronous put() operations so that the IO latency will not slow down inference in addition to blocking get() operations.
The local disk backend also has a prefetch() operation that will preemptively move KV caches from the disk to CPU RAM offloading storage
(i.e. ``LMCACHE_LOCAL_CPU=True`` should be set, see :doc:`CPU RAM <./cpu_ram>`) for specified tokens (these KV caches are also still kept in the disk).
Architecture Overview
---------------------
The following diagram shows the overall architecture of the Local Disk Backend:
.. mermaid::
%%{init: {'theme': 'base', 'themeVariables': { 'fontSize': '18px', 'fontFamily': 'arial', 'primaryColor': '#e3f2fd', 'primaryTextColor': '#000', 'primaryBorderColor': '#1976d2', 'lineColor': '#424242', 'secondaryColor': '#f5f5f5', 'tertiaryColor': '#ffffff', 'background': '#ffffff', 'clusterBkg': '#f8f9fa', 'clusterBorder': '#495057' }}}%%
flowchart TB
subgraph Engine["<b>LMCache Engine</b>"]
E["<b>Request Save/Load Operations</b>"]
end
subgraph LDB["<b>LocalDiskBackend</b>"]
subgraph Meta["<b>Metadata Dictionary</b>"]
Dict["<b>self.dict: CacheEngineKey → DiskCacheMetadata</b>
(path, size, shape, dtype, pinned, positions)"]
end
subgraph Policy["<b>Cache Policy</b>"]
CP["<b>Configurable Policy</b>
(LRU, LFU, FIFO, MRU)
Decides what to evict"]
end
subgraph Worker["<b>LocalDiskWorker</b>"]
PQ["<b>Priority Queue Executor (configurable workers, default 4)</b>"]
P0["<b>Priority 0: PREFETCH</b>"]
P1["<b>Priority 1: DELETE</b>"]
P2["<b>Priority 2: PUT</b>"]
end
CPU["<b>LocalCPUBackend</b>
(memory allocator)"]
end
subgraph Disk["<b>Local Filesystem</b>"]
Files["/cache/vllm@model@...@abc.pt
/cache/vllm@model@...@def.pt
/cache/vllm@model@...@ghi.pt"]
end
E --> Dict
Dict --> CP
CP --> PQ
PQ --> P0
PQ --> P1
PQ --> P2
Worker --> Files
CPU -.-> Worker
**Key Components:**
- **Metadata Dictionary**: Maps each ``CacheEngineKey`` to its disk metadata (file path, size, shape, dtype, pin status)
- **Cache Policy**: Configurable eviction policy (LRU, LFU, FIFO, or MRU) that tracks access patterns and decides which entries to evict when space is needed
- **LocalDiskWorker**: Async task executor with priority queue - prefetch tasks run first (priority 0), then deletes (priority 1), then saves (priority 2). The number of I/O worker threads is configurable via ``extra_config.disk_io_threads`` (default: 4).
- **Local Disk**: Filesystem where KV cache chunks are stored as individual ``.pt`` files
Save Flow (PUT)
~~~~~~~~~~~~~~~
.. mermaid::
%%{init: {'theme': 'base', 'flowchart': {'useMaxWidth': false, 'htmlLabels': true, 'nodeSpacing': 30, 'rankSpacing': 30}, 'themeVariables': { 'fontSize': '18px', 'fontFamily': 'arial', 'primaryColor': '#e3f2fd', 'primaryTextColor': '#000', 'primaryBorderColor': '#1976d2', 'lineColor': '#424242', 'secondaryColor': '#f5f5f5', 'tertiaryColor': '#ffffff', 'background': '#ffffff', 'clusterBkg': '#f8f9fa', 'clusterBorder': '#495057' }}}%%
flowchart LR
A["<b>MemoryObj</b><br/>(KV cache in CPU memory)"] --> B{<b>Disk space<br/>available?</b>}
B -->|"No"| C["<b>Evict via policy</b><br/>Delete .pt files"]
C --> B
B -->|"Yes"| D["<b>Track in put_tasks</b><br/>Queue async write<br/>(Priority 2 - lowest)"]
D --> E["<b>LocalDiskWorker</b><br/>write_file()"]
E --> F[("<b>Disk</b><br/>.pt file")]
F --> G["<b>Add to metadata dict</b>"]
style A fill:#e1f5fe
style F fill:#c8e6c9
style C fill:#ffcdd2
Load Flow (GET)
~~~~~~~~~~~~~~~
.. mermaid::
%%{init: {'theme': 'base', 'flowchart': {'useMaxWidth': false, 'htmlLabels': true, 'nodeSpacing': 30, 'rankSpacing': 30}, 'themeVariables': { 'fontSize': '18px', 'fontFamily': 'arial', 'primaryColor': '#e3f2fd', 'primaryTextColor': '#000', 'primaryBorderColor': '#1976d2', 'lineColor': '#424242', 'secondaryColor': '#f5f5f5', 'tertiaryColor': '#ffffff', 'background': '#ffffff', 'clusterBkg': '#f8f9fa', 'clusterBorder': '#495057' }}}%%
flowchart LR
A["<b>Request</b><br/>(CacheEngineKey)"] --> B{<b>Key exists<br/>in dict?</b>}
B -->|"No"| C["<b>Return None</b><br/>(cache miss)"]
B -->|"Yes"| D["<b>Update policy</b><br/>Mark as accessed"]
D --> E["<b>Allocate buffer</b><br/>via LocalCPUBackend"]
E --> F["<b>Read from disk</b><br/>read_file()"]
F --> G[("<b>Disk</b><br/>.pt file")]
G --> F
F --> H["<b>MemoryObj</b><br/>(KV cache ready)"]
style A fill:#e1f5fe
style H fill:#c8e6c9
style C fill:#ffcdd2
.. _local-storage-online-inference-example:
Online Inference Example
------------------------
This example is almost identical to the :doc:`CPU RAM <./cpu_ram>` example.
Let's feel the TTFT (time to first token) differential!
.. _local-storage-prerequisites:
**Prerequisites:**
- A Machine with at least one GPU. Adjust the max model length of your vllm instance depending on your GPU memory and the long context you want to use.
- vllm and lmcache installed (:doc:`Installation Guide <../../getting_started/installation>`)
- Hugging Face access to ``meta-llama/Meta-Llama-3.1-8B-Instruct``
.. code-block:: bash
export HF_TOKEN=your_hugging_face_token
- A few packages:
.. code-block:: bash
pip install openai transformers
**Step 0. Set up a directory for this example:**
.. code-block:: bash
mkdir lmcache-local-disk-example
cd lmcache-local-disk-example
**Step 1. Prepare a long context!**
We want a context long enough that vllm's prefix caching will not be able to hold the KV caches in
GPU memory and LMCache is necessary to keep KV caches in non-GPU memory:
.. code-block:: bash
# 382757 bytes
man bash > man-bash.txt
**Step 2. Start a vLLM server with Disk offloading enabled:**
*Generally, it is not recommended but we will disable CPU offloading to feel just the disk offloading latency.*
Create a an lmcache configuration file called: ``disk-offload.yaml``
Example ``config.yaml``:
.. code-block:: yaml
chunk_size: 256
local_cpu: false
max_local_cpu_size: 5.0
local_disk: "file:///local/disk_test/local_disk/"
max_local_disk_size: 5.0
If you don't want to use a config file, uncomment the first five environment variables
and then comment out the ``LMCACHE_CONFIG_FILE`` below:
.. code-block:: bash
# LMCACHE_CHUNK_SIZE=256 \
# LMCACHE_LOCAL_CPU=False \
# LMCACHE_MAX_LOCAL_CPU_SIZE=5.0 \
# LMCACHE_LOCAL_DISK="file:///local/disk_test/local_disk/" \
# LMCACHE_MAX_LOCAL_DISK_SIZE=5.0 \
LMCACHE_CONFIG_FILE="disk-offload.yaml" \
vllm serve \
meta-llama/Llama-3.1-8B-Instruct \
--max-model-len 16384 \
--kv-transfer-config \
'{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
- ``--kv-transfer-config``: This is the parameter that actually tells vLLM to use LMCache for KV cache offloading.
- ``kv_connector``: Specifies the LMCache connector for vLLM V1
- ``kv_role``: Set to "kv_both" for both storing and loading KV cache (important because we will run two queries and the first will produce/store a KV cache while the second will consume/load that KV cache)
**Step 3. Query TTFT improvements with LMCache:**
Once the Open AI compatible server is running on default vllm port 8000, let's query it twice with the same long context!
Create a script called ``query-twice.py`` and paste the following code:
.. code-block:: python
import time
from openai import OpenAI
from transformers import AutoTokenizer
client = OpenAI(
api_key="dummy-key", # required by OpenAI client even for local servers
base_url="http://localhost:8000/v1"
)
models = client.models.list()
model = models.data[0].id
# 119512 characters total
# 26054 tokens total
long_context = ""
with open("man-bash.txt", "r") as f:
long_context = f.read()
# a truncation of the long context for the --max-model-len 16384
# if you increase the --max-model-len, you can decrease the truncation i.e.
# use more of the long context
long_context = long_context[:70000]
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct")
question = "Summarize bash in 2 sentences."
prompt = f"{long_context}\n\n{question}"
print(f"Number of tokens in prompt: {len(tokenizer.encode(prompt))}")
def query_and_measure_ttft():
start = time.perf_counter()
ttft = None
chat_completion = client.chat.completions.create(
messages=[{"role": "user", "content": prompt}],
model=model,
temperature=0.7,
stream=True,
)
for chunk in chat_completion:
chunk_message = chunk.choices[0].delta.content
if chunk_message is not None:
if ttft is None:
ttft = time.perf_counter()
print(chunk_message, end="", flush=True)
print("\n") # New line after streaming
return ttft - start
print("Querying vLLM server with cold LMCache Disk Offload")
cold_ttft = query_and_measure_ttft()
print(f"Cold TTFT: {cold_ttft:.3f} seconds")
print("\nQuerying vLLM server with warm LMCache Disk Offload")
warm_ttft = query_and_measure_ttft()
print(f"Warm TTFT: {warm_ttft:.3f} seconds")
print(f"\nTTFT Improvement: {(cold_ttft - warm_ttft):.3f} seconds \
({(cold_ttft/warm_ttft):.1f}x faster)")
Then run:
.. code-block:: bash
python query-twice.py
Since we're in streaming mode, you'll be able to feel the TTFT differential in
real time!
Note that if we were to enable ``LMCACHE_LOCAL_CPU=True``, we would just be using
the same example from :doc:`CPU RAM <./cpu_ram>` since the CPU RAM is checked before
the disk by LMCache. In practice, the disk will be capable of storing a larger
quantity of KV caches so the CPU RAM offloading will only be able to store a
subset of the disk's KV caches.
**Example Output:**
.. code-block:: text
Number of tokens in prompt: 15376
Querying vLLM server with cold LMCache Disk Offload
Bash is a Unix shell and command-line interpreter that reads and executes
commands from standard input or a file, incorporating features from the
Korn and C shells. It is a conformant implementation of the IEEE POSIX
specification and can be configure to be POSIX-conformant by default,
supporting a wide range of options, built-in commands,
and features for scripting, job control, and interactive use.
Cold TTFT: 6.314 seconds
Querying vLLM server with warm LMCache Disk Offload
Bash is a Unix shell and command-line interpreter that reads and
executes commands from the standard input or a file, and is designed
to be a conformant implementation of the IEEE POSIX specification. It
is a powerful tool for automating tasks, managing files and directories,
and interacting with other programs and services, with features such as
scripting, conditional statements, loops, and functions.
Warm TTFT: 0.148 seconds
TTFT Improvement: 6.166 seconds (42.6x faster)
If you look at the logs of your vLLM server, you should see (the logs are truncated for cleanliness):
.. code-block:: text
# Cold LMCache Miss and then Store
LMCache INFO: Reqid: chatcmpl-8676f9b9ebf04c79a5d47b9ada7b65fd, Total tokens 15410,
LMCache hit tokens: 0, need to load: 0
# you should see 8 of these storing logs total
# 2048 tokens is a multiple of the chunk size
LMCache INFO: Storing KV cache for 2048 out of 12288 tokens for request
chatcmpl-8676f9b9ebf04c79a5d47b9ada7b65fd
LMCache INFO: Storing KV cache for 2048 out of 14336 tokens for request
chatcmpl-8676f9b9ebf04c79a5d47b9ada7b65fd
LMCache INFO: Storing KV cache for 1074 out of 15410 tokens for request
chatcmpl-8676f9b9ebf04c79a5d47b9ada7b65fd
# Warm LMCache Hit!!
LMCache INFO: Reqid: chatcmpl-136d9dac1ba94bd4b4ae85007e8ad437, Total tokens 15410,
LMCache hit tokens: 15409, need to load: 1
Check out your KV Cache in your SSD:
.. code-block:: bash
ls "$HOME/local/disk_test/local_disk/"
.. _local-storage-tips:
Tips:
-----
- If you want to run the ``query-twice.py`` script multiple times, you'll need to either restart the vLLM LMCache server or change the prefix of the context you pass in since you've already warmed LMCache.
- The max model length here was decided by running an L4 with only 23GB of GPU memory. If you have more memory, you can increase the max model length and modify ``query-twice.py`` to use more of the long context. LMCache TTFT improvement becomes more pronounced as the context length increases!
docs/source/kv_cache/storage_backends/maru.rst
+118
View File
@@ -0,0 +1,118 @@
Maru
====
.. warning::
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/l2_storage/index`.
.. _maru-overview:
Overview
--------
`Maru <https://github.com/xcena-dev/maru>`_ is a high-performance KV cache storage engine built on CXL shared memory,
designed for LLM inference scenarios where multiple instances need to share a KV cache with minimal latency.
.. image:: ../../assets/maru-kvcache.png
:alt: KV Cache Sharing: Without vs With Maru
For architecture details, see the `Maru documentation <https://xcena-dev.github.io/maru/>`_.
Quick Start
-----------
Install Maru:
.. code-block:: bash
git clone https://github.com/xcena-dev/maru.git
cd maru
./install.sh
This installs ``maru-server``, ``maru-resourced``, and the ``maru`` Python package.
Deploy Model With Maru
~~~~~~~~~~~~~~~~~~~~~~
**Prerequisites:** CXL device (``/dev/dax*``), Python 3.12+, vLLM and LMCache installed.
**1. Start the Maru Server**
.. code-block:: bash
maru-server
**2. Create configuration file** (``maru-config.yaml``):
.. code-block:: yaml
chunk_size: 256
local_cpu: False
max_local_cpu_size: 0
save_unfull_chunk: True
# Maru backend
maru_path: "maru://localhost:5555"
maru_pool_size: 4
**3. Start vLLM with Maru**
.. code-block:: bash
LMCACHE_CONFIG_FILE="maru-config.yaml" \
vllm serve \
meta-llama/Llama-3.1-8B-Instruct \
--max-model-len 65536 \
--kv-transfer-config \
'{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
Configuration
-------------
**LMCache Parameters:**
.. list-table::
:header-rows: 1
:widths: 25 15 60
* - Parameter
- Default
- Description
* - ``maru_path``
- Required
- Maru server URL (format: ``maru://host:port``)
* - ``maru_pool_size``
- ``4.0``
- CXL memory pool size per instance in GB (e.g., ``4``, ``0.5``)
**Advanced Parameters (via extra_config):**
.. list-table::
:header-rows: 1
:widths: 25 15 60
* - Parameter
- Default
- Description
* - ``maru_instance_id``
- auto UUID
- Unique client instance identifier
* - ``maru_timeout_ms``
- 5000
- ZMQ RPC socket timeout in milliseconds
* - ``maru_use_async_rpc``
- true
- Async DEALER-ROUTER RPC (``false`` for synchronous REQ-REP)
* - ``maru_max_inflight``
- 64
- Max concurrent async RPC requests
* - ``maru_eager_map``
- true
- Pre-map all shared regions on connect
Additional Resources
--------------------
- `Maru GitHub Repository <https://github.com/xcena-dev/maru>`_
- `Maru Documentation <https://xcena-dev.github.io/maru/>`_
docs/source/kv_cache/storage_backends/mock.rst
+70
View File
@@ -0,0 +1,70 @@
Mock
====
.. warning::
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/l2_storage/index`.
LMCache provides a mock remote connector that allows you to manually set the peeking latency, read throughput, and write throughput inside of the remote url. It will create copies of your KV cache in unmanaged local RAM.
Configuration
-------------
Create a configuration file (e.g., ``mock.yaml``) with the following content:
.. code-block:: yaml
chunk_size: 256
local_cpu: False
max_local_cpu_size: 10
remote_url: "mock://100/?peeking_latency=1&read_throughput=2&write_throughput=2"
The ``remote_url`` format is ``mock://SIZE/?peeking_latency=LATENCY&read_throughput=READ_GBPS&write_throughput=WRITE_GBPS`` where:
- ``SIZE``: Maximum storage size
- ``peeking_latency``: Latency for peeking operations (in milliseconds)
- ``read_throughput``: Read throughput (in GB/s)
- ``write_throughput``: Write throughput (in GB/s)
Usage
-----
Deploy a serving engine with the mock remote backend:
.. code-block:: bash
LMCACHE_CONFIG_FILE=mock.yaml vllm serve meta-llama/Llama-3.1-8B-Instruct \
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}' \
--disable-log-requests \
--no-enable-prefix-caching
Check the retrieval (storing is async so the throughput there is meaningless) logs on the second query to confirm that the throughput is slightly lower than 2 GB/s (the CPU <-> GPU allocation/transfer also has overhead).
Example Query
-------------
Send a test request:
.. code-block:: bash
curl -X POST http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"prompt": "'"$(printf 'Elaborate the significance of KV cache in language models. %.0s' {1..1000})"'",
"max_tokens": 10
}'
Expected Logs
-------------
You should see logs similar to the following:
.. code-block:: text
(EngineCore_0 pid=586318) [2025-09-03 05:06:41,751] LMCache INFO: Reqid: cmpl-b34e7c5b2f3e46a592722db2c27f6fc0-0, Total tokens 12002, LMCache hit tokens: 12002, need to load: 12001 (vllm_v1_adapter.py:1049:lmcache.integration.vllm.vllm_v1_adapter)
(EngineCore_0 pid=586318) [2025-09-03 05:06:42,736] LMCache INFO: Retrieved 12002 out of total 12002 out of total 12002 tokens. size: 1.651 gb, cost 980.6983 ms, throughput: 1.8939 GB/s; (cache_engine.py:503:lmcache.v1.cache_engine)
The logs confirm that the throughput is slightly lower than 2 GB/s due to CPU <-> GPU allocation/transfer overhead.
docs/source/kv_cache/storage_backends/mooncake.rst
+255
View File
@@ -0,0 +1,255 @@
Mooncake
========
.. warning::
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/l2_storage/mooncake_store`.
.. _mooncake-overview:
Overview
--------
`Mooncake <https://github.com/kvcache-ai/Mooncake>`_ is an open-source distributed KV cache storage system designed specifically for LLM inference scenarios.
The system creates a distributed memory pool by aggregating memory space contributed by various client nodes, enabling efficient resource utilization across clusters.
By pooling underutilized DRAM and SSD resources from multiple nodes, the system forms a unified distributed storage service that maximizes resource efficiency.
.. image:: ../../assets/mooncake-store-preview.png
:alt: Mooncake Architecture Diagram
Key Features
~~~~~~~~~~~~
- **Distributed memory pooling**: Aggregates memory contributions from multiple client nodes into a unified storage pool
- **High bandwidth utilization**: Supports striping and parallel I/O transfer of large objects, fully utilizing multi-NIC aggregated bandwidth
- **RDMA optimization**: Built on Transfer Engine with support for TCP, RDMA (InfiniBand/RoCEv2/eRDMA/NVIDIA GPUDirect)
- **Dynamic resource scaling**: Supports dynamically adding and removing nodes for elastic resource management
For detailed architecture information, see the `Mooncake Architecture Guide <https://kvcache-ai.github.io/Mooncake/design/mooncake-store.html>`_.
Quick Start
-----------
Install Mooncake via pip:
.. code-block:: bash
pip install mooncake-transfer-engine
This package includes all necessary components:
- ``mooncake_master``: Master service that manages cluster metadata and coordinates distributed storage operations
- ``mooncake_http_metadata_server``: HTTP-based metadata server used by the underlying transfer engine for connection establishment
- Mooncake Python bindings
For production deployments or custom builds, see the `Build Instructions <https://kvcache-ai.github.io/Mooncake/getting_started/build.html>`_.
Setup and Deployment
~~~~~~~~~~~~~~~~~~~~
**Prerequisites:**
- Machine with at least one GPU for vLLM inference
- RDMA-capable network hardware and drivers (recommended) or TCP network
- Python 3.8+ with pip
- vLLM and LMCache installed
**Step 1: Start Infrastructure Services**
Start the Mooncake master service (with built‑in HTTP metadata server):
.. code-block:: bash
# Master service (use -v=1 for verbose logging)
# The flag enables the integrated HTTP metadata server
mooncake_master --enable_http_metadata_server=1
Expected output:
.. code-block:: text
Master service started on port 50051
HTTP metrics server started on port 9003
Master Metrics: Storage: 0.00 B / 0.00 B | Keys: 0 | ...
**Step 2: Create Configuration File**
Create your ``mooncake-config.yaml``:
.. code-block:: yaml
# LMCache Configuration
local_cpu: False
remote_url: "mooncakestore://localhost:50051/"
max_local_cpu_size: 2 # small local buffer
numa_mode: "auto" # reduce tail latency with multi-NUMA/multi-NIC
pre_caching_hash_algorithm: sha256_cbor_64bit
# Mooncake Configuration (via extra_config)
extra_config:
use_exists_sync: true
save_chunk_meta: False # Enable chunk metadata optimization
local_hostname: "localhost"
metadata_server: "http://localhost:8080/metadata"
protocol: "rdma"
device_name: "" # leave empty; autodetect device(s)
global_segment_size: 21474836480 # 20 GiB per worker
master_server_address: "localhost:50051"
local_buffer_size: 0 # rely on LMCache local_cpu as the buffer
mooncake_prefer_local_alloc: true # prefer local segment if available
**Step 3: Start vLLM with Mooncake**
.. code-block:: bash
# If you see persistent misses (no Mooncake hits), make sure
# PYTHONHASHSEED is fixed across processes (e.g., export PYTHONHASHSEED=0).
LMCACHE_CONFIG_FILE="mooncake-config.yaml" \
vllm serve \
meta-llama/Llama-3.1-8B-Instruct \
--max-model-len 65536 \
--kv-transfer-config \
'{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
**Step 4: Verify the Setup**
Test the integration with a sample request:
.. code-block:: bash
curl -X POST "http://localhost:8000/v1/completions" \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"prompt": "The future of AI is",
"max_tokens": 100,
"temperature": 0.7
}'
**Debugging Tips:**
1. **Enable verbose logging:**
.. code-block:: bash
mooncake_master -v=1
2. **Check service status:**
.. code-block:: bash
# Check if services are running
ps aux | grep mooncake
netstat -tlnp | grep -E "(8080|50051)"
3. **Monitor metrics:**
Access metrics at ``http://localhost:9003`` when master service is running.
Configuration
-------------
**LMCache Parameters:**
.. list-table::
:header-rows: 1
:widths: 25 15 60
* - Parameter
- Default
- Description
* - ``chunk_size``
- 256
- Number of tokens per KV chunk
* - ``remote_url``
- Required
- Mooncake store connection URL (format: ``mooncakestore://host:port/``).
* - ``remote_serde``
- "naive"
- Serialization method for remote storage
* - ``local_cpu``
- False
- Enable/disable local CPU caching (set to False for pure Mooncake evaluation)
* - ``max_local_cpu_size``
- Required
- Maximum local CPU cache size in GB (required even when local_cpu is False)
* - ``numa_mode``
- "auto"
- NUMA binding mode. "auto" is recommended on multi‑NIC/multi‑NUMA systems to reduce tail latency.
* - ``pre_caching_hash_algorithm``
- "sha256_cbor_64bit"
- Hash used for pre-caching keying. For cross‑process consistency, fix ``PYTHONHASHSEED`` (e.g., export ``PYTHONHASHSEED=0``).
**Mooncake Parameters (via extra_config):**
.. list-table::
:header-rows: 1
:widths: 25 15 60
* - Parameter
- Default
- Description
* - ``local_hostname``
- Required
- Hostname/IP of the local node for Mooncake client identification
* - ``metadata_server``
- Required
- HTTP metadata server address. When starting master with ``--enable_http_metadata_server=1``, it exposes this endpoint.
* - ``master_server_address``
- Required
- Mooncake master service address (host:port format)
* - ``protocol``
- "rdma"
- Communication protocol ("rdma" for high performance; "tcp" for compatibility)
* - ``device_name``
- ""
- RDMA device specification (e.g., "erdma_0,erdma_1" or "mlx5_0,mlx5_1"). Leave empty for autodetection in most setups.
* - ``global_segment_size``
- 21474836480
- **Memory size contributed by each vLLM worker** in bytes (e.g., 20 GiB recommended)
* - ``local_buffer_size``
- 0
- Local buffer size in bytes used by Mooncake. Behavior depends on ``save_chunk_meta``:
- When ``save_chunk_meta: False`` (recommended), LMCache uses its local CPU backend for zero‑copy RDMA, so Mooncake's ``local_buffer_size`` can be ``0``.
- When ``save_chunk_meta: True``, Mooncake uses its own local buffer; set this to a proper value (e.g., several GiB).
- Note: Some RDMA NICs have memory registration limits; registering LMCache's large CPU buffer can fail on constrained devices. In those cases, consider enabling ``save_chunk_meta: True`` and sizing ``local_buffer_size`` instead.
* - ``transfer_timeout``
- 1
- Timeout for transfer operations in seconds
* - ``storage_root_dir``
- ""
- The root directory for persistence (e.g., "/mnt/mooncake")
* - ``save_chunk_meta``
- False
- Whether to save chunk metadata alongside data. Set to ``False`` to enable the optimized zero‑copy path in LMCache.
* - ``use_exists_sync``
- False
- Use synchronous existence checks to avoid async scheduling overhead in hot paths.
* - ``mooncake_prefer_local_alloc``
- False
- Prefer allocating on the local segment when possible.
.. important::
**Understanding global_segment_size**: This parameter defines the amount of memory each vLLM worker contributes to the distributed memory pool.
The total cluster memory available for KV cache storage will be: ``number_of_vllm_workers × global_segment_size``.
Adjust this value based on your available system memory and expected cache requirements.
.. tip::
If you consistently get misses (no Mooncake hits), ensure all processes use the same hashing seed: ``export PYTHONHASHSEED=0``. This keeps pre‑caching keys consistent across processes.
.. note::
RDMA device(s) usually do not need to be specified; leaving ``device_name`` empty works for most deployments.
Additional Resources
--------------------
- `Mooncake Store Architecture <https://kvcache-ai.github.io/Mooncake/design/mooncake-store.html>`_
- `Mooncake Store Deployment Guide <https://kvcache-ai.github.io/Mooncake/deployment/mooncake-store-deployment-guide.html>`_
- `Mooncake Store Python API Reference <https://kvcache-ai.github.io/Mooncake/python-api-reference/mooncake-store.html>`_
- `Transfer Engine Documentation <https://kvcache-ai.github.io/Mooncake/design/transfer-engine/index.html>`_
- `Build Instructions <https://kvcache-ai.github.io/Mooncake/getting_started/build.html>`_
- `GitHub Repository <https://github.com/kvcache-ai/Mooncake>`_
- `LMCache Integration Guide <https://kvcache-ai.github.io/Mooncake/getting_started/examples/lmcache-integration.html>`_
docs/source/kv_cache/storage_backends/nixl.rst
+275
View File
@@ -0,0 +1,275 @@
Nixl
====
.. warning::
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/l2_storage/nixl`.
.. _nixl-overview:
Overview
--------
NIXL (NVIDIA Inference Xfer Library) is a high-performance library designed for accelerating point to point communications in AI inference frameworks. It provides an abstraction over various types of memory (CPU and GPU) and storage through a modular plug-in architecture, enabling efficient data transfer and coordination between different components of the inference pipeline.
LMCache supports using NIXL as a storage backend, allowing using NIXL to save either GPU or CPU memory into storage.
Prerequisites
~~~~~~~~~~~~~
- **LMCache**: Install with ``pip install lmcache``
- **NIXL**: Install from `NIXL GitHub repository <https://github.com/ai-dynamo/nixl>`_
- **Model Access**: Valid Hugging Face token (HF_TOKEN) for Llama 3.1 8B Instruct
Ways to configure LMCache NIXL Offloading
-----------------------------------------
**Configuration File**:
Passed in through ``LMCACHE_CONFIG_FILE=lmcache-config.yaml``
Example ``lmcache-config.yaml`` for POSIX backend:
.. code-block:: yaml
chunk_size: 256
nixl_buffer_device: cpu
local_cpu_use_hugepages: true # optional, requires pre-allocated hugepages
extra_config:
enable_nixl_storage: true
nixl_backend: POSIX
nixl_pool_size: 64
nixl_path: /mnt/nixl/cache/
use_direct_io: true
Key settings:
- ``nixl_buffer_size``: buffer size for NIXL transfers. **GPU mode only** (``nixl_buffer_device: cuda``). Setting this with ``nixl_buffer_device: cpu`` is a configuration error and will be rejected — in CPU mode NIXL shares ``LocalCPUBackend``'s pinned pool, which is sized by ``max_local_cpu_size``.
- ``max_local_cpu_size``: size of ``LocalCPUBackend``'s pinned pool in GiB. In CPU mode, this pool is shared with NIXL and must accommodate both the hot cache and concurrent NIXL I/O in flight. Must be > 0 when ``nixl_buffer_device: cpu``. Default: ``5.0``.
- ``nixl_pool_size``: number of descriptors opened at init time for nixl backend. Set to 0 for dynamic mode.
- ``nixl_path``: directory (or list of directories) under which the storage files will be saved (e.g. /mnt/nixl/). Needed for NIXL backends that store to file. When using a list of paths with ``path_sharding``, paths will be selected based on the sharding strategy.
- ``nixl_buffer_device``: dictates where the memory managed by NIXL should be on. "cpu" or "cuda" is supported for "GDS", "GDS_MT", and "OBJ" backends - for "POSIX", "HF3FS", "AZURE_BLOB" & "DOCA_MEMOS", must be "cpu". In CPU mode, NIXL shares ``LocalCPUBackend``'s pinned buffer; ``LocalCPUBackend`` is always created when ``nixl_buffer_device: cpu``, regardless of the ``local_cpu`` setting. ``local_cpu: false`` still suppresses hot-cache promotions — the backend acts as a staging buffer only, mirroring how ``local_disk`` already uses ``LocalCPUBackend``.
- ``nixl_backend``: configuration of which nixl backend to use for storage.
- ``nixl_path_sharding``: strategy for selecting path when multiple paths are provided. Currently only "by_gpu" is supported, which selects paths based on GPU device ID.
- ``local_cpu_use_hugepages``: whether to use Linux hugepages (2 MiB) for ``LocalCPUBackend``'s pinned pool (which NIXL shares in CPU mode). Requires pre-allocated hugepages (``sysctl vm.nr_hugepages``). Default: ``false``. **Deprecated alias:** ``extra_config.nixl_use_hugepages`` — accepted with a warning and copied into this field; will be removed in a future release.
.. note::
In CPU mode, the shared paged allocator consumes one full page per object. With ``save_unfull_chunk: true`` (only valid in static mode — dynamic mode rejects it; see "Dynamic Mode" → "Restrictions" below), partial chunks still occupy a full page each, so effective capacity degrades proportionally to the fraction of unfull last chunks across active sequences.
.. note::
``enable_p2p: true`` is rejected together with ``nixl_buffer_device: cpu``. The combination is structurally supported — both backends share ``LocalCPUBackend``'s pinned pool, each runs its own NIXL agent over it, and allocations route through ``LocalCPUBackend.allocate()`` — but it has not been exercised end-to-end and has no CI coverage. Use ``enable_p2p: true`` with ``nixl_buffer_device: cuda`` instead, or disable ``enable_p2p`` when running the NIXL CPU shared pool.
- ``nixl_presence_cache``: whether to keep an in-DRAM presence cache of keys known to exist, so repeated existence checks for the same key are answered locally instead of via a NIXL ``query_memory`` call. Applies to the dynamic backend (``nixl_pool_size: 0``). Default: ``false``.
- ``nixl_presence_cache_only``: when ``true``, the dynamic NIXL backend treats the local presence cache (and in-progress put set) as authoritative for existence checks. If a key is not known locally, lookup reports a miss **without** issuing a NIXL ``query_memory`` check. This requires ``nixl_presence_cache: true`` and can intentionally produce **false negatives** for objects that exist in the underlying storage but are absent from local presence metadata — giving "DRAM-only metadata" semantics where a process restart always yields a logically empty cache. Default: ``false``.
.. note::
This is a **lookup/existence-check** mode, not a "never touch the underlying storage" policy. It gates ``contains`` / ``batched_contains`` (and the async variant); direct retrieval still reads from the underlying storage. In normal operation a retrieval is only issued for a key that lookup already reported as present, so locally-unknown keys are not fetched. The option is only consulted by the dynamic backend (``nixl_pool_size: 0``); it is accepted but unused for static configurations.
.. note::
Supported backends are: ["GDS", "GDS_MT", "POSIX", "HF3FS", "OBJ", "AZURE_BLOB", "DOCA_MEMOS"].
Backend specific params should be provided via ``extra_config.nixl_backend_params``. Please refer to NIXL documentation for specifics.
Example ``lmcache-config.yaml`` for POSIX backend with multipath support:
.. code-block:: yaml
chunk_size: 256
nixl_buffer_size: 1073741824 # 1GB
nixl_buffer_device: cpu
extra_config:
enable_nixl_storage: true
nixl_backend: POSIX
nixl_pool_size: 64
nixl_path:
- /mnt/nixl/cache0/
- /mnt/nixl/cache1/
- /mnt/nixl/cache2/
nixl_path_sharding: by_gpu
use_direct_io: True
Example ``lmcache-config.yaml`` for OBJ backend using S3 API:
.. code-block:: yaml
chunk_size: 256
nixl_buffer_device: cpu
max_local_cpu_size: 1 # GiB
extra_config:
enable_nixl_storage: true
nixl_backend: OBJ
nixl_pool_size: 64
nixl_path: /mnt/nixl/cache/
nixl_backend_params:
access_key: <your_access_key>
secret_key: <your_secret_key>
bucket: <your_bucket>
region: <your_region>
Example ``lmcache-config.yaml`` for POSIX backend using liburing:
.. note::
using POSIX backend with liburing requires NIXL to be built with liburing support.
.. code-block:: yaml
chunk_size: 256
nixl_buffer_device: cpu
max_local_cpu_size: 1 # GiB
extra_config:
enable_nixl_storage: true
nixl_backend: POSIX
nixl_pool_size: 64
nixl_path: /mnt/nixl/cache/
use_direct_io: True
nixl_backend_params:
use_uring: "true"
Example ``lmcache-config.yaml`` for AZURE_BLOB backend to offload using Azure Blob Storage API:
.. code-block:: yaml
chunk_size: 256
nixl_buffer_device: cpu
max_local_cpu_size: 1 # GiB
extra_config:
enable_nixl_storage: true
nixl_backend: AZURE_BLOB
nixl_pool_size: 64
nixl_path: /mnt/nixl/cache/
nixl_backend_params:
account_url: https://<your_azure_storage_account_name>.blob.core.windows.net
container_name: <your_container_name>
Per-Worker Endpoint Distribution
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When using the OBJ backend with multiple tensor-parallel (TP) workers, you can
distribute workers across multiple object-storage endpoints by providing a list of
endpoints via ``nixl_endpoint_list``. Each worker selects an endpoint in
round-robin order based on its ``local_worker_id`` (the worker ID within its host).
.. code-block:: yaml
extra_config:
enable_nixl_storage: true
nixl_backend: OBJ
nixl_pool_size: 64
nixl_path: /mnt/nixl/cache/
nixl_endpoint_list:
- https://node-0.object-storage:9021
- https://node-1.object-storage:9021
- https://node-2.object-storage:9021
nixl_backend_params:
access_key: <your_access_key>
secret_key: <your_secret_key>
bucket: <your_bucket>
region: <your_region>
.. note::
When ``nixl_endpoint_list`` is set, any ``endpoint_override`` value in
``nixl_backend_params`` is ignored (a warning is logged).
``nixl_endpoint_list`` is only honored for the OBJ backend; it is ignored
for all other backends (including DOCA_MEMOS, AZURE_BLOB, and the file
backends).
Dynamic Mode
~~~~~~~~~~~~~
Nixl Storage Backend also supports a dynamic mode, which creates nixl storage descriptors on demand instead of at init time.
In order to use dynamic mode, extra_config.nixl_pool_size should be set to 0.
Restrictions
^^^^^^^^^^^^
- Dynamic mode is supported for object backends ("OBJ", "AZURE_BLOB", "DOCA_MEMOS") and file backends ("POSIX", "GDS", "GDS_MT", "HF3FS").
- save_unfull_chunk must be set to False.
Example ``lmcache-config.yaml`` for OBJ backend with dynamic mode:
.. code-block:: yaml
chunk_size: 256
local_cpu: False
save_unfull_chunk: False
enable_async_loading: False # set to True to test async loading
nixl_buffer_device: cpu
max_local_cpu_size: 3 # GiB
extra_config:
enable_nixl_storage: true
nixl_backend: OBJ
nixl_pool_size: 0
nixl_presence_cache: False
nixl_async_put: False
nixl_backend_params:
access_key: <your_access_key>
secret_key: <your_secret_key>
bucket: <your_bucket>
region: <your_region>
endpoint_override: https://url-to-object-storage
ca_bundle: path to self-signed certificate # remove this line if not using self-signed certificate
Example ``lmcache-config.yaml`` for AZURE_BLOB backend with dynamic mode:
.. code-block:: yaml
chunk_size: 256
local_cpu: False
save_unfull_chunk: False
enable_async_loading: False # set to True to test async loading
nixl_buffer_device: cpu
max_local_cpu_size: 3 # GiB
extra_config:
enable_nixl_storage: true
nixl_backend: AZURE_BLOB
nixl_pool_size: 0
nixl_presence_cache: False
nixl_async_put: False
nixl_backend_params:
account_url: https://<your_azure_storage_account_name>.blob.core.windows.net
container_name: <your_container_name>
DOCA_MEMOS Backend (NVIDIA CMX)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
``DOCA_MEMOS`` stores KV cache on NVIDIA CMX (Context Memory Storage), a
BlueField-4 context-memory tier accessed through NIXL. It is an object-style
backend (like ``OBJ``), supported in both static (``nixl_pool_size`` > 0) and
dynamic (``nixl_pool_size`` = 0) mode. ``nixl_buffer_device`` must be ``cpu``.
``nixl_endpoint_list`` is not supported for DOCA_MEMOS.
Object names are 128-bit lowercase-hex strings: the NIXL DOCA_MEMOS plugin
passes object names as strings and hex-decodes them on the device side, so
each name is exactly 32 hex characters. In dynamic mode this name is a
truncated SHA-256 of the cache key, so names are opaque (they carry no
model/chunk debug information) and uniqueness is probabilistic at 128 bits.
.. code-block:: yaml
chunk_size: 256
nixl_buffer_device: cpu
max_local_cpu_size: 1 # GiB
extra_config:
enable_nixl_storage: true
nixl_backend: DOCA_MEMOS
nixl_pool_size: 64
nixl_backend_params:
# refer to NIXL DOCA_MEMOS plugin docs for connection params
docs/source/kv_cache/storage_backends/redis.rst
+455
View File
@@ -0,0 +1,455 @@
Redis
=====
.. warning::
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/l2_storage/resp`.
.. _redis-overview:
Overview
--------
Redis is an in-memory key-value store and is a supported option for remote KV Cache offloading in LMCache.
Some other remote backends are :doc:`Mooncake <./mooncake>`, :doc:`Valkey <./valkey>`, and :doc:`InfiniStore <./infinistore>`.
This guide will mainly focus on single-node Redis but also shows you how to set up Redis Sentinels and an LMCache Server.
Two ways to configure LMCache Redis Offloading:
-----------------------------------------------
**1. Environment Variables:**
.. code-block:: bash
# 256 Tokens per KV Chunk
export LMCACHE_CHUNK_SIZE=256
# Redis host
export LMCACHE_REMOTE_URL="redis://your-redis-host:6379"
# Redis Sentinel hosts (for high availability)
# export LMCACHE_REMOTE_URL="redis-sentinel://localhost:26379,localhost:26380,localhost:26381"
# LMCache Server host
# export LMCACHE_REMOTE_URL="lm://localhost:65432"
# How to serialize and deserialize KV cache on remote transmission
export LMCACHE_REMOTE_SERDE="naive" # "naive" (default) or "cachegen"
**2. Configuration File**:
Passed in through ``LMCACHE_CONFIG_FILE=your-lmcache-config.yaml``
Example ``config.yaml``:
.. code-block:: yaml
# 256 Tokens per KV Chunk
chunk_size: 256
# Redis host
remote_url: "redis://your-redis-host:6379"
# Redis Sentinel hosts (for high availability)
# remote_url: "redis-sentinel://localhost:26379,localhost:26380,localhost:26381"
# LMCache Server host
# remote_url: "lm://localhost:65432"
# How to serialize and deserialize KV cache on remote transmission
remote_serde: "naive" # "naive" (default) or "cachegen"
Dynamic Plugin Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: yaml
# 256 Tokens per KV Chunk
chunk_size: 256
local_cpu: true
remote_storage_plugins:
- "redis"
extra_config:
remote_storage_plugin.redis.redis_url: "redis://your-redis-host:6379"
Remote Storage Explanation:
----------------------------
LMCache's backend is obeys the natural memory hierarchy of prioritizing CPU RAM offloading, then Local Storage
offloading, and finally remote offloading.
For LMCache to know how to create a connector to a remote backend, you must specify in
``remote_url`` a connector type followed by one or most host:port pairs (depending on what connector type is used).
If ``remote_url`` is set to ``None``, LMCache will not use any remote storage.
Examples of ``remote_url``'s:
.. code-block:: yaml
remote_url: "redis://your-redis-host:6379"
remote_url: "redis-sentinel://localhost:26379,localhost:26380,localhost:26381"
remote_url: "lm://localhost:65432"
remote_url: "infinistore://127.0.0.1:12345"
remote_url: "mooncakestore://127.0.0.1:50051"
Remote Storage Example
-----------------------
.. _redis-prerequisites:
**Prerequisites:**
- A Machine with at least one GPU. You can adjust the max model length of your vllm instance depending on your GPU memory.
- vllm and lmcache installed (:doc:`Installation Guide <../../getting_started/installation>`)
- Hugging Face access to ``meta-llama/Meta-Llama-3.1-8B-Instruct``
.. code-block:: bash
export HF_TOKEN=your_hugging_face_token
**Step 0. Set up a directory for this example:**
.. code-block:: bash
mkdir lmcache-redis-offload-example
cd lmcache-redis-offload-example
**Step 1. Start a Redis server:**
.. code-block:: bash
# Ubuntu / Debian Installation
sudo apt-get install redis
redis-server # starts the server on default port 6379
Check if Redis is running:
.. code-block:: bash
redis-cli ping
Expected Response:
.. code-block:: text
PONG
**Optional: Setting up Sentinels:**
To enable high availability with Redis, you can configure Redis sentinels to
monitor the master and automatically fail over to a replica if needed.
**Step 1a. Start a Redis replica:**
.. code-block:: bash
redis-server --port 6380 --replicaof 127.0.0.1 6379
**Step 1b. Create Sentinel configuration files:**
Create three files: ``sentinel-26379.conf``, ``sentinel-26380.conf``, and ``sentinel-26381.conf``, with contents like this:
.. code-block:: ini
port 26379 # Use 26380 and 26381 in other files respectively
sentinel monitor mymaster 127.0.0.1 6379 1
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 10000
sentinel parallel-syncs mymaster 1
**Step 1c. Start each Sentinel:**
.. code-block:: bash
redis-server sentinel-26379.conf --sentinel
redis-server sentinel-26380.conf --sentinel
redis-server sentinel-26381.conf --sentinel
**Step 1d. Make sure the Sentinels are tracking the master:**
.. code-block:: bash
redis-cli -p 26379 sentinel master mymaster
redis-cli -p 26380 sentinel master mymaster
redis-cli -p 26381 sentinel master mymaster
**Step 1e. Verify everything is running:**
.. code-block:: bash
ps aux | grep redis
You should see something like this (without the comments):
.. code-block:: text
# Master (read-write)
user 60816 0.1 0.0 69804 11132 ? Sl 04:11 0:00 redis-server *:6379
# Replica (read-only mirror of 6379)
user 60903 0.1 0.0 80048 10928 ? Sl 04:12 0:00 redis-server *:6380
# Sentinels (monitor the master and hold quorums to decide when to failover)
user 61301 0.1 0.0 67244 10944 ? Sl 04:14 0:00 redis-server *:26379 [sentinel]
user 61382 0.1 0.0 67244 10944 ? Sl 04:14 0:00 redis-server *:26380 [sentinel]
user 61462 0.1 0.0 67244 10944 ? Sl 04:15 0:00 redis-server *:26381 [sentinel]
**Alternative: Starting an LMCache Server:**
The ``lmcache_server`` CLI entrypoint starts a remote LMCache server and comes with
the ``lmcache`` package.
.. code-block:: bash
lmcache_server <host> <port> <device>
lmcache_server localhost 65432
Currently, the only supported device is "cpu" (which is the default, so you don't need to specify it).
**Step 2. Start a vLLM server with remote offloading enabled:**
Create a an lmcache configuration file called: ``redis-offload.yaml``
.. code-block:: yaml
# disabling CPU RAM offload not recommended (on by default) but
# if you want to confirm that the remote backend works by itself
# local_cpu: false
chunk_size: 256
remote_url: "redis://localhost:6379"
remote_serde: "naive"
If you don't want to use a config file, uncomment the first three environment variables
and then comment out the ``LMCACHE_CONFIG_FILE`` below:
.. code-block:: bash
# disabling CPU RAM offload not recommended (on by default) but
# if you want to confirm that the remote backend works by itself
# LMCACHE_LOCAL_CPU=False \
# LMCACHE_CHUNK_SIZE=256 \
# LMCACHE_REMOTE_URL="redis://localhost:6379" \
# LMCACHE_REMOTE_SERDE="naive"
LMCACHE_CONFIG_FILE="redis-offload.yaml" \
vllm serve \
meta-llama/Llama-3.1-8B-Instruct \
--max-model-len 16384 \
--kv-transfer-config \
'{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
**Optional: Sentinels**
Create a an lmcache configuration file called: ``redis-sentinel-offload.yaml``
.. code-block:: yaml
chunk_size: 256
remote_url: "redis-sentinel://localhost:26379,localhost:26380,localhost:26381"
remote_serde: "naive"
If you don't want to use a config file, uncomment the first three environment variables
and then comment out the ``LMCACHE_CONFIG_FILE`` below:
.. code-block:: bash
# LMCACHE_CHUNK_SIZE=256 \
# LMCACHE_REMOTE_URL="redis-sentinel://localhost:26379,localhost:26380,localhost:26381" \
# LMCACHE_REMOTE_SERDE="naive"
LMCACHE_CONFIG_FILE="redis-sentinel-offload.yaml" \
vllm serve \
meta-llama/Llama-3.1-8B-Instruct \
--max-model-len 16384 \
--kv-transfer-config \
'{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
**Alternative: LMCache Server**
Create a an lmcache configuration file called: ``lmcache-server-offload.yaml``
.. code-block:: yaml
chunk_size: 256
remote_url: "lm://localhost:65432"
remote_serde: "naive"
If you don't want to use a config file, uncomment the first three environment variables
and then comment out the ``LMCACHE_CONFIG_FILE`` below:
.. code-block:: bash
# LMCACHE_CHUNK_SIZE=256 \
# LMCACHE_REMOTE_URL="lm://localhost:65432" \
# LMCACHE_REMOTE_SERDE="naive"
LMCACHE_CONFIG_FILE="lmcache-server-offload.yaml" \
vllm serve \
meta-llama/Llama-3.1-8B-Instruct \
--max-model-len 16384 \
--kv-transfer-config \
'{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
**Step 3. Viewing and Managing LMCache Entries in Redis:**
If you would like to feel the TTFT speed up with offloading and KV Cache reuse, feel free to use the same
``query-twice.py`` script and ``man-bash.txt`` long context as in :doc:`CPU RAM <./cpu_ram>` and :doc:`Local Storage <./local_storage>`.
Here, we are instead going to demonstrate how to search for and modify LMCache KV Chunk entries in Redis.
Please note that the official LMCache way to achieve this redis-specific functionality of viewing and modifying LMCache KV Chunks is available in :doc:`LMCache Controller <../../kv_cache_management/index>`.
Let's warm/populate LMCache first with ``curl`` this time:
.. code-block:: bash
curl -X 'POST' \
'http://127.0.0.1:8000/v1/chat/completions' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [
{"role": "system", "content": "You are a helpful AI coding assistant."},
{"role": "user", "content": "Write a segment tree implementation in python"}
],
"max_tokens": 150
}'
LMCache stores data in Redis using a structured key format. Each key contains the following information in a delimited format:
.. code-block:: text
model_name@world_size@worker_id@chunk_hash
- `model_name`: Name of the language model
- `world_size`: Total number of workers in distributed deployment
- `worker_id`: ID of the worker that created this cache entry, in the range of [0, world_size - 1]
- `chunk_hash`: Hash of the token chunk (SHA-256 based)
For example, a typical key might look like:
.. code-block:: text
vllm@meta-llama/Llama-3.1-8B-Instruct@1@0@a1b2c3d4e5f6...
**Using redis-cli to View LMCache Data**
To inspect and manage LMCache entries in Redis:
.. code-block:: bash
redis-cli -h localhost -p 6379
**Optional: If you are using sentinels, first find the master port:**
.. code-block:: bash
redis-cli -p 26379 sentinel get-master-addr-by-name mymaster
redis-cli -h localhost -p <master-port>
**List LMCache keys:**
Notice (from the suffixes of the keys) that each LMCache KV Chunk has two entries: ``kv_bytes`` and ``metadata``
.. code-block:: bash
# Show all keys
localhost:6379> KEYS *
1) "vllm@meta-llama/Llama-3.1-8B-Instruct@1@0@02783dafec...kv_bytes"
2) "vllm@meta-llama/Llama-3.1-8B-Instruct@1@0@02783dafec...metadata"
# Show keys for a specific model
localhost:6379> KEYS *Llama-3.1-8B-Instruct*
1) "vllm@meta-llama/Llama-3.1-8B-Instruct@1@0@02783dafec...kv_bytes"
2) "vllm@meta-llama/Llama-3.1-8B-Instruct@1@0@02783dafec...metadata"
**Delete LMCache entries:**
.. code-block:: bash
localhost:6379> DEL *
Delete a specific LMCache entry:
.. code-block:: bash
localhost:6379> DEL "vllm@meta-llama/Llama-3.1-8B-Instruct@1@0@02783dafec...kv_bytes"
localhost:6379> KEYS *
1) "vllm@meta-llama/Llama-3.1-8B-Instruct@1@0@02783dafec...metadata"
**Check if a key exists:**
.. code-block:: bash
localhost:6379> EXISTS "vllm@meta-llama/Llama-3.1-8B-Instruct@1@0@02783dafec...kv_bytes"
**View memory usage for a key:**
Notice that the ``kv_bytes`` entry is what is exactly holding the KV Chunk and is much
larger than the ``metadata`` entry.
.. code-block:: bash
localhost:6379> MEMORY USAGE "vllm@meta-llama/Llama-3.1-8B-Instruct@1@0@02783dafec...metadata"
(integer) 198
localhost:6379> MEMORY USAGE "vllm@meta-llama/Llama-3.1-8B-Instruct@1@0@02783dafec...kv_bytes"
(integer) 7340200
**Delete specific keys:**
.. code-block:: bash
# Delete a single key
localhost:6379> DEL "vllm@meta-llama/Llama-3.1-8B-Instruct@1@0@02783dafec...kv_bytes"
.. code-block:: bash
# Delete all keys matching a pattern
redis-cli -h localhost -p 6379 --scan --pattern "vllm@meta-llama/Llama-3.1-8B-Instruct*" \
| xargs redis-cli -h localhost -p 6379 DEL
**Monitor Redis in real-time:**
.. code-block:: bash
localhost:6379> MONITOR
**Get Redis stats for LMCache:**
.. code-block:: bash
# Get memory stats
localhost:6379> INFO memory
# Get statistics about operations
localhost:6379> INFO stats
This tutorial utilized the ``redis-cli`` to directly peak into a remote backend and manipualte
KV Chunks.
Once again, please refer to the :doc:`LMCache Controller <../../kv_cache_management/index>`
for the official LMCache way of controlling and routing your KV Caches in your LMCache instances.
**Step 4. Clean up:**
.. code-block:: bash
redis-cli shutdown
# Optional:
# Shut down the Redis replica (if started)
redis-cli -p 6380 shutdown
# Shut down all Redis Sentinels (if started)
redis-cli -p 26379 shutdown
redis-cli -p 26380 shutdown
redis-cli -p 26381 shutdown
# (Optional) Remove temporary files or configs
rm -f sentinel-26379.conf sentinel-26380.conf sentinel-26381.conf
# Confirm no Redis processes are still running
ps aux | grep redis
docs/source/kv_cache/storage_backends/resp.rst
+456
View File
@@ -0,0 +1,456 @@
RESP (Native Redis/Valkey)
==========================
.. _resp-overview:
Overview
--------
The RESP backend is a high-performance native C++ storage connector for Redis and Valkey servers,
using the RESP2 wire protocol over TCP. It is designed for maximum throughput on KV cache
store and retrieval operations, achieving **6+ GB/s** on reads with optimal configuration.
Key advantages over the standard Redis connector:
- **Multi-threaded C++ I/O**: Worker threads operate in parallel with zero-copy buffer passing and full GIL release
- **Batched tiling**: Large batch operations are automatically split across worker threads for maximum parallelism
- **eventfd-based completions**: The kernel wakes Python on completion -- no polling overhead
- **Dual-mode support**: The same C++ connector works in both non-MP mode (via ``ConnectorClientBase``) and MP mode (via ``NativeConnectorL2Adapter`` as an L2 adapter)
The native C++ source lives in ``csrc/storage_backends/redis/``. See :doc:`Adding Native Connectors <../../developer_guide/extending_lmcache/native_connectors>` for the full architecture.
Prerequisites
-------------
- LMCache installed from source (``pip install -e .``) to compile the C++ extension
- A Redis 8.2+ or Valkey server (Redis 8.2 recommended for IO threads support)
- A machine with at least one GPU for vLLM inference
Redis Server Setup
------------------
.. important::
Redis version and server configuration have a **major** impact on throughput.
Using Redis 8.2 with IO threads yields ~6 GB/s reads vs ~1.5 GB/s with Redis 6.0 defaults.
**Build Redis 8.2 from source (recommended):**
.. code-block:: bash
git clone https://github.com/redis/redis.git
cd redis
git checkout 8.2
make -j
**Start the server with IO threads enabled:**
.. code-block:: bash
./src/redis-server \
--protected-mode no \
--save '' \
--appendonly no \
--io-threads 4 \
--port 6379
.. list-table:: Recommended Server Flags
:header-rows: 1
:widths: 30 70
* - Flag
- Why
* - ``--protected-mode no``
- Allow connections from other hosts (use auth in production)
* - ``--save '' --appendonly no``
- Disable persistence -- KV cache is ephemeral, persistence wastes bandwidth
* - ``--io-threads 4``
- Enable multi-threaded I/O for parallel read/write handling
* - ``--port 6379``
- Default port (adjust if running multiple instances)
.. tip::
The number of ``--io-threads`` should roughly match the number of physical cores
available to the Redis process. 4 is a good starting point; benchmark with your
hardware to find the optimum.
Chunk Size Selection and Throughput Tuning
------------------------------------------
The chunk size (in tokens) determines how many bytes each Redis key-value pair holds.
This is the **single most important parameter** for throughput.
**The sweet spot is ~4 MB per chunk.** Both smaller and larger chunks degrade throughput:
.. list-table:: Chunk Size vs Throughput (Redis 8.2, 8 workers)
:header-rows: 1
:widths: 20 20 20 20
* - Chunk Size
- Total Data
- SET Throughput
- GET Throughput
* - 1 MB (500 keys)
- 500 MB
- ~3.5 GB/s
- ~5.2 GB/s
* - **4 MB (500 keys)**
- **2 GB**
- **~4.4 GB/s**
- **~5.9 GB/s**
* - 8 MB (200 keys)
- 1.6 GB
- ~4.2 GB/s
- ~1.4 GB/s
**Why 4 MB?**
- Below ~2 MB, per-key overhead (RESP framing, TCP round-trips) dominates
- Above ~4 MB, Redis server-side memory allocation and TCP window sizes become bottlenecks
- At 4 MB, the balance between amortized overhead and memory pressure is optimal
**Calculating chunk size in tokens:**
The chunk size in bytes depends on the model's hidden dimension, number of KV heads,
number of layers, and dtype:
.. code-block:: text
bytes_per_token = 2 * num_kv_heads * head_dim * num_layers * dtype_bytes
For ``meta-llama/Llama-3.1-8B-Instruct`` with BFloat16:
.. code-block:: text
bytes_per_token = 2 * 8 * 128 * 32 * 2 = 131,072 bytes (~128 KB)
chunk_size_tokens = 4 MB / 128 KB = 32 tokens
# But typically chunk_size is set as token count in config:
chunk_size: 16 # ~2 MB per chunk (conservative)
chunk_size: 32 # ~4 MB per chunk (optimal for throughput)
.. note::
The bytes-per-token calculation varies by model architecture. Larger models
(e.g., 70B) have more layers and larger hidden dimensions, so fewer tokens
are needed per chunk to reach the 4 MB sweet spot.
Throughput Sweep
~~~~~~~~~~~~~~~~
To find the optimal configuration for your hardware, use the included benchmark:
.. code-block:: bash
cd examples/kv_cache_reuse/remote_backends/resp
# Sweep chunk sizes
for mb in 0.5 1 2 4 8; do
echo "=== Chunk: ${mb} MB ==="
python benchmark_resp_client.py \
--host 127.0.0.1 --port 6379 \
--chunk-mb $mb --num-workers 8 --num-keys 500
done
# Sweep worker counts
for w in 1 2 4 8 16; do
echo "=== Workers: $w ==="
python benchmark_resp_client.py \
--host 127.0.0.1 --port 6379 \
--chunk-mb 4 --num-workers $w --num-keys 500
done
Expected output:
.. code-block:: text
Redis RESP Client Benchmark
Server: 127.0.0.1:6379, Workers: 8
Chunk size: 4096KB, Keys: 500
------------------------------------------------------------
Batch SET: 4.36 GB/s (1.95 GB written)
Batch GET: 5.91 GB/s (1.95 GB read)
Batch EXISTS: 143528 ops/s (500/500 hits)
------------------------------------------------------------
All tests passed
Environment Variable Configuration
------------------------------------
Sensitive credentials (and optionally host/port) can be provided via environment
variables instead of config files or CLI arguments. This prevents secrets from
appearing in logged configuration at startup.
.. list-table::
:header-rows: 1
:widths: 35 65
* - Variable
- Description
* - ``LMCACHE_RESP_USERNAME``
- Redis ACL username. Used as default when ``username`` is not set in config/JSON.
* - ``LMCACHE_RESP_PASSWORD``
- Redis AUTH password. Used as default when ``password`` is not set in config/JSON.
* - ``LMCACHE_RESP_HOST``
- Redis hostname or IP. Used as default when ``host`` is not set in config/JSON/URL.
* - ``LMCACHE_RESP_PORT``
- Redis port. Used as default when ``port`` is not set in config/JSON/URL.
Config files (non-MP) and ``--l2-adapter`` JSON (MP) take precedence over
environment variables. Environment variables serve as defaults — they are used
when the corresponding config value is empty or unset. They are read at adapter
creation time inside the adapter itself, so they are **never stored in the
config object** and **never printed in startup logs**.
**Example — MP mode with env vars:**
.. code-block:: bash
export LMCACHE_RESP_USERNAME="default"
export LMCACHE_RESP_PASSWORD="secret"
lmcache server \
--l1-size-gb 10 \
--eviction-policy LRU \
--chunk-size 16 \
--l2-adapter '{"type": "resp", "host": "localhost", "port": 6379, "num_workers": 8}' \
--port 6555
**Example — Non-MP mode with env vars:**
.. code-block:: bash
export LMCACHE_RESP_USERNAME="default"
export LMCACHE_RESP_PASSWORD="secret"
LMCACHE_CONFIG_FILE=resp-config.yaml \
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}' \
--no-enable-prefix-caching \
--load-format dummy
.. tip::
For production deployments, always use environment variables for credentials
rather than embedding them in config files or CLI arguments.
Non-MP Mode (Single Process)
-----------------------------
In non-MP mode, the RESP connector is used directly as a remote storage backend
via the ``RESPClient`` asyncio wrapper.
**Configuration file** (``resp-config.yaml``):
.. code-block:: yaml
chunk_size: 16
remote_url: "resp://localhost:6379"
remote_serde: "naive"
Credentials can be set via environment variables (recommended) or in the config file
under ``extra_config`` (see `Environment Variable Configuration`_ above).
**Launch vLLM:**
.. code-block:: bash
LMCACHE_CONFIG_FILE=resp-config.yaml \
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}' \
--no-enable-prefix-caching \
--load-format dummy
.. note::
``save_unfull_chunk`` must be off (default) and chunk metadata saving must be
disabled for optimal throughput with the native RESP connector.
MP Mode (Multiprocess)
-----------------------
In MP mode, LMCache runs as a separate server process communicating with vLLM over
ZMQ. The RESP connector serves as an L2 adapter with variable-size chunk support.
**Step 1: Start Redis** (see `Redis Server Setup`_ above)
**Step 2: Launch LMCache MP Server:**
.. code-block:: bash
lmcache server \
--l1-size-gb 10 \
--eviction-policy LRU \
--chunk-size 16 \
--l2-adapter '{"type": "resp", "host": "localhost", "port": 6379, "num_workers": 8}' \
--port 6555
**Step 3: Launch vLLM with LMCache MP Connector:**
.. code-block:: bash
PORT=8000
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--kv-transfer-config '{
"kv_connector": "LMCacheMPConnector",
"kv_role": "kv_both",
"kv_connector_extra_config": {
"lmcache.mp.host": "tcp://localhost",
"lmcache.mp.port": 6555
}
}' \
--no-enable-prefix-caching \
--port $PORT \
--load-format dummy
L2 Adapter Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~
The ``--l2-adapter`` JSON accepts these fields:
.. list-table::
:header-rows: 1
:widths: 20 10 10 60
* - Field
- Type
- Default
- Description
* - ``type``
- str
- (required)
- Must be ``"resp"``
* - ``host``
- str
- (required)
- Redis/Valkey hostname or IP
* - ``port``
- int
- (required)
- Redis/Valkey port
* - ``num_workers``
- int
- 8
- C++ worker threads for parallel I/O
* - ``username``
- str
- ``""``
- Redis ACL username (leave empty for no auth). Falls back to ``LMCACHE_RESP_USERNAME`` env var if empty.
* - ``password``
- str
- ``""``
- Redis AUTH password (leave empty for no auth). Falls back to ``LMCACHE_RESP_PASSWORD`` env var if empty.
* - ``max_capacity_gb``
- float
- 0
- Maximum L2 storage capacity in GB for client-side usage tracking. Required for L2 eviction. Set to 0 (default) to disable usage tracking.
L2 Eviction
~~~~~~~~~~~~
To enable automatic eviction of least-recently-used keys when the Redis backend fills up,
set ``max_capacity_gb`` and add an ``"eviction"`` block:
.. code-block:: bash
lmcache server \
--l1-size-gb 10 \
--eviction-policy LRU \
--chunk-size 16 \
--l2-adapter '{
"type": "resp",
"host": "localhost",
"port": 6379,
"num_workers": 8,
"max_capacity_gb": 10,
"eviction": {
"eviction_policy": "LRU",
"trigger_watermark": 0.8,
"eviction_ratio": 0.2
}
}' \
--port 6555
This configures a 10 GB capacity limit. When usage exceeds 80% (``trigger_watermark``),
the eviction controller will delete the least-recently-used ~20% of stored keys
(``eviction_ratio``) using the Redis ``DEL`` command.
.. note::
``max_capacity_gb`` enables **client-side** size tracking. It does not configure
the Redis server's ``maxmemory`` setting. You should set ``max_capacity_gb`` to
match or be slightly below your Redis server's available memory.
Testing the Setup
------------------
Send the same prompt twice. The first request stores KV cache to Redis; the second retrieves it.
.. code-block:: bash
PORT=8000
PROMPT="$(printf 'Elaborate the significance of KV cache in language models. %.0s' {1..1000})"
# First request: store
curl -s -X POST http://localhost:${PORT}/v1/completions \
-H "Content-Type: application/json" \
-d '{"model":"meta-llama/Llama-3.1-8B-Instruct","prompt":"'"$PROMPT"'","max_tokens":10}'
# Second request with same prefix: retrieve from Redis
curl -s -X POST http://localhost:${PORT}/v1/completions \
-H "Content-Type: application/json" \
-d '{"model":"meta-llama/Llama-3.1-8B-Instruct","prompt":"'"$PROMPT"'","max_tokens":10}'
Verify data was stored:
.. code-block:: bash
redis-cli -p 6379 DBSIZE
Clear state between runs:
.. code-block:: bash
redis-cli -p 6379 FLUSHALL
Best Practices
--------------
**Server deployment:**
- Use Redis 8.2+ with ``--io-threads 4`` (or more, matching available cores)
- Disable persistence (``--save '' --appendonly no``) for KV cache workloads
- Pin Redis to its own NUMA node if running on multi-socket systems
- For production, enable authentication with ``--requirepass`` and supply credentials via ``LMCACHE_RESP_USERNAME`` / ``LMCACHE_RESP_PASSWORD`` environment variables to keep them out of logs
**Client tuning:**
- Start with ``num_workers: 8`` and increase if the server has spare CPU and you're not saturating the network
- More workers help when chunk sizes are smaller (more keys per batch = more parallelism needed)
- On NUMA systems, ensure the LMCache process runs on the same NUMA node as the NIC
**Chunk size:**
- Target ~4 MB per chunk for maximum throughput
- Calculate the token count using your model's per-token byte size (see formula above)
- If unsure, run the benchmark sweep to find the optimum for your specific hardware
**Network:**
- Use localhost or loopback for single-machine deployments
- For cross-machine setups, ensure low-latency networking (ideally <100 us RTT)
- The RESP connector uses TCP; RDMA is not currently supported (consider :doc:`Mooncake <./mooncake>` for RDMA)
Additional Resources
--------------------
- Benchmark script: ``examples/kv_cache_reuse/remote_backends/resp/benchmark_resp_client.py``
- C++ source: ``csrc/storage_backends/redis/``
- Native connector architecture: ``csrc/storage_backends/README.md``
- Developer guide for adding new native connectors: :doc:`Adding Native Connectors <../../developer_guide/extending_lmcache/native_connectors>`
docs/source/kv_cache/storage_backends/s3.rst
+155
View File
@@ -0,0 +1,155 @@
S3 Backend
==========
Example Configurations
----------------------
Basic S3 Configuration
~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: yaml
chunk_size: 256
local_cpu: False
save_unfull_chunk: False
remote_url: "s3://your-bucket-name"
remote_serde: "naive"
blocking_timeout_secs: 10
extra_config:
s3_region: "us-east-1"
s3_num_io_threads: 64
save_chunk_meta: False
S3 Express One Zone
~~~~~~~~~~~~~~~~~~~
.. code-block:: yaml
chunk_size: 256
local_cpu: False
save_unfull_chunk: False
remote_url: "s3://{BUCKET_NAME}.s3express-{AZ_ID}.{REGION}.amazonaws.com"
remote_serde: "naive"
blocking_timeout_secs: 10
extra_config:
save_chunk_meta: False
s3_num_io_threads: 64
s3_prefer_http2: True
s3_region: "{REGION}"
s3_enable_s3express: True
CoreWeave (S3-compatible)
~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: yaml
chunk_size: 256
local_cpu: False
max_local_cpu_size: 50
save_unfull_chunk: False
enable_async_loading: True
remote_url: "s3://test-127.cwlota.com"
remote_serde: "naive"
blocking_timeout_secs: 10
extra_config:
s3_num_io_threads: 320
s3_prefer_http2: False
s3_region: "US-WEST-04A"
s3_enable_s3express: False
save_chunk_meta: False
disable_tls: True
aws_access_key_id: "your-access-key-id"
aws_secret_access_key: "your-secret-access-key"
**Note**: `cwlota.com` is CoreWeave's S3-compatible Cloud Storage that caches for GPU locality. You can set `disable_tls: True` for non-AWS services.
Check out the blog post between LMCache, Cohere, and CoreWeave: https://blog.lmcache.ai/en/2025/10/29/breaking-the-memory-barrier-how-lmcache-and-coreweave-power-efficient-llm-inference-for-cohere/
Configuration Parameters
------------------------
* **remote_url**: S3 bucket URL (`s3://bucket-name`)
* **save_unfull_chunk**: Save partial chunks (default: False, **must be False for S3**)
* **enable_async_loading**: Async loading (default: False)
* **blocking_timeout_secs**: Timeout seconds (default: 10)
S3-Specific (in extra_config)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* **s3_region**: AWS region for S3 client (required)
* **s3_num_io_threads**: Number of IO threads for the AWS CRT client to spawn. Benefits taper out after exceeding the number of CPU cores. This is also a way to restrict the number of outgoing requests in case your S3-compatible object store has a rate-limiting gateway.
* **s3_prefer_http2**: Enable HTTP/2 with ALPN negotiation (["h2", "http/1.1"])
* **s3_enable_s3express**: Enable S3 Express One Zone support in AWS CRT client
* **save_chunk_meta**: Whether to save chunk metadata in the object store along with your data (False required for S3)
* **aws_access_key_id**: AWS access key ID (or log in with `aws configure` in your environment)
* **aws_secret_access_key**: AWS secret access key (or log in with `aws configure` in your environment)
**Tips:**::
- Use same region for compute and S3
- Consider S3 Express One Zone for less redundancy but better performance
MP Mode Configuration
---------------------
In multi-process (MP) mode, S3 is configured as an L2 adapter via a JSON
spec passed to the LMCache server, rather than through ``remote_url`` +
``extra_config``. Each ``--l2-adapter`` argument takes a JSON object
whose ``"type": "s3"`` field selects the S3 adapter.
.. code-block:: json
{
"type": "s3",
"s3_endpoint": "s3://my-bucket",
"s3_region": "us-east-1",
"s3_num_io_threads": 64,
"s3_prefer_http2": true,
"s3_enable_s3express": false,
"disable_tls": false,
"max_capacity_gb": 500,
"eviction": {
"eviction_policy": "LRU",
"trigger_watermark": 0.85,
"eviction_ratio": 0.2
}
}
S3 L2 Adapter Fields
~~~~~~~~~~~~~~~~~~~~
* **type** (required): must be ``"s3"``.
* **s3_endpoint** (required): bucket URL. Accepts either ``s3://bucket`` or
the bare host form (e.g. ``bucket.s3.us-east-1.amazonaws.com``).
* **s3_region** (required): AWS region for the S3 client.
* **s3_num_io_threads**: number of CRT IO threads (default ``64``).
* **s3_prefer_http2**: attempt HTTP/2 via ALPN negotiation (default ``true``).
* **s3_enable_s3express**: enable S3 Express One Zone signing (default ``false``).
* **disable_tls**: bypass TLS, for non-AWS HTTP endpoints (default ``false``).
* **aws_access_key_id**, **aws_secret_access_key**: optional static credentials.
When omitted the adapter uses the AWS default credentials chain
(``aws configure``, environment variables, IRSA, etc.).
* **max_capacity_gb**: capacity used by ``get_usage()`` for watermark-based
L2 eviction. Set to ``0`` (default) to disable usage tracking — ``get_usage()``
then returns ``(-1.0, -1.0)`` and no automatic eviction is triggered.
* **eviction**: optional sub-dict enabling the L2 eviction controller for
this adapter. See :class:`L2AdapterConfigBase` ``_parse_eviction_config``
for the full schema. When present, keys that are currently being loaded
(reference-counted by the lookup-and-lock path) are skipped by
``delete()``.
Differences vs Non-MP S3
~~~~~~~~~~~~~~~~~~~~~~~~
* The MP adapter honors first-class eviction: it implements ``delete()``
(real S3 ``DeleteObject``), refcounted ``submit_unlock``, and
``get_usage()`` driven by ``max_capacity_gb``.
* Keys are identified by ``ObjectKey`` (``model_name`` + ``kv_rank`` +
``chunk_hash``) rather than ``CacheEngineKey``. The wire-format object
name is ``<model>@<kv_rank_hex>@<chunk_hash_hex>``, which is **not**
compatible with the non-MP naming. A bucket populated by non-MP LMCache
cannot be read directly by MP LMCache and vice versa.
* Unfull chunks are rejected (same constraint as non-MP).
docs/source/kv_cache/storage_backends/sagemaker_hyperpod.rst
+76
View File
@@ -0,0 +1,76 @@
SageMaker Hyperpod
==================
.. warning::
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/l2_storage/index`.
Prerequisites
-------------
Create an Amazon SageMaker HyperPod cluster with tiered storage enabled by following the instructions at:
https://docs.aws.amazon.com/sagemaker/latest/dg/managed-tier-checkpointing-setup.html
This enables the ai-toolkit daemon that provides shared memory access for LMCache.
Example Configuration
---------------------
.. code-block:: yaml
chunk_size: 256
local_cpu: True
max_local_cpu_size: 5
remote_url: "sagemaker-hyperpod://$NODE_IP:9200"
Configuration Parameters
------------------------
SageMaker Hyperpod-Specific (in extra_config)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* **sagemaker_hyperpod_bucket**: Bucket name for KV storage namespace (default: "lmcache")
* **sagemaker_hyperpod_shared_memory_name**: Name of shared memory segment (default: "shared_memory"). Set to None to disable shared memory.
* **sagemaker_hyperpod_max_concurrent_requests**: Maximum concurrent HTTP requests allowed in-flight at any moment (application-level throttling, default: 100, minimum: 1). This limit is per LMCache engine instance. With multiple workers (e.g., high TP), each worker creates its own engine with separate limits.
* **sagemaker_hyperpod_max_connections**: Maximum total TCP connections in the connection pool per LMCache engine across all daemons (default: 256, minimum: 1). For typical single-daemon setups, this effectively limits connections from one engine to one daemon. With N workers per node, total connections to the daemon = N × this value.
* **sagemaker_hyperpod_max_connections_per_host**: Maximum TCP connections per LMCache engine to a single daemon address (IP:port) (default: 128, minimum: 1). "Host" refers to the daemon's network address, not the client machine. For today's typical single-daemon setup, this has similar effect as max_connections. This parameter enables future multi-daemon configurations where one engine connects to multiple daemons for load balancing. With N workers per node connecting to the same daemon, total connections = N × this value. Reduce proportionally for high TP setups (e.g., set to 16 for 8 workers to achieve ~128 total connections).
* **sagemaker_hyperpod_timeout_ms**: Timeout for lease acquisition requests in milliseconds (default: 5000, minimum: 100)
* **sagemaker_hyperpod_lease_ttl_s**: Server-side lease timeout in seconds (default: 30.0)
* **sagemaker_hyperpod_put_stream_chunk_bytes**: Chunk size for streaming PUT requests in bytes (default: 65536, minimum: 1024)
* **sagemaker_hyperpod_use_https**: Enable HTTPS instead of HTTP (default: False). **Note**: Ignored if ``remote_url`` already contains ``http://`` or ``https://`` protocol.
* **save_chunk_meta**: Whether to save chunk metadata with data (set False for performance)
Kubernetes Deployment Requirements
-----------------------------------
Environment Variable for Node IP
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Add the ``NODE_IP`` environment variable to resolve the local node's IP address:
.. code-block:: yaml
env:
- name: NODE_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
/dev/shm Volume Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
SageMaker Hyperpod requires /dev/shm for high-performance shared memory operations:
.. code-block:: yaml
volumeMounts:
- name: dshm
mountPath: /dev/shm/shared_memory
subPath: shared_memory
volumes:
- name: dshm
hostPath:
path: /dev/shm
docs/source/kv_cache/storage_backends/valkey.rst
+205
View File
@@ -0,0 +1,205 @@
Valkey
======
.. warning::
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/l2_storage/resp`.
Overview
--------
Valkey is an open source (BSD) high-performance key/value datastore and is a supported option for remote KV Cache offloading in LMCache.
Some other remote backends are :doc:`Mooncake <./mooncake>`, :doc:`Redis <./redis>`, and :doc:`InfiniStore <./infinistore>`.
Prerequisites
-------------
To use this connector, you need valkey-glide 2.0 or higher.
.. code-block:: shell
# Install Valkey-GLIDE (Minimum 2.0.0 or higher)
$ pip install valkey-glide
Configuration Reference
-----------------------
The following ``extra_config`` keys are supported:
.. list-table::
:header-rows: 1
:widths: 30 15 55
* - Key
- Default
- Description
* - ``valkey_num_workers``
- ``8``
- Number of worker threads, each with its own GLIDE client connection.
* - ``valkey_mode``
- ``"standalone"``
- ``"standalone"`` or ``"cluster"``. Cluster mode auto-discovers topology from a seed node.
* - ``tls_enable``
- ``false``
- Enable TLS. Required for ElastiCache Serverless.
* - ``valkey_username``
- ``""``
- Authentication username.
* - ``valkey_password``
- ``""``
- Authentication password.
* - ``valkey_database``
- None
- Database ID (standalone mode only, ignored in cluster mode).
* - ``valkey_enable_ttl``
- ``false``
- Feature flag. When ``true``, every key is written with an expiry (see ``valkey_ttl_sec``) so Valkey/Redis ``volatile-*`` eviction policies can reclaim L2 cache keys once the node reaches ``maxmemory``. When ``false`` (default), keys are persisted without a TTL.
* - ``valkey_ttl_sec``
- ``86400``
- Key TTL in seconds, applied only when ``valkey_enable_ttl`` is ``true``. Must be a positive integer. If the flag is enabled but this key is omitted, it defaults to ``86400`` (24 hours).
* - ``request_timeout``
- ``5.0``
- GLIDE request timeout in seconds. Also used as the Python-side Future timeout.
* - ``connection_timeout``
- ``10.0``
- GLIDE connection timeout in seconds for initial client connections.
Example Configurations
----------------------
Standalone-mode
~~~~~~~~~~~~~~~~
Basic Valkey Configuration (Standalone-mode)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: yaml
chunk_size: 256
remote_url: "valkey://<your host>:6379"
remote_serde: "naive"
extra_config:
valkey_username: "Your username"
valkey_password: "Your password"
Standalone-mode Valkey Configuration with database
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: yaml
chunk_size: 256
remote_url: "valkey://<your host>:6379"
remote_serde: "naive"
extra_config:
valkey_username: "Your username"
valkey_password: "Your password"
valkey_database: 0
Standalone-mode Valkey Configuration with key TTL (volatile eviction)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Enable a per-key TTL so a node configured with a ``volatile-lru`` /
``volatile-lfu`` eviction policy can reclaim L2 cache keys once it reaches
``maxmemory``. Without a TTL such policies never evict the KV cache keys,
which can choke the remote cache.
.. code-block:: yaml
chunk_size: 256
remote_url: "valkey://<your host>:6379"
remote_serde: "naive"
extra_config:
valkey_username: "Your username"
valkey_password: "Your password"
valkey_enable_ttl: true
valkey_ttl_sec: 3600 # keys expire after 1 hour
Cluster-mode
~~~~~~~~~~~~~~~~
Cluster-mode Valkey Configuration (Endpoint)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For example, the configuration endpoint in ElastiCache is as follows:
<cache-name>.<identifier>.clustercfg.<region>.cache.amazonaws.com
You need to add this DNS name in the <your host>.
.. code-block:: yaml
chunk_size: 256
remote_url: "valkey://<your host>:6379"
remote_serde: "naive"
extra_config:
valkey_mode: "cluster"
valkey_username: "Your username"
valkey_password: "Your password"
Cluster-mode Valkey Configuration (Nodes)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Nodes are deployed directly and configured in cluster mode without connecting them via DNS names (CNAME).
In this scenario, you simply input multiple IP hosts and ports.
Example: 172.0.0.1:7001, 172.0.0.2:7002 ... 172.0.0.6:7006
.. code-block:: yaml
chunk_size: 256
remote_url: "valkey://<your host 1>:<your port 1>, <your host 2>:<your port 2>, ... <your host N>:<your port N>"
remote_serde: "naive"
extra_config:
valkey_mode: "cluster"
valkey_username: "Your username"
valkey_password: "Your password"
Cluster-mode Valkey Configuration with numbered databases (Valkey 9.0+ and Valkey-GLIDE 2.1+)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Valkey connector supports numbered databases in both the Endpoint using DNS and the Nodes method using IP and port pairs.
.. code-block:: yaml
chunk_size: 256
remote_url: "valkey://<your host>:6379"
remote_serde: "naive"
extra_config:
valkey_mode: "cluster"
valkey_username: "Your username"
valkey_password: "Your password"
valkey_database: 1
TLS / ElastiCache Serverless
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ElastiCache Serverless requires TLS. Set ``tls_enable: true``:
.. code-block:: yaml
chunk_size: 256
remote_url: "valkey://<serverless-endpoint>:6379"
remote_serde: "naive"
extra_config:
valkey_mode: "cluster"
tls_enable: true
valkey_num_workers: 32
Performance Tuning
~~~~~~~~~~~~~~~~~~
For large models (e.g., 70B with TP=8), increase the worker count for higher throughput:
.. code-block:: yaml
chunk_size: 256
remote_url: "valkey://<your host>:6379"
remote_serde: "naive"
pre_caching_hash_algorithm: sha256_cbor_64bit # Required for TP>1
extra_config:
valkey_mode: "cluster"
valkey_num_workers: 32
docs/source/kv_cache/storage_backends/weka.rst
+137
View File
@@ -0,0 +1,137 @@
Weka
====
.. warning::
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/l2_storage/index`.
.. _weka-overview:
Overview
--------
WekaFS is a high-performance, distributed filesystem and is a supported option for KV Cache offloading in
LMCache. Even though the local filesystem backend can work with a WekaFS mount, this particular backend is
optimized for Weka's characteristics. It leverages GPUDirect Storage for I/O and it allows data-sharing
between multiple LMCache instances.
Ways to configure LMCache WEKA Offloading
-----------------------------------------
**1. Environment Variables:**
.. code-block:: bash
# 256 Tokens per KV Chunk
export LMCACHE_CHUNK_SIZE=256
# Path to Weka Mount
export LMCACHE_GDS_PATH="/mnt/weka/cache"
# GDS Buffer Size in MiB
export LMCACHE_GDS_BUFFER_SIZE="8192"
# Disabling CPU RAM offload is sometimes recommended as the
# CPU can get in the way of GPUDirect operations
export LMCACHE_LOCAL_CPU=False
# GDS I/O Threads
export LMCACHE_EXTRA_CONFIG='{"disk_io_threads": 32}'
**2. Configuration File**:
Passed in through ``LMCACHE_CONFIG_FILE=your-lmcache-config.yaml``
Example ``config.yaml``:
.. code-block:: yaml
# 256 Tokens per KV Chunk
chunk_size: 256
# Disable local CPU
local_cpu: false
# Path to Weka Mount
gds_path: "/mnt/weka/cache"
# GDS Buffer Size in MiB
gds_buffer_size: 8192
# GDS I/O Threads
extra_config:
disk_io_threads: 32
GDS Buffer Size Explanation
---------------------------
The backend currently pre-registers buffer space to speed up GDS operations. This buffer space
is registered in VRAM so options like ``--gpu-memory-utilization`` from ``vllm`` should be considered
when setting it. For example, a good rule of thumb for H100 which generally has 80GiBs of VRAM would
be to start with 8GiB and set ``--gpu-memory-utilization 0.85`` and depending on your workflow fine-tune
it from there.
Setup Example
-------------
.. _weka-prerequisites:
**Prerequisites:**
- A Machine with at least one GPU. You can adjust the max model length of your vllm instance depending on your GPU memory.
- Weka already installed and mounted.
- vllm and lmcache installed (:doc:`Installation Guide <../../getting_started/installation>`)
- Hugging Face access to ``meta-llama/Llama-3.1-8B-Instruct``
.. code-block:: bash
export HF_TOKEN=your_hugging_face_token
**Step 1. Create cache directory under your Weka mount:**
To find all your WekaFS mounts run:
.. code-block:: bash
mount -t wekafs
For the sake of this example let's say that the above returns:
.. code-block:: text
10.27.1.1/default on /mnt/weka type wekafs (rw,relatime,writecache,inode_bits=auto,readahead_kb=32768,dentry_max_age_positive=1000,dentry_max_age_negative=0,container_name=client)
Then create a directory under it (the name here is arbitrary):
.. code-block:: bash
mkdir /mnt/weka/cache
**Step 2. Start a vLLM server with Weka offloading enabled:**
Create a an lmcache configuration file called: ``weka-offload.yaml``
.. code-block:: yaml
local_cpu: false
chunk_size: 256
gds_path: "/mnt/weka/cache"
gds_buffer_size: 8192
extra_config:
disk_io_threads: 32
If you don't want to use a config file, uncomment the first three environment variables
and then comment out the ``LMCACHE_CONFIG_FILE`` below:
.. code-block:: bash
# LMCACHE_LOCAL_CPU=False \
# LMCACHE_CHUNK_SIZE=256 \
# LMCACHE_GDS_PATH="/mnt/weka/cache" \
# LMCACHE_GDS_BUFFER_SIZE=8192 \
# LMCACHE_EXTRA_CONFIG='{"disk_io_threads": 32}' \
LMCACHE_CONFIG_FILE="weka-offload.yaml" \
vllm serve \
meta-llama/Llama-3.1-8B-Instruct \
--max-model-len 65536 \
--kv-transfer-config \
'{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'