wehub-resource-sync
457 lines
14 KiB
ReStructuredText
457 lines
14 KiB
ReStructuredText
RESP (Native Redis/Valkey)
|
|
==========================
|
|
|
|
.. _resp-overview:
|
|
|
|
Overview
|
|
--------
|
|
|
|
The RESP backend is a high-performance native C++ storage connector for Redis and Valkey servers,
|
|
using the RESP2 wire protocol over TCP. It is designed for maximum throughput on KV cache
|
|
store and retrieval operations, achieving **6+ GB/s** on reads with optimal configuration.
|
|
|
|
Key advantages over the standard Redis connector:
|
|
|
|
- **Multi-threaded C++ I/O**: Worker threads operate in parallel with zero-copy buffer passing and full GIL release
|
|
- **Batched tiling**: Large batch operations are automatically split across worker threads for maximum parallelism
|
|
- **eventfd-based completions**: The kernel wakes Python on completion -- no polling overhead
|
|
- **Dual-mode support**: The same C++ connector works in both non-MP mode (via ``ConnectorClientBase``) and MP mode (via ``NativeConnectorL2Adapter`` as an L2 adapter)
|
|
|
|
The native C++ source lives in ``csrc/storage_backends/redis/``. See :doc:`Adding Native Connectors <../../developer_guide/extending_lmcache/native_connectors>` for the full architecture.
|
|
|
|
|
|
Prerequisites
|
|
-------------
|
|
|
|
- LMCache installed from source (``pip install -e .``) to compile the C++ extension
|
|
- A Redis 8.2+ or Valkey server (Redis 8.2 recommended for IO threads support)
|
|
- A machine with at least one GPU for vLLM inference
|
|
|
|
|
|
Redis Server Setup
|
|
------------------
|
|
|
|
.. important::
|
|
Redis version and server configuration have a **major** impact on throughput.
|
|
Using Redis 8.2 with IO threads yields ~6 GB/s reads vs ~1.5 GB/s with Redis 6.0 defaults.
|
|
|
|
**Build Redis 8.2 from source (recommended):**
|
|
|
|
.. code-block:: bash
|
|
|
|
git clone https://github.com/redis/redis.git
|
|
cd redis
|
|
git checkout 8.2
|
|
make -j
|
|
|
|
**Start the server with IO threads enabled:**
|
|
|
|
.. code-block:: bash
|
|
|
|
./src/redis-server \
|
|
--protected-mode no \
|
|
--save '' \
|
|
--appendonly no \
|
|
--io-threads 4 \
|
|
--port 6379
|
|
|
|
.. list-table:: Recommended Server Flags
|
|
:header-rows: 1
|
|
:widths: 30 70
|
|
|
|
* - Flag
|
|
- Why
|
|
* - ``--protected-mode no``
|
|
- Allow connections from other hosts (use auth in production)
|
|
* - ``--save '' --appendonly no``
|
|
- Disable persistence -- KV cache is ephemeral, persistence wastes bandwidth
|
|
* - ``--io-threads 4``
|
|
- Enable multi-threaded I/O for parallel read/write handling
|
|
* - ``--port 6379``
|
|
- Default port (adjust if running multiple instances)
|
|
|
|
.. tip::
|
|
The number of ``--io-threads`` should roughly match the number of physical cores
|
|
available to the Redis process. 4 is a good starting point; benchmark with your
|
|
hardware to find the optimum.
|
|
|
|
|
|
Chunk Size Selection and Throughput Tuning
|
|
------------------------------------------
|
|
|
|
The chunk size (in tokens) determines how many bytes each Redis key-value pair holds.
|
|
This is the **single most important parameter** for throughput.
|
|
|
|
**The sweet spot is ~4 MB per chunk.** Both smaller and larger chunks degrade throughput:
|
|
|
|
.. list-table:: Chunk Size vs Throughput (Redis 8.2, 8 workers)
|
|
:header-rows: 1
|
|
:widths: 20 20 20 20
|
|
|
|
* - Chunk Size
|
|
- Total Data
|
|
- SET Throughput
|
|
- GET Throughput
|
|
* - 1 MB (500 keys)
|
|
- 500 MB
|
|
- ~3.5 GB/s
|
|
- ~5.2 GB/s
|
|
* - **4 MB (500 keys)**
|
|
- **2 GB**
|
|
- **~4.4 GB/s**
|
|
- **~5.9 GB/s**
|
|
* - 8 MB (200 keys)
|
|
- 1.6 GB
|
|
- ~4.2 GB/s
|
|
- ~1.4 GB/s
|
|
|
|
**Why 4 MB?**
|
|
|
|
- Below ~2 MB, per-key overhead (RESP framing, TCP round-trips) dominates
|
|
- Above ~4 MB, Redis server-side memory allocation and TCP window sizes become bottlenecks
|
|
- At 4 MB, the balance between amortized overhead and memory pressure is optimal
|
|
|
|
**Calculating chunk size in tokens:**
|
|
|
|
The chunk size in bytes depends on the model's hidden dimension, number of KV heads,
|
|
number of layers, and dtype:
|
|
|
|
.. code-block:: text
|
|
|
|
bytes_per_token = 2 * num_kv_heads * head_dim * num_layers * dtype_bytes
|
|
|
|
For ``meta-llama/Llama-3.1-8B-Instruct`` with BFloat16:
|
|
|
|
.. code-block:: text
|
|
|
|
bytes_per_token = 2 * 8 * 128 * 32 * 2 = 131,072 bytes (~128 KB)
|
|
chunk_size_tokens = 4 MB / 128 KB = 32 tokens
|
|
|
|
# But typically chunk_size is set as token count in config:
|
|
chunk_size: 16 # ~2 MB per chunk (conservative)
|
|
chunk_size: 32 # ~4 MB per chunk (optimal for throughput)
|
|
|
|
.. note::
|
|
The bytes-per-token calculation varies by model architecture. Larger models
|
|
(e.g., 70B) have more layers and larger hidden dimensions, so fewer tokens
|
|
are needed per chunk to reach the 4 MB sweet spot.
|
|
|
|
|
|
Throughput Sweep
|
|
~~~~~~~~~~~~~~~~
|
|
|
|
To find the optimal configuration for your hardware, use the included benchmark:
|
|
|
|
.. code-block:: bash
|
|
|
|
cd examples/kv_cache_reuse/remote_backends/resp
|
|
|
|
# Sweep chunk sizes
|
|
for mb in 0.5 1 2 4 8; do
|
|
echo "=== Chunk: ${mb} MB ==="
|
|
python benchmark_resp_client.py \
|
|
--host 127.0.0.1 --port 6379 \
|
|
--chunk-mb $mb --num-workers 8 --num-keys 500
|
|
done
|
|
|
|
# Sweep worker counts
|
|
for w in 1 2 4 8 16; do
|
|
echo "=== Workers: $w ==="
|
|
python benchmark_resp_client.py \
|
|
--host 127.0.0.1 --port 6379 \
|
|
--chunk-mb 4 --num-workers $w --num-keys 500
|
|
done
|
|
|
|
Expected output:
|
|
|
|
.. code-block:: text
|
|
|
|
Redis RESP Client Benchmark
|
|
Server: 127.0.0.1:6379, Workers: 8
|
|
Chunk size: 4096KB, Keys: 500
|
|
------------------------------------------------------------
|
|
Batch SET: 4.36 GB/s (1.95 GB written)
|
|
Batch GET: 5.91 GB/s (1.95 GB read)
|
|
Batch EXISTS: 143528 ops/s (500/500 hits)
|
|
------------------------------------------------------------
|
|
All tests passed
|
|
|
|
|
|
Environment Variable Configuration
|
|
------------------------------------
|
|
|
|
Sensitive credentials (and optionally host/port) can be provided via environment
|
|
variables instead of config files or CLI arguments. This prevents secrets from
|
|
appearing in logged configuration at startup.
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
:widths: 35 65
|
|
|
|
* - Variable
|
|
- Description
|
|
* - ``LMCACHE_RESP_USERNAME``
|
|
- Redis ACL username. Used as default when ``username`` is not set in config/JSON.
|
|
* - ``LMCACHE_RESP_PASSWORD``
|
|
- Redis AUTH password. Used as default when ``password`` is not set in config/JSON.
|
|
* - ``LMCACHE_RESP_HOST``
|
|
- Redis hostname or IP. Used as default when ``host`` is not set in config/JSON/URL.
|
|
* - ``LMCACHE_RESP_PORT``
|
|
- Redis port. Used as default when ``port`` is not set in config/JSON/URL.
|
|
|
|
Config files (non-MP) and ``--l2-adapter`` JSON (MP) take precedence over
|
|
environment variables. Environment variables serve as defaults — they are used
|
|
when the corresponding config value is empty or unset. They are read at adapter
|
|
creation time inside the adapter itself, so they are **never stored in the
|
|
config object** and **never printed in startup logs**.
|
|
|
|
**Example — MP mode with env vars:**
|
|
|
|
.. code-block:: bash
|
|
|
|
export LMCACHE_RESP_USERNAME="default"
|
|
export LMCACHE_RESP_PASSWORD="secret"
|
|
|
|
lmcache server \
|
|
--l1-size-gb 10 \
|
|
--eviction-policy LRU \
|
|
--chunk-size 16 \
|
|
--l2-adapter '{"type": "resp", "host": "localhost", "port": 6379, "num_workers": 8}' \
|
|
--port 6555
|
|
|
|
**Example — Non-MP mode with env vars:**
|
|
|
|
.. code-block:: bash
|
|
|
|
export LMCACHE_RESP_USERNAME="default"
|
|
export LMCACHE_RESP_PASSWORD="secret"
|
|
|
|
LMCACHE_CONFIG_FILE=resp-config.yaml \
|
|
vllm serve meta-llama/Llama-3.1-8B-Instruct \
|
|
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}' \
|
|
--no-enable-prefix-caching \
|
|
--load-format dummy
|
|
|
|
.. tip::
|
|
For production deployments, always use environment variables for credentials
|
|
rather than embedding them in config files or CLI arguments.
|
|
|
|
|
|
Non-MP Mode (Single Process)
|
|
-----------------------------
|
|
|
|
In non-MP mode, the RESP connector is used directly as a remote storage backend
|
|
via the ``RESPClient`` asyncio wrapper.
|
|
|
|
**Configuration file** (``resp-config.yaml``):
|
|
|
|
.. code-block:: yaml
|
|
|
|
chunk_size: 16
|
|
remote_url: "resp://localhost:6379"
|
|
remote_serde: "naive"
|
|
|
|
Credentials can be set via environment variables (recommended) or in the config file
|
|
under ``extra_config`` (see `Environment Variable Configuration`_ above).
|
|
|
|
**Launch vLLM:**
|
|
|
|
.. code-block:: bash
|
|
|
|
LMCACHE_CONFIG_FILE=resp-config.yaml \
|
|
vllm serve meta-llama/Llama-3.1-8B-Instruct \
|
|
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}' \
|
|
--no-enable-prefix-caching \
|
|
--load-format dummy
|
|
|
|
.. note::
|
|
``save_unfull_chunk`` must be off (default) and chunk metadata saving must be
|
|
disabled for optimal throughput with the native RESP connector.
|
|
|
|
|
|
MP Mode (Multiprocess)
|
|
-----------------------
|
|
|
|
In MP mode, LMCache runs as a separate server process communicating with vLLM over
|
|
ZMQ. The RESP connector serves as an L2 adapter with variable-size chunk support.
|
|
|
|
**Step 1: Start Redis** (see `Redis Server Setup`_ above)
|
|
|
|
**Step 2: Launch LMCache MP Server:**
|
|
|
|
.. code-block:: bash
|
|
|
|
lmcache server \
|
|
--l1-size-gb 10 \
|
|
--eviction-policy LRU \
|
|
--chunk-size 16 \
|
|
--l2-adapter '{"type": "resp", "host": "localhost", "port": 6379, "num_workers": 8}' \
|
|
--port 6555
|
|
|
|
**Step 3: Launch vLLM with LMCache MP Connector:**
|
|
|
|
.. code-block:: bash
|
|
|
|
PORT=8000
|
|
vllm serve meta-llama/Llama-3.1-8B-Instruct \
|
|
--kv-transfer-config '{
|
|
"kv_connector": "LMCacheMPConnector",
|
|
"kv_role": "kv_both",
|
|
"kv_connector_extra_config": {
|
|
"lmcache.mp.host": "tcp://localhost",
|
|
"lmcache.mp.port": 6555
|
|
}
|
|
}' \
|
|
--no-enable-prefix-caching \
|
|
--port $PORT \
|
|
--load-format dummy
|
|
|
|
|
|
L2 Adapter Configuration
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
The ``--l2-adapter`` JSON accepts these fields:
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
:widths: 20 10 10 60
|
|
|
|
* - Field
|
|
- Type
|
|
- Default
|
|
- Description
|
|
* - ``type``
|
|
- str
|
|
- (required)
|
|
- Must be ``"resp"``
|
|
* - ``host``
|
|
- str
|
|
- (required)
|
|
- Redis/Valkey hostname or IP
|
|
* - ``port``
|
|
- int
|
|
- (required)
|
|
- Redis/Valkey port
|
|
* - ``num_workers``
|
|
- int
|
|
- 8
|
|
- C++ worker threads for parallel I/O
|
|
* - ``username``
|
|
- str
|
|
- ``""``
|
|
- Redis ACL username (leave empty for no auth). Falls back to ``LMCACHE_RESP_USERNAME`` env var if empty.
|
|
* - ``password``
|
|
- str
|
|
- ``""``
|
|
- Redis AUTH password (leave empty for no auth). Falls back to ``LMCACHE_RESP_PASSWORD`` env var if empty.
|
|
* - ``max_capacity_gb``
|
|
- float
|
|
- 0
|
|
- Maximum L2 storage capacity in GB for client-side usage tracking. Required for L2 eviction. Set to 0 (default) to disable usage tracking.
|
|
|
|
L2 Eviction
|
|
~~~~~~~~~~~~
|
|
|
|
To enable automatic eviction of least-recently-used keys when the Redis backend fills up,
|
|
set ``max_capacity_gb`` and add an ``"eviction"`` block:
|
|
|
|
.. code-block:: bash
|
|
|
|
lmcache server \
|
|
--l1-size-gb 10 \
|
|
--eviction-policy LRU \
|
|
--chunk-size 16 \
|
|
--l2-adapter '{
|
|
"type": "resp",
|
|
"host": "localhost",
|
|
"port": 6379,
|
|
"num_workers": 8,
|
|
"max_capacity_gb": 10,
|
|
"eviction": {
|
|
"eviction_policy": "LRU",
|
|
"trigger_watermark": 0.8,
|
|
"eviction_ratio": 0.2
|
|
}
|
|
}' \
|
|
--port 6555
|
|
|
|
This configures a 10 GB capacity limit. When usage exceeds 80% (``trigger_watermark``),
|
|
the eviction controller will delete the least-recently-used ~20% of stored keys
|
|
(``eviction_ratio``) using the Redis ``DEL`` command.
|
|
|
|
.. note::
|
|
``max_capacity_gb`` enables **client-side** size tracking. It does not configure
|
|
the Redis server's ``maxmemory`` setting. You should set ``max_capacity_gb`` to
|
|
match or be slightly below your Redis server's available memory.
|
|
|
|
|
|
Testing the Setup
|
|
------------------
|
|
|
|
Send the same prompt twice. The first request stores KV cache to Redis; the second retrieves it.
|
|
|
|
.. code-block:: bash
|
|
|
|
PORT=8000
|
|
PROMPT="$(printf 'Elaborate the significance of KV cache in language models. %.0s' {1..1000})"
|
|
|
|
# First request: store
|
|
curl -s -X POST http://localhost:${PORT}/v1/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"model":"meta-llama/Llama-3.1-8B-Instruct","prompt":"'"$PROMPT"'","max_tokens":10}'
|
|
|
|
# Second request with same prefix: retrieve from Redis
|
|
curl -s -X POST http://localhost:${PORT}/v1/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"model":"meta-llama/Llama-3.1-8B-Instruct","prompt":"'"$PROMPT"'","max_tokens":10}'
|
|
|
|
Verify data was stored:
|
|
|
|
.. code-block:: bash
|
|
|
|
redis-cli -p 6379 DBSIZE
|
|
|
|
Clear state between runs:
|
|
|
|
.. code-block:: bash
|
|
|
|
redis-cli -p 6379 FLUSHALL
|
|
|
|
|
|
Best Practices
|
|
--------------
|
|
|
|
**Server deployment:**
|
|
|
|
- Use Redis 8.2+ with ``--io-threads 4`` (or more, matching available cores)
|
|
- Disable persistence (``--save '' --appendonly no``) for KV cache workloads
|
|
- Pin Redis to its own NUMA node if running on multi-socket systems
|
|
- For production, enable authentication with ``--requirepass`` and supply credentials via ``LMCACHE_RESP_USERNAME`` / ``LMCACHE_RESP_PASSWORD`` environment variables to keep them out of logs
|
|
|
|
**Client tuning:**
|
|
|
|
- Start with ``num_workers: 8`` and increase if the server has spare CPU and you're not saturating the network
|
|
- More workers help when chunk sizes are smaller (more keys per batch = more parallelism needed)
|
|
- On NUMA systems, ensure the LMCache process runs on the same NUMA node as the NIC
|
|
|
|
**Chunk size:**
|
|
|
|
- Target ~4 MB per chunk for maximum throughput
|
|
- Calculate the token count using your model's per-token byte size (see formula above)
|
|
- If unsure, run the benchmark sweep to find the optimum for your specific hardware
|
|
|
|
**Network:**
|
|
|
|
- Use localhost or loopback for single-machine deployments
|
|
- For cross-machine setups, ensure low-latency networking (ideally <100 us RTT)
|
|
- The RESP connector uses TCP; RDMA is not currently supported (consider :doc:`Mooncake <./mooncake>` for RDMA)
|
|
|
|
|
|
Additional Resources
|
|
--------------------
|
|
|
|
- Benchmark script: ``examples/kv_cache_reuse/remote_backends/resp/benchmark_resp_client.py``
|
|
- C++ source: ``csrc/storage_backends/redis/``
|
|
- Native connector architecture: ``csrc/storage_backends/README.md``
|
|
- Developer guide for adding new native connectors: :doc:`Adding Native Connectors <../../developer_guide/extending_lmcache/native_connectors>`
|