lmcache--lmcache/examples/serde/turboquant

TurboQuant Serde End-to-End Example

This example demonstrates the per-adapter serde feature: the L2 disk adapter quantizes KV cache to TurboQuant before writing to disk, and dequantizes back to the original dtype on prefetch.

What it does

1. Starts an lmcache server with:
   • L1: 20 GB CPU memory cache, LRU eviction
   • L2: filesystem (disk) adapter at /tmp/lmcache_turboquant_serde_disk
   • Serde: TurboQuant (TurboQuant preset) attached to the L2 adapter
2. Starts vLLM connected via LMCacheMPConnector
3. Sends an inference request — KV is computed, written to L1, then asynchronously serialized (TurboQuant) and stored to L2 disk
4. Calls the lmcache HTTP API to force-clear L1 (CPU cache)
5. Re-sends the same request — L1 misses, L2 prefetch fires, the serialized bytes are loaded from disk and deserialized back into KV-shaped buffers, then vLLM resumes from cache

Files

• run_serde_turboquant_example.sh — full end-to-end: lmcache server + vllm serve + real inference, then clear L1 and re-infer to hit the L2 path.

Quick sanity check (no vLLM required)

The pytest suite includes TurboQuant serde tests that exercise direct CUDA, MockL2, and filesystem-backed L2 round-trips without needing vLLM:

CUDA_VISIBLE_DEVICES=0 python -m pytest tests/v1/distributed/serde/test_turboquant.py -q -s

Requirements

• vLLM installed (vllm serve works)
• lmcache CLI installed (lmcache server --help works)
• 1 GPU (default CUDA_VISIBLE_DEVICES=0)
• PyTorch + Triton CUDA environment

Run

./run_serde_turboquant_example.sh

You can override defaults via environment variables:

MODEL="meta-llama/Llama-3.1-8B-Instruct" \
GPU_DEVICE=0 \
L1_SIZE_GB=20 \
LMCACHE_PORT=6555 \
VLLM_PORT=8000 \
TQ_PRESET=turboquant_k8v4 \
TQ_BLOCK_SIZE=16 \
./run_serde_turboquant_example.sh

Server output is streamed to stdout. Logs are also saved under /tmp/lmcache_turboquant_serde_example/{lmcache,vllm}.log (override with TMP_DIR).

L2 adapter config syntax

The serde is attached per-adapter via a serde sub-dict in the --l2-adapter JSON. Each adapter independently decides whether to use serde.

{
  "type": "fs",
  "base_path": "/tmp/lmcache_turboquant_serde_disk",
  "serde": {
    "type": "turboquant",
    "preset": "turboquant_k8v4",
    "block_size": 16
  }
}

To disable serde for an adapter, omit the serde field.

Adding a custom serde

1. Implement Serializer and Deserializer from lmcache.v1.distributed.serde

2. Register a factory:

   from lmcache.v1.distributed.serde import (
       AsyncSerdeProcessor,
       register_serde_factory,
   )
   
   def _create_my_serde(config: dict):
       return AsyncSerdeProcessor(MySerializer(), MyDeserializer())
   
   register_serde_factory("mine", _create_my_serde)
   

3. Reference it in the adapter config: "serde": {"type": "mine", ...}