wehub-resource-sync
66 lines
3.2 KiB
Markdown
66 lines
3.2 KiB
Markdown
# Dynamo + LMCache MP integration
|
|
|
|
Recipes for running [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) vLLM
|
|
serving with KV cache offloaded to an **LMCache multiprocess (mp) server**, in
|
|
both **aggregated** and **disaggregated** modes.
|
|
|
|
In mp mode LMCache runs as an out-of-process cache engine. vLLM workers attach
|
|
to it through the `LMCacheMPConnector` and share KV tensors over CUDA IPC, so
|
|
`sharedMemory` is disabled (LMCache owns `/dev/shm`) and `hostIPC` is enabled.
|
|
|
|
## Layout
|
|
|
|
| Path | What it is |
|
|
|------|------------|
|
|
| [`deploy/lmcache_engine.yaml`](deploy/lmcache_engine.yaml) | `LMCacheEngine` CR for the shared MP server. Apply **before** the workers. |
|
|
| [`deploy/agg_lmcache_mp.yaml`](deploy/agg_lmcache_mp.yaml) | Kubernetes `DynamoGraphDeployment`, aggregated (single worker). |
|
|
| [`deploy/disagg_lmcache_mp.yaml`](deploy/disagg_lmcache_mp.yaml) | Kubernetes `DynamoGraphDeployment`, disaggregated (prefill + decode workers). |
|
|
| [`launch/agg_lmcache_mp.sh`](launch/agg_lmcache_mp.sh) | Local single-node launch script, aggregated (1 GPU). |
|
|
| [`launch/disagg_lmcache_mp.sh`](launch/disagg_lmcache_mp.sh) | Local single-node launch script, disaggregated (2 GPUs). |
|
|
|
|
## The LMCacheEngine (shared prerequisite)
|
|
|
|
[`deploy/lmcache_engine.yaml`](deploy/lmcache_engine.yaml) declares the MP
|
|
server that both the aggregated and disaggregated workers attach to. The
|
|
LMCache operator reconciles the CR into:
|
|
|
|
- a per-node **manager DaemonSet** pod (the LMCache server that holds the CPU
|
|
KV pool), and
|
|
- a **ConfigMap** named `<metadata.name>-connection` (i.e. `lmcache-mp-connection`),
|
|
which the vLLM workers mount at `/etc/lmcache` and read as
|
|
`--kv-transfer-config`.
|
|
|
|
It must live in the **same namespace** as the workers (`default`) so the
|
|
connection ConfigMap is created where the workers can mount it.
|
|
|
|
**Version pinning**: the server's bundled lmcache must be wire-compatible with
|
|
the worker's. The worker image `vllm-runtime:1.2.0-deepseek-v4-cuda13-dev.3`
|
|
ships lmcache `0.4.4` (vLLM `0.20.1`), and the recipe pairs it with the
|
|
guide-validated server build `nightly-2026-04-25` (lmcache `0.4.5.dev31`, a
|
|
pre-stable build wire-compatible with that worker). Replace `my-tag` in each
|
|
manifest accordingly.
|
|
|
|
## Usage
|
|
|
|
The `deploy/` manifests are applied with `kubectl` against a cluster that
|
|
already has the Dynamo platform and the LMCache operator installed. Apply the
|
|
`LMCacheEngine` first, then one of the worker manifests:
|
|
|
|
```bash
|
|
kubectl apply -n default -f deploy/lmcache_engine.yaml
|
|
kubectl apply -n default -f deploy/disagg_lmcache_mp.yaml # or agg_lmcache_mp.yaml
|
|
```
|
|
|
|
See the [Kubernetes deployment guide](../../docs/source/mp/deployment.rst) for
|
|
the full step-by-step (platform install, operator, namespace, HF token Secret,
|
|
then `kubectl apply`).
|
|
|
|
The `launch/` scripts are the single-node local equivalents. They run **inside
|
|
the Dynamo `vllm-runtime` container** and source Dynamo's
|
|
`examples/common/{gpu_utils,launch_utils}.sh` helpers, so run them from a Dynamo
|
|
checkout (or copy them into `examples/backends/vllm/launch/`).
|
|
|
|
> The deploy manifests pin the worker image to `my-tag` and the
|
|
> `LMCacheEngine` image separately; keep the two wire-compatible (see the
|
|
> validated pairings in the deployment guide).
|