lmcache--lmcache/docs/source/mp/deployment.rst

Deployment Guide
================

This page covers deploying LMCache multiprocess mode in Docker and Kubernetes
environments, along with production best practices.

.. contents::
   :local:
   :depth: 2

Docker
------

**LMCache container:**

.. code-block:: bash

    docker run --runtime nvidia --gpus all \
        --network host \
        --ipc host \
        lmcache/standalone:nightly \
        /opt/venv/bin/lmcache server \
        --l1-size-gb 60 --eviction-policy LRU --max-workers 4 --port 6555

**vLLM container:**

.. code-block:: bash

    docker run --runtime nvidia --gpus all \
        --network host \
        --ipc host \
        lmcache/vllm-openai:latest-nightly \
        Qwen/Qwen3-14B \
        --kv-transfer-config \
        '{"kv_connector":"LMCacheMPConnector", "kv_role":"kv_both", "kv_connector_extra_config": {"lmcache.mp.port": 6555}}'

Required Docker flags:

- ``--network host`` -- Allows the vLLM container to reach LMCache on localhost.
- ``--ipc host`` -- Required for CUDA IPC shared memory transfers between
  containers.
- ``--runtime nvidia --gpus all`` -- GPU access via the NVIDIA container
  runtime.

**HTTP server variant:**

For health-check and cache management API support (useful with container
orchestrators), use the HTTP server entry point:

.. code-block:: bash

    docker run --runtime nvidia --gpus all \
        --network host \
        --ipc host \
        lmcache/standalone:nightly \
        /opt/venv/bin/lmcache server \
        --l1-size-gb 60 --eviction-policy LRU --max-workers 4 --port 6555

Kubernetes
----------

LMCache is designed for a **DaemonSet + Deployment** pattern: one LMCache
server per node (DaemonSet) shared by multiple vLLM pods (Deployment).

Example YAML files are provided in ``examples/multi_process/``.

Prerequisites
~~~~~~~~~~~~~

- Kubernetes cluster with GPU support (NVIDIA GPU Operator installed)
- At least 4 GPUs per node
- ``kubectl`` configured to access your cluster

Step-by-Step
~~~~~~~~~~~~

**Step 1: Create namespace**

.. code-block:: bash

    kubectl create namespace multi-process

**Step 2: Deploy LMCache DaemonSet**

.. code-block:: bash

    kubectl apply -f examples/multi_process/lmcache-daemonset.yaml

**Step 3: Deploy vLLM**

.. code-block:: bash

    kubectl apply -f examples/multi_process/vllm-deployment.yaml

.. note::
   The default model is ``Qwen/Qwen3-14B``.  For gated models (e.g., Llama),
   create a Secret with your Hugging Face token:

   .. code-block:: bash

       kubectl create secret generic vllm-secrets \
         --from-literal=hf_token=your_hf_token_here \
         -n multi-process

   Then add the ``HF_TOKEN`` environment variable to the vLLM container spec.

**Step 4: Monitor deployment**

.. code-block:: bash

    # DaemonSet status
    kubectl get daemonset -n multi-process
    kubectl get pods -n multi-process -l app=lmcache-server

    # vLLM status
    kubectl get pods -n multi-process -l app=vllm-deployment -w

    # LMCache logs (for a specific node)
    VLLM_NODE=$(kubectl get pod -n multi-process -l app=vllm-deployment \
        -o jsonpath='{.items[0].spec.nodeName}')
    LMCACHE_POD=$(kubectl get pod -n multi-process -l app=lmcache-server \
        --field-selector spec.nodeName=$VLLM_NODE \
        -o jsonpath='{.items[0].metadata.name}')
    kubectl logs -n multi-process $LMCACHE_POD -f

**Step 5: Send test requests**

.. code-block:: bash

    kubectl port-forward -n multi-process deployment/vllm-deployment 8000:8000

    curl -X POST http://localhost:8000/v1/completions \
        -H "Content-Type: application/json" \
        -d "{
            \"model\": \"Qwen/Qwen3-14B\",
            \"prompt\": \"$(printf 'Explain the significance of KV cache in language models.%.0s' {1..100})\",
            \"max_tokens\": 10
        }"

Architecture Notes
~~~~~~~~~~~~~~~~~~

- **DaemonSet uses ``hostNetwork: true``** so vLLM pods discover the LMCache
  server via ``status.hostIP``.
- **Both containers mount ``/dev/shm``** from the host to enable CUDA IPC
  memory sharing.
- **GPUs are NOT requested in the DaemonSet** -- this allows GPUs to remain
  exclusively allocated to vLLM pods.  The NVIDIA container runtime
  automatically provides GPU access for IPC-based memory transfers.
- **Multiple vLLM pods** on the same node automatically connect to the same
  LMCache DaemonSet instance.

.. note::
   LMCache pods on nodes without GPUs will crash with CUDA initialization
   errors.  This is expected -- LMCache only needs to run on GPU nodes where
   vLLM pods are scheduled.

Health Checking (HTTP Server)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For Kubernetes liveness/readiness probes, deploy the HTTP server variant
instead.  Use the ``/healthcheck`` endpoint:

.. code-block:: yaml

    livenessProbe:
      httpGet:
        path: /healthcheck
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 30
    readinessProbe:
      httpGet:
        path: /healthcheck
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 10

Monitoring Integration
~~~~~~~~~~~~~~~~~~~~~~

Prometheus metrics are enabled by default on port 9090.  Add a
``ServiceMonitor`` or Prometheus scrape annotation to collect metrics from the
LMCache DaemonSet pods.  See :doc:`observability/index` for metric details.

Cleanup
~~~~~~~

.. code-block:: bash

    kubectl delete -f examples/multi_process/vllm-deployment.yaml
    kubectl delete -f examples/multi_process/lmcache-daemonset.yaml
    kubectl delete namespace multi-process

Production Best Practices
-------------------------

**Worker count (``--max-workers``, ``--max-gpu-workers``, ``--max-cpu-workers``):**
``--max-workers`` sets both the GPU affinity pool and CPU normal pool sizes
(default 1).  Use ``--max-gpu-workers`` to override the GPU pool independently
--- set it to at least the number of vLLM instances sharing the cache server so
each instance gets its own dedicated thread.  Use ``--max-cpu-workers`` to
override the CPU pool for lookup and other non-GPU operations.

**L1 memory sizing (``--l1-size-gb``):**
Allocate as much CPU memory as available after accounting for the OS and vLLM.
A larger L1 cache means fewer L2 round-trips.

**Eviction tuning:**

- ``--eviction-trigger-watermark 0.8`` (default) triggers eviction when L1 is
  80% full.
- ``--eviction-ratio 0.2`` (default) frees 20% of allocated memory per
  eviction cycle.
- Lower the watermark or increase the ratio if you observe frequent evictions
  under steady load.

**Logging:**
Use ``LMCACHE_LOG_LEVEL=DEBUG`` during initial setup to verify L2 store/load
activity.  Switch to ``INFO`` (default) for production to reduce log volume.

Transfer Mode (``--supported-transfer-mode``, ``--shm-name``)
-------------------------------------------------------------

LMCache supports two worker → server transfer paths: an
**lmcache-driven** path (server pulls/pushes via CUDA IPC or CPU SHM,
used for STORE/RETRIEVE) and an **engine-driven** path
(PREPARE/COMMIT, used by CPU-only or non-CUDA accelerator workers).
The server picks which paths to load via ``--supported-transfer-mode``:

- ``auto`` *(default)* -- load both paths.  Workers of either device
  type can connect without manual configuration; the server has no
  upfront knowledge of the connecting worker's device.
- ``lmcache_driven`` -- load only the server-driven transfer path.
  Supports CUDA devices (IPC) and CPU devices (SHM).  Use to skip
  allocating the engine-driven prepare/commit resources (pickle codec).
- ``engine_driven`` -- load only the engine-driven path.  Use when
  serving CPU-only or non-CUDA accelerator workers.

When the engine-driven path is loaded (``auto`` or ``engine_driven``),
LMCache by default creates a shared-memory (SHM) pool for KV transfers
between the server and vLLM workers.  The ``--shm-name`` option lets
you control this behavior:

.. list-table::
   :header-rows: 1
   :widths: 30 70

   * - Value
     - Effect
   * - *(not set)* (default)
     - Auto-allocate a SHM pool (current default behavior).
   * - ``""`` (empty string)
     - Disable the SHM pool entirely and fall back to the pickle-based
       transfer path.  Useful when ``/dev/shm`` is unavailable or when
       running without ``--ipc host`` in Docker.
   * - ``"my_pool"`` (any non-empty name)
     - Use that exact name for the SHM segment instead of the
       auto-generated one.  Handy when you need a deterministic,
       human-readable segment name for monitoring or debugging.

**Examples:**

.. code-block:: bash

    # Force pickle (no SHM):
    lmcache server --l1-size-gb 60 --eviction-policy LRU --shm-name ""

    # Named SHM segment:
    lmcache server --l1-size-gb 60 --eviction-policy LRU --shm-name "lmcache_pool"