wehub/lmcache--lmcache

Fork 0

Files

T

wehub-resource-sync f213ec8976 chore: import upstream snapshot with attribution

2026-07-13 12:24:33 +08:00

31 KiB

Raw Permalink Blame History

LMCache Kubernetes Operator Design

Overview

LMCache multiprocess mode runs as a separate server process that vLLM instances connect to for KV cache offloading. Today this is deployed manually via raw K8s manifests (a DaemonSet for LMCache + a Deployment for vLLM). A K8s operator would automate lifecycle management, enforce best practices (hostIPC, resource sizing), and expose connection info for vLLM discovery.

Current Pain Points the Operator Solves

Manual hostIPC setup and node-local service discovery
No automated connection info propagation to vLLM
No Prometheus ServiceMonitor integration
No validation of configuration parameters

How the Operator Addresses These

The operator introduces a single CRD (LMCacheEngine) that declaratively captures the full LMCache server configuration. The controller reconciles this CR into the underlying K8s resources (DaemonSet, Service, ConfigMap, ServiceMonitor), automatically injecting the required pod-level settings that are easy to forget in hand-written manifests.

Auto-injected pod settings eliminate manual boilerplate. The controller always sets hostIPC: true and --host 0.0.0.0 — settings that the current lmcache-daemonset.yaml requires users to specify by hand. Getting any of these wrong (e.g., forgetting hostIPC) causes silent connectivity failures or CUDA IPC errors that are hard to debug.

A node-local Service with connection ConfigMap provides a stable discovery contract for vLLM. The operator creates a ClusterIP Service with internalTrafficPolicy=Local, which ensures kube-proxy routes traffic only to the LMCache pod on the same node. A ConfigMap (<name>-connection) contains the kv-transfer-config JSON pointing to this service's cluster DNS name. vLLM deployments simply mount the ConfigMap — no downward API or shell substitution needed. When the LMCache CR changes, the ConfigMap updates automatically and vLLM pods pick up the new values on restart.

Prometheus integration is declarative. When prometheus.serviceMonitor.enabled is set, the operator creates a ServiceMonitor CR that the Prometheus Operator discovers automatically. Without the operator, users must manually create ServiceMonitor resources and keep labels/ports in sync with the DaemonSet.

CRD validation catches misconfigurations at apply time. OpenAPI schema validation and a validating webhook enforce constraints (e.g., l1.sizeGB > 0, eviction.triggerWatermark in (0.0, 1.0]) before any pods are created. Today, invalid CLI flags only surface as runtime crashes inside the container.

Resource sizing is auto-computed from the L1 cache size. The operator derives memoryRequest (l1.sizeGB + 5 GiB) and memoryLimit (1.5x request) automatically, eliminating the mental math that currently leads to either OOM kills (under-provisioned) or wasted node capacity (over-provisioned). Users can override with explicit values.

API Group & CRD

apiVersion: lmcache.ai/v1alpha1
kind: LMCacheEngine

Group lmcache.ai
v1alpha1 — alpha maturity; shape will evolve as L2 backends stabilize
LMCacheEngine — represents the per-node cache engine. Future CRDs can cover other concerns (e.g., LMCacheKeyManager for global key management, LMCacheMonitor for engine state monitoring)

CRD Spec (Complete)

apiVersion: lmcache.ai/v1alpha1
kind: LMCacheEngine
metadata:
  name: string
  namespace: string
spec:
  # -- Container image --
  image:
    repository: string        # default: "lmcache/standalone"
    tag: string               # default: "nightly"
    pullPolicy: string        # default: IfNotPresent
  imagePullSecrets: []LocalObjectReference

  # -- Server config (maps to server.py argparse) --
  server:
    port: int                 # default: 5555
    chunkSize: int            # default: 256 tokens
    maxWorkers: int           # default: 1
    hashAlgorithm: string     # default: blake3 (builtin | sha256_cbor | blake3)

  # -- L1 cache (maps to L1MemoryManagerConfig + L1ManagerConfig) --
  # Internal tuning knobs (useLazy, alignBytes, writeTTLSeconds,
  # readTTLSeconds) use server defaults and can be overridden via the
  # env escape hatch if needed.
  l1:
    sizeGB: float             # REQUIRED

  # -- Eviction (maps to EvictionConfig) --
  eviction:
    policy: string            # default: "LRU" (only supported value)
    triggerWatermark: float   # default: 0.8 (range 0.0-1.0)
    evictionRatio: float      # default: 0.2 (range 0.0-1.0)

  # -- Monitoring (maps to PrometheusConfig from mp_observability/config.py) --
  # Note: the CRD uses `enabled: true` by default; the CLI equivalent is
  # the absence of the `--disable-prometheus` flag.
  prometheus:
    enabled: bool             # default: true  (CLI: omit --disable-prometheus)
    port: int                 # default: 9090
    serviceMonitor:
      enabled: bool           # default: false
      interval: string        # default: "30s"
      labels: map[string]string
  # ServiceMonitor is a CRD from the Prometheus Operator (kube-prometheus-stack).
  # When enabled, the operator creates a ServiceMonitor resource that tells
  # Prometheus to automatically discover and scrape LMCache metrics endpoints.
  # Without it, you'd need to manually configure Prometheus scrape targets.
  # If you don't use the Prometheus Operator, leave serviceMonitor.enabled=false
  # and use the pod annotations (prometheus.io/scrape, prometheus.io/port) instead.

  # -- L2 storage backend (single adapter) --
  # Currently only one L2 adapter is supported at a time.
  # LMCache MP mode supports multiple adapters, but this is not yet
  # fully tested. Once validated, the operator will support multiple.
  # Exactly one of resp or raw must be set.
  l2Backend:
    # Option A: Native RESP (Redis/Valkey) adapter
    resp:
      host: string              # REQUIRED
      port: int                 # REQUIRED, 1-65535
      numWorkers: int           # default: 8
      maxCapacityGB: float      # default: 0 (disabled)
      authSecretRef:            # optional, Secret with "username"/"password" keys
        name: string
    # Option B: Raw escape hatch for other adapter types
    raw:
      type: string              # adapter type name (nixl_store, fs, mock, raw_block, etc.)
      config: map[string]any    # type-specific config as free-form map

  # -- Connection-injection webhook defaults (optional) --
  # Read by the LMCache mutating webhook for pods bound to this engine. When
  # unset, the webhook wires only the connection (--kv-transfer-config, hostIPC,
  # PYTHONHASHSEED). Set injection.payloadImage to ALSO stage an lmcache code
  # tree into the vLLM container (emptyDir + init container + readOnly mount +
  # PYTHONPATH=/lmcache-payload), reusing the CacheBlend payload-staging
  # mechanism so the vLLM client and the engine server run one lmcache build.
  injection:
    payloadImage:               # SEPARATE image: ships lmcache under /payload
      repository: string        # REQUIRED (no valid default)
      tag: string               # default: latest
      pullPolicy: string        # default: IfNotPresent
    imagePullSecrets: []LocalObjectReference  # for a private payload image
    targetContainer: string     # default: first container

  # -- Resources (auto-computed, no user input needed) --
  # The operator derives resource requests/limits from l1.sizeGB:
  #   memoryRequest = ceil(l1.sizeGB + 5) Gi
  #   memoryLimit   = ceil(memoryRequest * 1.5) Gi
  #   cpuRequest    = "4"
  # To override, use the resourceOverrides escape hatch below.
  resourceOverrides: ResourceRequirements  # optional, raw K8s resources override

  # -- Logging --
  logLevel: string            # default: INFO (DEBUG|INFO|WARNING|ERROR)

  # -- Scheduling --
  nodeSelector: map[string]string
  affinity: Affinity
  tolerations: []Toleration
  # nodeSelector determines which nodes get an LMCache instance.
  # Use nodeSelector: {nvidia.com/gpu.present: "true"} to target all GPU
  # nodes. When new GPU nodes join the cluster, the DaemonSet controller
  # automatically schedules an LMCache pod on them.

  # -- Overrides --
  env: []EnvVar               # additional environment variables
  volumes: []Volume           # additional volumes (e.g. for L2 disk backend)
  volumeMounts: []VolumeMount
  podAnnotations: map[string]string
  podLabels: map[string]string
  serviceAccountName: string
  priorityClassName: string

  # -- Security --
  privileged: bool            # default: false; run the engine container in
                              # privileged mode. Needed only on clusters where
                              # the engine cannot see the node GPUs otherwise
                              # (see "Auto-managed Pod Settings" below).

  # -- Extra CLI flags --
  extraArgs: []string         # appended last, can override any auto-generated flag

Auto-managed Pod Settings (not in CRD spec)

The operator always injects these into the pod spec:

hostIPC: true — required for CUDA IPC between LMCache and vLLM. LMCache uses CudaIPCWrapper which calls PyTorch's _share_cuda_() to get a GPU driver-level IPC handle. The handle is serialized and sent over ZMQ TCP. The receiving process reconstructs the tensor via cudaIpcOpenMemHandle at the driver level. This call requires both processes to share the same IPC namespace — without hostIPC: true, cudaIpcOpenMemHandle fails with cudaErrorMapBufferObjectFailed. Both the LMCache pods and vLLM pods must have hostIPC: true.
runtimeClassName: nvidia — uses the NVIDIA container runtime, which injects the host's NVIDIA driver libraries and device files into the container. This is required for CUDA to function inside the pod.
NVIDIA_VISIBLE_DEVICES=all — gives the engine access to all GPUs on the node for CUDA IPC and custom data transfer kernels without claiming any via nvidia.com/gpu resource requests (which would make those GPUs unavailable to the serving engine). The combination of runtimeClassName: nvidia + NVIDIA_VISIBLE_DEVICES=all lets the container see all GPUs without consuming device-plugin resources, so the serving engine (e.g., vLLM) can still request all GPUs on the node. On most clusters this is sufficient; on some, the engine cannot see the GPUs unless the pod is also privileged — set spec.privileged: true to run the engine container privileged (default false).
NVIDIA_VISIBLE_DEVICES=all and NVIDIA_DRIVER_CAPABILITIES=all — env vars that instruct the NVIDIA container runtime to expose all GPUs and all driver capabilities to the container.
--host 0.0.0.0 — always passed as a container arg. The server defaults to --host localhost which only binds to loopback; the server must bind to all interfaces so the node-local Service can route traffic to it.
No hostNetwork — the operator does not use hostNetwork. Instead, it creates a ClusterIP Service with internalTrafficPolicy=Local. kube-proxy ensures that traffic to the service is routed only to the LMCache pod on the same node. This avoids occupying host ports and reduces the privileged surface area.
No /dev/shm emptyDir mount — the operator intentionally does not mount an emptyDir at /dev/shm. With hostIPC: true, the container already sees the host's /dev/shm. Mounting an emptyDir would shadow the host's /dev/shm with a private tmpfs, breaking CUDA IPC (cudaIpcOpenMemHandle fails because IPC handles written by one pod are invisible to others). If your workload needs a larger /dev/shm for non-IPC purposes, add it via spec.volumes / spec.volumeMounts.

Security implications: The LMCache pods run with hostIPC: true (required for CUDA IPC), which exposes the host's IPC namespace. When spec.privileged: true is set they additionally run privileged, granting full device access to the container. Only deploy in trusted environments. Clusters using Pod Security Standards must allow the privileged profile for the LMCache namespace whenever hostIPC/privileged are in use — the baseline and restricted profiles reject these settings.

Validation Rules

Field	Rule
`l1.sizeGB`	Required, must be `> 0`
`eviction.policy`	Must be `"LRU"` (if set)
`eviction.triggerWatermark`	Must be in `(0.0, 1.0]`
`eviction.evictionRatio`	Must be in `(0.0, 1.0]`
`server.port`	Must be in `[1024, 65535]`

CRD Status

status:
  phase: Pending | Running | Degraded | Failed
  observedGeneration: int64

  desiredInstances: int
  readyInstances: int

  # Per-node connection info (for kubectl visibility)
  endpoints:
    - nodeName: string
      hostIP: string
      podName: string
      port: int
      metricsPort: int
      ready: bool

  # Standard conditions
  conditions:
    - type: Available          # at least one instance ready
    - type: AllInstancesReady  # all desired instances ready
    - type: ConfigValid        # spec validation passed
  # Connection ConfigMap is always named <metadata.name>-connection;
  # no need to store the ref in status.

Examples

Minimal Deployment

apiVersion: lmcache.ai/v1alpha1
kind: LMCacheEngine
metadata:
  name: my-cache
spec:
  l1:
    sizeGB: 60

This deploys a DaemonSet with 60GB L1 cache, LRU eviction, blake3 hashing, port 5555, auto-computed 65Gi memory request / 98Gi limit, Prometheus on 9090, and a connection ConfigMap for vLLM. The operator auto-injects hostIPC and --host 0.0.0.0, and creates a node-local Service for vLLM discovery.

Production Deployment (all GPU nodes)

apiVersion: lmcache.ai/v1alpha1
kind: LMCacheEngine
metadata:
  name: production-cache
  namespace: llm-serving
spec:
  # Target all GPU nodes — new GPU nodes automatically get an LMCache pod
  nodeSelector:
    nvidia.com/gpu.present: "true"

  image:
    repository: lmcache/standalone
    tag: v0.1.0
    pullPolicy: IfNotPresent

  server:
    port: 6555
    chunkSize: 256
    maxWorkers: 4
    hashAlgorithm: blake3

  l1:
    sizeGB: 60

  eviction:
    triggerWatermark: 0.8
    evictionRatio: 0.2

  prometheus:
    enabled: true
    port: 9090
    serviceMonitor:
      enabled: true
      labels:
        release: kube-prometheus-stack

  logLevel: INFO
  podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9090"
  priorityClassName: system-node-critical

Resources Created by the Operator

For LMCacheEngine named production-cache:

Resource	Name	Purpose
DaemonSet	`production-cache`	Runs LMCache server pods
Service (ClusterIP, `internalTrafficPolicy=Local`)	`production-cache`	Node-local service discovery for vLLM
ConfigMap	`production-cache-connection`	kv-transfer-config JSON pointing to the lookup Service
Service (headless)	`production-cache-metrics`	Prometheus scrape target
ServiceMonitor (optional)	`production-cache`	Prometheus Operator integration

ConfigMap Content (for vLLM discovery)

{
  "kv_connector": "LMCacheMPConnector",
  "kv_connector_module_path": "lmcache.integration.vllm.lmcache_mp_connector",
  "kv_role": "kv_both",
  "kv_connector_extra_config": {
    "lmcache.mp.host": "tcp://<name>.<namespace>.svc.cluster.local",
    "lmcache.mp.port": "<spec.server.port, default 5555>"
  }
}

The ConfigMap uses the lookup Service's cluster DNS name. Because the Service has internalTrafficPolicy=Local, kube-proxy routes traffic only to the LMCache pod on the same node as the vLLM pod. vLLM pods mount this ConfigMap and pass the JSON to --kv-transfer-config — no downward API or shell variable substitution required. The explicit kv_connector_module_path makes the external LMCache MP connector path load-bearing and avoids silent fallback to an older vendored builtin connector path.

Auto-injected Pod Template Details

In addition to the auto-managed pod settings above (hostIPC, --host 0.0.0.0), the operator injects:

Container command: /opt/venv/bin/lmcache server
Container args: serialized from spec fields
Env: LMCACHE_LOG_LEVEL from spec.logLevel
Probes:
- Startup: TCP on server port, initialDelay=5s, period=5s, failureThreshold=30
- Liveness: TCP on server port, period=10s
- Readiness: TCP on server port, period=5s

Reconciliation Logic

OnEvent(LMCacheEngine create/update/delete):

1. VALIDATE spec -> set condition ConfigValid
2. COMPUTE derived values (unless overridden):
   - memoryRequest = ceil(l1.sizeGB + 5) Gi
   - memoryLimit = ceil(memoryRequest * 1.5) Gi
   - containerArgs from all spec fields
3. RECONCILE DaemonSet (CreateOrUpdate, ownerRef)
   - Always inject: hostIPC, runtimeClassName: nvidia, NVIDIA_VISIBLE_DEVICES=all, --host 0.0.0.0 (privileged only when spec.privileged=true)
4. RECONCILE node-local lookup Service (internalTrafficPolicy=Local)
5. RECONCILE headless Service for metrics
6. RECONCILE connection ConfigMap
7. RECONCILE ServiceMonitor (if enabled)
8. UPDATE status:
   - Query workload for ready/desired counts
   - Enumerate pods -> build endpoints list
   - Set phase: Running | Degraded | Pending | Failed
   - Set conditions, observedGeneration

Secondary watches: DaemonSet, Pods (readiness changes → update endpoints), Nodes (new GPU node → DaemonSet auto-schedules)

Deletion / cleanup: every child resource the operator creates (DaemonSet, lookup Service, metrics Service, connection ConfigMap, managed RESP auth Secret, optional ServiceMonitor) carries an ownerReference to the LMCacheEngine, so Kubernetes garbage collection cascade-deletes them when the CR goes away. No finalizer is used. An earlier design added a lmcache.ai/cleanup finalizer to mirror that GC behavior, but it was a no-op that only created deadlocks when the controller pod was not running (e.g. during cluster issues or a single-step kubectl delete -k config/default). The reconciler now actively strips that legacy finalizer from any CR it sees, so migration from older operator versions is automatic. Finalizers will return when we need to clean up state K8s GC cannot reach (Redis L2 keys, federation deregistration, etc.).

CacheBlend: `CacheBlendEngine` CRD + Injection Webhook

CacheBlend reuses cached KV at shifted positions. It has two halves the operator manages together: a GPU-resident blend engine (server side) and a vLLM-side plugin that must be loaded into the serving container. The operator ships both as a second CRD plus a mutating admission webhook.

This implements what was previously deferred as a future blend.enabled field on LMCacheEngine. It is instead a separate CacheBlendEngine kind (with its own controller) plus an injection webhook — cleaner separation, and no behavior change to LMCacheEngine.

`CacheBlendEngine` CRD

Group lmcache.lmcache.ai, v1alpha1, kind CacheBlendEngine (shortName cbe). The spec mirrors LMCacheEngineSpec (image, server, l1, eviction, prometheus, l2Backend, scheduling, overrides, imagePullSecrets) and adds:

blend.checkLayer (default 1) and blend.recompRatio (default 0.15) — CB tunables fed to the vLLM connector.
injection — what the webhook injects into vLLM pods: payloadImage (an ImageSpec — repository/tag/pullPolicy, like spec.image — for the private lmcache-cacheblend init-container image; set repository explicitly, the inherited engine-image default is not a valid payload), imagePullSecrets (appended to the vLLM pod so the private payload image can pull — the Secret must exist in the vLLM pod's namespace), targetContainer (default: first container), and cudagraph (eager|piecewise|full_decode_only, default eager).
server.chunkSize defaults to 256 and is validated to equal 256 (the blend matcher requires chunk_size == vLLM --block-size * 4).

The blend engine (controller)

CacheBlendEngineReconciler mirrors LMCacheEngineReconciler and reconciles a DaemonSet running lmcache server --engine-type blend (plus --l1-align-bytes 16777216), a node-local lookup Service, a metrics Service, and a <name>-connection ConfigMap. GPU model is identical to LMCacheEngine: runtimeClassName: nvidia + NVIDIA_VISIBLE_DEVICES=all + hostIPC: true (and privileged when spec.privileged=true), with no nvidia.com/gpu device-plugin claim — the engine shares the vLLM GPU rather than reserving one, because the blend server scatters re-RoPE'd KV directly into vLLM's paged KV over same-device CUDA IPC. The engine resource builders are the same name/spec-keyed cores used by LMCacheEngine.

The <name>-connection ConfigMap carries the CBKVConnector kv-transfer-config (vs LMCacheMPConnector for LMCacheEngine) — same node-local tcp:// host/port shape, plus the cb.* tunables:

{
  "kv_connector": "CBKVConnector",
  "kv_connector_module_path": "lmcache_cacheblend.connector",
  "kv_role": "kv_both",
  "kv_connector_extra_config": {
    "lmcache.mp.host": "tcp://<name>.<namespace>.svc.cluster.local",
    "lmcache.mp.port": "<server.port>",
    "cb.check_layer": <blend.checkLayer>,
    "cb.recomp_ratio": <blend.recompRatio>
  }
}

Co-location works exactly like LMCacheEngine: one engine per GPU node (DaemonSet), and the node-local Service (internalTrafficPolicy: Local) routes a vLLM pod to the same-node engine. The control-plane RPC is TCP via that Service; the data-plane KV write is CUDA IPC on the shared GPU.

The injection webhook

A mutating admission webhook (/mutate--v1-pod, CREATE, failurePolicy: Ignore) injects the lmcache-cacheblend plugin into opted-in pods so a stock vLLM image needs no rebuild. A pod opts in with label lmcache.ai/cacheblend-inject: "true" and binds to an engine with annotation lmcache.ai/cacheblend-engine: <name>. The webhook then applies:

Mutation	What
pod `hostIPC: true`	required for CUDA IPC with the node-local engine
`cb-plugin` emptyDir + payload init container	the busybox payload `cp -a`'s the pure-Python plugin tree onto the shared volume
readOnly mount + `PYTHONPATH=/cb-plugin` on the vLLM container	vLLM discovers the plugin via its `vllm.general_plugins` entry point
append required vLLM args	`--attention-backend CUSTOM`, `--kv-transfer-config <from the connection ConfigMap>`, `--block-size 64`, `--pipeline-parallel-size 1`, `--no-enable-chunked-prefill`, `--no-async-scheduling`, `--enforce-eager` (or the configured cudagraph)
append `injection.imagePullSecrets`	so the private payload image can pull
stamp `lmcache.ai/cacheblend-injected: "true"`	idempotency guard

The webhook skips (stamping lmcache.ai/cacheblend-skip-reason) when: the target container overrides command (a sh -c wrapper — appended args wouldn't reach vllm serve); the user already supplies --kv-transfer-config (not clobbered); the named engine's connection ConfigMap doesn't exist; the engine's injection.payloadImage resolves to an empty reference (payload-image-unset); or the requested targetContainer/cacheblend-container annotation names a container that does not exist on the pod (target-container-not-found). It does not gate on engine readiness — like LMCacheEngine, the connector connects when the engine comes up. Args are emitted in two-token form (--attention-backend CUSTOM); the replace-not-duplicate dedup still recognizes a user-supplied --flag=value.

Shared with the LMCache injector. The emptyDir + payload init container + readOnly mount + PYTHONPATH staging (rows 2–3 above) is generic — the standard LMCacheEngine injector (/mutate-lmcache--v1-pod) reuses it (under internal/webhook/payload_staging.go) to optionally stage an lmcache build when LMCacheEngine.spec.injection.payloadImage is set. There the staging is additive and optional: the connection wiring always runs, and an unset (or absent) injection.payloadImage simply stages nothing — it never skips the whole injection. The LMCache staging uses its own names (lmcache-payload volume, /lmcache-payload mount) so both injectors can fire on one pod.

Prerequisites

cert-manager — the webhook's serving cert is a cert-manager Issuer + Certificate (caBundle injected via inject-ca-from); install it before make deploy.
make deploy, not make run — make run sets ENABLE_WEBHOOKS=false and installs no MutatingWebhookConfiguration; it is controller-only. The webhook needs the operator running as an in-cluster pod.
Pod Security Standards — the injected hostIPC/privileged is rejected by the baseline/restricted profiles, so the engine's and the vLLM pod's namespaces must be labeled pod-security.kubernetes.io/enforce=privileged.

Resources created (for a `CacheBlendEngine` named `cb`)

Resource	Name	Purpose
DaemonSet	`cb`	`lmcache server --engine-type blend` on GPU nodes
Service (node-local)	`cb`	same-node discovery for vLLM (`CBKVConnector`)
Service (headless)	`cb-metrics`	Prometheus scrape target
ConfigMap	`cb-connection`	`CBKVConnector` kv-transfer-config
MutatingWebhookConfiguration	(operator-wide)	injects the plugin into opted-in vLLM pods

Coordinator: `LMCacheCoordinator` CRD

The coordinator (lmcache/v1/mp_coordinator/) is the fleet-level HTTP service that engine servers register/heartbeat against, and which drives L2 quota eviction and global CacheBlend lookups. Unlike the engines (one DaemonSet pod per GPU node), the coordinator is a single fleet-wide service, so LMCacheCoordinator reconciles a Deployment + ClusterIP Service instead of a DaemonSet. The controller carries no finalizer — owner-reference GC cascade-deletes the children.

`LMCacheCoordinatorSpec`

The spec mirrors MPCoordinatorConfig (lmcache/v1/mp_coordinator/config.py); the controller renders each field into the matching lmcache coordinator CLI flag: host, port (9300), instanceTimeout (30), healthCheckInterval (10), evictionCheckInterval (5), evictionRatio (0.2), triggerWatermark (1.0), blendChunkSize (256), blendProbeStride (1). It also carries replicas, image, prometheus, and the usual pod-shaping fields (resources, scheduling, env, etc.).

The global-CacheBlend knobs (blendChunkSize, blendProbeStride) render into the --blend-chunk-size / --blend-probe-stride flags and default to 256 / 1. Note blendChunkSize must equal the blend servers' chunk size.

Metrics caveat: the coordinator process exposes only /healthz, not a Prometheus /metrics endpoint. ServiceMonitor support is wired for parity but defaults disabled; enabling it is only useful once a metrics endpoint is added to the coordinator app.

Connecting engines to a coordinator

LMCacheEngineSpec and CacheBlendEngineSpec gained a coordinator block (CoordinatorConnectionSpec) that maps to the server's coordinator-client flags (lmcache/v1/multiprocess/config.py: add_coordinator_args):

ref — name of an LMCacheCoordinator in the same namespace. The controller resolves it to the coordinator's Service URL (http://<name>.<ns>.svc:<port>) before building the DaemonSet, so BuildContainerArgs stays a pure function.
url — explicit escape hatch for a coordinator the operator does not manage. Exactly one of ref/url is required (enforced in validation).
advertiseIP — do not set this in almost every case. It defaults to the pod IP via the downward API env LMCACHE_COORDINATOR_ADVERTISE_IP, which is correct for normal in-cluster deployments. Only override it if you know exactly what you are doing (e.g. the coordinator runs outside the cluster and must reach the server through a specific externally-routable address); an incorrect value silently breaks coordinator-to-server connectivity.
heartbeatInterval, l2EventReporting, l2EventFlushInterval.

When coordinator is unset, the server emits no --coordinator-url and does not register (unchanged behavior).

Resources created (for an `LMCacheCoordinator` named `coord`)

Resource	Name	Purpose
Deployment	`coord`	`lmcache coordinator` HTTP server (port 9300)
Service (ClusterIP)	`coord`	fleet-wide discovery endpoint for engines
Service (headless)	`coord-metrics`	Prometheus scrape target (only if `prometheus.serviceMonitor.enabled`)
ServiceMonitor	`coord`	Prometheus scrape config (gated, disabled by default)

Future Extensibility

L2 backends: The RESP (Redis/Valkey) adapter is natively supported with typed CRD fields and Secret-based auth injection. Other adapter types (nixl_store, fs, mock, mooncake_store, raw_block) can be configured via the raw escape hatch. Currently only a single L2 adapter is supported at a time. LMCache MP mode is designed to support multiple adapters in cascade, but this is not yet fully tested — once validated, the operator will support multiple adapters.
Blend mode: Implemented as the separate CacheBlendEngine CRD + injection webhook — see CacheBlend above. (This supersedes the earlier idea of a blend.enabled field on LMCacheEngine.)
Update strategy: Future spec.updateStrategy field for RollingUpdate/OnDelete control on the DaemonSet.
Coordinator: Implemented as the separate LMCacheCoordinator CRD (Deployment + Service) with engine-side coordinator connection wiring — see Coordinator above.
Additional CRDs: LMCacheKeyManager (global key management), LMCacheMonitor (engine state monitoring), LMCacheFederation (cross-cluster P2P topology).

Key Source Files Referenced

File	What it defines
`lmcache/v1/distributed/config.py`	`L1MemoryManagerConfig`, `L1ManagerConfig`, `EvictionConfig`, `StorageManagerConfig`, argparse
`lmcache/v1/mp_observability/config.py`	`PrometheusConfig`, `add_prometheus_args`, `parse_args_to_prometheus_config`
`lmcache/v1/multiprocess/server.py`	`MPCacheServer`, server CLI entry point, argparse (lines 629–653)
`lmcache/v1/multiprocess/http_server.py`	HTTP server with `/healthcheck` endpoint (FastAPI + ZMQ)
`lmcache/v1/mp_coordinator/config.py`	`MPCoordinatorConfig` (coordinator knobs mapped by `LMCacheCoordinator`)
`lmcache/cli/commands/coordinator.py`	`lmcache coordinator` CLI (flags rendered into the coordinator Deployment)
`lmcache/v1/multiprocess/config.py`	`add_coordinator_args` (engine `coordinator` connection flags)
`lmcache/v1/distributed/l2_adapters/config.py`	L2 adapter registry pattern, `L2AdapterConfigBase`, `L2AdaptersConfig`
`examples/multi_process/lmcache-daemonset.yaml`	Reference DaemonSet manifest
`examples/multi_process/vllm-deployment.yaml`	Reference vLLM deployment with kv-transfer-config

31 KiB Raw Permalink Blame History Unescape Escape