wehub-resource-sync
38 lines
1.4 KiB
ReStructuredText
38 lines
1.4 KiB
ReStructuredText
Kubernetes Deployment
|
|
=====================
|
|
|
|
For Kubernetes deployment of vLLM with LMCache integration, we recommend using the `vLLM Production Stack <https://github.com/vllm-project/production-stack>`_ project. This is a specialized production-ready implementation for K8S-native cluster-wide deployment for vllm & lmcache.
|
|
|
|
For a quick start guide, please refer to the official `documentation <https://docs.vllm.ai/projects/production-stack/en/latest/getting_started/quickstart.html>`_
|
|
|
|
and replace the Helm values file with (`values-05-cpu-offloading.yaml <https://github.com/vllm-project/production-stack/blob/main/tutorials/assets/values-05-cpu-offloading.yaml>`_):
|
|
|
|
.. code-block:: yaml
|
|
|
|
servingEngineSpec:
|
|
runtimeClassName: ""
|
|
modelSpec:
|
|
- name: "mistral"
|
|
repository: "lmcache/vllm-openai"
|
|
tag: "latest"
|
|
modelURL: "mistralai/Mistral-7B-Instruct-v0.2"
|
|
replicaCount: 1
|
|
requestCPU: 10
|
|
requestMemory: "40Gi"
|
|
requestGPU: 1
|
|
pvcStorage: "50Gi"
|
|
pvcAccessMode:
|
|
- ReadWriteOnce
|
|
vllmConfig:
|
|
maxModelLen: 32000
|
|
|
|
lmcacheConfig:
|
|
enabled: true
|
|
cpuOffloadingBufferSize: "20"
|
|
|
|
hf_token: <hf-token>
|
|
|
|
OR
|
|
|
|
refer to a detailed `step-by-step tutorial <https://github.com/vllm-project/production-stack/blob/main/tutorials/05-offload-kv-cache.md>`_ on how to offload KV cache with LMCache in the production stack.
|