wehub-resource-sync
77 lines
2.6 KiB
ReStructuredText
77 lines
2.6 KiB
ReStructuredText
.. _docker_deployment:
|
|
|
|
Docker deployment
|
|
=================
|
|
|
|
.. warning::
|
|
|
|
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/deployment`.
|
|
|
|
|
|
**Prerequisites:** Docker Engine 27.0+
|
|
|
|
See :ref:`installation_guide` for pulling images.
|
|
|
|
Running the container
|
|
---------------------
|
|
|
|
.. code-block:: bash
|
|
|
|
IMAGE=<IMAGE_NAME>:<TAG>
|
|
docker run --runtime nvidia --gpus all \
|
|
--env "HF_TOKEN=<REPLACE_WITH_YOUR_HF_TOKEN>" \
|
|
--env "LMCACHE_CHUNK_SIZE=256" \
|
|
--env "LMCACHE_LOCAL_CPU=True" \
|
|
--env "LMCACHE_MAX_LOCAL_CPU_SIZE=5" \
|
|
--volume ~/.cache/huggingface:/root/.cache/huggingface \
|
|
--network host \
|
|
$IMAGE \
|
|
meta-llama/Llama-3.1-8B-Instruct --kv-transfer-config \
|
|
'{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}'
|
|
|
|
See the `docker run example <https://github.com/LMCache/LMCache/tree/dev/docker>`_ for more details.
|
|
|
|
ROCm (AMD)
|
|
----------
|
|
|
|
The `AMD Infinity hub <https://hub.docker.com/r/rocm/vllm-dev>`__ for vLLM offers a prebuilt,
|
|
optimized image for the AMD Instinct™ MI300X. See
|
|
`LLM inference performance validation on AMD Instinct MI300X <https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference/benchmark-docker/vllm.html?model=pyt_vllm_llama-3.1-8b>`__
|
|
for full instructions.
|
|
|
|
Validated environment: ``rocm/vllm-dev:nightly_0624_rc2_0624_rc2_20250620``, MI300X, vLLM V1.
|
|
|
|
.. code-block:: bash
|
|
|
|
docker run -it \
|
|
--network=host \
|
|
--group-add=video \
|
|
--ipc=host \
|
|
--cap-add=SYS_PTRACE \
|
|
--security-opt seccomp=unconfined \
|
|
--device /dev/kfd \
|
|
--device /dev/dri \
|
|
-v <path_to_your_models>:/app/model \
|
|
-e HF_HOME="/app/model" \
|
|
--name lmcache_rocm \
|
|
rocm/vllm-dev:nightly_0624_rc2_0624_rc2_20250620 \
|
|
bash
|
|
|
|
XPU (Intel)
|
|
-----------
|
|
|
|
The `Intel vLLM XPU hub <https://hub.docker.com/r/intel/vllm>`__ offers a prebuilt, optimized docker image designed for validating inference performance on the Intel GPU accelerators like ARC770, B60/B70, and future products.
|
|
|
|
Validated environment: ``intel/vllm:0.17.0-xpu``, Intel B60 GPU, vLLM V1.
|
|
|
|
.. code-block:: bash
|
|
|
|
docker run --privileged \
|
|
-it --rm --name vllm-xpu \
|
|
-u root \
|
|
--ipc=host --net=host \
|
|
--cap-add=ALL \
|
|
--device /dev/dri:/dev/dri \
|
|
-v /dev/dri/by-path:/dev/dri/by-path \
|
|
--entrypoint /bin/bash intel/vllm:0.17.0-xpu
|