wehub-resource-sync
266 lines
9.5 KiB
ReStructuredText
266 lines
9.5 KiB
ReStructuredText
.. _installation_guide:
|
||
|
||
Installation
|
||
============
|
||
|
||
**Prerequisites:** Linux · Python 3.9–3.13 · NVIDIA GPU (compute 7.0+) · CUDA 12.1+ · `uv <https://astral.sh/uv>`_
|
||
|
||
Install LMCache
|
||
---------------
|
||
|
||
.. tab-set::
|
||
|
||
.. tab-item:: Python (pip / uv)
|
||
|
||
.. tab-set::
|
||
|
||
.. tab-item:: Stable
|
||
|
||
.. tab-set::
|
||
|
||
.. tab-item:: CUDA 13.0
|
||
|
||
.. code-block:: bash
|
||
|
||
uv venv --python 3.12
|
||
source .venv/bin/activate
|
||
uv pip install lmcache
|
||
|
||
.. important::
|
||
|
||
You're all set! You can now start using LMCache. For hands-on guides and more
|
||
usage examples, see the :ref:`quickstart_examples` section.
|
||
|
||
.. note::
|
||
|
||
NIXL support (e.g. for disaggregated prefill and P2P KV
|
||
sharing) is an optional extra:
|
||
|
||
.. code-block:: bash
|
||
|
||
uv pip install lmcache[nixl]
|
||
|
||
.. tab-item:: CUDA 12.9
|
||
|
||
The CUDA 12.9 wheel is published to a dedicated
|
||
`GitHub Release <https://github.com/LMCache/LMCache/releases>`__ rather than PyPI.
|
||
|
||
.. code-block:: bash
|
||
|
||
uv venv --python 3.12
|
||
source .venv/bin/activate
|
||
VERSION=0.4.3 # replace with target release
|
||
uv pip install lmcache==${VERSION} \
|
||
--extra-index-url https://download.pytorch.org/whl/cu129 \
|
||
--find-links https://github.com/LMCache/LMCache/releases/expanded_assets/v${VERSION}-cu129 \
|
||
--index-strategy unsafe-best-match
|
||
|
||
.. note::
|
||
|
||
``--extra-index-url https://download.pytorch.org/whl/cu129`` ensures the CUDA 12.9
|
||
build of PyTorch is resolved. Without it, pip may select a mismatched CUDA variant.
|
||
|
||
.. tab-item:: Nightly
|
||
|
||
Nightly wheels are built from the latest ``dev`` branch each day at 07:30 UTC
|
||
and published to GitHub Releases. No version pinning required — ``--pre``
|
||
picks the latest nightly automatically.
|
||
|
||
.. tab-set::
|
||
|
||
.. tab-item:: CUDA 13.0
|
||
|
||
.. code-block:: bash
|
||
|
||
uv venv --python 3.12
|
||
source .venv/bin/activate
|
||
uv pip install lmcache --pre \
|
||
--extra-index-url https://download.pytorch.org/whl/cu130 \
|
||
--find-links https://github.com/LMCache/LMCache/releases/expanded_assets/nightly \
|
||
--index-strategy unsafe-best-match
|
||
|
||
.. tab-item:: CUDA 12.9
|
||
|
||
.. code-block:: bash
|
||
|
||
uv venv --python 3.12
|
||
source .venv/bin/activate
|
||
uv pip install lmcache --pre \
|
||
--extra-index-url https://download.pytorch.org/whl/cu129 \
|
||
--find-links https://github.com/LMCache/LMCache/releases/expanded_assets/nightly-cu129 \
|
||
--index-strategy unsafe-best-match
|
||
|
||
.. tab-item:: From Source
|
||
|
||
``--no-build-isolation`` ensures the kernels are compiled against the same torch
|
||
already installed in your environment, preventing undefined symbol errors at runtime.
|
||
|
||
.. tab-set::
|
||
|
||
.. tab-item:: CUDA 13.0
|
||
|
||
.. code-block:: bash
|
||
|
||
git clone https://github.com/LMCache/LMCache.git
|
||
cd LMCache
|
||
|
||
uv venv --python 3.12
|
||
source .venv/bin/activate
|
||
|
||
uv pip install -r requirements/build.txt
|
||
uv pip install vllm # pulls in required torch version (cu13)
|
||
uv pip install -e . --no-build-isolation
|
||
|
||
.. tab-item:: CUDA 12.9
|
||
|
||
.. code-block:: bash
|
||
|
||
git clone https://github.com/LMCache/LMCache.git
|
||
cd LMCache
|
||
|
||
uv venv --python 3.12
|
||
source .venv/bin/activate
|
||
|
||
uv pip install -r requirements/build.txt
|
||
# Pin vLLM (and torch) to the cu12.9 wheel index so the local
|
||
# CUDA 12 toolchain matches what the extensions are built against.
|
||
uv pip install vllm \
|
||
--extra-index-url https://download.pytorch.org/whl/cu129 \
|
||
--index-strategy unsafe-best-match
|
||
# LMCACHE_CUDA_MAJOR=12 makes setup.py pick cupy-cuda12x
|
||
# for install_requires instead of the cu13 default.
|
||
LMCACHE_CUDA_MAJOR=12 \
|
||
uv pip install -e . --no-build-isolation
|
||
|
||
.. tab-item:: ROCm
|
||
|
||
.. code-block:: bash
|
||
|
||
git clone https://github.com/LMCache/LMCache.git
|
||
cd LMCache
|
||
|
||
uv venv --python 3.12
|
||
source .venv/bin/activate
|
||
|
||
# Need to install these packages manually to avoid build isolation
|
||
uv pip install -r requirements/build.txt
|
||
|
||
# Install torch from the ROCm wheel index
|
||
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/rocm7.0
|
||
|
||
# Build LMCache. BUILD_WITH_HIP=1 makes setup.py pick cupy-rocm-7-0 automatically.
|
||
# PYTORCH_ROCM_ARCH selects the target GPU(s):
|
||
# gfx942 -> MI300X / MI325X
|
||
# gfx950 -> MI350X / MI355X
|
||
# Comma-separate to build a fat binary for multiple archs.
|
||
PYTORCH_ROCM_ARCH="gfx942,gfx950" \
|
||
TORCH_DONT_CHECK_COMPILER_ABI=1 \
|
||
CXX=hipcc \
|
||
BUILD_WITH_HIP=1 \
|
||
uv pip install -e . --no-build-isolation
|
||
|
||
.. tab-item:: Intel XPU
|
||
|
||
.. code-block:: bash
|
||
|
||
git clone https://github.com/LMCache/LMCache.git
|
||
cd LMCache
|
||
|
||
uv venv --python 3.12
|
||
source .venv/bin/activate
|
||
|
||
# Need to install these packages manually to avoid build isolation
|
||
uv pip install -r requirements/build.txt
|
||
|
||
# Build LMCache with SYCL backend.
|
||
BUILD_WITH_SYCL=1 uv pip install --no-build-isolation -e .
|
||
|
||
|
||
.. tab-item:: Docker
|
||
|
||
.. tab-set::
|
||
|
||
.. tab-item:: Stable
|
||
|
||
.. tab-set::
|
||
|
||
.. tab-item:: CUDA 13.0
|
||
|
||
.. code-block:: bash
|
||
|
||
docker pull lmcache/vllm-openai
|
||
|
||
.. tab-item:: CUDA 12.9
|
||
|
||
.. code-block:: bash
|
||
|
||
docker pull lmcache/vllm-openai:latest-cu129
|
||
|
||
.. tab-item:: Nightly
|
||
|
||
.. tab-set::
|
||
|
||
.. tab-item:: CUDA 13.0
|
||
|
||
.. code-block:: bash
|
||
|
||
docker pull lmcache/vllm-openai:latest-nightly
|
||
|
||
.. tab-item:: CUDA 12.9
|
||
|
||
.. code-block:: bash
|
||
|
||
docker pull lmcache/vllm-openai:latest-nightly-cu129
|
||
|
||
.. tab-item:: ROCm
|
||
|
||
.. code-block:: bash
|
||
|
||
docker pull rocm/vllm-dev:nightly_0624_rc2_0624_rc2_20250620
|
||
|
||
.. tab-item:: Intel XPU
|
||
|
||
.. code-block:: bash
|
||
|
||
docker pull intel/vllm:0.17.0-xpu
|
||
|
||
See :ref:`docker_deployment` for running the container and ROCm images.
|
||
|
||
.. tab-item:: CLI Only
|
||
|
||
Lightweight CLI-only package for querying or benchmarking a remote LMCache server.
|
||
No CUDA required, works on any OS.
|
||
|
||
.. code-block:: bash
|
||
|
||
pip install lmcache-cli
|
||
|
||
.. note::
|
||
|
||
``lmcache-cli`` and ``lmcache`` ship the same ``lmcache`` CLI command.
|
||
Do not install both in the same environment.
|
||
|
||
Build the Docker Image
|
||
----------------------
|
||
|
||
Instead of pulling a prebuilt image, you can build the LMCache (integrated with
|
||
vLLM) image yourself from the provided Dockerfile, located in
|
||
`docker/ <https://github.com/LMCache/LMCache/tree/dev/docker>`_.
|
||
|
||
From the root of the LMCache repository:
|
||
|
||
.. code-block:: bash
|
||
|
||
docker build --tag <IMAGE_NAME>:<TAG> --target image-build --file docker/Dockerfile .
|
||
|
||
Replace ``<IMAGE_NAME>`` and ``<TAG>`` with your desired image name and tag. See
|
||
the example build file in `docker/ <https://github.com/LMCache/LMCache/tree/dev/docker>`_
|
||
for an explanation of all build arguments.
|
||
|
||
Verify Installation
|
||
-------------------
|
||
|
||
.. code-block:: bash
|
||
|
||
python -c "import lmcache.c_ops"
|