chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,157 @@
|
||||
Benchmarking
|
||||
============
|
||||
|
||||
This is a simple tutorial on how to deploy and benchmark LMCache using the
|
||||
``lmcache bench engine`` CLI.
|
||||
|
||||
The ``lmcache bench engine`` command is a flexible traffic simulator that
|
||||
sends configurable workloads to your inference engine and reports TTFT,
|
||||
decoding speed, and throughput metrics. This tutorial walks through a
|
||||
long-document Q&A benchmark that exercises LMCache's CPU offloading path.
|
||||
|
||||
For the full CLI reference -- including every flag, every workload type, and
|
||||
config-file usage -- see :doc:`/cli/bench`.
|
||||
|
||||
Long Doc QA workload
|
||||
--------------------
|
||||
|
||||
The ``long-doc-qa`` workload simulates repeated Q&A over long synthetic
|
||||
documents: a warmup round primes the KV cache with each document, then a
|
||||
benchmark round dispatches the questions. The number of documents is
|
||||
derived from ``--kv-cache-volume`` and the model's tokens-per-GB rather
|
||||
than set directly. See :doc:`/cli/bench` for the full flag list.
|
||||
|
||||
Example
|
||||
-------
|
||||
|
||||
To measure the benefit of LMCache, run the **same benchmark against two
|
||||
setups** and compare the results:
|
||||
|
||||
- **Setup A (baseline)** -- vLLM alone.
|
||||
- **Setup B (with LMCache)** -- vLLM plus a standalone LMCache server.
|
||||
|
||||
The steps below reproduce both runs on ``Qwen/Qwen3-8B``. Adjust the sizes
|
||||
to match your hardware.
|
||||
|
||||
Setup A: vLLM alone (baseline)
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
vllm serve Qwen/Qwen3-8B
|
||||
|
||||
Setup B: vLLM with LMCache
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Run LMCache as a standalone service and point vLLM at it with the
|
||||
``LMCacheMPConnector``. See :doc:`/getting_started/quickstart` for the full MP-mode
|
||||
walkthrough.
|
||||
|
||||
**Start the LMCache server:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache server \
|
||||
--l1-size-gb 66 --eviction-policy LRU
|
||||
|
||||
The ZMQ port defaults to **5555** (used by vLLM) and the HTTP frontend
|
||||
defaults to **8080** (used by ``lmcache bench engine --lmcache-url``).
|
||||
|
||||
**Start vLLM with the MP connector in a separate terminal:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
vllm serve Qwen/Qwen3-8B \
|
||||
--kv-transfer-config \
|
||||
'{"kv_connector": "LMCacheMPConnector", "kv_role": "kv_both"}'
|
||||
|
||||
Run the benchmark
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
To make the comparison fair, capture the benchmark settings **once** in a
|
||||
config file, then replay the same config against both setups.
|
||||
|
||||
**Step 1 -- export a shared config.** With the LMCache server from Setup B
|
||||
still running, launch ``lmcache bench engine`` in interactive mode:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache bench engine --lmcache-url http://localhost:8080
|
||||
|
||||
Interactive mode triggers because ``--engine-url`` and ``--workload`` are
|
||||
missing, and ``--lmcache-url`` auto-detects ``tokens-per-gb-kvcache`` from
|
||||
the server. Walk through the prompts and pick:
|
||||
|
||||
- Engine URL: ``http://localhost:8000``
|
||||
- Workload: ``long-doc-qa``
|
||||
- Model: auto-detected from the engine (or type ``Qwen/Qwen3-8B``)
|
||||
- KV cache volume (GB): ``10``
|
||||
- ``ldqa-query-per-document``: ``1``
|
||||
- ``ldqa-shuffle-policy``: ``tile``
|
||||
- ``ldqa-num-inflight-requests``: ``4``
|
||||
- Leave the rest at their defaults.
|
||||
|
||||
At the **summary** step, choose **"Export configuration for later use and
|
||||
exit"** and save to ``bench_config.json``. The file looks like:
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"model": "Qwen/Qwen3-8B",
|
||||
"workload": "long-doc-qa",
|
||||
"kv_cache_volume": 10.0,
|
||||
"tokens_per_gb_kvcache": 46020,
|
||||
"ldqa_document_length": 10000,
|
||||
"ldqa_query_per_document": 1,
|
||||
"ldqa_shuffle_policy": "tile",
|
||||
"ldqa_num_inflight_requests": 4
|
||||
}
|
||||
|
||||
Note that the exported config stores ``tokens_per_gb_kvcache`` (resolved
|
||||
from ``--lmcache-url``) but **not** the engine URL or the LMCache URL, so
|
||||
the same file is portable across environments.
|
||||
|
||||
**Step 2 -- replay against each setup.** Point ``--engine-url`` at whichever
|
||||
vLLM you want to benchmark and pass the shared config:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache bench engine \
|
||||
--engine-url http://localhost:8000 \
|
||||
--config bench_config.json
|
||||
|
||||
Run this once against Setup A's vLLM and once against Setup B's
|
||||
vLLM-plus-LMCache, recording the metrics from each run.
|
||||
|
||||
Results
|
||||
~~~~~~~
|
||||
|
||||
Pull the headline numbers out of each run's
|
||||
``Engine Benchmark Result (long-doc-qa)`` summary:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 40 30 30
|
||||
|
||||
* - Metric
|
||||
- Setup A (vLLM)
|
||||
- Setup B (+ LMCache)
|
||||
* - Successful requests
|
||||
- 46
|
||||
- 46
|
||||
* - Benchmark duration (s)
|
||||
- 23.47
|
||||
- 13.79
|
||||
* - Mean TTFT (ms)
|
||||
- 757.00
|
||||
- 185.00
|
||||
|
||||
That's a **75%** reduction in Mean TTFT (757 ms → 185 ms) and a **41%**
|
||||
reduction in benchmark duration (23.47 s → 13.79 s) from LMCache offloading.
|
||||
|
||||
.. note::
|
||||
Without LMCache, once the benchmark's working set overflows the GPU KV
|
||||
cache the second round has to recompute every prefix, so TTFT and
|
||||
throughput don't improve even when content repeats. LMCache keeps the
|
||||
evicted blocks on CPU RAM and restores them on demand -- that's where
|
||||
the speedup comes from.
|
||||
@@ -0,0 +1,16 @@
|
||||
Getting Started
|
||||
===============
|
||||
|
||||
New to LMCache? Start here -- an overview of multiprocess mode, installation, a
|
||||
quickstart, the configuration reference, and tools for benchmarking and sizing
|
||||
the cache.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
/mp/index
|
||||
installation
|
||||
quickstart
|
||||
/mp/configuration
|
||||
benchmarking
|
||||
kv_cache_calculator
|
||||
@@ -0,0 +1,265 @@
|
||||
.. _installation_guide:
|
||||
|
||||
Installation
|
||||
============
|
||||
|
||||
**Prerequisites:** Linux · Python 3.9–3.13 · NVIDIA GPU (compute 7.0+) · CUDA 12.1+ · `uv <https://astral.sh/uv>`_
|
||||
|
||||
Install LMCache
|
||||
---------------
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Python (pip / uv)
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Stable
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: CUDA 13.0
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
uv pip install lmcache
|
||||
|
||||
.. important::
|
||||
|
||||
You're all set! You can now start using LMCache. For hands-on guides and more
|
||||
usage examples, see the :ref:`quickstart_examples` section.
|
||||
|
||||
.. note::
|
||||
|
||||
NIXL support (e.g. for disaggregated prefill and P2P KV
|
||||
sharing) is an optional extra:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
uv pip install lmcache[nixl]
|
||||
|
||||
.. tab-item:: CUDA 12.9
|
||||
|
||||
The CUDA 12.9 wheel is published to a dedicated
|
||||
`GitHub Release <https://github.com/LMCache/LMCache/releases>`__ rather than PyPI.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
VERSION=0.4.3 # replace with target release
|
||||
uv pip install lmcache==${VERSION} \
|
||||
--extra-index-url https://download.pytorch.org/whl/cu129 \
|
||||
--find-links https://github.com/LMCache/LMCache/releases/expanded_assets/v${VERSION}-cu129 \
|
||||
--index-strategy unsafe-best-match
|
||||
|
||||
.. note::
|
||||
|
||||
``--extra-index-url https://download.pytorch.org/whl/cu129`` ensures the CUDA 12.9
|
||||
build of PyTorch is resolved. Without it, pip may select a mismatched CUDA variant.
|
||||
|
||||
.. tab-item:: Nightly
|
||||
|
||||
Nightly wheels are built from the latest ``dev`` branch each day at 07:30 UTC
|
||||
and published to GitHub Releases. No version pinning required — ``--pre``
|
||||
picks the latest nightly automatically.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: CUDA 13.0
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
uv pip install lmcache --pre \
|
||||
--extra-index-url https://download.pytorch.org/whl/cu130 \
|
||||
--find-links https://github.com/LMCache/LMCache/releases/expanded_assets/nightly \
|
||||
--index-strategy unsafe-best-match
|
||||
|
||||
.. tab-item:: CUDA 12.9
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
uv pip install lmcache --pre \
|
||||
--extra-index-url https://download.pytorch.org/whl/cu129 \
|
||||
--find-links https://github.com/LMCache/LMCache/releases/expanded_assets/nightly-cu129 \
|
||||
--index-strategy unsafe-best-match
|
||||
|
||||
.. tab-item:: From Source
|
||||
|
||||
``--no-build-isolation`` ensures the kernels are compiled against the same torch
|
||||
already installed in your environment, preventing undefined symbol errors at runtime.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: CUDA 13.0
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
git clone https://github.com/LMCache/LMCache.git
|
||||
cd LMCache
|
||||
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
|
||||
uv pip install -r requirements/build.txt
|
||||
uv pip install vllm # pulls in required torch version (cu13)
|
||||
uv pip install -e . --no-build-isolation
|
||||
|
||||
.. tab-item:: CUDA 12.9
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
git clone https://github.com/LMCache/LMCache.git
|
||||
cd LMCache
|
||||
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
|
||||
uv pip install -r requirements/build.txt
|
||||
# Pin vLLM (and torch) to the cu12.9 wheel index so the local
|
||||
# CUDA 12 toolchain matches what the extensions are built against.
|
||||
uv pip install vllm \
|
||||
--extra-index-url https://download.pytorch.org/whl/cu129 \
|
||||
--index-strategy unsafe-best-match
|
||||
# LMCACHE_CUDA_MAJOR=12 makes setup.py pick cupy-cuda12x
|
||||
# for install_requires instead of the cu13 default.
|
||||
LMCACHE_CUDA_MAJOR=12 \
|
||||
uv pip install -e . --no-build-isolation
|
||||
|
||||
.. tab-item:: ROCm
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
git clone https://github.com/LMCache/LMCache.git
|
||||
cd LMCache
|
||||
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
|
||||
# Need to install these packages manually to avoid build isolation
|
||||
uv pip install -r requirements/build.txt
|
||||
|
||||
# Install torch from the ROCm wheel index
|
||||
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/rocm7.0
|
||||
|
||||
# Build LMCache. BUILD_WITH_HIP=1 makes setup.py pick cupy-rocm-7-0 automatically.
|
||||
# PYTORCH_ROCM_ARCH selects the target GPU(s):
|
||||
# gfx942 -> MI300X / MI325X
|
||||
# gfx950 -> MI350X / MI355X
|
||||
# Comma-separate to build a fat binary for multiple archs.
|
||||
PYTORCH_ROCM_ARCH="gfx942,gfx950" \
|
||||
TORCH_DONT_CHECK_COMPILER_ABI=1 \
|
||||
CXX=hipcc \
|
||||
BUILD_WITH_HIP=1 \
|
||||
uv pip install -e . --no-build-isolation
|
||||
|
||||
.. tab-item:: Intel XPU
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
git clone https://github.com/LMCache/LMCache.git
|
||||
cd LMCache
|
||||
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
|
||||
# Need to install these packages manually to avoid build isolation
|
||||
uv pip install -r requirements/build.txt
|
||||
|
||||
# Build LMCache with SYCL backend.
|
||||
BUILD_WITH_SYCL=1 uv pip install --no-build-isolation -e .
|
||||
|
||||
|
||||
.. tab-item:: Docker
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Stable
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: CUDA 13.0
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
docker pull lmcache/vllm-openai
|
||||
|
||||
.. tab-item:: CUDA 12.9
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
docker pull lmcache/vllm-openai:latest-cu129
|
||||
|
||||
.. tab-item:: Nightly
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: CUDA 13.0
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
docker pull lmcache/vllm-openai:latest-nightly
|
||||
|
||||
.. tab-item:: CUDA 12.9
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
docker pull lmcache/vllm-openai:latest-nightly-cu129
|
||||
|
||||
.. tab-item:: ROCm
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
docker pull rocm/vllm-dev:nightly_0624_rc2_0624_rc2_20250620
|
||||
|
||||
.. tab-item:: Intel XPU
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
docker pull intel/vllm:0.17.0-xpu
|
||||
|
||||
See :ref:`docker_deployment` for running the container and ROCm images.
|
||||
|
||||
.. tab-item:: CLI Only
|
||||
|
||||
Lightweight CLI-only package for querying or benchmarking a remote LMCache server.
|
||||
No CUDA required, works on any OS.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pip install lmcache-cli
|
||||
|
||||
.. note::
|
||||
|
||||
``lmcache-cli`` and ``lmcache`` ship the same ``lmcache`` CLI command.
|
||||
Do not install both in the same environment.
|
||||
|
||||
Build the Docker Image
|
||||
----------------------
|
||||
|
||||
Instead of pulling a prebuilt image, you can build the LMCache (integrated with
|
||||
vLLM) image yourself from the provided Dockerfile, located in
|
||||
`docker/ <https://github.com/LMCache/LMCache/tree/dev/docker>`_.
|
||||
|
||||
From the root of the LMCache repository:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
docker build --tag <IMAGE_NAME>:<TAG> --target image-build --file docker/Dockerfile .
|
||||
|
||||
Replace ``<IMAGE_NAME>`` and ``<TAG>`` with your desired image name and tag. See
|
||||
the example build file in `docker/ <https://github.com/LMCache/LMCache/tree/dev/docker>`_
|
||||
for an explanation of all build arguments.
|
||||
|
||||
Verify Installation
|
||||
-------------------
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python -c "import lmcache.c_ops"
|
||||
@@ -0,0 +1,18 @@
|
||||
KV Cache Size Calculator
|
||||
========================
|
||||
|
||||
Use this interactive calculator to:
|
||||
|
||||
- **Calculate KV Cache Size**: Estimate the memory required for caching a specific number of tokens
|
||||
- **Calculate Maximum Tokens**: Find out how many tokens you can cache given your available GPU RAM
|
||||
|
||||
This helps you plan memory requirements for your LMCache deployment.
|
||||
|
||||
.. raw:: html
|
||||
|
||||
<div style="margin: 20px 0;">
|
||||
<iframe src="../_static/kv_cache_calculator.html"
|
||||
style="width: 100%; height: 1400px; border: none;"
|
||||
title="KV Cache Size Calculator">
|
||||
</iframe>
|
||||
</div>
|
||||
@@ -0,0 +1,480 @@
|
||||
.. _quickstart:
|
||||
|
||||
Quickstart
|
||||
==========
|
||||
|
||||
This guide helps you get LMCache running end-to-end in a couple of minutes. Use the tabs below to switch the engine. Steps are the same; only the libraries and launch commands change.
|
||||
|
||||
.. tab-set::
|
||||
:sync-group: engine
|
||||
|
||||
.. tab-item:: vLLM
|
||||
|
||||
**Install LMCache**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
uv pip install lmcache vllm
|
||||
|
||||
LMCache supports two deployment modes with vLLM:
|
||||
|
||||
- **Multiprocess (MP) mode** -- **recommended.** LMCache runs as a
|
||||
standalone service and vLLM attaches via ``LMCacheMPConnector``.
|
||||
Scales better, exposes management/observability endpoints, and
|
||||
supports sharing one cache across multiple engine instances.
|
||||
- **In-process mode** -- LMCache runs inside the vLLM process via
|
||||
``LMCacheConnectorV1``. Single command, convenient for quick
|
||||
single-node experiments.
|
||||
|
||||
.. tab-set::
|
||||
:sync-group: vllm-mode
|
||||
|
||||
.. tab-item:: MP mode (recommended)
|
||||
:sync: mp
|
||||
|
||||
Start the LMCache server. ``--host`` / ``--port`` set the ZMQ
|
||||
address vLLM connects to; they are spelled out here so the two
|
||||
commands line up (these are also the defaults):
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# chunk-size 16 is an illustrative demo value so a short
|
||||
# prompt produces visible cache traffic; use the default
|
||||
# (256) in production.
|
||||
lmcache server \
|
||||
--host localhost --port 5555 \
|
||||
--l1-size-gb 20 --eviction-policy LRU --chunk-size 16
|
||||
|
||||
The ZMQ port (``--port``, default **5555**) accepts connections
|
||||
from vLLM; the HTTP frontend (default **8080**) serves the
|
||||
management and metrics endpoints. See :doc:`../mp/configuration`
|
||||
for the full list of ``lmcache server`` and connector options.
|
||||
|
||||
Start vLLM with the MP connector in a separate terminal. Point the
|
||||
connector at the server above via ``lmcache.mp.host`` /
|
||||
``lmcache.mp.port`` in ``kv_connector_extra_config`` -- the host
|
||||
**must** carry a ZMQ transport prefix such as ``tcp://``:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
vllm serve Qwen/Qwen3-8B \
|
||||
--port 8000 --kv-transfer-config \
|
||||
'{"kv_connector":"LMCacheMPConnector", "kv_role":"kv_both", "kv_connector_extra_config": {"lmcache.mp.host": "tcp://localhost", "lmcache.mp.port": 5555}}'
|
||||
|
||||
.. note::
|
||||
**Where does** ``LMCacheMPConnector`` **resolve to?** This depends on your vLLM version:
|
||||
|
||||
- **vLLM < 0.20.0** -- ``"kv_connector":"LMCacheMPConnector"`` always
|
||||
resolves to vLLM's built-in
|
||||
``vllm.distributed.kv_transfer.kv_connector.v1.LMCacheMPConnector``;
|
||||
there is no way to redirect it to the LMCache-shipped implementation.
|
||||
|
||||
- **vLLM >= 0.20.0** -- ``"kv_connector":"LMCacheMPConnector"`` still
|
||||
defaults to vLLM's built-in connector, but you can opt in to the
|
||||
LMCache-shipped implementation
|
||||
(:mod:`lmcache.integration.vllm.lmcache_mp_connector`) by adding
|
||||
``kv_connector_module_path``:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
vllm serve Qwen/Qwen3-8B \
|
||||
--port 8000 --kv-transfer-config \
|
||||
'{"kv_connector":"LMCacheMPConnector", "kv_connector_module_path":"lmcache.integration.vllm.lmcache_mp_connector", "kv_role":"kv_both"}'
|
||||
|
||||
The LMCache-shipped connector tracks the latest LMCache server
|
||||
protocol and ships fixes/features ahead of the version vendored
|
||||
into vLLM, so prefer it whenever you are on vLLM 0.20.0 or newer.
|
||||
|
||||
**Test** -- open a new terminal and send two requests whose
|
||||
prompts share a prefix:
|
||||
|
||||
**First request**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl http://localhost:8000/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen3-8B",
|
||||
"prompt": "Qwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts",
|
||||
"max_tokens": 100,
|
||||
"temperature": 0.7
|
||||
}'
|
||||
|
||||
**Second request**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl http://localhost:8000/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen3-8B",
|
||||
"prompt": "Qwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts (MoE) models",
|
||||
"max_tokens": 100,
|
||||
"temperature": 0.7
|
||||
}'
|
||||
|
||||
**You should see LMCache logs like this** -- in MP mode the
|
||||
store/retrieve logs come from the standalone ``lmcache server``
|
||||
process, one entry per chunk.
|
||||
|
||||
**First request** -- cache is empty, so every aligned chunk is
|
||||
offloaded:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
[2026-04-22 19:49:56,316] LMCache INFO: Stored 16 tokens in 0.023 seconds (server.py:390:lmcache.v1.multiprocess.server)
|
||||
[2026-04-22 19:49:56,555] LMCache INFO: Stored 16 tokens in 0.005 seconds (server.py:390:lmcache.v1.multiprocess.server)
|
||||
[2026-04-22 19:49:56,691] LMCache INFO: Stored 16 tokens in 0.005 seconds (server.py:390:lmcache.v1.multiprocess.server)
|
||||
...
|
||||
|
||||
**Second request** -- the shared prefix is retrieved from CPU
|
||||
RAM; only the new tail is stored:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
[2026-04-22 19:50:04,686] LMCache INFO: Retrieved 16 tokens in 0.003 seconds (server.py:573:lmcache.v1.multiprocess.server)
|
||||
[2026-04-22 19:50:04,832] LMCache INFO: Stored 16 tokens in 0.005 seconds (server.py:390:lmcache.v1.multiprocess.server)
|
||||
[2026-04-22 19:50:04,968] LMCache INFO: Stored 16 tokens in 0.005 seconds (server.py:390:lmcache.v1.multiprocess.server)
|
||||
...
|
||||
|
||||
For request-level statistics (hit ratio, bytes transferred) see
|
||||
:doc:`../mp/observability/index`.
|
||||
|
||||
.. tab-item:: In-process mode
|
||||
:sync: inproc
|
||||
|
||||
Start vLLM with LMCache embedded in the engine process:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# The chunk size here is only for illustration purpose, use default one (256) later
|
||||
LMCACHE_CHUNK_SIZE=8 \
|
||||
vllm serve Qwen/Qwen3-8B \
|
||||
--port 8000 --kv-transfer-config \
|
||||
'{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
|
||||
|
||||
.. note::
|
||||
To customize further, create a config file. See
|
||||
:doc:`../api_reference/configurations` for all options.
|
||||
|
||||
**Alternative simpler command:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
vllm serve <MODEL NAME> \
|
||||
--kv-offloading-backend lmcache \
|
||||
--kv-offloading-size <SIZE IN GB> \
|
||||
--disable-hybrid-kv-cache-manager
|
||||
|
||||
The ``--disable-hybrid-kv-cache-manager`` flag is mandatory.
|
||||
All configuration options from the
|
||||
:doc:`../api_reference/configurations` page still apply.
|
||||
|
||||
**Test** -- open a new terminal and send two requests whose
|
||||
prompts share a prefix:
|
||||
|
||||
**First request**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl http://localhost:8000/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen3-8B",
|
||||
"prompt": "Qwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts",
|
||||
"max_tokens": 100,
|
||||
"temperature": 0.7
|
||||
}'
|
||||
|
||||
**Second request**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl http://localhost:8000/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen3-8B",
|
||||
"prompt": "Qwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts (MoE) models",
|
||||
"max_tokens": 100,
|
||||
"temperature": 0.7
|
||||
}'
|
||||
|
||||
**You should see LMCache logs like this** -- in-process mode
|
||||
emits the logs inline with the vLLM engine core.
|
||||
|
||||
**First request** -- prompt is offloaded to LMCache:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
(EngineCore_DP0 pid=458469) [2025-09-30 00:08:43,982] LMCache INFO: Stored 31 out of total 31 tokens. size: 0.0040 gb, cost 1.95 ms, throughput: 1.98 GB/s; offload_time: 1.88 ms, put_time: 0.07 ms
|
||||
|
||||
**Second request** -- hits the cache and stores the new tail:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
Reqid: cmpl-6709d8795d3c4464b01999c9f3fffede-0, Total tokens 32, LMCache hit tokens: 24, need to load: 8
|
||||
(EngineCore_DP0 pid=494270) [2025-09-30 01:12:36,502] LMCache INFO: Retrieved 8 out of 24 required tokens (from 32 total tokens). size: 0.0011 gb, cost 0.55 ms, throughput: 1.98 GB/s;
|
||||
(EngineCore_DP0 pid=494270) [2025-09-30 01:12:36,509] LMCache INFO: Storing KV cache for 8 out of 32 tokens (skip_leading_tokens=24)
|
||||
(EngineCore_DP0 pid=494270) [2025-09-30 01:12:36,510] LMCache INFO: Stored 8 out of total 8 tokens. size: 0.0011 gb, cost 0.43 ms, throughput: 2.57 GB/s; offload_time: 0.40 ms, put_time: 0.03 ms
|
||||
|
||||
- **Total tokens 32**: The new prompt has 32 tokens after tokenization.
|
||||
- **LMCache hit tokens: 24**: 24 tokens (full 8-token chunks) were found in the cache from the first request that stored 31 tokens.
|
||||
- **Need to load: 8**: vLLM auto prefix caching uses block size 16; 16 tokens already sit in GPU RAM, so LMCache only loads 24-16=8.
|
||||
- **Why 24 hit tokens instead of 31?** LMCache hashes every 8 tokens (8, 16, 24, 31). It matches page-aligned chunks, so it uses the 24-token hash.
|
||||
- **Stored another 8 tokens**: The new 8 tokens form a full chunk and are stored for future reuse.
|
||||
|
||||
.. tab-item:: SGLang
|
||||
|
||||
.. note::
|
||||
The SGLang integration now defaults to MP (multi-process) mode.
|
||||
Please refer to `examples/sgl_integration/README.md`_ for the current setup instructions.
|
||||
|
||||
.. _examples/sgl_integration/README.md: https://github.com/LMCache/LMCache/blob/dev/examples/sgl_integration/README.md
|
||||
|
||||
**Install SGLang**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
uv pip install --prerelease=allow lmcache "sglang"
|
||||
|
||||
**Start SGLang with LMCache**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
cat > lmc_config.yaml <<'EOF'
|
||||
chunk_size: 8 # demo only; use 256 for production
|
||||
local_cpu: true
|
||||
use_layerwise: true
|
||||
max_local_cpu_size: 10 # GB
|
||||
EOF
|
||||
|
||||
export LMCACHE_CONFIG_FILE=$PWD/lmc_config.yaml
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-8B \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--enable-lmcache
|
||||
|
||||
.. note::
|
||||
Configure LMCache via the config file. See :doc:`../api_reference/configurations` for the full list.
|
||||
|
||||
**Test** -- open a new terminal and send two requests whose prompts
|
||||
share a prefix:
|
||||
|
||||
**First request**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl http://localhost:30000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen3-8B",
|
||||
"messages": [{"role": "user", "content": "Qwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts"}],
|
||||
"max_tokens": 100,
|
||||
"temperature": 0.7
|
||||
}'
|
||||
|
||||
**Second request**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl http://localhost:30000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen3-8B",
|
||||
"messages": [{"role": "user", "content": "Qwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts (MoE) models"}],
|
||||
"max_tokens": 100,
|
||||
"temperature": 0.7
|
||||
}'
|
||||
|
||||
**You should see LMCache logs like this:**
|
||||
|
||||
**First request** -- prompt plus generated tokens are stored:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
Prefill batch, #new-seq: 1, #new-token: 35, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0,
|
||||
Decode batch, #running-req: 1, #token: 74, token usage: 0.00, cuda graph: True, gen throughput (token/s): 1.63, #queue-req: 0,
|
||||
Decode batch, #running-req: 1, #token: 114, token usage: 0.00, cuda graph: True, gen throughput (token/s): 87.95, #queue-req: 0,
|
||||
LMCache INFO: Stored 128 out of total 135 tokens. size: 0.0195 GB, cost 12.8890 ms, throughput: 1.5153 GB/s (cache_engine.py:623:lmcache.v1.cache_engine)
|
||||
|
||||
**Second request** -- Radix Cache and LMCache share the prefix; only the new portion is stored:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
Prefill batch, #new-seq: 1, #new-token: 10, #cached-token: 30, token usage: 0.00, #running-req: 0, #queue-req: 0,
|
||||
Decode batch, #running-req: 1, #token: 64, token usage: 0.00, cuda graph: True, gen throughput (token/s): 8.29, #queue-req: 0,
|
||||
Decode batch, #running-req: 1, #token: 104, token usage: 0.00, cuda graph: True, gen throughput (token/s): 87.95, #queue-req: 0,
|
||||
Decode batch, #running-req: 1, #token: 144, token usage: 0.00, cuda graph: True, gen throughput (token/s): 87.89, #queue-req: 0,
|
||||
LMCache INFO: Stored 112 out of total 140 tokens. size: 0.0171 GB, cost 11.1986 ms, throughput: 1.5261 GB/s (cache_engine.py:623:lmcache.v1.cache_engine)
|
||||
|
||||
- **Total tokens 140**: SGLang stores KV cache for both prefill and decode tokens together, so total = 40 prompt + 100 generated = 140 tokens.
|
||||
- **Cached tokens: 30**: SGLang's Radix Attention Cache reused 30 tokens from the first request.
|
||||
- **LMCache hit tokens: 24**: LMCache detected 24 tokens (3 full 8-token chunks) stored from the first request. Since Radix Cache already provides 30 tokens in GPU memory, these 24 tokens don't need to be loaded from LMCache or stored again.
|
||||
- **New tokens: 10**: Only 10 prompt tokens need prefill computation (40 prompt - 30 cached = 10).
|
||||
- **Stored 112 out of 140**: 24 tokens (3 full chunks) are already in LMCache and skipped. Of the remaining 116 tokens, 112 (14 full 8-token chunks) are stored.
|
||||
|
||||
.. tab-item:: TensorRT-LLM
|
||||
|
||||
.. note::
|
||||
This integration depends on the connector preset registry from
|
||||
`NVIDIA/TensorRT-LLM PR #12626
|
||||
<https://github.com/NVIDIA/TensorRT-LLM/pull/12626>`_ and the
|
||||
matching LMCache adapter, neither of which has shipped in a
|
||||
stable release yet. Until they do, install both from source:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
|
||||
# LMCache from source (dev branch)
|
||||
uv pip install git+https://github.com/LMCache/LMCache.git@dev
|
||||
|
||||
# TensorRT-LLM from source — see NVIDIA's build guide:
|
||||
# https://nvidia.github.io/TensorRT-LLM/installation/build-from-source-linux.html
|
||||
|
||||
Once both ship in a stable release, the install command will be:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
uv pip install lmcache "tensorrt_llm>=<version>" \
|
||||
--extra-index-url https://pypi.nvidia.com
|
||||
|
||||
LMCache integrates with TensorRT-LLM via TRT-LLM's
|
||||
**KV Cache Connector** API and supports two deployment modes:
|
||||
|
||||
- **In-process mode** (``connector: lmcache``) -- LMCache runs as
|
||||
a singleton inside the TRT-LLM process. Simplest setup; no
|
||||
extra service to manage.
|
||||
- **MP mode** (``connector: lmcache-mp``) -- LMCache runs as a
|
||||
standalone server. Multiple TRT-LLM workers on the same node
|
||||
can share the cache, and the cache survives a TRT-LLM crash.
|
||||
|
||||
.. tab-set::
|
||||
:sync-group: trtllm-mode
|
||||
|
||||
.. tab-item:: In-process mode
|
||||
:sync: inproc
|
||||
|
||||
Configure LMCache via env vars:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
export PYTHONHASHSEED=0 # required — chunk hashing depends on stable hash()
|
||||
export LMCACHE_CHUNK_SIZE=256
|
||||
export LMCACHE_LOCAL_CPU=True
|
||||
export LMCACHE_MAX_LOCAL_CPU_SIZE=2.0 # GiB
|
||||
|
||||
Build the TRT-LLM ``LLM`` with ``connector: lmcache``:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from tensorrt_llm import LLM, SamplingParams
|
||||
from tensorrt_llm.llmapi.llm_args import (
|
||||
KvCacheConfig, KvCacheConnectorConfig,
|
||||
)
|
||||
|
||||
llm = LLM(
|
||||
model="Qwen/Qwen2-1.5B-Instruct",
|
||||
backend="pytorch",
|
||||
kv_cache_config=KvCacheConfig(enable_block_reuse=True),
|
||||
kv_connector_config=KvCacheConnectorConfig(connector="lmcache"),
|
||||
)
|
||||
|
||||
out = llm.generate(["Your prompt here"], SamplingParams(max_tokens=64))
|
||||
print(out[0].outputs[0].text)
|
||||
|
||||
.. tab-item:: MP mode
|
||||
:sync: mp
|
||||
|
||||
``PYTHONHASHSEED=0`` must be set in **both** terminals --
|
||||
chunk hashing depends on a stable ``hash()``, and the
|
||||
server and client must agree on the seed.
|
||||
|
||||
Start the LMCache server:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
export PYTHONHASHSEED=0
|
||||
lmcache server \
|
||||
--l1-size-gb 10 --eviction-policy LRU --chunk-size 256
|
||||
|
||||
In a separate terminal, point TRT-LLM at the server via
|
||||
``server_url``:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
export PYTHONHASHSEED=0
|
||||
python run_trtllm.py
|
||||
|
||||
where ``run_trtllm.py`` contains:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from tensorrt_llm import LLM, SamplingParams
|
||||
from tensorrt_llm.llmapi.llm_args import (
|
||||
KvCacheConfig, KvCacheConnectorConfig,
|
||||
)
|
||||
|
||||
llm = LLM(
|
||||
model="Qwen/Qwen2-1.5B-Instruct",
|
||||
backend="pytorch",
|
||||
kv_cache_config=KvCacheConfig(enable_block_reuse=True),
|
||||
kv_connector_config=KvCacheConnectorConfig(
|
||||
connector="lmcache-mp",
|
||||
server_url="tcp://localhost:5555",
|
||||
),
|
||||
)
|
||||
|
||||
out = llm.generate(["Your prompt here"], SamplingParams(max_tokens=64))
|
||||
print(out[0].outputs[0].text)
|
||||
|
||||
.. note::
|
||||
The TRT-LLM adapter reads :class:`LMCacheEngineConfig` the
|
||||
same way the vLLM adapter does: ``LMCACHE_CONFIG_FILE`` for
|
||||
a YAML file, otherwise individual ``LMCACHE_*`` environment
|
||||
variables. See :doc:`../api_reference/configurations` for
|
||||
all options.
|
||||
|
||||
🎉 **You now have LMCache caching and reusing KV caches across all three engines.**
|
||||
|
||||
More MP server options
|
||||
----------------------
|
||||
|
||||
The vLLM MP example above runs ``lmcache server`` locally on the default
|
||||
ports. Common variations:
|
||||
|
||||
**Custom port or remote host** -- by default the connector talks to
|
||||
``localhost:5555``. To use a different port, or a server on another host,
|
||||
pass ``lmcache.mp.host`` / ``lmcache.mp.port`` in
|
||||
``kv_connector_extra_config``:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
vllm serve Qwen/Qwen3-8B --kv-transfer-config \
|
||||
'{"kv_connector":"LMCacheMPConnector", "kv_role":"kv_both", "kv_connector_extra_config": {"lmcache.mp.host": "tcp://10.0.0.1", "lmcache.mp.port": 6555}}'
|
||||
|
||||
**CPU-only (no GPU)** -- the server runs with a ``StubCPUDevice`` and shares
|
||||
KV tensors with vLLM over POSIX shared memory. Start ``lmcache server``
|
||||
normally, then set ``lmcache.mp.mp_transfer_mode=lmcache_driven`` on the vLLM
|
||||
side to enable the zero-copy SHM handle path (the default ``auto`` routing
|
||||
maps non-CUDA devices to ``engine_driven``, which uses the worker-side
|
||||
gather/scatter copy path instead).
|
||||
|
||||
**Docker** -- see :doc:`../production/docker_deployment`.
|
||||
|
||||
**HTTP management endpoints** (health, clear-cache, status) -- see
|
||||
:doc:`../mp/http_api`.
|
||||
|
||||
Next Steps
|
||||
----------
|
||||
|
||||
- **Performance Testing**: Try the :doc:`benchmarking` section to experience LMCache's performance benefits with more comprehensive examples
|
||||
- **Production**: Deploy LMCache with Docker or Kubernetes, plus observability and tuning -- see :doc:`../mp/deployment`
|
||||
@@ -0,0 +1,213 @@
|
||||
.. _disaggregated_prefill:
|
||||
|
||||
Example: Disaggregated prefill
|
||||
==============================
|
||||
|
||||
.. warning::
|
||||
|
||||
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/disaggregated_prefill`.
|
||||
|
||||
|
||||
With LMCache as a KV cache transfer library, we can run disaggregated prefill with vLLM.
|
||||
Right now, LMCache uses NIXL as a transport layer to enable fast KV cache transfer via NVLink, RDMA, or TCP.
|
||||
|
||||
This guide demonstrates how to run LMCache with disaggregated prefill using a single prefiller and decoder setup (1P1D) on a single machine.
|
||||
The architecture splits the LLM inference into two stages: prefill and decode, running on separate GPUs for better resource utilization.
|
||||
|
||||
Prerequisites
|
||||
-------------
|
||||
|
||||
Before you begin, ensure you have:
|
||||
|
||||
* At least 2 GPUs
|
||||
* Python packages installed:
|
||||
* ``lmcache`` (0.2.1 or above)
|
||||
* ``nixl`` (Install instructions `here <https://github.com/ai-dynamo/nixl>`_)
|
||||
* ``vllm`` (latest main branch)
|
||||
* ``httpx``, ``fastapi``, and ``uvicorn``
|
||||
* A valid Hugging Face token (``HF_TOKEN``) with access to Llama 3.1 8B models
|
||||
|
||||
* (Recommended) A machine with NVLink or RDMA enabled GPUs
|
||||
|
||||
.. note::
|
||||
|
||||
You can use ``ucx_perftest`` to check the GPU-GPU memory transfer and verify the NVLink or RDMA connection.
|
||||
Please refer to this link: `UCX Performance Test <https://ucx-py.readthedocs.io/en/latest/ucx-debug.html>`_.
|
||||
|
||||
Architecture Overview
|
||||
---------------------
|
||||
|
||||
The disaggregated prefill setup consists of three main components:
|
||||
|
||||
1. **Prefiller Server (Port 8100)**: Handles the prefill phase of LLM inference
|
||||
2. **Decoder Server (Port 8200)**: Manages the decoding/generation phase
|
||||
3. **Proxy Server (Port 9000)**: Coordinates between prefiller and decoder
|
||||
|
||||
Configuration
|
||||
-------------
|
||||
|
||||
1. **Prefiller Server Configuration** (``lmcache-prefiller-config.yaml``):
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
local_cpu: False
|
||||
|
||||
# PD-related configurations
|
||||
enable_pd: True
|
||||
transfer_channel: "nixl" # Using NIXL for transfer
|
||||
pd_role: "sender" # Prefiller acts as KV cache sender
|
||||
pd_proxy_host: "localhost" # Host where proxy server is running
|
||||
pd_proxy_port: 7500 # Port where proxy server is listening
|
||||
pd_buffer_size: 1073741824 # 1GB buffer for KV cache transfer
|
||||
pd_buffer_device: "cuda" # Use GPU memory for buffer
|
||||
|
||||
2. **Decoder Server Configuration** (``lmcache-decoder-config.yaml``):
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
local_cpu: False
|
||||
|
||||
# PD-related configurations
|
||||
enable_pd: True
|
||||
transfer_channel: "nixl" # Using NIXL for transfer
|
||||
pd_role: "receiver" # Decoder acts as KV cache receiver
|
||||
pd_peer_host: "localhost" # Host where decoder is listening
|
||||
pd_peer_init_port: 7300 # Port where initialization happens
|
||||
pd_peer_alloc_port: 7400 # Port for memory allocation
|
||||
pd_buffer_size: 1073741824 # 1GB buffer for KV cache transfer
|
||||
pd_buffer_device: "cuda" # Use GPU memory for buffer
|
||||
|
||||
Step-by-Step Setup
|
||||
------------------
|
||||
|
||||
1. **Environment Setup**
|
||||
|
||||
Set your Hugging Face token before running the vLLM servers.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
export HF_TOKEN=your_hugging_face_token
|
||||
|
||||
2. **Launch the vLLM + LMCache Inference Servers**
|
||||
|
||||
You can launch the components individually:
|
||||
|
||||
a. Launch Decoder (on GPU 1):
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
UCX_TLS=cuda_ipc,cuda_copy,tcp \
|
||||
LMCACHE_CONFIG_FILE=lmcache-decoder-config.yaml \
|
||||
CUDA_VISIBLE_DEVICES=1 \
|
||||
vllm serve meta-llama/Llama-3.1-8B-Instruct \
|
||||
--port 7200 \
|
||||
--disable-log-requests \
|
||||
--kv-transfer-config \
|
||||
'{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_consumer","kv_connector_extra_config": {"discard_partial_chunks": false, "lmcache_rpc_port": "consumer1"}}'
|
||||
|
||||
b. Launch Prefiller (on GPU 0):
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
UCX_TLS=cuda_ipc,cuda_copy,tcp \
|
||||
LMCACHE_CONFIG_FILE=lmcache-prefiller-config.yaml \
|
||||
CUDA_VISIBLE_DEVICES=0 \
|
||||
vllm serve meta-llama/Llama-3.1-8B-Instruct \
|
||||
--port 7100 \
|
||||
--disable-log-requests \
|
||||
--kv-transfer-config \
|
||||
'{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_producer","kv_connector_extra_config": {"discard_partial_chunks": false, "lmcache_rpc_port": "producer1"}}'
|
||||
|
||||
c. Launch a proxy server to coordinate between prefiller and decoder:
|
||||
|
||||
The code for the proxy server is available `in vLLM repo <https://github.com/vllm-project/vllm/blob/main/examples/others/lmcache/disagg_prefill_lmcache_v1/disagg_proxy_server.py>`_.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python3 ../disagg_proxy_server.py \
|
||||
--host localhost \
|
||||
--port 9100 \
|
||||
--prefiller-host localhost \
|
||||
--prefiller-port 7100 \
|
||||
--num-prefillers 1 \
|
||||
--decoder-host localhost \
|
||||
--decoder-port 7200 \
|
||||
--decoder-init-port 7300 \
|
||||
--decoder-alloc-port 7400 \
|
||||
--proxy-host localhost \
|
||||
--proxy-port 7500 \
|
||||
--num-decoders 1
|
||||
|
||||
.. note::
|
||||
|
||||
The ``UCX_TLS`` environment variable is used to specify the transport layer for UCX (the example uses NVLink)
|
||||
The ``CUDA_VISIBLE_DEVICES`` environment variable is used to specify the GPUs to use for the servers.
|
||||
|
||||
|
||||
3. **Verify Setup**
|
||||
|
||||
The servers are ready when you can access:
|
||||
|
||||
* Prefiller: ``http://localhost:7100/v1/completions``
|
||||
* Decoder: ``http://localhost:7200/v1/completions``
|
||||
* Proxy: ``http://localhost:9100/v1/completions``
|
||||
|
||||
Usage
|
||||
-----
|
||||
|
||||
Send requests to the proxy server (port 9000) using either the completions or chat completions endpoint:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl http://localhost:9000/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "meta-llama/Llama-3.1-8B-Instruct",
|
||||
"prompt": "Tell me a story",
|
||||
"max_tokens": 100
|
||||
}'
|
||||
|
||||
You can also test the setup with the following command, which runs vLLM's serving benchmark:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
git clone https://github.com/vllm-project/vllm.git
|
||||
cd vllm/benchmarks
|
||||
vllm bench serve --port 9000 --seed $(date +%s) \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--dataset-name random --random-input-len 5000 --random-output-len 200 \
|
||||
--num-prompts 50 --burstiness 100 --request-rate 1
|
||||
|
||||
Monitoring
|
||||
----------
|
||||
|
||||
The prefiller instance will log the throughput of KV cache transfer:
|
||||
|
||||
LMCache INFO: Store 5271 tokens takes: 6.5000 ms, throughput: 98.9889 GB/s; offload_time: 2.6594 ms, put_time: 3.4539 ms (cache_engine.py:190:lmcache.v1.cache_engine)
|
||||
|
||||
The decoder instance will log how many tokens are fetched from the LMCache:
|
||||
|
||||
LMCache INFO: Reqid: cmpl-b8bf01cbe47e4d108732ceeb4158d310-0, Total tokens 5170, LMCache hit tokens: 5169, need to load: 5169 (vllm_v1_adapter.py:543:lmcache.integration.vllm.vllm_v1_adapter)
|
||||
|
||||
The proxy server will log the TTFT of the prefiller node:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
===============================
|
||||
Num requests: 49
|
||||
Prefill node TTFT stats:
|
||||
- Average (ms): 0.1530598815606565
|
||||
- Median (ms): 0.15739011764526367
|
||||
- 99th Percentile (ms): 0.1643616008758545
|
||||
===============================
|
||||
|
||||
|
||||
Troubleshooting
|
||||
---------------
|
||||
|
||||
Common issues and solutions:
|
||||
|
||||
1. **GPU Requirements**: Ensure you have at least 2 GPUs available
|
||||
2. **Port Conflicts**: Check if ports used above are available
|
||||
3. **HF Token**: Verify your token starts with ``hf_`` and has necessary model access
|
||||
4. **CUDA Errors**: Ensure CUDA_VISIBLE_DEVICES is set correctly for each server
|
||||
@@ -0,0 +1,72 @@
|
||||
.. _quickstart_examples:
|
||||
|
||||
More Examples
|
||||
=============
|
||||
|
||||
.. warning::
|
||||
|
||||
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance.
|
||||
|
||||
|
||||
This section provides quick examples to help you get started with LMCache's key features.
|
||||
|
||||
KV Cache Offloading
|
||||
-------------------
|
||||
|
||||
KV cache offloading allows you to move KV caches from GPU memory to CPU memory or other storage devices. This feature is particularly useful when:
|
||||
|
||||
- There are requests shares the same prefix (e.g., long system prompt, reusing chat history in chat applications, or caching offline-processed data).
|
||||
- The GPU memory is limited to save all the KV caches.
|
||||
|
||||
By offloading KV caches, LMCache can reduce both time-to-first-token (TTFT) and GPU cycles.
|
||||
|
||||
See :ref:`offload_kv_cache` for more details.
|
||||
|
||||
KV Cache Sharing
|
||||
----------------
|
||||
|
||||
KV cache sharing enables sharing the KV cache across different LLM instances. This feature is beneficial when:
|
||||
|
||||
- There are multiple LLM instances running in the same system.
|
||||
- The requests that share the same prefix may go to different LLM instances.
|
||||
|
||||
Sharing KV caches also reduces TTFT and GPU computation by eliminating redundant calculations across different LLM instances.
|
||||
|
||||
See :ref:`share_kv_cache` for more details.
|
||||
|
||||
Disaggregated Prefill
|
||||
---------------------
|
||||
|
||||
Disaggregated prefill separates the prefill and decode phases across different compute resources. This approach:
|
||||
|
||||
- Allows specialized hardware allocation for each phase of inference
|
||||
- Enables more efficient resource utilization in distributed settings
|
||||
- Improves overall system throughput by optimizing for the different computational patterns of prefill vs. decode
|
||||
|
||||
This architecture is particularly valuable in large-scale deployment scenarios where maximizing resource efficiency and keeping a stable generation speed are both important.
|
||||
|
||||
See :ref:`disaggregated_prefill` for more details.
|
||||
|
||||
Standalone Starter
|
||||
------------------
|
||||
|
||||
The LMCache Standalone Starter allows you to run LMCacheEngine as a standalone service without vLLM or GPU dependencies. This is particularly useful for:
|
||||
|
||||
- Testing and development environments
|
||||
- CPU-only deployments
|
||||
- Distributed cache scenarios
|
||||
- Integration with custom applications
|
||||
|
||||
See :ref:`standalone_starter` for more details.
|
||||
|
||||
Detailed Examples
|
||||
-----------------
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
offload_kv_cache
|
||||
share_kv_cache
|
||||
disaggregated_prefill
|
||||
multimodality
|
||||
standalone_starter
|
||||
@@ -0,0 +1,225 @@
|
||||
Example: Multimodal KV Cache Support
|
||||
====================================
|
||||
|
||||
.. warning::
|
||||
|
||||
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance.
|
||||
|
||||
|
||||
Quick Start Example (Audio Model):
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
We going to be running audio inference with ``ultravox-v0_5-llama-3_2-1b`` and using LMCache to speed up the TTFT after the first request.
|
||||
|
||||
**Install and Serve:**
|
||||
|
||||
``pip install lmcache vllm[audio] openai``
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
vllm serve fixie-ai/ultravox-v0_5-llama-3_2-1b \
|
||||
--max-model-len 4096 --trust-remote-code \
|
||||
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}'
|
||||
|
||||
|
||||
**Benchmark:**
|
||||
|
||||
Save as ``audio_query.py``
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
import base64
|
||||
|
||||
import requests
|
||||
from openai import OpenAI
|
||||
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
from openai import APIConnectionError, OpenAI
|
||||
from openai.pagination import SyncPage
|
||||
from openai.types.model import Model
|
||||
|
||||
import time
|
||||
|
||||
def get_first_model(client: OpenAI) -> str:
|
||||
"""
|
||||
Get the first model from the vLLM server.
|
||||
"""
|
||||
try:
|
||||
models: SyncPage[Model] = client.models.list()
|
||||
except APIConnectionError as e:
|
||||
raise RuntimeError(
|
||||
"Failed to get the list of models from the vLLM server at "
|
||||
f"{client.base_url} with API key {client.api_key}. Check\n"
|
||||
"1. the server is running\n"
|
||||
"2. the server URL is correct\n"
|
||||
"3. the API key is correct"
|
||||
) from e
|
||||
|
||||
if len(models.data) == 0:
|
||||
raise RuntimeError(f"No models found on the vLLM server at {client.base_url}")
|
||||
|
||||
return models.data[0].id
|
||||
|
||||
# Modify OpenAI's API key and API base to use vLLM's API server.
|
||||
openai_api_key = "EMPTY"
|
||||
openai_api_base = "http://localhost:8000/v1"
|
||||
|
||||
client = OpenAI(
|
||||
# defaults to os.environ.get("OPENAI_API_KEY")
|
||||
api_key=openai_api_key,
|
||||
base_url=openai_api_base,
|
||||
)
|
||||
|
||||
|
||||
def encode_base64_content_from_url(content_url: str) -> str:
|
||||
"""Encode a content retrieved from a remote url to base64 format."""
|
||||
|
||||
with requests.get(content_url) as response:
|
||||
response.raise_for_status()
|
||||
result = base64.b64encode(response.content).decode("utf-8")
|
||||
|
||||
return result
|
||||
# Audio input inference
|
||||
def run_audio(model: str) -> None:
|
||||
from vllm.assets.audio import AudioAsset
|
||||
|
||||
audio_url = AudioAsset("winning_call").url
|
||||
audio_base64 = encode_base64_content_from_url(audio_url)
|
||||
|
||||
# OpenAI-compatible schema (`input_audio`)
|
||||
chat_completion_from_base64 = client.chat.completions.create(
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "What's in this audio?"},
|
||||
{
|
||||
"type": "input_audio",
|
||||
"input_audio": {
|
||||
# Any format supported by librosa is supported
|
||||
"data": audio_base64,
|
||||
"format": "wav",
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
model=model,
|
||||
max_completion_tokens=64,
|
||||
)
|
||||
|
||||
result = chat_completion_from_base64.choices[0].message.content
|
||||
print("Chat completion output from input audio:", result)
|
||||
|
||||
# HTTP URL
|
||||
chat_completion_from_url = client.chat.completions.create(
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "What's in this audio?"},
|
||||
{
|
||||
"type": "audio_url",
|
||||
"audio_url": {
|
||||
# Any format supported by librosa is supported
|
||||
"url": audio_url
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
model=model,
|
||||
max_completion_tokens=64,
|
||||
)
|
||||
|
||||
result = chat_completion_from_url.choices[0].message.content
|
||||
print("Chat completion output from audio url:", result)
|
||||
|
||||
# base64 URL
|
||||
chat_completion_from_base64 = client.chat.completions.create(
|
||||
messages=[
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "What's in this audio?"},
|
||||
{
|
||||
"type": "audio_url",
|
||||
"audio_url": {
|
||||
# Any format supported by librosa is supported
|
||||
"url": f"data:audio/ogg;base64,{audio_base64}"
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
],
|
||||
model=model,
|
||||
max_completion_tokens=64,
|
||||
)
|
||||
|
||||
result = chat_completion_from_base64.choices[0].message.content
|
||||
print("Chat completion output from base64 encoded audio:", result)
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
model = get_first_model(client)
|
||||
run_audio(model)
|
||||
end_time = time.time()
|
||||
print(f"Time taken: {end_time - start_time} seconds")
|
||||
|
||||
|
||||
**Run and see TTFT speedup:**
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# first time:
|
||||
python audio_query.py
|
||||
|
||||
# second time:
|
||||
python audio_query.py
|
||||
|
||||
|
||||
**Retrieval and speed up in logs:**
|
||||
|
||||
1. After First Request:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
[2025-08-05 09:58:06,965] LMCache INFO: Reqid: chatcmpl-dd6e8a131f2b455fa3cd133a9bfab26f, Total tokens 201, LMCache hit tokens: 201, need to load: 8 (vllm_v1_adapter.py:803:lmcache.integration.vllm.vllm_v1_adapter)
|
||||
[2025-08-05 09:58:06,967] LMCache INFO: Retrieved 201 out of 201 out of total 201 tokens (cache_engine.py:500:lmcache.v1.cache_engine)
|
||||
[2025-08-05 09:58:07,178] LMCache INFO: Storing KV cache for 256 out of 256 tokens (skip_leading_tokens=0) for request chatcmpl-dd6e8a131f2b455fa3cd133a9bfab26f (vllm_v1_adapter.py:709:lmcache.integration.vllm.vllm_v1_adapter)
|
||||
[2025-08-05 09:58:07,178] LMCache INFO: Stored 256 out of total 256 tokens. size: 0.0078 gb, cost 0.5096 ms, throughput: 15.3291 GB/s; offload_time: 0.4897 ms, put_time: 0.0200 ms (cache_engine.py:251:lmcache.v1.cache_engine)
|
||||
|
||||
*Example Output:*
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
Chat completion output from input audio: It seems like you're excitedly sharing your thoughts and predictions about a game you're about to watch. The audio appears to be a stream of text messages or social media updates. The words and phrases you've copied seem to indicate that you're a sports fan, particularly in Major League Baseball (MLB).
|
||||
|
||||
Are
|
||||
Chat completion output from audio url: It appears to be a enthusiastic and excited baseball comment from an individual. The language used, such as "And the one pitch on the way to Edgar Martinez has swung on and line down the line for a base hit," suggests a strong amateur athlete's excitement and commentary. The reference to the playoff qualification and the praise for
|
||||
Chat completion output from base64 encoded audio: It seems like you're excited about a sports game, possibly the California Athletics (now known as the Los Angeles Angels), given the reference to Edgar Martinez and the Birds (no team by that name in the AL) in the mixed messages.
|
||||
|
||||
However, I'm not seeing any audio in the conversation. Are you referring to
|
||||
Time taken: 37.96290421485901 seconds
|
||||
|
||||
2. After Second Request:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
[2025-08-05 09:58:07,371] LMCache INFO: Reqid: chatcmpl-2a130545a6a24f33b41e219ef0807a61, Total tokens 201, LMCache hit tokens: 201, need to load: 8 (vllm_v1_adapter.py:803:lmcache.integration.vllm.vllm_v1_adapter)
|
||||
[2025-08-05 09:58:07,372] LMCache INFO: Retrieved 201 out of 201 out of total 201 tokens (cache_engine.py:500:lmcache.v1.cache_engine)
|
||||
[2025-08-05 09:58:07,558] LMCache INFO: Storing KV cache for 256 out of 256 tokens (skip_leading_tokens=0) for request chatcmpl-2a130545a6a24f33b41e219ef0807a61 (vllm_v1_adapter.py:709:lmcache.integration.vllm.vllm_v1_adapter)
|
||||
[2025-08-05 09:58:07,558] LMCache INFO: Stored 256 out of total 256 tokens. size: 0.0078 gb, cost 0.4962 ms, throughput: 15.7450 GB/s; offload_time: 0.4782 ms, put_time: 0.0179 ms (cache_engine.py:251:lmcache.v1.cache_engine)
|
||||
|
||||
*Example Output:*
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
Chat completion output from input audio: It seems like you're extremely excited about the possibility of the San Francisco Giants winning the American League championship and playing in the World Series. The audio is filled with emotions and a sense of optimism, with you enthusiastically expressing your thoughts and feelings. It's clear that this is a significant moment for you, particularly given the fact
|
||||
Chat completion output from audio url: I can tell you're excited about a baseball game. It seems like you're reliving a moment during the middle of a game, especially the highlight of a six runs game for the Golden Giants. The audio appears to include a local sports radio talk show style broadcast, with a narrator (or DJs) discussing the importance
|
||||
Chat completion output from base64 encoded audio: It seems like you're having a lively discussion about baseball, specifically about the Arizona Diamondbacks and their chances of winning the American League championship. You're using colloquial expressions and slang, such as "the Oone hitter," " rejoice," and "waving him in." These cues suggest that you're engaged in
|
||||
Time taken: 5.39893364906311 seconds
|
||||
@@ -0,0 +1,480 @@
|
||||
.. _offload_kv_cache:
|
||||
|
||||
Example: Offload KV cache to CPU
|
||||
================================
|
||||
|
||||
.. warning::
|
||||
|
||||
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance.
|
||||
|
||||
|
||||
In this example, we will show you how to offload KV cache to CPU memory.
|
||||
|
||||
.. note::
|
||||
Besides CPU memory, LMCache also supports offloading KV cache to many different destinations.
|
||||
See :ref:`getting_started/quickstart/offload_kv_cache:Supported offloading destinations` for more details.
|
||||
|
||||
Prerequisites
|
||||
-------------
|
||||
|
||||
Before you begin, make sure you have:
|
||||
|
||||
- vLLM v1 with LMCache installed (see :doc:`Installation <../installation>`)
|
||||
- A GPU that can run a LLM
|
||||
|
||||
|
||||
Use CPU offloading in offline inference
|
||||
---------------------------------------
|
||||
|
||||
This section demonstrates how to use CPU memory offloading in offline inference scenarios using LMCache with vLLM.
|
||||
The example script we use here is available in `vLLM examples <https://github.com/vllm-project/vllm/blob/main/examples/others/lmcache/cpu_offload_lmcache.py>`_.
|
||||
See the `examples README <https://github.com/vllm-project/vllm/tree/main/examples/others/lmcache#2-cpu-offload-examples>`_ to understand how to run the script for vLLM v1.
|
||||
|
||||
First, set up the necessary environment variables for LMCache:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import os
|
||||
|
||||
# Set token chunk size to 256
|
||||
os.environ["LMCACHE_CHUNK_SIZE"] = "256"
|
||||
# Enable CPU memory backend
|
||||
os.environ["LMCACHE_LOCAL_CPU"] = "True"
|
||||
# Set CPU memory limit to 5GB
|
||||
os.environ["LMCACHE_MAX_LOCAL_CPU_SIZE"] = "5.0"
|
||||
|
||||
Next, configure vLLM with LMCache integration:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from vllm import LLM, SamplingParams
|
||||
from vllm.config import KVTransferConfig
|
||||
|
||||
# Configure KV cache transfer to use LMCache
|
||||
ktc = KVTransferConfig(
|
||||
kv_connector="LMCacheConnectorV1",
|
||||
kv_role="kv_both",
|
||||
)
|
||||
|
||||
# Initialize LLM with LMCache configuration
|
||||
# Adjust gpu_memory_utilization based on your GPU memory
|
||||
llm = LLM(model="Qwen/Qwen3-8B",
|
||||
kv_transfer_config=ktc,
|
||||
max_model_len=8000,
|
||||
gpu_memory_utilization=0.8)
|
||||
|
||||
Now you can run inference with automatic KV cache offloading:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# Create example prompts with shared prefix
|
||||
shared_prompt = "Hello, how are you?" * 1000
|
||||
prompts = [
|
||||
shared_prompt + "Hello, my name is",
|
||||
]
|
||||
|
||||
# Define sampling parameters
|
||||
sampling_params = SamplingParams(temperature=0, top_p=0.95, max_tokens=10)
|
||||
|
||||
# Run inference
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
for output in outputs:
|
||||
generated_text = output.outputs[0].text
|
||||
print(f"Generated text: {generated_text!r}")
|
||||
|
||||
When the inference is complete, clean up the LMCache backend:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from lmcache.v1.cache_engine import LMCacheEngineBuilder
|
||||
from lmcache.integration.vllm.utils import ENGINE_NAME
|
||||
|
||||
LMCacheEngineBuilder.destroy(ENGINE_NAME)
|
||||
|
||||
During inference, LMCache will automatically handle storing and managing KV cache in CPU memory. You can monitor this through the logs, which will show messages like::
|
||||
|
||||
LMCache INFO: Storing KV cache for 6006 out of 6006 tokens for request 0
|
||||
|
||||
This indicates that the KV cache has been successfully offloaded to CPU memory.
|
||||
|
||||
.. note::
|
||||
- Adjust ``gpu_memory_utilization`` based on your GPU's available memory
|
||||
- The CPU offloading buffer size can be adjusted through ``LMCACHE_MAX_LOCAL_CPU_SIZE``
|
||||
|
||||
Use CPU offloading in online inference
|
||||
--------------------------------------
|
||||
|
||||
This section demonstrates how to use CPU memory offloading in online serving scenarios.
|
||||
|
||||
First, create a configuration file named ``lmcache_config.yaml`` with the following content:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
chunk_size: 256
|
||||
local_cpu: true
|
||||
max_local_cpu_size: 5
|
||||
|
||||
.. note::
|
||||
LMCache supports extensive configuration through a ``lmcache_config.yaml`` file where you can customize chunk sizes, memory limits, storage backends, and more. We'll cover advanced configuration options in later examples. For now, let's run a minimal example with default configuration.
|
||||
|
||||
Launch the vLLM server with LMCache integration using environment variables. Here's an example command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
LMCACHE_CONFIG_FILE=lmcache_config.yaml \
|
||||
vllm serve \
|
||||
Qwen/Qwen3-8B \
|
||||
--kv-transfer-config \
|
||||
'{"kv_connector":"LMCacheConnectorV1",
|
||||
"kv_role":"kv_both"
|
||||
}'
|
||||
|
||||
Key parameters explained:
|
||||
|
||||
- ``LMCACHE_CONFIG_FILE``: Path to the LMCache configuration file.
|
||||
- ``--kv-transfer-config``: Configures LMCache integration
|
||||
- ``kv_connector``: Specifies the LMCache connector
|
||||
- ``kv_role``: Set to "kv_both" for both storing and loading KV cache
|
||||
|
||||
Once the server is running, you can send requests to it using curl. Here's an example of how to send a request to the vLLM server with LMCache integration:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl http://localhost:8000/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen3-8B",
|
||||
"prompt": "<|im_start|>system\nYou are a helpful AI assistant.<|im_end|>\n<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n",
|
||||
"max_tokens": 100,
|
||||
"temperature": 0.7
|
||||
}'
|
||||
|
||||
You should see the following logs:
|
||||
|
||||
.. code-block:: text
|
||||
:emphasize-lines: 1
|
||||
|
||||
LMCache INFO: Storing KV cache for 31 out of 31 tokens for request cmpl-274bcaa80837444dbf9fbba4155d2620-0 (vllm_v1_adapter.py:497:lmcache.integration.vllm.vllm_v1_adapter)
|
||||
|
||||
Once you send the same curl request again, you should see the following logs:
|
||||
|
||||
.. code-block:: text
|
||||
:emphasize-lines: 1
|
||||
|
||||
LMCache INFO: Reqid: cmpl-4ddf8863a6ac4dc3b6a952f2a107e9b2-0, Total tokens 31, LMCache hit tokens: 30, need to load: 14 (vllm_v1_adapter.py:543:lmcache.integration.vllm.vllm_v1_adapter)
|
||||
|
||||
|
||||
Example: CPU offloading benefits
|
||||
--------------------------------
|
||||
|
||||
This section demonstrates the performance benefits of using CPU offloading with LMCache. We'll use a script that generates multiple prompts and compare the performance with and without LMCache.
|
||||
|
||||
Prerequisites (Setup)
|
||||
~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
- A CUDA GPU. The example picks a model that fits the GPU automatically:
|
||||
|
||||
- ``Qwen/Qwen3-8B`` (bf16) when the GPU has ~36 GiB or more (e.g. A100-80G, H100).
|
||||
- ``Qwen/Qwen3-8B-FP8`` with ``kv_cache_dtype="fp8"`` when the GPU has ~24 GiB
|
||||
and supports native FP8 (Ada Lovelace / Hopper, ``sm_89+``; e.g. L4, L40, RTX 4090).
|
||||
- ``Qwen/Qwen3-1.7B`` as the fallback for smaller GPUs (~10 GiB and up),
|
||||
including Ampere 24 GiB cards (RTX A5000, RTX 3090) where FP8 is unsupported.
|
||||
|
||||
- Sufficient CPU memory. The example clamps the LMCache pinned host buffer to
|
||||
fit your system RAM and ``RLIMIT_MEMLOCK`` (``ulimit -l``), so it also works
|
||||
on smaller hosts without manual tuning.
|
||||
|
||||
Example script
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
Save the following script as ``cpu-offloading.py``:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""
|
||||
This file demonstrates the example usage of cpu offloading
|
||||
with LMCache in vLLM v1.
|
||||
|
||||
Note that lmcache needs to be installed to run this example.
|
||||
Learn more about LMCache in https://github.com/LMCache/LMCache.
|
||||
"""
|
||||
import os
|
||||
import torch
|
||||
import argparse
|
||||
import time
|
||||
from lmcache.v1.cache_engine import LMCacheEngineBuilder
|
||||
from lmcache.integration.vllm.utils import ENGINE_NAME
|
||||
from vllm import LLM, SamplingParams
|
||||
from vllm.config import KVTransferConfig
|
||||
|
||||
def parse_arguments() -> argparse.Namespace:
|
||||
"""Parse command line arguments."""
|
||||
parser = argparse.ArgumentParser(description="CPU offloading example with LMCache")
|
||||
parser.add_argument("--num-prompts", type=int, default=10,
|
||||
help="Number of prompts to generate (default: 10)")
|
||||
parser.add_argument("--num-tokens", type=int, default=10000,
|
||||
help="Number of tokens per prompt (default: 10000)")
|
||||
parser.add_argument("--enable-lmcache", action="store_true",
|
||||
help="Enable LMCache for CPU offloading (default: True)")
|
||||
return parser.parse_args()
|
||||
|
||||
def pick_cpu_size_gb(workload_gb: float) -> float:
|
||||
"""
|
||||
Clamp the LMCache pinned host buffer to fit system RAM and RLIMIT_MEMLOCK.
|
||||
|
||||
cudaHostAlloc pins pages, so the buffer cannot exceed total RAM nor the
|
||||
per-process memlock limit (`ulimit -l`). On hosts where either is small,
|
||||
the original "1.5 GB per 10k tokens" formula fails with cudaErrorMemoryAllocation.
|
||||
|
||||
Args:
|
||||
workload_gb: Desired buffer size for the workload, in GiB.
|
||||
Returns:
|
||||
float: A buffer size in GiB that fits both caps, never below 1.0.
|
||||
"""
|
||||
import psutil
|
||||
|
||||
ram_gib = psutil.virtual_memory().total / (1024 ** 3)
|
||||
try:
|
||||
import resource
|
||||
memlock_soft, _ = resource.getrlimit(resource.RLIMIT_MEMLOCK)
|
||||
memlock_gib = (
|
||||
float("inf")
|
||||
if memlock_soft == resource.RLIM_INFINITY
|
||||
else memlock_soft / (1024 ** 3)
|
||||
)
|
||||
except ImportError:
|
||||
# `resource` is POSIX-only; on Windows treat memlock as unbounded.
|
||||
memlock_gib = float("inf")
|
||||
return max(min(workload_gb, ram_gib * 0.5, memlock_gib * 0.9), 1.0)
|
||||
|
||||
def setup_lmcache_environment(num_prompts: int, num_tokens: int) -> None:
|
||||
"""
|
||||
Configure LMCache environment variables.
|
||||
Args:
|
||||
num_prompts: Number of prompts to process
|
||||
num_tokens: Number of tokens per prompt
|
||||
"""
|
||||
workload_gb = num_prompts * num_tokens * 1.5 / 10000 # 1.5 GB per 10k tokens
|
||||
cpu_size = pick_cpu_size_gb(workload_gb)
|
||||
|
||||
env_vars = {
|
||||
"LMCACHE_CHUNK_SIZE": "256", # Set tokens per chunk
|
||||
"LMCACHE_LOCAL_CPU": "True", # Enable local CPU backend
|
||||
"LMCACHE_MAX_LOCAL_CPU_SIZE": str(cpu_size) # CPU memory limit (GB)
|
||||
}
|
||||
for key, value in env_vars.items():
|
||||
os.environ[key] = value
|
||||
|
||||
def pick_model_and_kwargs() -> tuple[str, dict]:
|
||||
"""
|
||||
Pick a Qwen model that fits the current GPU's memory and compute capability.
|
||||
|
||||
Tiers:
|
||||
- >= 36 GiB -> Qwen/Qwen3-8B (bf16)
|
||||
- >= 20 GiB and sm >= 89 -> Qwen/Qwen3-8B-FP8 (native FP8)
|
||||
- >= 10 GiB -> Qwen/Qwen3-1.7B
|
||||
- otherwise -> RuntimeError
|
||||
|
||||
Returns:
|
||||
tuple[str, dict]: (model id, extra kwargs to pass to ``LLM``).
|
||||
Raises:
|
||||
RuntimeError: If no CUDA GPU is visible or it is too small.
|
||||
"""
|
||||
if not torch.cuda.is_available():
|
||||
raise RuntimeError("No GPU available")
|
||||
|
||||
total_gib = torch.cuda.get_device_properties(0).total_memory / (1024 ** 3)
|
||||
major, minor = torch.cuda.get_device_capability(0)
|
||||
sm = major * 10 + minor
|
||||
has_fp8 = sm >= 89 # Ada Lovelace / Hopper
|
||||
|
||||
if total_gib >= 36:
|
||||
return "Qwen/Qwen3-8B", {}
|
||||
if total_gib >= 20 and has_fp8:
|
||||
print(f"[fallback] GPU {total_gib:.1f} GiB sm_{sm}: using Qwen3-8B-FP8")
|
||||
return "Qwen/Qwen3-8B-FP8", {"kv_cache_dtype": "fp8"}
|
||||
if total_gib >= 10:
|
||||
print(f"[fallback] GPU {total_gib:.1f} GiB sm_{sm}: using Qwen3-1.7B")
|
||||
return "Qwen/Qwen3-1.7B", {}
|
||||
raise RuntimeError(
|
||||
f"GPU has {total_gib:.1f} GiB; need at least 10 GiB for Qwen3-1.7B"
|
||||
)
|
||||
|
||||
def create_test_prompts(num_prompts: int = 10, num_tokens: int = 1000) -> list[str]:
|
||||
"""
|
||||
Create test prompts with index prefix and dummy body.
|
||||
Args:
|
||||
num_prompts: Number of prompts to generate
|
||||
num_tokens: Approximate number of tokens per prompt (using 'Hi ' as token unit)
|
||||
Returns:
|
||||
list: List of prompts with format '[index] Hi Hi Hi...'
|
||||
"""
|
||||
prompts = []
|
||||
dummy_text = "Hi " * num_tokens
|
||||
|
||||
for i in range(num_prompts):
|
||||
prompt = f"[Prompt {i}] {dummy_text} how are you?"
|
||||
prompts.append(prompt)
|
||||
|
||||
return prompts
|
||||
|
||||
def initialize_llm(max_len: int = 16384, enable_lmcache: bool = True) -> LLM:
|
||||
"""
|
||||
Initialize the LLM with a model auto-selected for the current GPU.
|
||||
Args:
|
||||
max_len: Maximum sequence length
|
||||
enable_lmcache: Whether to wire up the LMCache KV connector
|
||||
Returns:
|
||||
LLM: Configured LLM instance
|
||||
"""
|
||||
model_name, extra_kwargs = pick_model_and_kwargs()
|
||||
|
||||
ktc = KVTransferConfig(
|
||||
kv_connector="LMCacheConnectorV1",
|
||||
kv_role="kv_both",
|
||||
) if enable_lmcache else None
|
||||
|
||||
return LLM(
|
||||
model=model_name,
|
||||
kv_transfer_config=ktc,
|
||||
max_model_len=max_len,
|
||||
enable_prefix_caching=False,
|
||||
gpu_memory_utilization=0.9,
|
||||
**extra_kwargs,
|
||||
)
|
||||
|
||||
def generate_and_print_output(
|
||||
llm: LLM,
|
||||
prompts: list[str],
|
||||
sampling_params: SamplingParams,
|
||||
) -> float:
|
||||
"""
|
||||
Generate text and print the results.
|
||||
Args:
|
||||
llm: LLM instance
|
||||
prompts: List of input prompts
|
||||
sampling_params: Configured sampling parameters
|
||||
Returns:
|
||||
float: Time taken for generation in seconds
|
||||
"""
|
||||
start_time = time.time()
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
end_time = time.time()
|
||||
|
||||
for output in outputs:
|
||||
generated_text = output.outputs[0].text
|
||||
print(f"Generated text: {generated_text!r}")
|
||||
|
||||
return end_time - start_time
|
||||
|
||||
def main() -> None:
|
||||
"""Main execution function."""
|
||||
# Parse command line arguments
|
||||
args = parse_arguments()
|
||||
|
||||
# Setup environment if LMCache is enabled
|
||||
if args.enable_lmcache:
|
||||
setup_lmcache_environment(args.num_prompts, args.num_tokens)
|
||||
|
||||
# Create prompts and sampling parameters
|
||||
prompts = create_test_prompts(num_prompts=args.num_prompts, num_tokens=args.num_tokens)
|
||||
sampling_params = SamplingParams(temperature=0, top_p=0.95, max_tokens=1)
|
||||
|
||||
# Initialize model
|
||||
llm = initialize_llm(enable_lmcache=args.enable_lmcache)
|
||||
|
||||
# First run
|
||||
print("\nFirst run:")
|
||||
first_run_time = generate_and_print_output(llm, prompts, sampling_params)
|
||||
print(f"First run time: {first_run_time:.2f} seconds")
|
||||
|
||||
# Second run
|
||||
print("\nSecond run:")
|
||||
second_run_time = generate_and_print_output(llm, prompts, sampling_params)
|
||||
print(f"Second run time: {second_run_time:.2f} seconds")
|
||||
|
||||
# Print speedup
|
||||
if first_run_time > 0:
|
||||
speedup = first_run_time / second_run_time
|
||||
print(f"\nSpeedup (first run / second run): {speedup:.2f}x")
|
||||
|
||||
# Cleanup if LMCache was enabled
|
||||
if args.enable_lmcache:
|
||||
LMCacheEngineBuilder.destroy(ENGINE_NAME)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
Running the Example
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
1. First, run the script without LMCache:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python cpu-offloading.py
|
||||
|
||||
You'll see output like:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
Speedup (first run / second run): 1.00x
|
||||
|
||||
Without LMCache, there's no speedup between runs even if vLLM has prefix caching enabled.
|
||||
This is because the KV cache exceeds GPU memory and can't be reused.
|
||||
|
||||
2. Now, run with LMCache enabled:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python cpu-offloading.py --enable-lmcache
|
||||
|
||||
You'll see output like:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
Speedup (first run / second run): 7.43x
|
||||
|
||||
The significant speedup in the second case demonstrates how LMCache effectively manages KV cache offloading to CPU memory.
|
||||
When the total size of KV cache exceeds GPU memory, LMCache allows you to store and reuse the cache from CPU memory,
|
||||
resulting in much faster subsequent generations for prompts with shared prefixes.
|
||||
|
||||
|
||||
Supported offloading destinations
|
||||
---------------------------------
|
||||
|
||||
LMCache now supports offloading KV cache to the following destinations:
|
||||
|
||||
- :doc:`CPU memory <../../kv_cache/storage_backends/cpu_ram>`
|
||||
- :doc:`Local file system <../../kv_cache/storage_backends/local_storage>`
|
||||
- :doc:`Mooncake Storage <../../kv_cache/storage_backends/mooncake>`
|
||||
- :doc:`InfiniStore <../../kv_cache/storage_backends/infinistore>`
|
||||
- :doc:`Redis <../../kv_cache/storage_backends/redis>`
|
||||
- :doc:`ValKey <../../kv_cache/storage_backends/valkey>`
|
||||
|
||||
Troubleshooting
|
||||
---------------
|
||||
|
||||
If you encounter the following error:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
(EngineCore_DP0 pid=55437) ERROR 10-04 14:44:47 [core.py:708] RuntimeError:
|
||||
Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
|
||||
|
||||
You can resolve this issue using one of the following methods:
|
||||
|
||||
- Set ``VLLM_WORKER_MULTIPROC_METHOD=spawn`` in the environment variables.
|
||||
- Or update the Python code to guard usage of vllm behind a if ``__name__ == '__main__':`` block.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
if __name__ == '__main__':
|
||||
from vllm import LLM, SamplingParams
|
||||
from vllm.config import KVTransferConfig
|
||||
from lmcache.v1.cache_engine import LMCacheEngineBuilder
|
||||
from lmcache.integration.vllm.utils import ENGINE_NAME
|
||||
main()
|
||||
|
||||
For details, please refer to the `vLLM Troubleshooting Guide: Python multiprocessing <https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing>`_.
|
||||
@@ -0,0 +1,270 @@
|
||||
.. _share_kv_cache:
|
||||
|
||||
Example: Share KV cache across multiple LLMs
|
||||
============================================
|
||||
|
||||
.. warning::
|
||||
|
||||
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance.
|
||||
|
||||
|
||||
LMCache should be able to reduce the generation time of the second and following calls.
|
||||
|
||||
We have examples for the following types of across-instance KV cache sharing:
|
||||
|
||||
- KV cache sharing through a centralized cache server: ``centralized_sharing``
|
||||
- KV cache sharing through p2p cache transfer: ``p2p_sharing``
|
||||
|
||||
Prerequisites
|
||||
-------------
|
||||
|
||||
Your server should have at least 2 GPUs.
|
||||
|
||||
For Centralized sharing, this will use the port 8000 and 8001 (for vLLM) and port 65432 (for LMCache).
|
||||
|
||||
For P2P sharing:
|
||||
|
||||
- `NIXL <https://github.com/ai-dynamo/nixl>`_ installed on the host.
|
||||
- Port 8010 and 8011 for 2 vllms servers.
|
||||
- Port 8200 and 8202 for 2 p2p initialization connections.
|
||||
- Port 8201 and 8203 for 2 p2p lookup connections.
|
||||
- Port 8300 for controller pull requests.
|
||||
- Port 8400 for controller reply requests.
|
||||
- Port 8500 and 8501 for 2 LMCache workers.
|
||||
- Port 9000 for controller main port (arbitrary and can be changed) to start the controller.
|
||||
|
||||
Centralized KV cache sharing
|
||||
----------------------------
|
||||
|
||||
This section demonstrates how to share KV cache across multiple vLLM instances using a centralized LMCache server.
|
||||
|
||||
**Important**: For centralized cache sharing (which is cross-process cases), ensure all processes use the same `PYTHONHASHSEED` to keep the hash of the KV cache consistent across processes: ``export PYTHONHASHSEED=0``.
|
||||
|
||||
Setup centralized sharing
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
First, create a configuration file named ``lmcache_config.yaml`` with the following content:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
chunk_size: 256
|
||||
local_cpu: true
|
||||
remote_url: "lm://localhost:65432"
|
||||
remote_serde: "cachegen"
|
||||
|
||||
Run centralized sharing example
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
1. Start the LMCache centralized server,
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
lmcache_server localhost 65432
|
||||
|
||||
2. In a different terminal,
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
PYTHONHASHSEED=0 \
|
||||
LMCACHE_CONFIG_FILE=lmcache_config.yaml \
|
||||
CUDA_VISIBLE_DEVICES=0 \
|
||||
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--gpu-memory-utilization 0.8 \
|
||||
--port 8000 --kv-transfer-config \
|
||||
'{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
|
||||
|
||||
In another terminal,
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
PYTHONHASHSEED=0 \
|
||||
LMCACHE_CONFIG_FILE=lmcache_config.yaml \
|
||||
CUDA_VISIBLE_DEVICES=1 \
|
||||
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--gpu-memory-utilization 0.8 \
|
||||
--port 8001 \
|
||||
--kv-transfer-config \
|
||||
'{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
|
||||
|
||||
Wait until both engines are ready.
|
||||
|
||||
3. Send one request to the engine at port 8000,
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl -X POST http://localhost:8000/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
|
||||
"prompt": "Explain the significance of KV cache in language models.",
|
||||
"max_tokens": 10
|
||||
}'
|
||||
|
||||
4. Send the same request to the engine at port 8001,
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl -X POST http://localhost:8001/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
|
||||
"prompt": "Explain the significance of KV cache in language models.",
|
||||
"max_tokens": 10
|
||||
}'
|
||||
|
||||
The second request will automatically retrieve and reuse the KV cache from the first instance, significantly reducing generation time.
|
||||
|
||||
P2P KV cache sharing
|
||||
--------------------
|
||||
|
||||
This section demonstrates how to share KV cache across multiple vLLM instances using peer-to-peer transfer.
|
||||
|
||||
Configure LMCache instances
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Create two configuration files for the P2P sharing setup. The values that differ between the files are the ``lmcache_instance_id`` and the P2P/controller port assignments.
|
||||
|
||||
Instance 1 configuration (``p2p_example1.yaml``):
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
chunk_size: 256
|
||||
local_cpu: true
|
||||
max_local_cpu_size: 5
|
||||
enable_async_loading: True
|
||||
|
||||
# P2P configurations
|
||||
enable_p2p: true
|
||||
p2p_host: "localhost"
|
||||
p2p_init_ports: 8200
|
||||
p2p_lookup_ports: 8201
|
||||
transfer_channel: "nixl"
|
||||
|
||||
# Controller configurations
|
||||
enable_controller: true
|
||||
lmcache_instance_id: "lmcache_instance_1"
|
||||
controller_pull_url: "localhost:8300"
|
||||
controller_reply_url: "localhost:8400"
|
||||
lmcache_worker_ports: 8500
|
||||
|
||||
extra_config:
|
||||
lookup_backoff_time: 0.001
|
||||
|
||||
Instance 2 configuration (``p2p_example2.yaml``):
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
chunk_size: 256
|
||||
local_cpu: true
|
||||
max_local_cpu_size: 5
|
||||
enable_async_loading: True
|
||||
|
||||
# P2P configurations
|
||||
enable_p2p: true
|
||||
p2p_host: "localhost"
|
||||
p2p_init_ports: 8202
|
||||
p2p_lookup_ports: 8203
|
||||
transfer_channel: "nixl"
|
||||
|
||||
# Controller configurations
|
||||
enable_controller: true
|
||||
lmcache_instance_id: "lmcache_instance_2"
|
||||
controller_pull_url: "localhost:8300"
|
||||
controller_reply_url: "localhost:8400"
|
||||
lmcache_worker_ports: 8501
|
||||
|
||||
extra_config:
|
||||
lookup_backoff_time: 0.001
|
||||
|
||||
Save both files in the directory that you will mount into the container (referenced later as ``$YAML_FILES``).
|
||||
|
||||
Run the P2P sharing workflow
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
1. Configure the environment on the host and open a shell inside the container:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
docker pull vllm/vllm-openai:latest
|
||||
export WEIGHT_DIR="/models" # model weights directory
|
||||
export CONTAINER_NAME="lmcache_vllm" # container name
|
||||
export YAML_FILES="/path/to/yaml" # directory containing the YAML files
|
||||
docker run --name "$CONTAINER_NAME" \
|
||||
--detach \
|
||||
--ipc=host \
|
||||
--network host \
|
||||
--gpus all \
|
||||
--volume "$WEIGHT_DIR:$WEIGHT_DIR" \
|
||||
--volume "$YAML_FILES:$YAML_FILES" \
|
||||
--entrypoint "/bin/bash" \
|
||||
vllm/vllm-openai:latest -c "time sleep 452d"
|
||||
docker exec -it "$CONTAINER_NAME" /bin/bash
|
||||
pip install -U lmcache # update lmcache to the latest version
|
||||
|
||||
2. Start the LMCache controller and monitoring endpoints:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
PYTHONHASHSEED=123 lmcache_controller --host localhost --port 9000 --monitor-ports '{"pull": 8300, "reply": 8400}'
|
||||
|
||||
3. Launch two vLLM engines, each with its own LMCache worker configuration.
|
||||
|
||||
Start vLLM engine 1 on GPU 0:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
PYTHONHASHSEED=123 UCX_TLS=rc CUDA_VISIBLE_DEVICES=0 LMCACHE_CONFIG_FILE=p2p_example1.yaml \
|
||||
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--gpu-memory-utilization 0.8 \
|
||||
--port 8010 \
|
||||
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
|
||||
|
||||
Start vLLM engine 2 on GPU 1:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
PYTHONHASHSEED=123 UCX_TLS=rc CUDA_VISIBLE_DEVICES=1 LMCACHE_CONFIG_FILE=p2p_example2.yaml \
|
||||
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--gpu-memory-utilization 0.8 \
|
||||
--port 8011 \
|
||||
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
|
||||
|
||||
4. Populate the KV cache by sending a request to the first engine:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl -X POST http://localhost:8010/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{
|
||||
\"model\": \"meta-llama/Meta-Llama-3.1-8B-Instruct\",
|
||||
\"prompt\": \"$(printf 'Explain the significance of KV cache in language models.%.0s' {1..100})\",
|
||||
\"max_tokens\": 10
|
||||
}"
|
||||
|
||||
5. Send the same request to the second engine to demonstrate cache retrieval:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
curl -X POST http://localhost:8011/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{
|
||||
\"model\": \"meta-llama/Meta-Llama-3.1-8B-Instruct\",
|
||||
\"prompt\": \"$(printf 'Explain the significance of KV cache in language models.%.0s' {1..100})\",
|
||||
\"max_tokens\": 10
|
||||
}"
|
||||
|
||||
Expected output
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
When the second request successfully retrieves the cache from the first instance, the logs should include entries similar to:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
(EngineCore_DP0 pid=305) [2025-11-16 07:24:11,522] LMCache INFO: Got layout info from controller: ('lmcache_instance_2', 'LocalCPUBackend', 3, 'localhost:8202') (p2p_backend.py:196:lmcache.v1.storage_backend.p2p_backend)
|
||||
(EngineCore_DP0 pid=305) [2025-11-16 07:24:11,607] LMCache INFO: Established connection to peer_init_url localhost:8202. The peer_lookup_url: localhost:8203 (p2p_backend.py:349:lmcache.v1.storage_backend.p2p_backend)
|
||||
(EngineCore_DP0 pid=305) [2025-11-16 07:24:11,706] LMCache INFO: Responding to scheduler for lookup id cmpl-e9ec2875bf954bd298ca26d14e083b80-0 with retrieved length 768 (storage_manager.py:531:lmcache.v1.storage_backend.storage_manager)
|
||||
(EngineCore_DP0 pid=305) [2025-11-16 07:24:11,708] LMCache INFO: Reqid: cmpl-e9ec2875bf954bd298ca26d14e083b80-0, Total tokens 1002, LMCache hit tokens: 768, need to load: 768 (vllm_v1_adapter.py:1330:lmcache.integration.vllm.vllm_v1_adapter)
|
||||
(EngineCore_DP0 pid=305) [2025-11-16 07:24:11,724] LMCache INFO: Retrieved 768 out of 768 required tokens (from 768 total tokens). size: 0.0938 gb, cost 7.9816 ms, throughput: 11.7458 GB/s; (cache_engine.py:531:lmcache.v1.cache_engine)
|
||||
|
||||
|
||||
These logs indicate that the peer connection was established and the cache was transferred successfully.
|
||||
@@ -0,0 +1,303 @@
|
||||
.. _standalone_starter:
|
||||
|
||||
Standalone Starter
|
||||
==================
|
||||
|
||||
.. warning::
|
||||
|
||||
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance.
|
||||
|
||||
|
||||
The LMCache Standalone Starter allows you to run LMCacheEngine as a standalone service without vLLM or GPU dependencies. This is particularly useful for:
|
||||
|
||||
- Testing and development environments
|
||||
- CPU-only or P2P backend deployments
|
||||
|
||||
Quick Start
|
||||
-----------
|
||||
|
||||
Basic Usage
|
||||
~~~~~~~~~~~
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Start with default configuration
|
||||
python -m lmcache.v1.standalone
|
||||
|
||||
# Start with custom configuration file
|
||||
python -m lmcache.v1.standalone --config examples/cache_with_configs/example.yaml
|
||||
|
||||
# Start with environment variables
|
||||
export LMCACHE_CONFIG_FILE=examples/cache_with_configs/example.yaml
|
||||
python -m lmcache.v1.standalone
|
||||
|
||||
CPU-Only Mode
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python -m lmcache.v1.standalone \
|
||||
--config examples/cache_with_configs/example.yaml \
|
||||
--model_name my_model \
|
||||
--worker_id 0 \
|
||||
--world_size 1
|
||||
|
||||
Remote P2P Mode
|
||||
~~~~~~~~~~~~~
|
||||
TO be added
|
||||
|
||||
Configuration Section
|
||||
---------------------
|
||||
|
||||
The standalone starter supports multiple configuration sources with the following priority order:
|
||||
|
||||
1. **Command-line arguments** (highest priority)
|
||||
2. **Configuration file** (specified by ``--config`` or ``LMCACHE_CONFIG_FILE``)
|
||||
3. **Environment variables** (e.g., ``LMCACHE_CHUNK_SIZE=512``)
|
||||
4. **Default values** (lowest priority)
|
||||
|
||||
Parameter Details
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
**KV Cache Shape Specification**
|
||||
|
||||
The ``--kvcache_shape_spec`` parameter supports multi-layer group configurations:
|
||||
|
||||
- Format: ``(shape_string):dtype:layer_count;[...]``
|
||||
- shape_string: comma-separated shape (e.g., '2,2,256,4,16')
|
||||
- Examples:
|
||||
- Single group: ``(2,2,256,4,16):float16:2``
|
||||
- Multiple groups: ``(2,2,256,4,16):float16:2;(3,2,256,4,4):float32:3``
|
||||
|
||||
**Device Support**
|
||||
|
||||
- ``--device=cpu``: CPU-only mode (default)
|
||||
- ``--device=cuda``: CUDA GPU acceleration
|
||||
- ``--device=xpu``: XPU GPU acceleration
|
||||
|
||||
**MLA (Multi-Level Attention)**
|
||||
|
||||
- ``--use_mla``: Enable MLA for improved attention performance
|
||||
- Requires compatible model and configuration
|
||||
|
||||
**Cache Formats**
|
||||
|
||||
- ``--fmt=vllm``: vLLM-compatible format (default)
|
||||
- Supports other formats for different inference engines
|
||||
|
||||
Command-Line Parameters
|
||||
-----------------------
|
||||
|
||||
Basic Parameters
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
--config CONFIG_FILE # Path to configuration file
|
||||
--model_name MODEL_NAME # Model name for cache identification
|
||||
--worker_id WORKER_ID # Worker ID (default: 0)
|
||||
--world_size WORLD_SIZE # Total workers (default: 1)
|
||||
--kv_dtype {float16,float32,bfloat16,uint8} # KV cache data type
|
||||
--kv_shape KV_SHAPE # KV cache shape (default: "2,2,256,4,16")
|
||||
--kvcache_shape_spec SPEC # Multi-group KV shape specification
|
||||
--device {cpu,cuda,xpu} # Device to run on (default: cpu)
|
||||
--fmt FORMAT # Cache format (default: vllm)
|
||||
--use_mla # Enable MLA (Multi-Level Attention)
|
||||
|
||||
|
||||
Usage Examples
|
||||
--------------
|
||||
|
||||
Custom Configuration
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python -m lmcache.v1.standalone \
|
||||
--config examples/cache_with_configs/example.yaml \
|
||||
--chunk_size=512 \
|
||||
--max_local_cpu_size=4.0 \
|
||||
--model_name=custom_model
|
||||
|
||||
Multi-Layer Group Configuration
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python -m lmcache.v1.standalone \
|
||||
--config examples/cache_with_configs/example.yaml \
|
||||
--kvcache_shape_spec="(2,2,256,4,16):float16:2" \
|
||||
--kv_shape="2,2,256,4,16" \
|
||||
--model_name=multi_group_model \
|
||||
--device=cpu
|
||||
|
||||
GPU device
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python -m lmcache.v1.standalone \
|
||||
--config examples/cache_with_configs/example.yaml \
|
||||
--kvcache_shape_spec="(2,2,256,4,16):float16:2" \
|
||||
--kv_shape="2,2,256,4,16" \
|
||||
--kv_dtype=float16 \
|
||||
--device=cuda \
|
||||
--use_mla \
|
||||
--model_name=gpu_model
|
||||
|
||||
MLA Configuration
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python -m lmcache.v1.standalone \
|
||||
--config examples/cache_with_configs/example.yaml \
|
||||
--kv_shape="16,2,512,16,64" \
|
||||
--kv_dtype=bfloat16 \
|
||||
--use_mla \
|
||||
--fmt=vllm \
|
||||
--model_name=mla_model
|
||||
|
||||
Internal API Server
|
||||
-------------------
|
||||
|
||||
The standalone starter includes an internal API server for monitoring and management:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python -m lmcache.v1.standalone \
|
||||
--config examples/cache_with_configs/example.yaml \
|
||||
--chunk_size=512 \
|
||||
--max_local_cpu_size=4.0 \
|
||||
--model_name=custom_model \
|
||||
--internal_api_server_enabled=True
|
||||
|
||||
|
||||
Troubleshooting
|
||||
----------------
|
||||
|
||||
Common Issues
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
**Issue**: "No config file specified"
|
||||
**Solution**: Set ``LMCACHE_CONFIG_FILE`` or use ``--config`` parameter
|
||||
|
||||
**Issue**: "Failed to connect to controller"
|
||||
**Solution**: Start controller first: ``python -m lmcache.v1.api_server``
|
||||
|
||||
**Issue**: "Invalid KV shape specification"
|
||||
**Solution**: Check format: ``(shape):dtype:layer_count``, e.g., ``(2,2,256,4,16):float16:2``
|
||||
|
||||
**Issue**: "Device not available"
|
||||
**Solution**: Verify device support: use ``--device=cpu`` if GPU not available
|
||||
|
||||
**Issue**: "MLA configuration error"
|
||||
**Solution**: Ensure compatible model and check ``--use_mla`` parameter
|
||||
|
||||
Debug Mode
|
||||
~~~~~~~~~~
|
||||
|
||||
Enable debug logging for troubleshooting:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
export LMCACHE_LOG_LEVEL=DEBUG
|
||||
python -m lmcache.v1.standalone
|
||||
|
||||
Advanced Debugging
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
For detailed layer group information:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
export LMCACHE_LOG_LEVEL=DEBUG
|
||||
python -m lmcache.v1.standalone \
|
||||
--kvcache_shape_spec="(2,2,256,4,16):float16:2;(3,2,256,4,4):float32:3" \
|
||||
--device=cpu
|
||||
|
||||
Performance Tuning
|
||||
------------------
|
||||
|
||||
Memory Configuration
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# For systems with large memory
|
||||
--max_local_cpu_size=8.0
|
||||
|
||||
# For memory-constrained systems
|
||||
--max_local_cpu_size=1.0
|
||||
|
||||
Layer Group Optimization
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Optimize for mixed precision models
|
||||
--kvcache_shape_spec="(2,2,256,4,16):float16:2;(3,2,256,4,4):bfloat16:3"
|
||||
|
||||
# Optimize for different layer configurations
|
||||
--kvcache_shape_spec="(2,2,512,8,32):float16:4;(4,2,256,16,16):float32:2"
|
||||
|
||||
GPU Acceleration
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# GPU-optimized configuration
|
||||
--device=cuda --kv_dtype=float16 --use_mla
|
||||
|
||||
# Large model on GPU
|
||||
--device=cuda --kv_shape="64,2,512,64,128" --max_local_cpu_size=16.0
|
||||
|
||||
MLA Performance
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Enable MLA for attention optimization
|
||||
--use_mla --kv_dtype=bfloat16 --device=cuda
|
||||
|
||||
# MLA with custom shape
|
||||
--use_mla --kv_shape="32,2,1024,32,64" --fmt=vllm
|
||||
|
||||
|
||||
Best Practices
|
||||
--------------
|
||||
|
||||
1. **Use configuration files** for production deployments
|
||||
2. **Set appropriate cache sizes** based on available memory
|
||||
3. **Enable internal API** for monitoring and management
|
||||
4. **Monitor logs** for performance and error tracking
|
||||
5. **Use multi-layer group configurations** for complex model architectures
|
||||
6. **Enable MLA** for improved attention performance on supported hardware
|
||||
7. **Choose appropriate device** based on available resources (CPU/GPU/XPU)
|
||||
8. **Validate KV shape specifications** before deployment
|
||||
9. **Test with debug logging** when configuring new layer groups
|
||||
10. **Optimize chunk sizes** for specific hardware configurations
|
||||
|
||||
Multi-Layer Group Best Practices
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
- Use consistent chunk sizes across layer groups for optimal performance
|
||||
- Group layers with similar precision requirements together
|
||||
- Validate shape specifications in development environment first
|
||||
- Monitor memory usage when using multiple layer groups
|
||||
|
||||
MLA Configuration Tips
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
- Enable MLA only on supported hardware configurations
|
||||
- Use bfloat16 or float16 precision for best MLA performance
|
||||
- Test MLA performance impact before production deployment
|
||||
- Monitor attention performance metrics with MLA enabled
|
||||
|
||||
Related Documentation
|
||||
---------------------
|
||||
|
||||
- :doc:`../quickstart`
|
||||
- :doc:`../../api_reference/configurations`
|
||||
- :doc:`../../kv_cache/storage_backends/index`
|
||||
- :doc:`../../kv_cache_management/index`
|
||||
Reference in New Issue
Block a user