wehub-resource-sync
160 lines
9.5 KiB
ReStructuredText
160 lines
9.5 KiB
ReStructuredText
.. _kv_cache_events:
|
|
|
|
KV Cache Events
|
|
===============
|
|
|
|
.. warning::
|
|
|
|
This page documents the behavior of LMCache's in-process mode (deprecated). Please consider using :doc:`LMCache MP mode </mp/index>` for better feature support and performance. For the MP mode equivalent of this page, see :doc:`/mp/observability/index`.
|
|
|
|
|
|
KV cache events are actions or lifecycle events that occur when managing the KV cache during inference. These events can be used for KV-cache-aware routing.
|
|
|
|
LMCache supports KV cache events as follows:
|
|
|
|
- Generates storage KV cache events
|
|
- The events format is defined as per the `BlockStored class <https://github.com/vllm-project/vllm/blob/main/vllm/distributed/kv_events.py>`_ in vLLM
|
|
- LMCache passes the events to SGLang or vLLM to publish them using their messaging system
|
|
|
|
Prerequisites
|
|
-------------
|
|
|
|
The following prerequisites are required:
|
|
|
|
.. tab-set::
|
|
|
|
.. tab-item:: vLLM
|
|
|
|
- vLLM v0.13.0+
|
|
- LMCache v0.3.11+
|
|
|
|
.. tab-item:: SGLang
|
|
|
|
- SGLang vx.y.z+
|
|
- LMCache vx.y.z+
|
|
|
|
How to Generate KV Cache events
|
|
-------------------------------
|
|
|
|
.. tab-set::
|
|
|
|
.. tab-item:: vLLM
|
|
|
|
Before starting to generate KV events, you need to be aware of the following:
|
|
|
|
- You need to enable ``enable_kv_events`` for LMCache as events are not generated by default.
|
|
- If running more than 1 worker in vLLM, you need to use a non-default hashing algorithm (set ``pre_caching_hash_algorithm`` in LMCache) so that hashes generated per worker are the same. If not then you will have duplicate events for the same operation as events are generated per worker.
|
|
- LMCache sends the events to vLLM for publishing. To enable events to be published, you need to set the vLLM configuration setting ``--kv-events-config``. See `vLLM KV Events configuration <https://docs.vllm.ai/en/stable/api/vllm/config/kv_events/>`_ for more details.
|
|
|
|
The steps that follow give an example of how KV events can be generated, published and consumed:
|
|
|
|
1. Start vLLM with LMCache and model ``Qwen/Qwen3-0.6B`` as follows:
|
|
|
|
.. code-block:: bash
|
|
|
|
LMCACHE_CONFIG_FILE=lmcache_config.yaml \
|
|
vllm serve Qwen/Qwen3-0.6B --kv-transfer-config '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}' \
|
|
--disable-log-requests --no-enable-prefix-caching --kv-events-config '{"enable_kv_cache_events": "True", "publisher": "zmq", "topic": "kv-events"}'
|
|
|
|
Example of the LMCache configuration is as follows:
|
|
|
|
.. code-block:: yaml
|
|
|
|
chunk_size: 8 # demo only; use 256 for production
|
|
local_cpu: true
|
|
enable_kv_events: true
|
|
pre_caching_hash_algorithm: sha256_cbor_64bit
|
|
|
|
2. To be able to process the events that are published by vLLM, you need a client that subscribes to the publisher message channel and can consume the events. vLLM provides such a client example `KV Events Subscriber <https://docs.vllm.ai/en/stable/examples/online_serving/kv_events_subscriber/>`_. Run this python script in a separate terminal.
|
|
|
|
3. Prompt the model:
|
|
|
|
.. code-block:: bash
|
|
|
|
curl http://localhost:8000/v1/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"model": "Qwen/Qwen3-0.6B",
|
|
"prompt": "<|begin_of_text|><|system|>\nYou are a helpful AI assistant.\n<|user|>\nWhat is the capital of France?\n<|assistant|>",
|
|
"max_tokens": 100,
|
|
"temperature": 0.7
|
|
}'
|
|
|
|
4. You should receive a message in the client (that you started in step 2.) window, similar to the following:
|
|
|
|
.. code-block:: bash
|
|
|
|
Received event batch at 1765529395.2132685:
|
|
- BlockStored(block_hashes=[b'\x96\x95[h6\x1dE$v\x03\xe8\xf0\xc20\xcd\xe8\xa7#\x9cS\xe0\x16\xba\xab7\xf7z\x10P]\xfaT'], parent_block_hash=None, token_ids=[27, 91, 7265, 3575, 4326, 91, 1784, 91, 8948, 91, 397, 2610, 525, 264, 10950, 15235, 17847, 624, 27, 91, 872, 91, 397, 3838, 374, 279, 16158, 1685, 1370, 276, 5267, 27, 91, 77091, 91, 29], block_size=36, lora_id=None, medium='cpu')
|
|
|
|
This is the event generated after the cache store operation.
|
|
|
|
.. tab-item:: SGLang
|
|
|
|
Before starting to generate KV events, you need to be aware of the following:
|
|
|
|
- You need to enable ``enable_kv_events`` for LMCache as events are not generated by default.
|
|
- LMCache sends the events to SGLang for publishing. To enable events to be published, you need to set the SGLang configuration setting ``--kv-events-config``.
|
|
|
|
The steps that follow give an example of how KV events can be generated, published and consumed:
|
|
|
|
1. Start SGLang with LMCache and model ``Qwen/Qwen3-0.6B`` as follows:
|
|
|
|
.. code-block:: bash
|
|
|
|
export LMCACHE_CONFIG_FILE=lmcache_config.yaml
|
|
|
|
python -m sglang.launch_server \
|
|
--model-path Qwen/Qwen3-0.6B \
|
|
--enable-lmcache \
|
|
--kv-events-config '{"publisher": "zmq", "topic": "kv-events"}'
|
|
|
|
Example of the LMCache configuration is as follows:
|
|
|
|
.. code-block:: yaml
|
|
|
|
chunk_size: 8 # demo only; use 256 for production
|
|
local_cpu: true
|
|
use_layerwise: true
|
|
max_local_cpu_size: 10 # GB
|
|
enable_kv_events: true
|
|
|
|
2. To be able to process the events that are published by SGLang, you need a client that subscribes to the publisher message channel and can consume the events. vLLM provides such a client example `KV Events Subscriber <https://docs.vllm.ai/en/stable/examples/online_serving/kv_events_subscriber/>`_. To use this client for SGLang, you need to remove the properties ``medium`` and ``lora_name`` from the ``BlockStored`` class definition and ``medium`` from ``BlockRemoved`` class definition. Save the changes and run this updated python script in a separate terminal.
|
|
|
|
3. Prompt the model:
|
|
|
|
.. code-block:: bash
|
|
|
|
curl http://localhost:30000/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"model": "Qwen/Qwen3-0.6B",
|
|
"messages": [{"role": "user", "content": "Qwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts"}],
|
|
"max_tokens": 100,
|
|
"temperature": 0.7
|
|
}'
|
|
|
|
|
|
4. You should receive a message in the client (that you started in step 2.) window, similar to the following:
|
|
|
|
.. code-block:: bash
|
|
|
|
Received event batch at 1769014811.9058058:
|
|
- BlockStored(block_hashes=[-7651984371600085018], parent_block_hash=None, token_ids=[151644, 872, 198, 48, 16948, 18, 374, 279, 5535], block_size=8, lora_id=None)
|
|
- BlockStored(block_hashes=[1717827842932260036], parent_block_hash=-7651984371600085018, token_ids=[5535, 9471, 315, 3460, 4128, 4119, 304, 1207, 16948], block_size=8, lora_id=None)
|
|
- BlockStored(block_hashes=[-6563676647234339623], parent_block_hash=1717827842932260036, token_ids=[16948, 4013, 11, 10004, 264, 15817, 16182, 315, 27950], block_size=8, lora_id=None)
|
|
- BlockStored(block_hashes=[-5164197595219155465], parent_block_hash=-6563676647234339623, token_ids=[27950, 323, 20980, 8668, 18376, 15546, 151645, 198, 151644], block_size=8, lora_id=None)
|
|
- BlockStored(block_hashes=[8690007828157426740], parent_block_hash=-5164197595219155465, token_ids=[151644, 77091, 198, 151667, 198, 32313, 11, 279, 1196], block_size=8, lora_id=None)
|
|
- BlockStored(block_hashes=[5720773965762948853], parent_block_hash=8690007828157426740, token_ids=[1196, 9733, 1207, 16948, 18, 438, 279, 5535, 9471], block_size=8, lora_id=None)
|
|
- BlockStored(block_hashes=[-4465594513801548703], parent_block_hash=5720773965762948853, token_ids=[9471, 315, 3460, 4128, 4119, 304, 279, 1207, 16948], block_size=8, lora_id=None)
|
|
- BlockStored(block_hashes=[4010782427232237897], parent_block_hash=-4465594513801548703, token_ids=[16948, 4013, 323, 429, 432, 5707, 264, 15817, 16182], block_size=8, lora_id=None)
|
|
- BlockStored(block_hashes=[8472258105533326837], parent_block_hash=4010782427232237897, token_ids=[16182, 315, 27950, 323, 20980, 8668, 18376, 15546, 4119], block_size=8, lora_id=None)
|
|
- BlockStored(block_hashes=[-3602322156693524155], parent_block_hash=8472258105533326837, token_ids=[4119, 13, 6771, 752, 1191, 553, 48996, 279, 1207], block_size=8, lora_id=None)
|
|
- BlockStored(block_hashes=[-6413316389463734553], parent_block_hash=-3602322156693524155, token_ids=[1207, 16948, 4013, 13, 1207, 16948, 374, 264, 4013], block_size=8, lora_id=None)
|
|
- BlockStored(block_hashes=[-4080340760183068020], parent_block_hash=-6413316389463734553, token_ids=[4013, 315, 15235, 4119, 7881, 553, 54364, 13, 576], block_size=8, lora_id=None)
|
|
- BlockStored(block_hashes=[1557368444906237766], parent_block_hash=-4080340760183068020, token_ids=[576, 5535, 825, 11, 1207, 16948, 18, 11, 374], block_size=8, lora_id=None)
|
|
- BlockStored(block_hashes=[-2282733302929094006], parent_block_hash=1557368444906237766, token_ids=[374, 12824, 279, 5535, 11, 773, 429, 594, 4396], block_size=8, lora_id=None)
|
|
- BlockStored(block_hashes=[8695562889830890067], parent_block_hash=-2282733302929094006, token_ids=[4396, 382, 7039, 11, 279, 1196, 6801, 311, 1414], block_size=8, lora_id=None)
|
|
- BlockStored(block_hashes=[-6034740625096789744], parent_block_hash=8695562889830890067, token_ids=[1414, 911, 279, 15817, 16182, 315, 27950, 323, 20980], block_size=8, lora_id=None)
|
|
|
|
This is the event generated after the cache store operation. |