lmcache--lmcache/examples/kvcache_sdk

KV Cache SDK Examples

These examples show how to use the Python SDK to store and retrieve KV cache tensors through the LMCache MP HTTP API. The SDK path is memory-first: applications can retrieve a tensor, pair it with new token metadata, and store it again without writing the tensor through a local storage format. Store will return False if LMCache already have the KV Cache with the same token sequence.

End-to-end vLLM flow

First, start up the vLLM and LMCache server by running commands listed in the first cell of e2e_kv_edit.ipynb.

Then, starting from cell 2 of e2e_kv_edit.ipynb, it is doing below experiment flow:

1. Send a source prompt to vLLM so the normal connector stores KV in LMCache.
2. Retrieve the source KV cache into an in-memory tensor with retrieve().
3. Build a target token-ID prompt: the same length as the source prompt and identical apart from a few different synthetic leading tokens.
4. Store the source KV under the target prefix with store().
5. Send the target token IDs to vLLM so the target prefix hits the remapped KV.
6. Print retrieve counts, latencies, response previews, and whether the source and target outputs match.

The target prompt starts with different token IDs, so it does not rely on a serving-engine local prefix match. Because the prompts are identical apart from those leading tokens, reusing the source KV reconstructs the same final context for the target request, which should produce the same deterministic output.

The core SDK pattern used by the end-to-end example is:

import lmcache.sdk.kvcache as lmc_sdk

ctx = lmc_sdk.connect(
    url="tcp://localhost:6555",        # ZMQ message queue
    http_url="http://localhost:8080",  # HTTP config / status
    model_name="...",
)

kv = lmc_sdk.retrieve(ctx, tokens=source_tokens)
if kv is not None:
    lmc_sdk.store(ctx, kv=kv, tokens=target_tokens)

lmc_sdk.close(ctx)

Requirements

• An LMCache MP server running with HTTP enabled.
• A model already registered with that server. Check /status for the registered model_name, chunk_size, layer count, dtype, and hidden dim.
• A homogeneous KV_2LTD layout.

Use --cache-salt on all commands when storing and retrieving from a non-default namespace.