wehub-resource-sync
57 lines
2.2 KiB
Markdown
57 lines
2.2 KiB
Markdown
# KV Cache SDK Examples
|
|
|
|
These examples show how to use the Python SDK to store and retrieve
|
|
KV cache tensors through the LMCache MP HTTP API. The SDK path is memory-first:
|
|
applications can retrieve a tensor, pair it with new token metadata, and store
|
|
it again without writing the tensor through a local storage format. Store will
|
|
return False if LMCache already have the KV Cache with the same token sequence.
|
|
|
|
## End-to-end vLLM flow
|
|
|
|
First, start up the vLLM and LMCache server by running commands listed in the
|
|
first cell of `e2e_kv_edit.ipynb`.
|
|
|
|
Then, starting from cell 2 of `e2e_kv_edit.ipynb`, it is doing below experiment
|
|
flow:
|
|
1. Send a source prompt to vLLM so the normal connector stores KV in LMCache.
|
|
2. Retrieve the source KV cache into an in-memory tensor with `retrieve()`.
|
|
3. Build a target token-ID prompt: the same length as the source prompt and
|
|
identical apart from a few different synthetic leading tokens.
|
|
4. Store the source KV under the target prefix with `store()`.
|
|
5. Send the target token IDs to vLLM so the target prefix hits the remapped KV.
|
|
6. Print retrieve counts, latencies, response previews, and whether the source
|
|
and target outputs match.
|
|
|
|
The target prompt starts with different token IDs, so it does not rely on a
|
|
serving-engine local prefix match. Because the prompts are identical apart from
|
|
those leading tokens, reusing the source KV reconstructs the same final context
|
|
for the target request, which should produce the same deterministic output.
|
|
|
|
The core SDK pattern used by the end-to-end example is:
|
|
|
|
```python
|
|
import lmcache.sdk.kvcache as lmc_sdk
|
|
|
|
ctx = lmc_sdk.connect(
|
|
url="tcp://localhost:6555", # ZMQ message queue
|
|
http_url="http://localhost:8080", # HTTP config / status
|
|
model_name="...",
|
|
)
|
|
|
|
kv = lmc_sdk.retrieve(ctx, tokens=source_tokens)
|
|
if kv is not None:
|
|
lmc_sdk.store(ctx, kv=kv, tokens=target_tokens)
|
|
|
|
lmc_sdk.close(ctx)
|
|
```
|
|
|
|
### Requirements
|
|
|
|
- An LMCache MP server running with HTTP enabled.
|
|
- A model already registered with that server. Check `/status` for the
|
|
registered `model_name`, `chunk_size`, layer count, dtype, and hidden dim.
|
|
- A homogeneous `KV_2LTD` layout.
|
|
|
|
Use `--cache-salt` on all commands when storing and retrieving from a
|
|
non-default namespace.
|