wehub/lmcache--lmcache
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:24:33 +08:00
commit f213ec8976
2101 changed files with 494002 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/cache_controller/README.md
+15
View File
@@ -0,0 +1,15 @@
# Examples of Cache Controller APIs
LMCache offers various ochestration APIs which can be used for routing (e.g., KV cache lookup) or hot context migration (e.g., KV cache move/migration).
Here are a few examples:
- [KV cache clear](clear/)
- [KV cache compress](compress/)
- [KV cache lookup](lookup/)
- [KV cache move](move/)
- [KV cache pin](pin/)
Unsupported APIs (WIP):
- [KV cache decompress](decompress/)
- [KV cache unpin](unpin/)
examples/cache_controller/clear/README.md
+46
View File
@@ -0,0 +1,46 @@
# LMCache Clear
This is an example to demonstrate how to clear KV cache in an LMCacheEngine externally.
## Prerequisites
Your server should have at least 1 GPU.
This will use port 8000 for 1 vllm and port 8001 for LMCache. The controller occupies ports 9000 and 9001.
## Steps
1. Start the vllm engine at port 8000:
```bash
CUDA_VISIBLE_DEVICES=0 LMCACHE_CONFIG_FILE=example.yaml vllm serve meta-llama/Llama-3.1-8B-Instruct --max-model-len 4096 --gpu-memory-utilization 0.8 --port 8000 --kv-transfer-config '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
```
2. Start the lmcache controller at port 9000 and the monitor at port 9001:
```bash
lmcache_controller --host localhost --port 9000 --monitor-port 9001
```
3. Send a request to vllm engine:
```bash
curl -X POST http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"prompt": "Explain the significance of KV cache in language models.",
"max_tokens": 10
}'
```
4. Clear the KV cache in the system:
```bash
curl -X POST http://localhost:9000/clear \
-H "Content-Type: application/json" \
-d '{
"instance_id": "lmcache_default_instance",
"location": "LocalCPUBackend"
}'
```
You should be able to see a return message indicating the number of tokens' KV cache that has been successfully cleared in the system:
```plaintext
{"event_id": "xxx", "num_tokens": 12}
```
examples/cache_controller/clear/example.yaml
+13
View File
@@ -0,0 +1,13 @@
chunk_size: 256
local_cpu: True
max_local_cpu_size: 5
# cache controller configurations
enable_controller: True
lmcache_instance_id: "lmcache_default_instance"
controller_pull_url: "localhost:9001"
lmcache_worker_ports: 8001
# Peer identifiers
p2p_host: "localhost"
p2p_init_ports: 8200
examples/cache_controller/compress/README.md
+86
View File
@@ -0,0 +1,86 @@
# LMCache Compress
This is an example to demonstrate how to compress or decompress a request's KV cache externally.
## Prerequisites
Your server should have at least 1 GPU.
This will use port 8000 for vllm and port 8001 for the LMCache worker. The controller itself occupies port 9000 and 9001.
## Steps
1. Start vllm engine at port 8000
```bash
CUDA_VISIBLE_DEVICES=0 LMCACHE_CONFIG_FILE=example.yaml vllm serve meta-llama/Llama-3.1-8B-Instruct --gpu-memory-utilization 0.8 --port 8000 --kv-transfer-config '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
```
2. Start the lmcache controller at port 9000 and the monitor at port 9001:
```bash
lmcache_controller --host localhost --port 9000 --monitor-port 9001
```
3. Send a request to vllm engine:
```bash
curl -X POST http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"prompt": "Explain the significance of KV cache in language models.",
"max_tokens": 10
}'
```
LMCache will automatically offloads the KV cache to CPU.
4. Tokenize the prompt:
```bash
curl -X POST http://localhost:8000/tokenize \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"prompt": "Explain the significance of KV cache in language models."
}'
```
You should be able to see the returned token ids as:
```plaintext
{"count":12,"max_model_len":4096,"tokens":[128000,849,21435,279,26431,315,85748,6636,304,4221,4211,13],"token_strs":null}
```
5. Using Cachegen to compress request's KV cache:
```bash
curl -X POST http://localhost:9000/compress \
-H "Content-Type: application/json" \
-d '{
"instance_id": "lmcache_default_instance",
"method": "cachegen",
"location": "LocalCPUBackend",
"tokens": [128000, 849, 21435, 279, 26431, 315, 85748, 6636, 304, 4221, 4211, 13]
}'
```
You should be able to see a return message indicating the KV cache has started to be compressed
```plaintext
{"num_tokens": 12, "event_id": "xxx"}
```
`num_tokens: 12` means that there are 12 tokens's KV cache are being compressed in the system. The returned `event_id` can be used to check the status of the compress operation (this functionality is coming soon).
6. Using Cachegen to decompress request's KV cache:
```bash
curl -X POST http://localhost:9000/decompress \
-H "Content-Type: application/json" \
-d '{
"instance_id": "lmcache_default_instance",
"method": "cachegen",
"location": "LocalCPUBackend",
"tokens": [128000, 849, 21435, 279, 26431, 315, 85748, 6636, 304, 4221, 4211, 13]
}'
```
You should be able to see a return message indicating the KV cache has started to be decompressed
```plaintext
{"num_tokens": 12, "event_id": "xxx"}
```
`num_tokens: 12` means that there are 12 tokens's KV cache are being decompressed in the system. The returned `event_id` can be used to check the status of the decompress operation .
examples/cache_controller/compress/example.yaml
+13
View File
@@ -0,0 +1,13 @@
chunk_size: 256
local_cpu: True
max_local_cpu_size: 5
# cache controller configurations
enable_controller: True
lmcache_instance_id: "lmcache_default_instance"
controller_pull_url: "localhost:9001"
lmcache_worker_ports: 8001
# Peer identifiers
p2p_host: "localhost"
p2p_init_ports: 8200
examples/cache_controller/health/README.md
+26
View File
@@ -0,0 +1,26 @@
# LMCache Health Check
This example demonstrates how to check the health status of the LMCache controller.
## Prerequisites
- The LMCache controller must be running.
## Steps
1. Start the LMCache controller (if not already running):
```bash
PYTHONHASHSEED=123 lmcache_controller --host localhost --port 9000 --monitor-port 9001
```
2. Send a health check request to the controller's monitor port (9001 in this example):
```bash
curl -X POST http://localhost:9000/health -H "Content-Type: application/json" -d '{"instance_id":"lmcache_default_instance"}'
```
`lmcache_default_instance` indicates the `instance_id`.
3. The expected response is a JSON object indicating the error_codes:
```json
{"event_id":"health47ce328d-f27e-48ae-ab0c-c2218aabce95","error_codes":{"0":0,"1":0}}
```
`event_id` is an identifier of the controller operation, which can be ignored in this functionality.
error_codes formatted to worker_id to error_code pair, and error_code
`0` stand for health, `non zero` means error occurred.
examples/cache_controller/health/example.yaml
+13
View File
@@ -0,0 +1,13 @@
chunk_size: 256
local_cpu: True
max_local_cpu_size: 5
# cache controller configurations
enable_controller: True
lmcache_instance_id: "lmcache_default_instance"
controller_pull_url: "localhost:9001"
lmcache_worker_ports: 8001
# Peer identifiers
p2p_host: "localhost"
p2p_init_ports: 8200
examples/cache_controller/lookup/README.md
+64
View File
@@ -0,0 +1,64 @@
# LMCache Lookup
This is an example to demonstrate how to check the existence of a request's KV cache in an LMCacheEngine externally.
## Prerequisites
Your server should have at least 1 GPU.
This will use port 8000 for 1 vllm and port 8001 for LMCache. The controller occupies ports 9000 and 9001.
## Steps
1. Start the vllm engine at port 8000:
```bash
PYTHONHASHSEED=123 CUDA_VISIBLE_DEVICES=0 LMCACHE_CONFIG_FILE=example.yaml vllm serve meta-llama/Llama-3.1-8B-Instruct --gpu-memory-utilization 0.8 --port 8000 --kv-transfer-config '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
```
2. Start the lmcache controller at port 9000 and the monitor at port 9001:
```bash
PYTHONHASHSEED=123 lmcache_controller --host localhost --port 9000 --monitor-port 9001
```
3. Send a request to vllm engine:
```bash
curl -X POST http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"prompt": "Explain the significance of KV cache in language models.",
"max_tokens": 10
}'
```
4. Tokenize the prompt:
```bash
curl -X POST http://localhost:8000/tokenize \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"prompt": "Explain the significance of KV cache in language models."
}'
```
You should be able to see the returned token ids as:
```plaintext
{"count":12,"tokens":[128000,849,21435,279,26431,315,85748,6636,304,4221,4211,13],"token_strs":null}
```
5. Send a lookup request to lmcache controller:
```bash
curl -X POST http://localhost:9000/lookup \
-H "Content-Type: application/json" \
-d '{
"tokens": [128000, 849, 21435, 279, 26431, 315, 85748, 6636, 304, 4221, 4211, 13]
}'
```
The above request returns the cache information.
You should be able to see a return message:
```plaintext
{"event_id": "xxx", "lmcache_default_instance": ("LocalCPUBackend", 12)}
```
`lmcache_default_instance` indicates the `instance_id` and `("LocalCPUBackend", 12)` indicates the cache location within that instance and matched prefix length. `event_id` is an identifier of the controller operation, which can be ignored in this functionality.
examples/cache_controller/lookup/example.yaml
+13
View File
@@ -0,0 +1,13 @@
chunk_size: 256
local_cpu: True
max_local_cpu_size: 5
# cache controller configurations
enable_controller: True
lmcache_instance_id: "lmcache_default_instance"
controller_pull_url: "localhost:9001"
lmcache_worker_ports: 8001
# Peer identifiers
p2p_host: "localhost"
p2p_init_ports: 8200
examples/cache_controller/move/README.md
+68
View File
@@ -0,0 +1,68 @@
# LMCache Move/Migrate
This is an example to demonstrate how to move/migrate a request's KV cache across LMCacheEngines externally.
## Prerequisites
Your server should have at least 2 GPUs. [NIXL](https://github.com/ai-dynamo/nixl) is required to be installed.
This will use port 8000 and 8001 for 2 vllms and port 8500 and 8501 for the corresponding LMCache workers. Also, ports 8200, 8201, 8202 and 8203 are used for p2p KV cache transfer. The controller itself occupies port 9000, 8300 and 9400.
## Steps
1. Start two vllm engines at port 8000 and port 8001:
```bash
PYTHONHASHSEED=123 UCX_TLS=rc CUDA_VISIBLE_DEVICES=0 LMCACHE_CONFIG_FILE=instance1.yaml vllm serve meta-llama/Llama-3.1-8B-Instruct --gpu-memory-utilization 0.8 --port 8000 --kv-transfer-config '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
```
```bash
PYTHONHASHSEED=123 UCX_TLS=rc CUDA_VISIBLE_DEVICES=1 LMCACHE_CONFIG_FILE=instance2.yaml vllm serve meta-llama/Llama-3.1-8B-Instruct --gpu-memory-utilization 0.8 --port 8001 --kv-transfer-config '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
```
2. Start the lmcache controller at port 9000 and the monitor at port 9001:
```bash
PYTHONHASHSEED=123 lmcache_controller --host localhost --port 9000 --monitor-ports '{"pull": 8300, "reply": 8400}'
```
3. Send a request to vllm engine 1:
```bash
curl -X POST http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"prompt": "Explain the significance of KV cache in language models.",
"max_tokens": 10
}'
```
4. Tokenize the prompt:
```bash
curl -X POST http://localhost:8000/tokenize \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"prompt": "Explain the significance of KV cache in language models."
}'
```
You should be able to see the returned token ids as:
```plaintext
{"count":12,"tokens":[128000,849,21435,279,26431,315,85748,6636,304,4221,4211,13],"token_strs":null}
```
5. Move the request's KV cache from vllm engine 1's CPU to vllm engine 2's CPU using request's token ids:
```bash
curl -X POST http://localhost:9000/move \
-H "Content-Type: application/json" \
-d '{
"old_position": ["lmcache_instance_1", "LocalCPUBackend"],
"new_position": ["lmcache_instance_2", "LocalCPUBackend"],
"tokens": [128000, 849, 21435, 279, 26431, 315, 85748, 6636, 304, 4221, 4211, 13]
}'
```
You should be able to see a return message indicating the KV cache has started to be moved in the system:
```plaintext
{"num_tokens": 12, "event_id": "xxx"}
```
`num_tokens: 12` means that there are 12 tokens's KV cache are stored in the system. The returned `event_id` can be used to check the status of the move operation (this functionality is coming soon).
examples/cache_controller/move/instance1.yaml
+17
View File
@@ -0,0 +1,17 @@
chunk_size: 256
local_cpu: True
max_local_cpu_size: 5
# P2P configurations
enable_p2p: True
p2p_host: "localhost"
p2p_init_ports: 8200
p2p_lookup_ports: 8201
transfer_channel: "nixl"
# cache controller configurations
enable_controller: True
lmcache_instance_id: "lmcache_instance_1"
controller_pull_url: "localhost:8300"
controller_reply_url: "localhost:8400"
lmcache_worker_ports: 8500
examples/cache_controller/move/instance2.yaml
+17
View File
@@ -0,0 +1,17 @@
chunk_size: 256
local_cpu: True
max_local_cpu_size: 5
# P2P configurations
enable_p2p: True
p2p_host: "localhost"
p2p_init_ports: 8202
p2p_lookup_ports: 8203
transfer_channel: "nixl"
# cache controller configurations
enable_controller: True
lmcache_instance_id: "lmcache_instance_2"
controller_pull_url: "localhost:8300"
controller_reply_url: "localhost:8400"
lmcache_worker_ports: 8501
examples/cache_controller/pin/README.md
+62
View File
@@ -0,0 +1,62 @@
# LMCache Pin/Persistence
This is an example to demonstrate how to pin/persist a request's KV cache in an LMCacheEngine externally.
## Prerequisites
Your server should have at least 1 GPU.
This will use port 8000 for 1 vllm and port 8001 for LMCache. The controller occupies ports 9000 and 9001.
## Steps
1. Start the vllm engine at port 8000:
```bash
CUDA_VISIBLE_DEVICES=0 LMCACHE_CONFIG_FILE=example.yaml vllm serve meta-llama/Llama-3.1-8B-Instruct --gpu-memory-utilization 0.8 --port 8000 --kv-transfer-config '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
```
2. Start the lmcache controller at port 9000 and the monitor at port 9001:
```bash
lmcache_controller --host localhost --port 9000 --monitor-port 9001
```
3. Send a request to vllm engine:
```bash
curl -X POST http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"prompt": "Explain the significance of KV cache in language models.",
"max_tokens": 10
}'
```
4. Tokenize the prompt:
```bash
curl -X POST http://localhost:8000/tokenize \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"prompt": "Explain the significance of KV cache in language models."
}'
```
You should be able to see the returned token ids as:
```plaintext
{"count":12,"max_model_len":4096,"tokens":[128000,849,21435,279,26431,315,85748,6636,304,4221,4211,13],"token_strs":null}
```
5. Pin a request's KV cache in the system:
```bash
curl -X POST http://localhost:9000/pin \
-H "Content-Type: application/json" \
-d '{
"tokens": [128000, 849, 21435, 279, 26431, 315, 85748, 6636, 304, 4221, 4211, 13],
"instance_id": "lmcache_default_instance",
"location": "LocalCPUBackend"
}'
```
You should be able to see a return message indicating the number of tokens' KV cache that has been successfully pinned in the system:
```plaintext
{"event_id": "xxx", "num_tokens": 12}
```
examples/cache_controller/pin/example.yaml
+13
View File
@@ -0,0 +1,13 @@
chunk_size: 256
local_cpu: True
max_local_cpu_size: 5
# cache controller configurations
enable_controller: True
lmcache_instance_id: "lmcache_default_instance"
controller_pull_url: "localhost:9001"
lmcache_worker_ports: 8001
# Peer identifiers
p2p_host: "localhost"
p2p_init_ports: 8200