chore: import upstream snapshot with attribution

2026-07-13 12:38:16 +08:00
commit 94057c3d3e
7152 changed files with 2120455 additions and 0 deletions
@@ -0,0 +1,121 @@
+# SiMM as L3 KV Cache
+
+This document describes how to use SiMM as the L3 KV cache for SGLang.
+
+## About SiMM
+
+SiMM(Scalable In-Memory Middleware) is a distributed, high-performance, elastic cache acceleration layer for all AI workloads.
+
+For more details about SiMM, please refer to [SiMM project](https://github.com/scitix/SiMM) and [SiMM documents](https://github.com/scitix/SiMM/tree/main/docs).
+
+### SiMM & SGLang HiCache
+
+SiMM serves as a high-performance L3 storage backend for SGLang HiCache, enabling distributed KV cache storage across multiple servers with RDMA-baed transport. This integration addresses the capacity limitations of traditional GPU-only or GPU+CPU caching by providing virtually unlimited cache storage through a distributed memory pool.
+
+When a cache miss occurs in L1 and L2, HiCache automatically fetches the required KV cache from SiMM's distributed memory pool. The system uses intelligent prefetching strategies to minimize latency, and utilize RDMA technology and zero-copy technique to ensure high-bandwidth, low-latency data transfer between SGLang instances and SiMM data servers.
+
+## Install SiMM
+
+**from source**
+
+Clone SiMM project:
+
+```bash
+git clone https://github.com/scitix/SiMM --recursive
+```
+
+Install dependencies:
+
+```bash
+cd SiMM
+bash configure.sh
+```
+
+Build and install SiMM:
+
+```bash
+bash build.sh --mode=release --clean
+```
+
+For more details, please refer to [SiMM official installation guide](https://github.com/scitix/SiMM/blob/main/README.md).
+
+## Deployment
+
+**SiMM**
+
+Before launch `SGLang server` with SiMM, you should launch SiMM `cluster manager service` and `data server service`.
+
+You can visit [SiMM official deploy guide](https://github.com/scitix/SiMM/blob/main/docs/deploy_guide.md) and deploy SiMM on your K8S cluster with RDMA network.
+
+**Start the `SGLang server` with SiMM enabled:**
+
+There are three ways to configure SiMM:
+
+1. Via extra configuration passed through sglang parameters
+2. Using JSON configuration files
+3. Using environment variables
+
+SiMM loads configuration in the following priority order:
+
+1. If SiMM-specific options are provided in `--hicache-storage-backend-extra-config`, they are used first.
+2. If not, SiMM checks whether the environment variable `DEFAULT_SIMM_CONFIG_PATH_ENV` is set, and loads the JSON config file from that path.
+3. If neither of the above is provided, SiMM falls back to environment variables.
+
+**HiCache Related Parameters for SGLang Server**
+
+For a comprehensive overview of HiCache-related parameters, please refer to [this document](https://docs.sglang.io/advanced_features/hicache_design.html#related-parameters).
+
+
+Note that, for `--hicache-mem-layout {layer_first,page_first,page_first_direct}`, which specifies the memory layout for the host memory pool, `page_first` or `page_first_direct` are required if use SiMM backend.
+
+### Distributed Deployment
+
+**Using extra-config of sglang arguments to configure SiMM**
+
+```bash
+python -m sglang.launch_server \
+    --enable-hierarchical-cache \
+    --hicache-storage-backend simm \
+    --model-path [model_path] \
+    --hicache-storage-backend-extra-config '{"manager_address": "127.0.0.1:30001"}'
+```
+
+**Using JSON file to configure SiMM**
+
+SGLang server can load SiMM config from `SGLANG_HICACHE_SIMM_CONFIG_PATH`.
+
+```bash
+export SGLANG_HICACHE_SIMM_CONFIG_PATH=/sgl-workspace/sglang/benchmark/hicache/simm_config.json
+
+echo '{
+    "manager_address": "127.0.0.1:30001"
+}' > ${SGLANG_HICACHE_SIMM_CONFIG_PATH}
+
+python -m sglang.launch_server \
+    --enable-hierarchical-cache \
+    --hicache-storage-backend simm \
+    --model-path [model_path]
+```
+
+**Using env variables to configure SiMM**
+
+```bash
+SIMM_CLUSTER_MANAGER="127.0.0.1:30001"
+python -m sglang.launch_server \
+    --enable-hierarchical-cache \
+    --hicache-storage-backend simm \
+    --model-path [model_path]
+```
+
+## Test SiMM
+
+This test is intended for developers to quickly verify that the SiMM class interfaces are functioning correctly.
+
+First, start the `cluster manager service` and `data server service`. Then run the `test_hicache_simm.py`.
+
+```bash
+SIMM_CLUSTER_MANAGER="127.0.0.1:30001" \
+python3 [path of test_hicache_simm.py]
+```
+
+If all tests pass, the message "✅ All tests passed" will be printed at the end.