wehub-resource-sync
106 lines
3.2 KiB
ReStructuredText
106 lines
3.2 KiB
ReStructuredText
.. _recipe_glm5_2:
|
|
|
|
GLM 5.1/5.2
|
|
===========
|
|
|
|
A large Mixture-of-Experts model using **Dynamic Sparse Attention (DSA)**, shared
|
|
by the **GLM-5.1 / GLM-5.2** series. Like DeepSeek-V4-Flash, the sparse-attention path
|
|
splits the model's layers into more than one KV cache group; the
|
|
``LMCacheMPConnector`` stores and retrieves each group in its own block size, so
|
|
KV reuse works without extra flags.
|
|
|
|
Validated models
|
|
----------------
|
|
|
|
- `zai-org/GLM-5.2-FP8 <https://huggingface.co/zai-org/GLM-5.2-FP8>`_ (8 GPUs)
|
|
|
|
.. tab-set::
|
|
:sync-group: engine
|
|
|
|
.. tab-item:: vLLM
|
|
|
|
**Engine documentation:**
|
|
`GLM-5.2 in vLLM supported models
|
|
<https://docs.vllm.ai/en/latest/models/supported_models.html#text-generation>`_
|
|
(architecture ``GlmMoeDsaForCausalLM``). See also the
|
|
`vLLM GLM-5.2 recipe <https://recipes.vllm.ai/zai-org/GLM-5.2>`_.
|
|
|
|
**Status:** Validated with LMCache (vLLM 0.23.0 + LMCache 0.4.7).
|
|
|
|
Start the LMCache MP server:
|
|
|
|
.. code-block:: bash
|
|
|
|
lmcache server \
|
|
--port 6555 \
|
|
--max-workers 8 \
|
|
--l1-size-gb 100 \
|
|
--eviction-policy LRU \
|
|
--chunk-size 1024
|
|
|
|
|
|
|
|
|
Start vLLM with the LMCache MP connector (8 GPUs):
|
|
|
|
.. code-block:: bash
|
|
|
|
vllm serve zai-org/GLM-5.2-FP8 \
|
|
--tensor-parallel-size 8 \
|
|
--tool-call-parser glm47 \
|
|
--enable-auto-tool-choice \
|
|
--reasoning-parser glm45 \
|
|
--no-enable-prefix-caching \
|
|
--kv-transfer-config \
|
|
'{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both","kv_connector_extra_config":{"lmcache.mp.port":6555}}'
|
|
|
|
|
|
|
|
|
``--tool-call-parser glm47``, ``--enable-auto-tool-choice``, and
|
|
``--reasoning-parser glm45`` are GLM-5.2's serving requirements (see the
|
|
vLLM recipe). ``--no-enable-prefix-caching`` routes all KV reuse through
|
|
LMCache rather than vLLM's in-engine prefix cache. The server's
|
|
``--port 6555`` must match ``lmcache.mp.port`` in the connector config;
|
|
``--max-workers`` is set to the tensor-parallel size. Adjust
|
|
``--tensor-parallel-size`` to match your hardware. For the generic
|
|
LMCache + vLLM wiring (ports, remote hosts), see
|
|
:doc:`../getting_started/quickstart`.
|
|
|
|
If there are any issues with vLLM setup, please refer to the
|
|
`vLLM Recipes <https://docs.vllm.ai/projects/recipes/en/latest/index.html>`_
|
|
for more details.
|
|
|
|
.. tab-item:: SGLang
|
|
|
|
**Status:** Not validated with LMCache.
|
|
|
|
.. tab-item:: TRT-LLM
|
|
|
|
**Status:** Supported. See :doc:`../getting_started/quickstart` for TRT-LLM + LMCache setup.
|
|
|
|
CacheBlend support
|
|
------------------
|
|
|
|
Not validated.
|
|
|
|
Compression support
|
|
-------------------
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
:widths: 25 20 55
|
|
|
|
* - Method
|
|
- Status
|
|
- Notes
|
|
* - :doc:`CacheGen <../kv_cache_optimizations/compression/cachegen>`
|
|
- Not validated
|
|
-
|
|
|
|
Caveats
|
|
-------
|
|
|
|
- **Dynamic Sparse Attention KV groups.** GLM-5.2's DSA path splits the model's
|
|
layers into more than one KV cache group with different block geometries.
|
|
LMCache stores and retrieves each group in its own block size; no extra flags
|
|
are required beyond the launch commands above.
|