wehub-resource-sync
108 lines
3.4 KiB
ReStructuredText
108 lines
3.4 KiB
ReStructuredText
.. _recipe_deepseek_v4_flash:
|
|
|
|
DeepSeek-V4-Flash
|
|
=================
|
|
|
|
Validated models
|
|
----------------
|
|
|
|
- `deepseek-ai/DeepSeek-V4-Flash <https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash>`_
|
|
|
|
.. tab-set::
|
|
:sync-group: engine
|
|
|
|
.. tab-item:: vLLM
|
|
|
|
**Engine documentation:**
|
|
`DeepSeek-V4-Flash in vLLM supported models
|
|
<https://docs.vllm.ai/en/latest/models/supported_models.html#text-generation>`_
|
|
(architecture ``DeepseekV4ForCausalLM``).
|
|
|
|
**Status:** Validated with LMCache.
|
|
|
|
**Installing vLLM:** DeepSeek-V4-Flash needs the sparse-MLA attention
|
|
backends and the ``fp8_ds_mla`` KV cache kernels, so install vLLM by
|
|
following its own recipe rather than a bare ``pip install vllm``:
|
|
`vLLM DeepSeek-V4-Flash recipe
|
|
<https://docs.vllm.ai/projects/recipes/en/latest/index.html>`_
|
|
(also mirrored at https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash).
|
|
|
|
.. warning::
|
|
|
|
Use the **latest vLLM release**, not the ``main``/dev branch. The
|
|
current vLLM development branch is broken for DeepSeek-V4-Flash (the
|
|
``fp4`` MoE experts are misdispatched and the real weights fail to
|
|
load). Pin to the latest tagged release as the vLLM recipe instructs.
|
|
|
|
Start the LMCache MP server:
|
|
|
|
.. code-block:: bash
|
|
|
|
lmcache server --l1-size-gb 100 --eviction-policy LRU
|
|
|
|
|
|
|
|
|
Start vLLM with the LMCache MP connector (8 GPUs):
|
|
|
|
.. code-block:: bash
|
|
|
|
vllm serve deepseek-ai/DeepSeek-V4-Flash \
|
|
--tensor-parallel-size 8 \
|
|
--enable-expert-parallel \
|
|
--kv-cache-dtype fp8_ds_mla \
|
|
--trust-remote-code \
|
|
--tokenizer-mode deepseek_v4 \
|
|
--kv-transfer-config \
|
|
'{"kv_connector":"LMCacheMPConnector", "kv_role":"kv_both"}'
|
|
|
|
|
|
|
|
|
``--kv-cache-dtype fp8_ds_mla`` and ``--tokenizer-mode deepseek_v4`` are
|
|
required for this model; ``--enable-expert-parallel`` distributes the MoE
|
|
experts across the tensor-parallel ranks. Adjust
|
|
``--tensor-parallel-size`` to match your hardware. For the generic
|
|
LMCache + vLLM wiring (ports, remote hosts), see
|
|
:doc:`../getting_started/quickstart`.
|
|
|
|
If there are any issues with vLLM setup, please refer to the
|
|
`vLLM Recipes <https://docs.vllm.ai/projects/recipes/en/latest/index.html>`_
|
|
for more details.
|
|
|
|
.. tab-item:: SGLang
|
|
|
|
**Status:** Not validated with LMCache.
|
|
|
|
.. tab-item:: TRT-LLM
|
|
|
|
**Status:** Supported. See :doc:`../getting_started/quickstart` for TRT-LLM + LMCache setup.
|
|
|
|
CacheBlend support
|
|
------------------
|
|
|
|
Compression support
|
|
-------------------
|
|
|
|
.. list-table::
|
|
:header-rows: 1
|
|
:widths: 25 20 55
|
|
|
|
* - Method
|
|
- Status
|
|
- Notes
|
|
* - :doc:`CacheGen <../kv_cache_optimizations/compression/cachegen>`
|
|
- Not validated
|
|
-
|
|
|
|
Caveats
|
|
-------
|
|
|
|
- **Requires the latest vLLM release.** The vLLM dev branch is currently broken
|
|
for this model (see the warning above) -- use a tagged release installed via
|
|
the vLLM recipe.
|
|
- **Sparse-MLA hybrid KV cache.** DeepSeek-V4-Flash interleaves several KV
|
|
cache groups with different block geometries (the compressed MLA latents are
|
|
stored as ``fp8``/``uint8`` while the sparse-attention indexer groups are
|
|
``float32``), so the groups do not share a single block size. LMCache stores
|
|
and retrieves each group in its own block size; no extra flags are required
|
|
beyond ``--kv-cache-dtype fp8_ds_mla``.
|