Hybrid Attention Models ======================= Some models interleave more than one attention type across their layers — most commonly **sliding-window attention** on most layers and **full attention** on a few. vLLM serves these with its *hybrid KV cache manager*, which splits the model's layers into multiple **KV cache groups** (one per attention behavior). The LMCache multiprocess connector (``LMCacheMPConnector``) supports these hybrid models: it stores and retrieves the KV cache for every group, so prefix caching and KV reuse work the same way they do for plain models. .. contents:: :local: :depth: 2 Validated hybrid models ----------------------- Recipe pages for the validated hybrid-attention architectures: .. list-table:: :header-rows: 1 :widths: 34 34 32 * - Model - Attention layout - Recipe * - Gemma 3 - Sliding-window + full - :doc:`/recipes/gemma3` * - Gemma 4 - Sliding-window + full - :doc:`/recipes/gemma4` * - gpt-oss - Sliding-window + full - :doc:`/recipes/gpt_oss` * - Qwen3.5 / Qwen3.6 series - Mamba / GDN + full - :doc:`/recipes/qwen3_5` * - DeepSeek-V4-Flash - Sparse-MLA (multiple KV groups) - :doc:`/recipes/deepseek_v4_flash` * - GLM 5.1/5.2 - Dynamic Sparse Attention (multiple KV groups) - :doc:`/recipes/glm5_2` * - MiniMax-M3 - Sparse attention + lightning indexer (mixed KV formats in one group) - :doc:`/recipes/minimax_m3` .. toctree:: :hidden: :maxdepth: 1 /recipes/gemma3 /recipes/gemma4 /recipes/gpt_oss /recipes/qwen3_5 /recipes/deepseek_v4_flash /recipes/glm5_2 /recipes/minimax_m3 What Works ---------- Models whose layers all use **standard paged attention** — including hybrids that mix sliding-window and full attention — are supported with no special configuration. Examples: .. list-table:: :header-rows: 1 :widths: 35 30 35 * - Model family - Attention layout - Status * - Gemma 2 / Gemma 3 - Interleaved sliding-window + full - Supported * - gpt-oss - Interleaved sliding-window + full - Supported * - Qwen3.5 (and other Gated-DeltaNet hybrids) - Interleaved Mamba/GDN + full - Supported (see below) * - Llama, Qwen2/Qwen3 (dense), Mistral, … - Single attention type - Supported Just point vLLM at the LMCache server as usual (see :doc:`/getting_started/quickstart`); LMCache detects the model's KV cache groups automatically at registration time. .. note:: Because ``LMCacheMPConnector`` advertises hybrid support to vLLM, vLLM keeps its hybrid KV cache manager **enabled** for these models (it does not fall back to a single unified group). You do not need ``--no-disable-hybrid-kv-cache-manager`` or any related flag. Object-group separation ----------------------- At KV-cache registration LMCache buckets a hybrid model's layers into **object groups** — the unit it stores and retrieves as one object. By default (``--separate-object-groups``, on) each distinct cross-chunk attention window becomes its own object group: full-attention layers form one group, and each sliding-window size (mamba / GDN included) forms another. Pass ``--no-separate-object-groups`` to keep every layer in a single full-attention object group instead (the previous behavior). .. code-block:: bash # default: one object group per attention window lmcache server --chunk-size 256 --l1-size-gb 100 # opt out: a single full-attention object group for all layers lmcache server --chunk-size 256 --l1-size-gb 100 --no-separate-object-groups The flag is transparent to correctness — prefix caching and KV reuse behave the same either way, and a non-hybrid model (a single attention behavior) always resolves to one object group regardless of the setting. Separation organizes storage by attention window so that each group's cross-chunk window is tracked independently. Mamba / Linear-Attention Hybrids -------------------------------- Models that interleave **Mamba / Gated-DeltaNet (GDN) linear-attention layers** with full attention — the Qwen3.5 and Qwen3.6 series (``Qwen/Qwen3.5-0.8B``, ``Qwen/Qwen3.6-27B``, …), Qwen3-Next, and other GDN hybrids — are supported. Unlike a paged key/value cache, their linear-attention layers keep a recurrent **state cache** (a convolution + SSM state). LMCache reinterprets that state as an opaque page at registration time, so prefix caching and KV reuse work end to end without any model-specific transfer code. This section is the **general procedure for any such model**. The only per-model variable is the *unified block size* ``N`` (step 1); everything else is identical across models. .. _mamba-block-size: Step 1 — find the model's unified block size ``N`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``N`` is the **single number** that drives every other setting: the LMCache server's ``--chunk-size`` and vLLM's ``--max-num-batched-tokens`` are both derived from it (step 2). Get it wrong and LMCache raises at engine startup. For a Mamba / GDN hybrid, vLLM forces **one** block size across all KV cache groups, chosen large enough that an attention page is at least as big as a Mamba state page. It depends on the model's head dimensions and GDN state size, so it is **model-specific — never assume a value, read it from the model**. vLLM prints it once at startup:: INFO ... interface.py:670] Setting attention block size to 784 tokens to ensure that attention page size is >= mamba page size. You do not need LMCache, a full serving run, or the weights to be quantized to read it — just launch vLLM until the line appears, then stop. The snippet below does exactly that and prints ``N``: .. code-block:: bash MODEL=Qwen/Qwen3.6-27B LOG=$(mktemp) # Launch vLLM just far enough to size the KV cache; cheap settings only. vllm serve "$MODEL" \ --mamba-cache-mode align --enable-prefix-caching \ --max-model-len 8192 --gpu-memory-utilization 0.5 \ --port 8011 > "$LOG" 2>&1 & VLLM_PID=$! # Wait for the block-size line (or a fatal error), then stop vLLM. until grep -qiE "Setting attention block size|Error|Traceback" "$LOG"; do sleep 3 done grep -i "Setting attention block size" "$LOG" kill "$VLLM_PID" The number in ``to N tokens`` is your ``N``. Values grow with model size; for example: .. list-table:: :header-rows: 1 :widths: 50 25 25 * - Model - Unified block size ``N`` - GPUs * - ``Qwen/Qwen3.6-27B`` - 784 - 1 * - ``Qwen/Qwen3.5-0.8B`` - 544 - 1 Step 2 — derive the three required flags from ``N`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ #. **LMCache server** ``--chunk-size`` **= N** (or any multiple of ``N``). This is the rule the connector enforces: LMCache's chunk size must be a multiple of vLLM's unified block size, or registration fails:: lmcache server --chunk-size 784 --l1-size-gb 100 --eviction-policy LRU #. **vLLM** ``--max-num-batched-tokens`` **in [N, 2·N)** — setting it equal to ``N`` is the simple, always-valid choice. Outside this range LMCache raises at engine startup. ``align`` mode snapshots the Mamba state only at the *end* of each scheduler step, so each prefill step must advance exactly one block; a larger budget would let a step skip block boundaries, leaving no snapshot for LMCache to store at those prefixes. #. **vLLM** ``--mamba-cache-mode align --enable-prefix-caching`` — ``align`` is mandatory (GDN backends do not support the ``all`` mode):: vllm serve \ --enable-prefix-caching --mamba-cache-mode align \ --max-num-batched-tokens 784 \ --kv-transfer-config \ '{"kv_connector":"LMCacheMPConnector", "kv_role":"kv_both"}' So for a freshly-probed model the whole derivation is just: read ``N`` (step 1), then pass ``--chunk-size N`` to the server and ``--max-num-batched-tokens N`` to vLLM. No ``--no-disable-hybrid-kv-cache-manager`` or attention-backend flag is needed; ``LMCacheMPConnector`` advertises hybrid support and vLLM auto-selects the GDN backend. Caveats ^^^^^^^ - Generation is **not bit-exact** between a cached and a fresh run: GDN backends do not support vLLM's batch-invariant mode. Validate with a **score-level** comparison (see `Verifying Correctness`_), not a token-level diff. - The cached pages are **byte-opaque**, so content-aware features (CacheGen compression, CacheBlend) do not apply, and cache entries must not be shared across engines with different attention backends or kernel block sizes. - Several of these models are **vision-language** (they load a vision tower). The validated, supported path is **text** KV caching; image/video KV caching is not validated. - vLLM's Mamba prefix caching in ``align`` mode is marked experimental upstream. See the :doc:`Qwen3.5 / Qwen3.6 recipe <../recipes/qwen3_5>` for the validated end-to-end commands and the per-model block sizes. What Is Not Supported Yet ------------------------- - **DeepSeek-V4-style compressed / indexer caches** are not yet handled by the multiprocess connector. Verifying Correctness --------------------- To convince yourself that a hybrid model's KV is being cached and reused correctly, you can compare a cold run against a run served from LMCache: #. Run an evaluation (e.g. ``lm_eval`` on ``gsm8k``) against vLLM + LMCache. This computes the KV cache and **stores** it in LMCache. #. Reset *only* vLLM's local prefix cache, leaving the LMCache-managed cache intact (requires launching vLLM with ``VLLM_SERVER_DEV_MODE=1``):: curl -X POST http://localhost:8000/reset_prefix_cache Omit the ``reset_external=true`` query parameter so the LMCache cache is preserved. #. Re-run the same evaluation. vLLM now misses in its local cache, so the prefix KV is **retrieved** from LMCache. The score should match the first run. The project ships this as the ``hma_lm_eval`` continuous-integration test (see ``.buildkite/k3_tests/multiprocess``). See Also -------- - :doc:`/getting_started/quickstart` — launching the LMCache server and a vLLM client. - Design notes on how groups are detected and addressed: ``docs/design/integration/vllm/hybrid-kv-cache-groups.md`` in the source tree.