huggingface--transformers/docs/source/en/model_doc/qwen3_5_moe.md

This model was contributed to Hugging Face Transformers on 2026-02-09.

Qwen3.5 MoE is the sparse-expert variant of Qwen3.5. It keeps the same natively multimodal decoder and 3:1 Gated DeltaNet/Gated Attention backbone, but replaces dense FFNs with a 256-expert sparse mixture — 8 routed experts are activated per token, plus 1 shared expert — so total parameters scale well past the dense checkpoints while active compute per token stays much smaller.

Notable checkpoints include Qwen/Qwen3.5-35B-A3B (35B total/3B active), Qwen/Qwen3.5-122B-A10B, Qwen/Qwen3.5-397B-A17B, and Qwen/Qwen3.6-35B-A3B. Qwen3.6 checkpoints share the same architecture and model_type as Qwen3.5 and are loaded with the same classes. The text tower reuses Qwen3NextSparseMoeBlock and expert kernels from Qwen3-Next; the vision tower is inherited from Qwen3-VL.

You can find all the official Qwen3.5 MoE checkpoints under the Qwen organization.

Quickstart

import torch
from transformers import pipeline

pipe = pipeline(
    task="text-generation",
    model="Qwen/Qwen3.5-35B-A3B",
    device_map="auto",
)
print(pipe("The capital of France is", max_new_tokens=20)[0]["generated_text"])

import torch
from transformers import AutoTokenizer, Qwen3_5MoeForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-35B-A3B")
model = Qwen3_5MoeForCausalLM.from_pretrained(
    "Qwen/Qwen3.5-35B-A3B",
    device_map="auto",
)

inputs = tokenizer("Explain mixture-of-experts in one paragraph.", return_tensors="pt").to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(generated_ids[0], skip_special_tokens=True))

Usage tips and notes

• When training or fine-tuning, set output_router_logits=True so the forward returns router logits and the load-balancing auxiliary loss is added to the total loss (scaled by router_aux_loss_coef, default 0.001). Without it, experts can collapse to a few popular slots.

• [Qwen3_5MoeCausalLMOutputWithPast] includes a router_logits field. Downstream code that destructures model outputs by position needs to account for it or switch to keyword access.

• For Qwen3.5-35B-A3B, the text config uses hidden_size=2048 across 40 layers, 256 experts with 8 routed + 1 shared per token, and moe_intermediate_size=512 — very different shapes from the dense Qwen3.5 checkpoints, so weights are not interchangeable.

• Native context is 262,144 tokens. To reach the advertised ~1M context, enable YaRN rope scaling via the config's rope_scaling field — plain loading gives you the native window only.

• As with Qwen3.5, linear-attention layers depend on optional causal_conv1d (from Dao-AILab). Without it, the model silently falls back to slower and more memory hungry PyTorch ops.

• On NVIDIA GB10 (compute capability 12.1 / SM121) causal_conv1d and fla have no SM121 build, so the Gated DeltaNet path always uses the slow PyTorch reference. Passing use_kernels=True (pip install -U kernels) to [~PreTrainedModel.from_pretrained] swaps it for the same compute-capability-gated Hub kernel as the dense variant (Atlas-Inference/gdn, shared because Qwen3_5MoeGatedDeltaNet has the same core as Qwen3_5GatedDeltaNet); every other GPU keeps the existing path. The kernel is numerically faithful to the fallback (identical greedy output) and speeds up prefill. Measured on Qwen/Qwen3.6-35B-A3B (bf16, GB10/SM121, 1024-token prompt, greedy decode of 256 tokens):

  use_kernels	TTFT (prefill)	Decode
  False (PyTorch fallback)	0.73 s	16.3 tok/s
  True (Atlas-Inference/gdn)	0.53 s (1.38x faster)	16.7 tok/s

  Decode is roughly flat because the single-token DeltaNet recurrence is memory-bandwidth-bound; the win is on the chunked-prefill core and grows with prompt length. Loading the mapped kernel currently requires trust_remote_code=True until Atlas-Inference is added to the trusted-kernels allowlist.

Qwen3_5MoeConfig

autodoc Qwen3_5MoeConfig

Qwen3_5MoeTextConfig

autodoc Qwen3_5MoeTextConfig

Qwen3_5MoeVisionConfig

autodoc Qwen3_5MoeVisionConfig

Qwen3_5MoeVisionModel

autodoc Qwen3_5MoeVisionModel - forward

Qwen3_5MoeTextModel

autodoc Qwen3_5MoeTextModel - forward

Qwen3_5MoeModel

autodoc Qwen3_5MoeModel - forward

Qwen3_5MoeForCausalLM

autodoc Qwen3_5MoeForCausalLM - forward

Qwen3_5MoeForConditionalGeneration

autodoc Qwen3_5MoeForConditionalGeneration - forward