chore: import upstream snapshot with attribution

docs/code-docs/source/autoep.rst

@@ -0,0 +1,106 @@
+AutoEP (Automatic Expert Parallelism)
+=====================================
+
+AutoEP automatically detects MoE layers in Hugging Face models and replaces them
+with EP-enabled versions, requiring zero model code changes. It follows the
+pattern of AutoTP (Automatic Tensor Parallelism).
+
+This API is separate from the explicit ``deepspeed.moe.layer.MoE`` layer API.
+For the explicit DeepSpeed MoE layer API, see :doc:`moe`.
+
+**Built-in AutoEP presets:** ``mixtral`` (Mixtral), ``qwen3_moe`` (Qwen3-MoE),
+``qwen3_5_moe`` (Qwen3.5-MoE), ``deepseek_v2`` (DeepSeek-V2), and
+``deepseek_v3`` (DeepSeek-V3).
+
+The preset name means AutoEP knows the router, expert, and weight naming
+patterns for that model family. Running a Hugging Face model also requires a
+Transformers build that exposes the matching config/model classes,
+``model.config.model_type`` value, and fused expert layout.
+
+.. list-table:: AutoEP preset compatibility by Transformers version
+   :header-rows: 1
+
+   * - Preset
+     - Minimum Transformers version
+     - Notes
+   * - ``mixtral``
+     - ``5.0.0``
+     -
+   * - ``qwen3_moe``
+     - ``5.0.0``
+     - Also covers Qwen2-MoE when the installed Transformers build uses the
+       validated fused expert layout. Qwen3-MoE classes appear in ``4.51.3``,
+       but the tested ``4.x`` builds do not match the validated AutoEP layout.
+   * - ``qwen3_5_moe``
+     - ``5.2.0``
+     - Requires the Qwen3.5 text-backbone ``qwen3_5_moe_text`` model type;
+       for performance on Qwen3.5's Gated DeltaNet layers, install optimized
+       kernels. See the `Hugging Face Transformers kernel loading docs
+       <https://huggingface.co/docs/transformers/kernel_doc/loading_kernels>`__
+       and the `Qwen FlashQLA blog <https://qwen.ai/blog?id=flashqla>`__.
+   * - ``deepseek_v2``
+     - ``5.0.0``
+     - ``load_balance_coeff`` / expert-bias auxiliary-loss-free load balancing
+       is not currently supported; non-null values are rejected.
+   * - ``deepseek_v3``
+     - ``5.0.0``
+     - ``load_balance_coeff`` / expert-bias auxiliary-loss-free load balancing
+       is not currently supported; non-null values are rejected.
+
+**ZeRO compatibility:** Stages 0, 1, and 2, plus constrained Stage 3
+support. Stage 3 requires AutoEP-managed MoE layers and does not support native
+DeepSpeed MoE layers, AutoTP, tensor model parallelism from ``mpu``, sequence
+parallelism, MiCS, hpZeRO secondary tensor groups, non-1 expert tensor
+parallelism, or quantized gradients. Stage 3 AutoEP checkpoints are saved
+partition-natively in the ``zero_pp_rank_*`` shard files and support
+same-topology load, module-only loads (``load_module_only``),
+optimizer-state-free loads (``load_optimizer_states=False``), and Universal
+Checkpoint conversion. Optimizer-including Universal Checkpoint loads can
+resume with a different data-parallel world size, a different ``autoep_size``,
+or both, when the target ``autoep_size`` divides the model's expert count.
+Weights-only/module-only Universal Checkpoint loads use the converted
+``fp32.pt`` parameter files and support the same data-parallel and
+``autoep_size`` topology changes.
+
+**Usage:**
+
+.. code-block:: json
+
+    {
+        "expert_parallel": {
+            "enabled": true,
+            "autoep_size": 4,
+            "preset_model": "mixtral"
+        }
+    }
+
+**How it works:**
+
+1. During ``deepspeed.initialize()``, AutoEP scans the model for MoE layers
+   using preset-defined patterns (router name, expert name, weight shapes).
+2. Detected MoE blocks are replaced with ``AutoEPMoELayer``, which uses
+   TorchTitan's grouped GEMM kernels and AllToAll token dispatch.
+3. EP/EDP process groups are created automatically based on ``autoep_size``.
+4. Expert parameters are marked for expert-data-parallel gradient reduction;
+   router and shared-expert parameters use standard data-parallel reduction.
+
+**Constraints:**
+
+- ``autoep_size`` must divide ``num_experts`` for all detected MoE layers.
+- ``autoep_size=1`` is valid: all experts remain local (no AllToAll), useful
+  for functional testing on a single GPU.
+- AutoEP currently cannot be combined with AutoTP
+  (``tensor_parallel.autotp_size > 1``) or tensor model parallelism from
+  ``mpu``; support is planned as follow-up work.
+- AutoEP with ZeRO Stage 3 is supported only without sequence parallelism,
+  MiCS, hpZeRO secondary tensor groups, non-1 expert tensor parallelism, or
+  quantized gradients.
+- Regular checkpoint save/load requires matching ``autoep_size``. To change
+  ``autoep_size`` or data-parallel world size across runs for the same
+  AutoEP-detected model topology, convert the checkpoint to Universal
+  Checkpoint format and load it with ``checkpoint.load_universal``; see the
+  `Universal Checkpointing tutorial </tutorials/universal-checkpointing/>`__
+  for the detailed flow and constraints.
+- DeepSeek-V2 and DeepSeek-V3 AutoEP do not support load-balance expert bias
+  yet. The built-in DeepSeek presets disable it by default; explicit non-null
+  values fail.