chore: import upstream snapshot with attribution

2026-07-13 13:18:33 +08:00
commit 4ececc111a
2017 changed files with 331736 additions and 0 deletions
@@ -0,0 +1,69 @@
+---
+title: "ZenFlow"
+tags: training
+---
+
+ZenFlow is an extension of ZeRO-Offload that decouples and asynchronously updates gradients during training. It reduces CPU-induced stalls when using offload optimizers, enabling smoother and faster training. Like ZeRO-Offload, ZenFlow requires no code changes, only configuration updates in your DeepSpeed JSON file.
+
+We recommend that you read the tutorials on [Getting Started](/getting-started/) and [ZeRO](/tutorials/zero/) before stepping through this tutorial. ZenFlow builds on top of [ZeRO-Offload](/tutorials/zero-offload/), so shared setup details can be found there.
+
+## Configuration Changes
+
+To enable ZenFlow, simply add a `zenflow` section under the existing `zero_optimization` block in your DeepSpeed config:
+
+```json
+{
+  "zero_optimization": {
+    "stage": 2,
+    "offload_optimizer": {
+      "device": "cpu",
+      "pin_memory": true
+    },
+    "zenflow": {
+      "topk_ratio": 0.05,
+      "select_strategy": "auto",
+      "select_interval": "auto",
+      "update_interval": 4,
+      "full_warm_up_rounds": 0,
+      "overlap_step": true
+    }
+  }
+}
+```
+
+
+Each field in the `zenflow` block controls selective gradient update behavior:
+
+- `topk_ratio`: Fraction of the most important gradients to update on GPU (e.g., 0.05 means top 5% by importance).
+- `select_strategy`: Strategy for selecting important gradients (`"auto"`, `"step"`, or custom).
+- `select_interval`: How often to re-select important gradients (`"auto"` or integer like 1).
+- `update_interval`: How often to update unimportant gradients (`"auto"` or an integer like 4, meaning every 4 steps).
+- `full_warm_up_rounds`: Number of initial steps with full gradient updates before selection begins.
+- `overlap_step`: Whether to overlap communication with computation (`true` enables it).
+
+---
+
+**Recommended**: Use `"auto"` for `select_strategy`, `select_interval`, and `update_interval` to enable adaptive behavior with minimal tuning.
+
+You can continue using the same training setup and launch script as in the [ZeRO-Offload tutorial](/tutorials/zero-offload/), since ZenFlow builds directly on top of ZeRO Offload.
+
+## Quick Start: Fine-tuning Example
+
+A complete fine-tuning example using ZenFlow is available in [DeepSpeedExamples](https://github.com/microsoft/DeepSpeedExamples) -- [ZenFlow Fine-Tuning on GLUE](https://github.com/deepspeedai/DeepSpeedExamples/tree/master/training/DeepSpeed-ZenFlow)
+
+This example shows how to fine-tune a GPT model on the GLUE benchmark with:
+- CPU optimizer offload
+- ZenFlow asynchronous updates
+
+To run the example:
+
+```bash
+cd DeepSpeedExamples/training/DeepSpeed-ZenFlow
+bash finetune_gpt_glue.sh
+```
+
+Refer to the `README.md` in the folder for setup instructions, dataset preparation, and configuration details.
+
+---
+
+Congratulations! You have successfully enabled ZenFlow for stall-free offloading.