wehub-resource-sync
e06fe8e8c6
Secret Leaks / trufflehog (push) Failing after 1s
Build documentation / build (push) Failing after 1s
Build documentation / build_other_lang (push) Failing after 0s
CodeQL Security Analysis / CodeQL Analysis (push) Failing after 0s
PR CI / pr-ci (push) Failing after 1s
Slow tests on important models (on Push - A10) / Get all modified files (push) Failing after 1s
Slow tests on important models (on Push - A10) / Model CI (push) Has been skipped
Self-hosted runner (benchmark) / Benchmark (aws-g5-4xlarge-cache) (push) Has been cancelled
New model PR merged notification / Notify new model (push) Has been cancelled
Update Transformers metadata / build_and_package (push) Has been cancelled
2.0 KiB
2.0 KiB
Gradient checkpointing
The forward pass typically caches all intermediate activations for the backward pass to reuse. However, activations scale with batch size and sequence length. Gradient checkpointing only saves certain activations and discards the rest. This forces the backward pass to recompute some of the activations on-the-fly as they're needed.
Normal training:
Forward: [L1]→[L2]→[L3]→[L4] (save ALL activations)
Backward: ←uses cached activations everywhere
Gradient checkpointing:
Forward: [L1]→[L2]→[L3]→[L4] (save only at checkpoints, discard the rest)
Backward: ←reaches L2, recomputes L2→L3 from scratch, uses it, discards it
Training will be ~20% slower because some activations need to be recomputed, but it reduces activation memory.
Set gradient_checkpointing=True to enable.
Tip
Use with gradient accumulation to further reduce memory usage.
from transformers import TrainingArguments
args = TrainingArguments(
...,
gradient_checkpointing=True,
)
Next steps
- Read the GPU memory usage doc to understand what is driving memory usage on the GPU during training.
- See the Mixed precision training guide to learn how to use lower precision data types to further reduce memory and speed up training.
- See the Kernels guide to learn how to speed up training with custom fused kernels.