chore: import upstream snapshot with attribution

doc/source/train/deepspeed.rst

@@ -0,0 +1,136 @@
+.. _train-deepspeed:
+
+Get Started with DeepSpeed
+==========================
+
+The :class:`~ray.train.torch.TorchTrainer` can help you easily launch your `DeepSpeed <https://www.deepspeed.ai/>`_ training across a distributed Ray cluster. 
+DeepSpeed is an optimization library that enables efficient large-scale model training through techniques like ZeRO (Zero Redundancy Optimizer).
+
+Benefits of Using Ray Train with DeepSpeed
+------------------------------------------
+
+- **Simplified Distributed Setup**: Ray Train handles all the distributed environment setup for you
+- **Multi-Node Scaling**: Easily scale to multiple nodes with minimal code changes
+- **Checkpoint Management**: Built-in checkpoint saving and loading across distributed workers
+- **Seamless Integration**: Works with your existing DeepSpeed code
+
+Code example
+------------
+
+You can use your existing DeepSpeed training code with Ray Train's TorchTrainer. The integration is minimal and preserves your familiar DeepSpeed workflow:
+
+.. testcode::
+    :skipif: True
+
+    import deepspeed
+    from deepspeed.accelerator import get_accelerator
+
+    def train_func():
+        # Instantiate your model and dataset
+        model = ...
+        train_dataset = ...
+        eval_dataset = ...
+        deepspeed_config = {...} # Your DeepSpeed config
+
+        # Prepare everything for distributed training
+        model, optimizer, train_dataloader, lr_scheduler = deepspeed.initialize(
+            model=model,
+            model_parameters=model.parameters(),
+            training_data=tokenized_datasets["train"],
+            collate_fn=collate_fn,
+            config=deepspeed_config,
+        )
+
+        # Define the GPU device for the current worker
+        device = get_accelerator().device_name(model.local_rank)
+
+        # Start training
+        for epoch in range(num_epochs):
+            # Training logic
+            ...
+            
+            # Report metrics to Ray Train
+            ray.train.report(metrics={"loss": loss})
+
+    from ray.train.torch import TorchTrainer
+    from ray.train import ScalingConfig
+
+    trainer = TorchTrainer(
+        train_func,
+        scaling_config=ScalingConfig(...),
+        # If running in a multi-node cluster, this is where you
+        # should configure the run's persistent storage that is accessible
+        # across all worker nodes.
+        # run_config=ray.train.RunConfig(storage_path="s3://..."),
+        ...
+    )
+    result = trainer.fit()
+
+
+Complete Examples
+-----------------
+
+Below are complete examples of ZeRO-3 training with DeepSpeed. Each example shows a full implementation of fine-tuning
+a Bidirectional Encoder Representations from Transformers (BERT) model on the Microsoft Research Paraphrase Corpus (MRPC) dataset.
+
+Install the requirements:
+
+.. code-block:: bash
+
+    pip install deepspeed torch datasets transformers torchmetrics "ray[train]"
+
+.. tab-set::
+
+    .. tab-item:: Example with Ray Data
+
+        .. dropdown:: Show Code
+
+            .. literalinclude:: /../../python/ray/train/examples/deepspeed/deepspeed_torch_trainer.py
+                :language: python
+                :start-after: __deepspeed_torch_basic_example_start__
+                :end-before: __deepspeed_torch_basic_example_end__
+
+    .. tab-item:: Example with PyTorch DataLoader
+
+        .. dropdown:: Show Code
+
+            .. literalinclude:: /../../python/ray/train/examples/deepspeed/deepspeed_torch_trainer_no_raydata.py
+                :language: python
+                :start-after: __deepspeed_torch_basic_example_no_raydata_start__
+                :end-before: __deepspeed_torch_basic_example_no_raydata_end__
+
+.. tip::
+
+    To run DeepSpeed with pure PyTorch, you **don't need to** provide any additional Ray Train utilities
+    like :meth:`~ray.train.torch.prepare_model` or :meth:`~ray.train.torch.prepare_data_loader` in your training function. Instead,
+    keep using `deepspeed.initialize() <https://deepspeed.readthedocs.io/en/latest/initialize.html>`_ as usual to prepare everything
+    for distributed training.
+
+
+Fine-tune LLMs with DeepSpeed
+-----------------------------
+
+See this step-by-step guide for how to fine-tune large language models (LLMs) with Ray Train and DeepSpeed: :doc:`Fine-tune an LLM with Ray Train and DeepSpeed </_collections/train/examples/pytorch/deepspeed_finetune/README>`.
+
+
+Run DeepSpeed with Other Frameworks
+-----------------------------------
+
+Many deep learning frameworks have integrated with DeepSpeed, including Lightning, Transformers, Accelerate, and more. You can run all these combinations in Ray Train.
+
+Check the below examples for more details:
+
+.. list-table::
+   :header-rows: 1
+
+   * - Framework
+     - Example
+   * - Accelerate (:ref:`User Guide <train-hf-accelerate>`)
+     - `Fine-tune Llama-2 series models with DeepSpeed, Accelerate, and Ray Train. <https://github.com/ray-project/ray/tree/master/doc/source/templates/04_finetuning_llms_with_deepspeed>`_
+   * - Transformers (:ref:`User Guide <train-pytorch-transformers>`)
+     - :doc:`Fine-tune GPT-J-6b with DeepSpeed and Hugging Face Transformers <examples/deepspeed/gptj_deepspeed_fine_tuning>`
+   * - Lightning (:ref:`User Guide <train-pytorch-lightning>`)
+     - :doc:`Fine-tune vicuna-13b with DeepSpeed and PyTorch Lightning <examples/lightning/vicuna_13b_lightning_deepspeed_finetune>`
+
+
+For more information about DeepSpeed configuration options, refer to the `official DeepSpeed documentation <https://www.deepspeed.ai/docs/config-json/>`_.