wehub-resource-sync
296 lines
12 KiB
ReStructuredText
296 lines
12 KiB
ReStructuredText
.. _tune-distributed-ref:
|
|
|
|
Running Distributed Experiments with Ray Tune
|
|
==============================================
|
|
|
|
Tune is commonly used for large-scale distributed hyperparameter optimization. This page will overview how to setup and launch a distributed experiment along with :ref:`commonly used commands <tune-distributed-common>` for Tune when running distributed experiments.
|
|
|
|
.. contents::
|
|
:local:
|
|
:backlinks: none
|
|
|
|
Summary
|
|
-------
|
|
|
|
To run a distributed experiment with Tune, you need to:
|
|
|
|
1. First, :ref:`start a Ray cluster <cluster-index>` if you have not already.
|
|
2. Run the script on the head node, or use :ref:`ray submit <ray-submit-doc>`, or use :ref:`Ray Job Submission <jobs-overview>`.
|
|
|
|
.. tune-distributed-cloud:
|
|
|
|
Example: Distributed Tune on AWS VMs
|
|
------------------------------------
|
|
|
|
Follow the instructions below to launch nodes on AWS (using the Deep Learning AMI). See the :ref:`cluster setup documentation <cluster-index>`. Save the below cluster configuration (``tune-default.yaml``):
|
|
|
|
.. literalinclude:: /../../python/ray/tune/examples/tune-default.yaml
|
|
:language: yaml
|
|
:name: tune-default.yaml
|
|
|
|
``ray up`` starts Ray on the cluster of nodes.
|
|
|
|
.. code-block:: bash
|
|
|
|
ray up tune-default.yaml
|
|
|
|
``ray submit --start`` starts a cluster as specified by the given cluster configuration YAML file, uploads ``tune_script.py`` to the cluster, and runs ``python tune_script.py [args]``.
|
|
|
|
.. code-block:: bash
|
|
|
|
ray submit tune-default.yaml tune_script.py --start -- --ray-address=localhost:6379
|
|
|
|
.. image:: /images/tune-upload.png
|
|
:scale: 50%
|
|
:align: center
|
|
|
|
Analyze your results on TensorBoard by starting TensorBoard on the remote head machine.
|
|
|
|
.. code-block:: bash
|
|
|
|
# Go to http://localhost:6006 to access TensorBoard.
|
|
ray exec tune-default.yaml 'tensorboard --logdir=~/ray_results/ --port 6006' --port-forward 6006
|
|
|
|
|
|
Note that you can customize the directory of results by specifying: ``RunConfig(storage_path=..)``, taken in by ``Tuner``. You can then point TensorBoard to that directory to visualize results. You can also use `awless <https://github.com/wallix/awless>`_ for easy cluster management on AWS.
|
|
|
|
|
|
Running a Distributed Tune Experiment
|
|
-------------------------------------
|
|
|
|
Running a distributed (multi-node) experiment requires Ray to be started already.
|
|
You can do this on local machines or on the cloud.
|
|
|
|
Across your machines, Tune will automatically detect the number of GPUs and CPUs without you needing to manage ``CUDA_VISIBLE_DEVICES``.
|
|
|
|
To execute a distributed experiment, call ``ray.init(address=XXX)`` before ``Tuner.fit()``, where ``XXX`` is the Ray address, which defaults to ``localhost:6379``. The Tune python script should be executed only on the head node of the Ray cluster.
|
|
|
|
One common approach to modifying an existing Tune experiment to go distributed is to set an ``argparse`` variable so that toggling between distributed and single-node is seamless.
|
|
|
|
.. code-block:: python
|
|
|
|
import ray
|
|
import argparse
|
|
|
|
parser = argparse.ArgumentParser()
|
|
parser.add_argument("--address")
|
|
args = parser.parse_args()
|
|
ray.init(address=args.address)
|
|
|
|
tuner = tune.Tuner(...)
|
|
tuner.fit()
|
|
|
|
.. code-block:: bash
|
|
|
|
# On the head node, connect to an existing ray cluster
|
|
$ python tune_script.py --ray-address=localhost:XXXX
|
|
|
|
If you used a cluster configuration (starting a cluster with ``ray up`` or ``ray submit --start``), use:
|
|
|
|
.. code-block:: bash
|
|
|
|
ray submit tune-default.yaml tune_script.py -- --ray-address=localhost:6379
|
|
|
|
.. tip::
|
|
|
|
1. In the examples, the Ray address commonly used is ``localhost:6379``.
|
|
2. If the Ray cluster is already started, you should not need to run anything on the worker nodes.
|
|
|
|
|
|
Storage Options in a Distributed Tune Run
|
|
-----------------------------------------
|
|
|
|
In a distributed experiment, you should try to use :ref:`cloud checkpointing <tune-cloud-checkpointing>` to
|
|
reduce synchronization overhead. For this, you just have to specify a remote ``storage_path`` in the
|
|
:class:`RunConfig <ray.tune.RunConfig>`.
|
|
|
|
`my_trainable` is a user-defined :ref:`Tune Trainable <tune_60_seconds_trainables>` in the following example:
|
|
|
|
.. code-block:: python
|
|
|
|
from ray import tune
|
|
from my_module import my_trainable
|
|
|
|
tuner = tune.Tuner(
|
|
my_trainable,
|
|
run_config=tune.RunConfig(
|
|
name="experiment_name",
|
|
storage_path="s3://bucket-name/sub-path/",
|
|
)
|
|
)
|
|
tuner.fit()
|
|
|
|
For more details or customization, see our
|
|
:ref:`guide on configuring storage in a distributed Tune experiment <tune-storage-options>`.
|
|
|
|
|
|
.. _tune-distributed-spot:
|
|
|
|
Tune Runs on preemptible instances
|
|
-----------------------------------
|
|
|
|
Running on spot instances (or preemptible instances) can reduce the cost of your experiment.
|
|
You can enable spot instances in AWS via the following configuration modification:
|
|
|
|
.. code-block:: yaml
|
|
|
|
# Provider-specific config for worker nodes, e.g. instance type.
|
|
worker_nodes:
|
|
InstanceType: m5.large
|
|
ImageId: ami-0b294f219d14e6a82 # Deep Learning AMI (Ubuntu) Version 21.0
|
|
|
|
# Run workers on spot by default. Comment this out to use on-demand.
|
|
InstanceMarketOptions:
|
|
MarketType: spot
|
|
SpotOptions:
|
|
MaxPrice: 1.0 # Max Hourly Price
|
|
|
|
In GCP, you can use the following configuration modification:
|
|
|
|
.. code-block:: yaml
|
|
|
|
worker_nodes:
|
|
machineType: n1-standard-2
|
|
disks:
|
|
- boot: true
|
|
autoDelete: true
|
|
type: PERSISTENT
|
|
initializeParams:
|
|
diskSizeGb: 50
|
|
# See https://cloud.google.com/compute/docs/images for more images
|
|
sourceImage: projects/deeplearning-platform-release/global/images/family/tf-1-13-cpu
|
|
|
|
# Run workers on preemtible instances.
|
|
scheduling:
|
|
- preemptible: true
|
|
|
|
Spot instances may be pre-empted suddenly while trials are still running.
|
|
Tune allows you to mitigate the effects of this by preserving the progress of your model training through
|
|
:ref:`checkpointing <tune-trial-checkpoint>`.
|
|
|
|
.. literalinclude:: /../../python/ray/tune/tests/tutorial.py
|
|
:language: python
|
|
:start-after: __trainable_run_begin__
|
|
:end-before: __trainable_run_end__
|
|
|
|
|
|
Example for Using Tune with Spot instances (AWS)
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Here is an example for running Tune on spot instances. This assumes your AWS credentials have already been setup (``aws configure``):
|
|
|
|
1. Download a full example Tune experiment script here. This includes a Trainable with checkpointing: :download:`mnist_pytorch_trainable.py </../../python/ray/tune/examples/mnist_pytorch_trainable.py>`. To run this example, you will need to install the following:
|
|
|
|
.. code-block:: bash
|
|
|
|
$ pip install ray torch torchvision filelock
|
|
|
|
2. Download an example cluster yaml here: :download:`tune-default.yaml </../../python/ray/tune/examples/tune-default.yaml>`
|
|
3. Run ``ray submit`` as below to run Tune across them. Append ``[--start]`` if the cluster is not up yet. Append ``[--stop]`` to automatically shutdown your nodes after running.
|
|
|
|
.. code-block:: bash
|
|
|
|
ray submit tune-default.yaml mnist_pytorch_trainable.py --start -- --ray-address=localhost:6379
|
|
|
|
|
|
4. Optionally for testing on AWS or GCP, you can use the following to kill a random worker node after all the worker nodes are up
|
|
|
|
.. code-block:: bash
|
|
|
|
$ ray kill-random-node tune-default.yaml --hard
|
|
|
|
To summarize, here are the commands to run:
|
|
|
|
.. code-block:: bash
|
|
|
|
wget https://raw.githubusercontent.com/ray-project/ray/master/python/ray/tune/examples/mnist_pytorch_trainable.py
|
|
wget https://raw.githubusercontent.com/ray-project/ray/master/python/ray/tune/tune-default.yaml
|
|
ray submit tune-default.yaml mnist_pytorch_trainable.py --start -- --ray-address=localhost:6379
|
|
|
|
# wait a while until after all nodes have started
|
|
ray kill-random-node tune-default.yaml --hard
|
|
|
|
You should see Tune eventually continue the trials on a different worker node. See the :ref:`Fault Tolerance <tune-fault-tol>` section for more details.
|
|
|
|
You can also specify ``storage_path=...``, as part of ``RunConfig``, which is taken in by ``Tuner``, to upload results to cloud storage like S3, allowing you to persist results in case you want to start and stop your cluster automatically.
|
|
|
|
.. _tune-fault-tol:
|
|
|
|
Fault Tolerance of Tune Runs
|
|
----------------------------
|
|
|
|
Tune automatically restarts trials in the case of trial failures (if ``max_failures != 0``),
|
|
both in the single node and distributed setting.
|
|
|
|
For example, let's say a node is pre-empted or crashes while a trial is still executing on that node.
|
|
Assuming that a checkpoint for this trial exists (and in the distributed setting,
|
|
:ref:`some form of persistent storage is configured to access the trial's checkpoint <tune-storage-options>`),
|
|
Tune waits until available resources are available to begin executing the trial again from where it left off.
|
|
If no checkpoint is found, the trial will restart from scratch.
|
|
See :ref:`here for information on checkpointing <tune-trial-checkpoint>`.
|
|
|
|
|
|
If the trial or actor is then placed on a different node, Tune automatically pushes the previous checkpoint file
|
|
to that node and restores the remote trial actor state, allowing the trial to resume from the latest checkpoint
|
|
even after failure.
|
|
|
|
Recovering From Failures
|
|
~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Tune automatically persists the progress of your entire experiment (a ``Tuner.fit()`` session), so if an experiment crashes or is otherwise cancelled, it can be resumed through :meth:`~ray.tune.Tuner.restore`.
|
|
|
|
.. _tune-distributed-common:
|
|
|
|
Common Tune Commands
|
|
--------------------
|
|
|
|
Below are some commonly used commands for submitting experiments. Please see the :ref:`Clusters page <cluster-index>` to see find more comprehensive documentation of commands.
|
|
|
|
.. code-block:: bash
|
|
|
|
# Upload `tune_experiment.py` from your local machine onto the cluster. Then,
|
|
# run `python tune_experiment.py --address=localhost:6379` on the remote machine.
|
|
$ ray submit CLUSTER.YAML tune_experiment.py -- --address=localhost:6379
|
|
|
|
# Start a cluster and run an experiment in a detached tmux session,
|
|
# and shut down the cluster as soon as the experiment completes.
|
|
# In `tune_experiment.py`, set `RunConfig(storage_path="s3://...")`
|
|
# to persist results
|
|
$ ray submit CLUSTER.YAML --tmux --start --stop tune_experiment.py -- --address=localhost:6379
|
|
|
|
# To start or update your cluster:
|
|
$ ray up CLUSTER.YAML [-y]
|
|
|
|
# Shut-down all instances of your cluster:
|
|
$ ray down CLUSTER.YAML [-y]
|
|
|
|
# Run TensorBoard and forward the port to your own machine.
|
|
$ ray exec CLUSTER.YAML 'tensorboard --logdir ~/ray_results/ --port 6006' --port-forward 6006
|
|
|
|
# Run Jupyter Lab and forward the port to your own machine.
|
|
$ ray exec CLUSTER.YAML 'jupyter lab --port 6006' --port-forward 6006
|
|
|
|
# Get a summary of all the experiments and trials that have executed so far.
|
|
$ ray exec CLUSTER.YAML 'tune ls ~/ray_results'
|
|
|
|
# Upload and sync file_mounts up to the cluster with this command.
|
|
$ ray rsync-up CLUSTER.YAML
|
|
|
|
# Download the results directory from your cluster head node to your local machine on ``~/cluster_results``.
|
|
$ ray rsync-down CLUSTER.YAML '~/ray_results' ~/cluster_results
|
|
|
|
# Launching multiple clusters using the same configuration.
|
|
$ ray up CLUSTER.YAML -n="cluster1"
|
|
$ ray up CLUSTER.YAML -n="cluster2"
|
|
$ ray up CLUSTER.YAML -n="cluster3"
|
|
|
|
Troubleshooting
|
|
---------------
|
|
|
|
Sometimes, your program may freeze.
|
|
Run this to restart the Ray cluster without running any of the installation commands.
|
|
|
|
.. code-block:: bash
|
|
|
|
$ ray up CLUSTER.YAML --restart-only
|