chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,5 @@
|
||||
filegroup(
|
||||
name = "tune_tutorials",
|
||||
srcs = glob(["*.ipynb"]),
|
||||
visibility = ["//doc:__subpackages__"],
|
||||
)
|
||||
@@ -0,0 +1,24 @@
|
||||
.. _tune-guides:
|
||||
|
||||
===========
|
||||
User Guides
|
||||
===========
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
Running Basic Experiments <tune-run>
|
||||
tune-output
|
||||
Setting Trial Resources <tune-resources>
|
||||
Using Search Spaces <tune-search-spaces>
|
||||
tune-stopping
|
||||
tune-trial-checkpoints
|
||||
tune-storage
|
||||
tune-fault-tolerance
|
||||
Using Callbacks and Metrics <tune-metrics>
|
||||
tune_get_data_in_and_out
|
||||
../examples/tune_analyze_results
|
||||
../examples/pbt_guide
|
||||
Deploying Tune in the Cloud <tune-distributed>
|
||||
Tune Architecture <tune-lifecycle>
|
||||
Scalability Benchmarks <tune-scalability>
|
||||
@@ -0,0 +1,295 @@
|
||||
.. _tune-distributed-ref:
|
||||
|
||||
Running Distributed Experiments with Ray Tune
|
||||
==============================================
|
||||
|
||||
Tune is commonly used for large-scale distributed hyperparameter optimization. This page will overview how to setup and launch a distributed experiment along with :ref:`commonly used commands <tune-distributed-common>` for Tune when running distributed experiments.
|
||||
|
||||
.. contents::
|
||||
:local:
|
||||
:backlinks: none
|
||||
|
||||
Summary
|
||||
-------
|
||||
|
||||
To run a distributed experiment with Tune, you need to:
|
||||
|
||||
1. First, :ref:`start a Ray cluster <cluster-index>` if you have not already.
|
||||
2. Run the script on the head node, or use :ref:`ray submit <ray-submit-doc>`, or use :ref:`Ray Job Submission <jobs-overview>`.
|
||||
|
||||
.. tune-distributed-cloud:
|
||||
|
||||
Example: Distributed Tune on AWS VMs
|
||||
------------------------------------
|
||||
|
||||
Follow the instructions below to launch nodes on AWS (using the Deep Learning AMI). See the :ref:`cluster setup documentation <cluster-index>`. Save the below cluster configuration (``tune-default.yaml``):
|
||||
|
||||
.. literalinclude:: /../../python/ray/tune/examples/tune-default.yaml
|
||||
:language: yaml
|
||||
:name: tune-default.yaml
|
||||
|
||||
``ray up`` starts Ray on the cluster of nodes.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ray up tune-default.yaml
|
||||
|
||||
``ray submit --start`` starts a cluster as specified by the given cluster configuration YAML file, uploads ``tune_script.py`` to the cluster, and runs ``python tune_script.py [args]``.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ray submit tune-default.yaml tune_script.py --start -- --ray-address=localhost:6379
|
||||
|
||||
.. image:: /images/tune-upload.png
|
||||
:scale: 50%
|
||||
:align: center
|
||||
|
||||
Analyze your results on TensorBoard by starting TensorBoard on the remote head machine.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Go to http://localhost:6006 to access TensorBoard.
|
||||
ray exec tune-default.yaml 'tensorboard --logdir=~/ray_results/ --port 6006' --port-forward 6006
|
||||
|
||||
|
||||
Note that you can customize the directory of results by specifying: ``RunConfig(storage_path=..)``, taken in by ``Tuner``. You can then point TensorBoard to that directory to visualize results. You can also use `awless <https://github.com/wallix/awless>`_ for easy cluster management on AWS.
|
||||
|
||||
|
||||
Running a Distributed Tune Experiment
|
||||
-------------------------------------
|
||||
|
||||
Running a distributed (multi-node) experiment requires Ray to be started already.
|
||||
You can do this on local machines or on the cloud.
|
||||
|
||||
Across your machines, Tune will automatically detect the number of GPUs and CPUs without you needing to manage ``CUDA_VISIBLE_DEVICES``.
|
||||
|
||||
To execute a distributed experiment, call ``ray.init(address=XXX)`` before ``Tuner.fit()``, where ``XXX`` is the Ray address, which defaults to ``localhost:6379``. The Tune python script should be executed only on the head node of the Ray cluster.
|
||||
|
||||
One common approach to modifying an existing Tune experiment to go distributed is to set an ``argparse`` variable so that toggling between distributed and single-node is seamless.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import ray
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--address")
|
||||
args = parser.parse_args()
|
||||
ray.init(address=args.address)
|
||||
|
||||
tuner = tune.Tuner(...)
|
||||
tuner.fit()
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# On the head node, connect to an existing ray cluster
|
||||
$ python tune_script.py --ray-address=localhost:XXXX
|
||||
|
||||
If you used a cluster configuration (starting a cluster with ``ray up`` or ``ray submit --start``), use:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ray submit tune-default.yaml tune_script.py -- --ray-address=localhost:6379
|
||||
|
||||
.. tip::
|
||||
|
||||
1. In the examples, the Ray address commonly used is ``localhost:6379``.
|
||||
2. If the Ray cluster is already started, you should not need to run anything on the worker nodes.
|
||||
|
||||
|
||||
Storage Options in a Distributed Tune Run
|
||||
-----------------------------------------
|
||||
|
||||
In a distributed experiment, you should try to use :ref:`cloud checkpointing <tune-cloud-checkpointing>` to
|
||||
reduce synchronization overhead. For this, you just have to specify a remote ``storage_path`` in the
|
||||
:class:`RunConfig <ray.tune.RunConfig>`.
|
||||
|
||||
`my_trainable` is a user-defined :ref:`Tune Trainable <tune_60_seconds_trainables>` in the following example:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from ray import tune
|
||||
from my_module import my_trainable
|
||||
|
||||
tuner = tune.Tuner(
|
||||
my_trainable,
|
||||
run_config=tune.RunConfig(
|
||||
name="experiment_name",
|
||||
storage_path="s3://bucket-name/sub-path/",
|
||||
)
|
||||
)
|
||||
tuner.fit()
|
||||
|
||||
For more details or customization, see our
|
||||
:ref:`guide on configuring storage in a distributed Tune experiment <tune-storage-options>`.
|
||||
|
||||
|
||||
.. _tune-distributed-spot:
|
||||
|
||||
Tune Runs on preemptible instances
|
||||
-----------------------------------
|
||||
|
||||
Running on spot instances (or preemptible instances) can reduce the cost of your experiment.
|
||||
You can enable spot instances in AWS via the following configuration modification:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
# Provider-specific config for worker nodes, e.g. instance type.
|
||||
worker_nodes:
|
||||
InstanceType: m5.large
|
||||
ImageId: ami-0b294f219d14e6a82 # Deep Learning AMI (Ubuntu) Version 21.0
|
||||
|
||||
# Run workers on spot by default. Comment this out to use on-demand.
|
||||
InstanceMarketOptions:
|
||||
MarketType: spot
|
||||
SpotOptions:
|
||||
MaxPrice: 1.0 # Max Hourly Price
|
||||
|
||||
In GCP, you can use the following configuration modification:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
worker_nodes:
|
||||
machineType: n1-standard-2
|
||||
disks:
|
||||
- boot: true
|
||||
autoDelete: true
|
||||
type: PERSISTENT
|
||||
initializeParams:
|
||||
diskSizeGb: 50
|
||||
# See https://cloud.google.com/compute/docs/images for more images
|
||||
sourceImage: projects/deeplearning-platform-release/global/images/family/tf-1-13-cpu
|
||||
|
||||
# Run workers on preemtible instances.
|
||||
scheduling:
|
||||
- preemptible: true
|
||||
|
||||
Spot instances may be pre-empted suddenly while trials are still running.
|
||||
Tune allows you to mitigate the effects of this by preserving the progress of your model training through
|
||||
:ref:`checkpointing <tune-trial-checkpoint>`.
|
||||
|
||||
.. literalinclude:: /../../python/ray/tune/tests/tutorial.py
|
||||
:language: python
|
||||
:start-after: __trainable_run_begin__
|
||||
:end-before: __trainable_run_end__
|
||||
|
||||
|
||||
Example for Using Tune with Spot instances (AWS)
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Here is an example for running Tune on spot instances. This assumes your AWS credentials have already been setup (``aws configure``):
|
||||
|
||||
1. Download a full example Tune experiment script here. This includes a Trainable with checkpointing: :download:`mnist_pytorch_trainable.py </../../python/ray/tune/examples/mnist_pytorch_trainable.py>`. To run this example, you will need to install the following:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ pip install ray torch torchvision filelock
|
||||
|
||||
2. Download an example cluster yaml here: :download:`tune-default.yaml </../../python/ray/tune/examples/tune-default.yaml>`
|
||||
3. Run ``ray submit`` as below to run Tune across them. Append ``[--start]`` if the cluster is not up yet. Append ``[--stop]`` to automatically shutdown your nodes after running.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ray submit tune-default.yaml mnist_pytorch_trainable.py --start -- --ray-address=localhost:6379
|
||||
|
||||
|
||||
4. Optionally for testing on AWS or GCP, you can use the following to kill a random worker node after all the worker nodes are up
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ ray kill-random-node tune-default.yaml --hard
|
||||
|
||||
To summarize, here are the commands to run:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
wget https://raw.githubusercontent.com/ray-project/ray/master/python/ray/tune/examples/mnist_pytorch_trainable.py
|
||||
wget https://raw.githubusercontent.com/ray-project/ray/master/python/ray/tune/tune-default.yaml
|
||||
ray submit tune-default.yaml mnist_pytorch_trainable.py --start -- --ray-address=localhost:6379
|
||||
|
||||
# wait a while until after all nodes have started
|
||||
ray kill-random-node tune-default.yaml --hard
|
||||
|
||||
You should see Tune eventually continue the trials on a different worker node. See the :ref:`Fault Tolerance <tune-fault-tol>` section for more details.
|
||||
|
||||
You can also specify ``storage_path=...``, as part of ``RunConfig``, which is taken in by ``Tuner``, to upload results to cloud storage like S3, allowing you to persist results in case you want to start and stop your cluster automatically.
|
||||
|
||||
.. _tune-fault-tol:
|
||||
|
||||
Fault Tolerance of Tune Runs
|
||||
----------------------------
|
||||
|
||||
Tune automatically restarts trials in the case of trial failures (if ``max_failures != 0``),
|
||||
both in the single node and distributed setting.
|
||||
|
||||
For example, let's say a node is pre-empted or crashes while a trial is still executing on that node.
|
||||
Assuming that a checkpoint for this trial exists (and in the distributed setting,
|
||||
:ref:`some form of persistent storage is configured to access the trial's checkpoint <tune-storage-options>`),
|
||||
Tune waits until available resources are available to begin executing the trial again from where it left off.
|
||||
If no checkpoint is found, the trial will restart from scratch.
|
||||
See :ref:`here for information on checkpointing <tune-trial-checkpoint>`.
|
||||
|
||||
|
||||
If the trial or actor is then placed on a different node, Tune automatically pushes the previous checkpoint file
|
||||
to that node and restores the remote trial actor state, allowing the trial to resume from the latest checkpoint
|
||||
even after failure.
|
||||
|
||||
Recovering From Failures
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Tune automatically persists the progress of your entire experiment (a ``Tuner.fit()`` session), so if an experiment crashes or is otherwise cancelled, it can be resumed through :meth:`~ray.tune.Tuner.restore`.
|
||||
|
||||
.. _tune-distributed-common:
|
||||
|
||||
Common Tune Commands
|
||||
--------------------
|
||||
|
||||
Below are some commonly used commands for submitting experiments. Please see the :ref:`Clusters page <cluster-index>` to see find more comprehensive documentation of commands.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Upload `tune_experiment.py` from your local machine onto the cluster. Then,
|
||||
# run `python tune_experiment.py --address=localhost:6379` on the remote machine.
|
||||
$ ray submit CLUSTER.YAML tune_experiment.py -- --address=localhost:6379
|
||||
|
||||
# Start a cluster and run an experiment in a detached tmux session,
|
||||
# and shut down the cluster as soon as the experiment completes.
|
||||
# In `tune_experiment.py`, set `RunConfig(storage_path="s3://...")`
|
||||
# to persist results
|
||||
$ ray submit CLUSTER.YAML --tmux --start --stop tune_experiment.py -- --address=localhost:6379
|
||||
|
||||
# To start or update your cluster:
|
||||
$ ray up CLUSTER.YAML [-y]
|
||||
|
||||
# Shut-down all instances of your cluster:
|
||||
$ ray down CLUSTER.YAML [-y]
|
||||
|
||||
# Run TensorBoard and forward the port to your own machine.
|
||||
$ ray exec CLUSTER.YAML 'tensorboard --logdir ~/ray_results/ --port 6006' --port-forward 6006
|
||||
|
||||
# Run Jupyter Lab and forward the port to your own machine.
|
||||
$ ray exec CLUSTER.YAML 'jupyter lab --port 6006' --port-forward 6006
|
||||
|
||||
# Get a summary of all the experiments and trials that have executed so far.
|
||||
$ ray exec CLUSTER.YAML 'tune ls ~/ray_results'
|
||||
|
||||
# Upload and sync file_mounts up to the cluster with this command.
|
||||
$ ray rsync-up CLUSTER.YAML
|
||||
|
||||
# Download the results directory from your cluster head node to your local machine on ``~/cluster_results``.
|
||||
$ ray rsync-down CLUSTER.YAML '~/ray_results' ~/cluster_results
|
||||
|
||||
# Launching multiple clusters using the same configuration.
|
||||
$ ray up CLUSTER.YAML -n="cluster1"
|
||||
$ ray up CLUSTER.YAML -n="cluster2"
|
||||
$ ray up CLUSTER.YAML -n="cluster3"
|
||||
|
||||
Troubleshooting
|
||||
---------------
|
||||
|
||||
Sometimes, your program may freeze.
|
||||
Run this to restart the Ray cluster without running any of the installation commands.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ ray up CLUSTER.YAML --restart-only
|
||||
@@ -0,0 +1,218 @@
|
||||
.. _tune-fault-tolerance-ref:
|
||||
|
||||
How to Enable Fault Tolerance in Ray Tune
|
||||
=========================================
|
||||
|
||||
Fault tolerance is an important feature for distributed machine learning experiments
|
||||
that can help mitigate the impact of node failures due to out of memory and out of disk issues.
|
||||
|
||||
With fault tolerance, users can:
|
||||
|
||||
- **Save time and resources by preserving training progress** even if a node fails.
|
||||
- **Access the cost savings of preemptible spot instance nodes** in the distributed setting.
|
||||
|
||||
.. seealso::
|
||||
|
||||
In a *distributed* Tune experiment, a prerequisite to enabling fault tolerance
|
||||
is configuring some form of persistent storage where all trial results and
|
||||
checkpoints can be consolidated. See :ref:`tune-storage-options`.
|
||||
|
||||
In this guide, we will cover how to enable different types of fault tolerance offered by Ray Tune.
|
||||
|
||||
|
||||
.. _tune-experiment-level-fault-tolerance:
|
||||
|
||||
Experiment-level Fault Tolerance in Tune
|
||||
----------------------------------------
|
||||
|
||||
At the experiment level, :meth:`Tuner.restore <ray.tune.Tuner.restore>`
|
||||
resumes a previously interrupted experiment from where it left off.
|
||||
|
||||
You should use :meth:`Tuner.restore <ray.tune.Tuner.restore>` in the following cases:
|
||||
|
||||
1. The driver script that calls :meth:`Tuner.fit() <ray.tune.Tuner.fit>` errors out (e.g., due to the head node running out of memory or out of disk).
|
||||
2. The experiment is manually interrupted with ``Ctrl+C``.
|
||||
3. The entire cluster, and the experiment along with it, crashes due to an ephemeral error such as the network going down or Ray object store memory filling up.
|
||||
|
||||
.. note::
|
||||
|
||||
:meth:`Tuner.restore <ray.tune.Tuner.restore>` is *not* meant for resuming a terminated
|
||||
experiment and modifying hyperparameter search spaces or stopping criteria.
|
||||
Rather, experiment restoration is meant to resume and complete the *exact job*
|
||||
that was previously submitted via :meth:`Tuner.fit <ray.tune.Tuner.fit>`.
|
||||
|
||||
For example, consider a Tune experiment configured to run for ``10`` training iterations,
|
||||
where all trials have already completed.
|
||||
:meth:`Tuner.restore <ray.tune.Tuner.restore>` cannot be used to restore the experiment,
|
||||
change the number of training iterations to ``20``, then continue training.
|
||||
|
||||
Instead, this should be achieved by starting a *new* experiment and initializing
|
||||
your model weights with a checkpoint from the previous experiment.
|
||||
See :ref:`this FAQ post <tune-iterative-experimentation>` for an example.
|
||||
|
||||
|
||||
.. note::
|
||||
|
||||
Bugs in your user-defined training loop cannot be fixed with restoration. Instead, the issue
|
||||
that caused the experiment to crash in the first place should be *ephemeral*,
|
||||
meaning that the retry attempt after restoring can succeed the next time.
|
||||
|
||||
|
||||
.. _tune-experiment-restore-example:
|
||||
|
||||
Restore a Tune Experiment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Let's say your initial Tune experiment is configured as follows.
|
||||
The actual training loop is just for demonstration purposes: the important detail is that
|
||||
:ref:`saving and loading checkpoints has been implemented in the trainable <tune-trial-checkpoint>`.
|
||||
|
||||
.. literalinclude:: /tune/doc_code/fault_tolerance.py
|
||||
:language: python
|
||||
:start-after: __ft_initial_run_start__
|
||||
:end-before: __ft_initial_run_end__
|
||||
|
||||
The results and checkpoints of the experiment are saved to ``~/ray_results/tune_fault_tolerance_guide``,
|
||||
as configured by :class:`~ray.tune.RunConfig`.
|
||||
If the experiment has been interrupted due to one of the reasons listed above, use this path to resume:
|
||||
|
||||
.. literalinclude:: /tune/doc_code/fault_tolerance.py
|
||||
:language: python
|
||||
:start-after: __ft_restored_run_start__
|
||||
:end-before: __ft_restored_run_end__
|
||||
|
||||
.. tip::
|
||||
|
||||
You can also restore the experiment from a cloud bucket path:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
tuner = tune.Tuner.restore(
|
||||
path="s3://cloud-bucket/tune_fault_tolerance_guide", trainable=trainable
|
||||
)
|
||||
|
||||
See :ref:`tune-storage-options`.
|
||||
|
||||
|
||||
Restore Configurations
|
||||
~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Tune allows configuring which trials should be resumed, based on their status when the experiment was interrupted:
|
||||
|
||||
- Unfinished trials left in the ``RUNNING`` state will be resumed by default.
|
||||
- Trials that have ``ERRORED`` can be resumed or retried from scratch.
|
||||
- ``TERMINATED`` trials *cannot* be resumed.
|
||||
|
||||
.. literalinclude:: /tune/doc_code/fault_tolerance.py
|
||||
:language: python
|
||||
:start-after: __ft_restore_options_start__
|
||||
:end-before: __ft_restore_options_end__
|
||||
|
||||
|
||||
.. _tune-experiment-autoresume-example:
|
||||
|
||||
Auto-resume
|
||||
~~~~~~~~~~~
|
||||
|
||||
When running in a production setting, one may want a *single script* that (1) launches the
|
||||
initial training run in the beginning and (2) restores the experiment if (1) already happened.
|
||||
|
||||
Use the :meth:`Tuner.can_restore <ray.tune.Tuner.can_restore>` utility to accomplish this:
|
||||
|
||||
.. literalinclude:: /tune/doc_code/fault_tolerance.py
|
||||
:language: python
|
||||
:start-after: __ft_restore_multiplexing_start__
|
||||
:end-before: __ft_restore_multiplexing_end__
|
||||
|
||||
Running this script the first time will launch the initial training run.
|
||||
Running this script the second time will attempt to resume from the outputs of the first run.
|
||||
|
||||
|
||||
Tune Experiment Restoration with Ray Object References (Advanced)
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Experiment restoration often happens in a different Ray session than the original run,
|
||||
in which case Ray object references are automatically garbage collected.
|
||||
If object references are saved along with experiment state (e.g., within each trial's config),
|
||||
then attempting to retrieve these objects will not work properly after restoration:
|
||||
the objects these references point to no longer exist.
|
||||
|
||||
To work around this, you must re-create these objects, put them in the Ray object store,
|
||||
and then pass the new object references to Tune.
|
||||
|
||||
Example
|
||||
*******
|
||||
|
||||
Let's say we have some large pre-trained model that we want to use in some way in our training loop.
|
||||
For example, this could be a image classification model used to calculate an Inception Score
|
||||
to evaluate the quality of a generative model.
|
||||
We may have multiple models that we want to tune over, where each trial samples one of the models to use.
|
||||
|
||||
.. literalinclude:: /tune/doc_code/fault_tolerance.py
|
||||
:language: python
|
||||
:start-after: __ft_restore_objrefs_initial_start__
|
||||
:end-before: __ft_restore_objrefs_initial_end__
|
||||
|
||||
To restore, we just need to re-specify the ``param_space`` via :meth:`Tuner.restore <ray.tune.Tuner.restore>`:
|
||||
|
||||
.. literalinclude:: /tune/doc_code/fault_tolerance.py
|
||||
:language: python
|
||||
:start-after: __ft_restore_objrefs_restored_start__
|
||||
:end-before: __ft_restore_objrefs_restored_end__
|
||||
|
||||
.. note::
|
||||
|
||||
If you're tuning over :ref:`Ray Data <data>`, you'll also need to re-specify them in the ``param_space``.
|
||||
Ray Data can contain object references, so the same problems described above apply.
|
||||
|
||||
See below for an example:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
ds_1 = ray.data.from_items([{"x": i, "y": 2 * i} for i in range(128)])
|
||||
ds_2 = ray.data.from_items([{"x": i, "y": 3 * i} for i in range(128)])
|
||||
|
||||
param_space = {
|
||||
"datasets": {"train": tune.grid_search([ds_1, ds_2])},
|
||||
}
|
||||
|
||||
tuner = tune.Tuner.restore(..., param_space=param_space)
|
||||
|
||||
.. _tune-trial-level-fault-tolerance:
|
||||
|
||||
Trial-level Fault Tolerance in Tune
|
||||
-----------------------------------
|
||||
|
||||
Trial-level fault tolerance deals with individual trial failures in the cluster, which can be caused by:
|
||||
|
||||
- Running with preemptible spot instances.
|
||||
- Ephemeral network connection issues.
|
||||
- Nodes running out of memory or out of disk space.
|
||||
|
||||
Ray Tune provides a way to configure failure handling of individual trials with the :class:`~ray.tune.FailureConfig`.
|
||||
|
||||
Assuming that we're using the ``trainable`` from the previous example that implements
|
||||
trial checkpoint saving and loading, here is how to configure :class:`~ray.tune.FailureConfig`:
|
||||
|
||||
.. literalinclude:: /tune/doc_code/fault_tolerance.py
|
||||
:language: python
|
||||
:start-after: __ft_trial_failure_start__
|
||||
:end-before: __ft_trial_failure_end__
|
||||
|
||||
When a trial encounters a runtime error, the above configuration will re-schedule that trial
|
||||
up to ``max_failures=3`` times.
|
||||
|
||||
Similarly, if a node failure occurs for node ``X`` (e.g., pre-empted or lost connection),
|
||||
this configuration will reschedule all trials that lived on node ``X`` up to ``3`` times.
|
||||
|
||||
|
||||
Summary
|
||||
-------
|
||||
|
||||
In this user guide, we covered how to enable experiment-level and trial-level fault tolerance in Ray Tune.
|
||||
|
||||
See the following resources for more information:
|
||||
|
||||
- :ref:`tune-storage-options`
|
||||
- :ref:`tune-distributed-ref`
|
||||
- :ref:`tune-trial-checkpoint`
|
||||
@@ -0,0 +1,192 @@
|
||||
How does Tune work?
|
||||
===================
|
||||
|
||||
This page provides an overview of Tune's inner workings.
|
||||
We describe in detail what happens when you call ``Tuner.fit()``, what the lifecycle of a Tune trial looks like
|
||||
and what the architectural components of Tune are.
|
||||
|
||||
.. tip:: Before you continue, be sure to have read :ref:`the Tune Key Concepts page <tune-60-seconds>`.
|
||||
|
||||
What happens in ``Tuner.fit``?
|
||||
------------------------------
|
||||
|
||||
When calling the following:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
space = {"x": tune.uniform(0, 1)}
|
||||
tuner = tune.Tuner(
|
||||
my_trainable,
|
||||
param_space=space,
|
||||
tune_config=tune.TuneConfig(num_samples=10),
|
||||
)
|
||||
results = tuner.fit()
|
||||
|
||||
The provided ``my_trainable`` is evaluated multiple times in parallel
|
||||
with different hyperparameters (sampled from ``uniform(0, 1)``).
|
||||
|
||||
Every Tune run consists of "driver process" and many "worker processes".
|
||||
The driver process is the python process that calls ``Tuner.fit()`` (which calls ``ray.init()`` underneath the hood).
|
||||
The Tune driver process runs on the node where you run your script (which calls ``Tuner.fit()``),
|
||||
while Ray Tune trainable "actors" run on any node (either on the same node or on worker nodes (distributed Ray only)).
|
||||
|
||||
.. note:: :ref:`Ray Actors <actor-guide>` allow you to parallelize an instance of a class in Python.
|
||||
When you instantiate a class that is a Ray actor, Ray will start a instance of that class on a separate process
|
||||
either on the same machine (or another distributed machine, if running a Ray cluster).
|
||||
This actor can then asynchronously execute method calls and maintain its own internal state.
|
||||
|
||||
The driver spawns parallel worker processes (:ref:`Ray actors <actor-guide>`)
|
||||
that are responsible for evaluating each trial using its hyperparameter configuration and the provided trainable.
|
||||
|
||||
While the Trainable is executing (:ref:`trainable-execution`), the Tune Driver communicates with each actor
|
||||
via actor methods to receive intermediate training results and pause/stop actors (see :ref:`trial-lifecycle`).
|
||||
|
||||
When the Trainable terminates (or is stopped), the actor is also terminated.
|
||||
|
||||
.. _trainable-execution:
|
||||
|
||||
The execution of a trainable in Tune
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Tune uses :ref:`Ray actors <actor-guide>` to parallelize the evaluation of multiple hyperparameter configurations.
|
||||
Each actor is a Python process that executes an instance of the user-provided Trainable.
|
||||
|
||||
The definition of the user-provided Trainable will be
|
||||
:ref:`serialized via cloudpickle <serialization-guide>`) and sent to each actor process.
|
||||
Each Ray actor will start an instance of the Trainable to be executed.
|
||||
|
||||
If the Trainable is a class, it will be executed iteratively by calling ``train/step``.
|
||||
After each invocation, the driver is notified that a "result dict" is ready.
|
||||
The driver will then pull the result via ``ray.get``.
|
||||
|
||||
If the trainable is a callable or a function, it will be executed on the Ray actor process on a separate execution thread.
|
||||
Whenever ``tune.report`` is called, the execution thread is paused and waits for the driver to pull a
|
||||
result (see `function_trainable.py <https://github.com/ray-project/ray/blob/master/python/ray/tune/trainable/function_trainable.py>`__.
|
||||
After pulling, the actor’s execution thread will automatically resume.
|
||||
|
||||
|
||||
Resource Management in Tune
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Before running a trial, the Ray Tune driver will check whether there are available
|
||||
resources on the cluster (see :ref:`resource-requirements`).
|
||||
It will compare the available resources with the resources required by the trial.
|
||||
|
||||
If there is space on the cluster, then the Tune Driver will start a Ray actor (worker).
|
||||
This actor will be scheduled and executed on some node where the resources are available.
|
||||
See :doc:`tune-resources` for more information.
|
||||
|
||||
.. _trial-lifecycle:
|
||||
|
||||
Lifecycle of a Tune Trial
|
||||
-------------------------
|
||||
|
||||
A trial's life cycle consists of 6 stages:
|
||||
|
||||
* **Initialization** (generation): A trial is first generated as a hyperparameter sample,
|
||||
and its parameters are configured according to what was provided in ``Tuner``.
|
||||
Trials are then placed into a queue to be executed (with status PENDING).
|
||||
|
||||
* **PENDING**: A pending trial is a trial to be executed on the machine.
|
||||
Every trial is configured with resource values. Whenever the trial’s resource values are available,
|
||||
Tune will run the trial (by starting a ray actor holding the config and the training function.
|
||||
|
||||
* **RUNNING**: A running trial is assigned a Ray Actor. There can be multiple running trials in parallel.
|
||||
See the :ref:`trainable execution <trainable-execution>` section for more details.
|
||||
|
||||
* **ERRORED**: If a running trial throws an exception, Tune will catch that exception and mark the trial as errored.
|
||||
Note that exceptions can be propagated from an actor to the main Tune driver process.
|
||||
If max_retries is set, Tune will set the trial back into "PENDING" and later start it from the last checkpoint.
|
||||
|
||||
* **TERMINATED**: A trial is terminated if it is stopped by a Stopper/Scheduler.
|
||||
If using the Function API, the trial is also terminated when the function stops.
|
||||
|
||||
* **PAUSED**: A trial can be paused by a Trial scheduler. This means that the trial’s actor will be stopped.
|
||||
A paused trial can later be resumed from the most recent checkpoint.
|
||||
|
||||
|
||||
Tune's Architecture
|
||||
-------------------
|
||||
|
||||
.. image:: ../../images/tune-arch.png
|
||||
|
||||
The blue boxes refer to internal components, while green boxes are public-facing.
|
||||
|
||||
Tune's main components consist of
|
||||
the :class:`~ray.tune.execution.tune_controller.TuneController`,
|
||||
:class:`~ray.tune.experiment.trial.Trial` objects,
|
||||
a :class:`~ray.tune.search.search_algorithm.SearchAlgorithm`,
|
||||
a :class:`~ray.tune.schedulers.trial_scheduler.TrialScheduler`,
|
||||
and a :class:`~ray.tune.trainable.trainable.Trainable`,
|
||||
|
||||
.. _trial-runner-flow:
|
||||
|
||||
This is an illustration of the high-level training flow and how some of the components interact:
|
||||
|
||||
*Note: This figure is horizontally scrollable*
|
||||
|
||||
.. figure:: ../../images/tune-trial-runner-flow-horizontal.png
|
||||
:class: horizontal-scroll
|
||||
|
||||
|
||||
TuneController
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
[`source code <https://github.com/ray-project/ray/blob/master/python/ray/tune/execution/tune_controller.py>`__]
|
||||
This is the main driver of the training loop. This component
|
||||
uses the TrialScheduler to prioritize and execute trials,
|
||||
queries the SearchAlgorithm for new
|
||||
configurations to evaluate, and handles the fault tolerance logic.
|
||||
|
||||
**Fault Tolerance**: The TuneController executes checkpointing if ``checkpoint_freq``
|
||||
is set, along with automatic trial restarting in case of trial failures (if ``max_failures`` is set).
|
||||
For example, if a node is lost while a trial (specifically, the corresponding
|
||||
Trainable of the trial) is still executing on that node and checkpointing
|
||||
is enabled, the trial will then be reverted to a ``"PENDING"`` state and resumed
|
||||
from the last available checkpoint when it is run.
|
||||
The TuneController is also in charge of checkpointing the entire experiment execution state
|
||||
upon each loop iteration. This allows users to restart their experiment
|
||||
in case of machine failure.
|
||||
|
||||
See the docstring at :class:`~ray.tune.execution.tune_controller.TuneController`.
|
||||
|
||||
Trial objects
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
[`source code <https://github.com/ray-project/ray/blob/master/python/ray/tune/experiment/trial.py>`__]
|
||||
This is an internal data structure that contains metadata about each training run. Each Trial
|
||||
object is mapped one-to-one with a Trainable object but are not themselves
|
||||
distributed/remote. Trial objects transition among
|
||||
the following states: ``"PENDING"``, ``"RUNNING"``, ``"PAUSED"``, ``"ERRORED"``, and
|
||||
``"TERMINATED"``.
|
||||
|
||||
See the docstring at :ref:`trial-docstring`.
|
||||
|
||||
SearchAlg
|
||||
~~~~~~~~~
|
||||
[`source code <https://github.com/ray-project/ray/tree/master/python/ray/tune/search>`__]
|
||||
The SearchAlgorithm is a user-provided object
|
||||
that is used for querying new hyperparameter configurations to evaluate.
|
||||
|
||||
SearchAlgorithms will be notified every time a trial finishes
|
||||
executing one training step (of ``train()``), every time a trial
|
||||
errors, and every time a trial completes.
|
||||
|
||||
TrialScheduler
|
||||
~~~~~~~~~~~~~~
|
||||
[`source code <https://github.com/ray-project/ray/tree/master/python/ray/tune/schedulers>`__]
|
||||
TrialSchedulers operate over a set of possible trials to run,
|
||||
prioritizing trial execution given available cluster resources.
|
||||
|
||||
TrialSchedulers are given the ability to kill or pause trials,
|
||||
and also are given the ability to reorder/prioritize incoming trials.
|
||||
|
||||
Trainables
|
||||
~~~~~~~~~~
|
||||
[`source code <https://github.com/ray-project/ray/blob/master/python/ray/tune/trainable/trainable.py>`__]
|
||||
These are user-provided objects that are used for
|
||||
the training process. If a class is provided, it is expected to conform to the
|
||||
Trainable interface. If a function is provided. it is wrapped into a
|
||||
Trainable class, and the function itself is executed on a separate thread.
|
||||
|
||||
Trainables will execute one step of ``train()`` before notifying the TrialRunner.
|
||||
@@ -0,0 +1,92 @@
|
||||
A Guide To Callbacks & Metrics in Tune
|
||||
======================================
|
||||
|
||||
.. _tune-callbacks:
|
||||
|
||||
How to work with Callbacks in Ray Tune?
|
||||
---------------------------------------
|
||||
|
||||
Ray Tune supports callbacks that are called during various times of the training process.
|
||||
Callbacks can be passed as a parameter to ``RunConfig``, taken in by ``Tuner``, and the sub-method you provide will be invoked automatically.
|
||||
|
||||
This simple callback just prints a metric each time a result is received:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from ray import tune
|
||||
from ray.tune import Callback
|
||||
|
||||
|
||||
class MyCallback(Callback):
|
||||
def on_trial_result(self, iteration, trials, trial, result, **info):
|
||||
print(f"Got result: {result['metric']}")
|
||||
|
||||
|
||||
def train_fn(config):
|
||||
for i in range(10):
|
||||
tune.report({"metric": i})
|
||||
|
||||
|
||||
tuner = tune.Tuner(
|
||||
train_fn,
|
||||
run_config=tune.RunConfig(callbacks=[MyCallback()]))
|
||||
tuner.fit()
|
||||
|
||||
For more details and available hooks, please :ref:`see the API docs for Ray Tune callbacks <tune-callbacks-docs>`.
|
||||
|
||||
|
||||
.. _tune-autofilled-metrics:
|
||||
|
||||
How to use log metrics in Tune?
|
||||
-------------------------------
|
||||
|
||||
You can log arbitrary values and metrics in both Function and Class training APIs:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
def trainable(config):
|
||||
for i in range(num_epochs):
|
||||
...
|
||||
tune.report({"acc": accuracy, "metric_foo": random_metric_1, "bar": metric_2})
|
||||
|
||||
class Trainable(tune.Trainable):
|
||||
def step(self):
|
||||
...
|
||||
# don't call report here!
|
||||
return dict(acc=accuracy, metric_foo=random_metric_1, bar=metric_2)
|
||||
|
||||
|
||||
.. tip::
|
||||
Note that ``tune.report()`` is not meant to transfer large amounts of data, like models or datasets.
|
||||
Doing so can incur large overheads and slow down your Tune run significantly.
|
||||
|
||||
Which Tune metrics get automatically filled in?
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Tune has the concept of auto-filled metrics.
|
||||
During training, Tune will automatically log the below metrics in addition to any user-provided values.
|
||||
All of these can be used as stopping conditions or passed as a parameter to Trial Schedulers/Search Algorithms.
|
||||
|
||||
* ``config``: The hyperparameter configuration
|
||||
* ``date``: String-formatted date and time when the result was processed
|
||||
* ``done``: True if the trial has been finished, False otherwise
|
||||
* ``episodes_total``: Total number of episodes (for RLlib trainables)
|
||||
* ``experiment_id``: Unique experiment ID
|
||||
* ``experiment_tag``: Unique experiment tag (includes parameter values)
|
||||
* ``hostname``: Hostname of the worker
|
||||
* ``iterations_since_restore``: The number of times ``tune.report`` has been
|
||||
called after restoring the worker from a checkpoint
|
||||
* ``node_ip``: Host IP of the worker
|
||||
* ``pid``: Process ID (PID) of the worker process
|
||||
* ``time_since_restore``: Time in seconds since restoring from a checkpoint.
|
||||
* ``time_this_iter_s``: Runtime of the current training iteration in seconds (i.e.
|
||||
one call to the trainable function or to ``_train()`` in the class API.
|
||||
* ``time_total_s``: Total runtime in seconds.
|
||||
* ``timestamp``: Timestamp when the result was processed
|
||||
* ``timesteps_since_restore``: Number of timesteps since restoring from a checkpoint
|
||||
* ``timesteps_total``: Total number of timesteps
|
||||
* ``training_iteration``: The number of times ``tune.report()`` has been
|
||||
called
|
||||
* ``trial_id``: Unique trial ID
|
||||
|
||||
All of these metrics can be seen in the ``Trial.last_result`` dictionary.
|
||||
@@ -0,0 +1,336 @@
|
||||
Logging and Outputs in Tune
|
||||
===========================
|
||||
|
||||
By default, Tune logs results for TensorBoard, CSV, and JSON formats.
|
||||
If you need to log something lower level like model weights or gradients, see :ref:`Trainable Logging <trainable-logging>`.
|
||||
You can learn more about logging and customizations here: :ref:`loggers-docstring`.
|
||||
|
||||
|
||||
.. _tune-logging:
|
||||
|
||||
How to configure logging in Tune?
|
||||
---------------------------------
|
||||
|
||||
Tune will log the results of each trial to a sub-folder under a specified local dir, which defaults to ``~/ray_results``.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# This logs to two different trial folders:
|
||||
# ~/ray_results/trainable_name/trial_name_1 and ~/ray_results/trainable_name/trial_name_2
|
||||
# trainable_name and trial_name are autogenerated.
|
||||
tuner = tune.Tuner(trainable, run_config=RunConfig(num_samples=2))
|
||||
results = tuner.fit()
|
||||
|
||||
You can specify the ``storage_path`` and ``trainable_name``:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# This logs to 2 different trial folders:
|
||||
# ./results/test_experiment/trial_name_1 and ./results/test_experiment/trial_name_2
|
||||
# Only trial_name is autogenerated.
|
||||
tuner = tune.Tuner(trainable,
|
||||
tune_config=tune.TuneConfig(num_samples=2),
|
||||
run_config=RunConfig(storage_path="./results", name="test_experiment"))
|
||||
results = tuner.fit()
|
||||
|
||||
|
||||
To learn more about Trials, see its detailed API documentation: :ref:`trial-docstring`.
|
||||
|
||||
.. _tensorboard:
|
||||
|
||||
How to log your Tune runs to TensorBoard?
|
||||
-----------------------------------------
|
||||
|
||||
Tune automatically outputs TensorBoard files during ``Tuner.fit()``.
|
||||
To visualize learning in tensorboard, install tensorboardX:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ pip install tensorboardX
|
||||
|
||||
Then, after you run an experiment, you can visualize your experiment with TensorBoard by specifying
|
||||
the output directory of your results.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ tensorboard --logdir=~/ray_results/my_experiment
|
||||
|
||||
If you are running Ray on a remote multi-user cluster where you do not have sudo access,
|
||||
you can run the following commands to make sure tensorboard is able to write to the tmp directory:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ export TMPDIR=/tmp/$USER; mkdir -p $TMPDIR; tensorboard --logdir=~/ray_results
|
||||
|
||||
.. image:: ../images/ray-tune-tensorboard.png
|
||||
|
||||
If using TensorFlow ``2.x``, Tune also automatically generates TensorBoard HParams output, as shown below:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
tuner = tune.Tuner(
|
||||
...,
|
||||
param_space={
|
||||
"lr": tune.grid_search([1e-5, 1e-4]),
|
||||
"momentum": tune.grid_search([0, 0.9])
|
||||
}
|
||||
)
|
||||
results = tuner.fit()
|
||||
|
||||
.. image:: ../../images/tune-hparams.png
|
||||
|
||||
|
||||
.. _tune-console-output:
|
||||
|
||||
How to control console output with Tune?
|
||||
----------------------------------------
|
||||
|
||||
User-provided fields will be outputted automatically on a best-effort basis.
|
||||
You can use a :ref:`Reporter <tune-reporter-doc>` object to customize the console output.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
== Status ==
|
||||
Memory usage on this node: 11.4/16.0 GiB
|
||||
Using FIFO scheduling algorithm.
|
||||
Resources requested: 4/12 CPUs, 0/0 GPUs, 0.0/3.17 GiB heap, 0.0/1.07 GiB objects
|
||||
Result logdir: /Users/foo/ray_results/myexp
|
||||
Number of trials: 4 (4 RUNNING)
|
||||
+----------------------+----------+---------------------+-----------+--------+--------+----------------+-------+
|
||||
| Trial name | status | loc | param1 | param2 | acc | total time (s) | iter |
|
||||
|----------------------+----------+---------------------+-----------+--------+--------+----------------+-------|
|
||||
| MyTrainable_a826033a | RUNNING | 10.234.98.164:31115 | 0.303706 | 0.0761 | 0.1289 | 7.54952 | 15 |
|
||||
| MyTrainable_a8263fc6 | RUNNING | 10.234.98.164:31117 | 0.929276 | 0.158 | 0.4865 | 7.0501 | 14 |
|
||||
| MyTrainable_a8267914 | RUNNING | 10.234.98.164:31111 | 0.068426 | 0.0319 | 0.9585 | 7.0477 | 14 |
|
||||
| MyTrainable_a826b7bc | RUNNING | 10.234.98.164:31112 | 0.729127 | 0.0748 | 0.1797 | 7.05715 | 14 |
|
||||
+----------------------+----------+---------------------+-----------+--------+--------+----------------+-------+
|
||||
|
||||
|
||||
.. _tune-log_to_file:
|
||||
|
||||
How to redirect Trainable logs to files in a Tune run?
|
||||
---------------------------------------------------------
|
||||
|
||||
In Tune, Trainables are run as remote actors. By default, Ray collects actors' stdout and stderr and prints them to
|
||||
the head process (see :ref:`ray worker logs <ray-worker-logs>` for more information).
|
||||
Logging that happens within Tune Trainables follows this handling by default.
|
||||
However, if you wish to collect Trainable logs in files for analysis, Tune offers the option
|
||||
``log_to_file`` for this.
|
||||
This applies to print statements, ``warnings.warn`` and ``logger.info`` etc.
|
||||
|
||||
By passing ``log_to_file=True`` to ``RunConfig``, which is taken in by ``Tuner``, stdout and stderr will be logged
|
||||
to ``trial_logdir/stdout`` and ``trial_logdir/stderr``, respectively:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
tuner = tune.Tuner(
|
||||
trainable,
|
||||
run_config=RunConfig(log_to_file=True)
|
||||
)
|
||||
results = tuner.fit()
|
||||
|
||||
If you would like to specify the output files, you can either pass one filename,
|
||||
where the combined output will be stored, or two filenames, for stdout and stderr,
|
||||
respectively:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
tuner = tune.Tuner(
|
||||
trainable,
|
||||
run_config=RunConfig(log_to_file="std_combined.log")
|
||||
)
|
||||
tuner.fit()
|
||||
|
||||
tuner = tune.Tuner(
|
||||
trainable,
|
||||
run_config=RunConfig(log_to_file=("my_stdout.log", "my_stderr.log")))
|
||||
results = tuner.fit()
|
||||
|
||||
The file names are relative to the trial's logdir. You can pass absolute paths,
|
||||
too.
|
||||
|
||||
Caveats
|
||||
^^^^^^^
|
||||
Logging that happens in distributed training workers (if you happen to use Ray Tune together with Ray Train)
|
||||
is not part of this ``log_to_file`` configuration.
|
||||
|
||||
Where to find ``log_to_file`` files?
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
If your Tune workload is configured with syncing to head node, then the corresponding ``log_to_file`` outputs
|
||||
can be located under each trial folder.
|
||||
If your Tune workload is instead configured with syncing to cloud, then the corresponding ``log_to_file``
|
||||
outputs are *NOT* synced to cloud and can only be found in the worker nodes that the corresponding trial happens.
|
||||
|
||||
.. note::
|
||||
This can cause problems when the trainable is moved across different nodes throughout its lifetime.
|
||||
This can happen with some schedulers or with node failures.
|
||||
We may prioritize enabling this if there are enough user requests.
|
||||
If this impacts your workflow, consider commenting on
|
||||
[this ticket](https://github.com/ray-project/ray/issues/32142).
|
||||
|
||||
|
||||
Leave us feedback on this feature
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
We know that logging and observability can be a huge performance boost for your workflow. Let us know what is your
|
||||
preferred way to interact with logging that happens in trainables. Leave you comments in
|
||||
[this ticket](https://github.com/ray-project/ray/issues/32142).
|
||||
|
||||
.. _trainable-logging:
|
||||
|
||||
How do you log arbitrary files from a Tune Trainable?
|
||||
-----------------------------------------------------
|
||||
|
||||
By default, Tune only logs the *training result dictionaries* and *checkpoints* from your Trainable.
|
||||
However, you may want to save a file that visualizes the model weights or model graph,
|
||||
or use a custom logging library that requires multi-process logging.
|
||||
For example, you may want to do this if you're trying to log images to TensorBoard.
|
||||
We refer to these saved files as **trial artifacts**.
|
||||
|
||||
.. note::
|
||||
|
||||
If :class:`SyncConfig(sync_artifacts=True) <ray.tune.SyncConfig>`, trial artifacts
|
||||
are uploaded periodically from each trial (or from each remote training worker for Ray Train)
|
||||
to the :class:`RunConfig(storage_path) <ray.tune.RunConfig>`.
|
||||
|
||||
See the :class:`~ray.tune.SyncConfig` API reference for artifact syncing configuration options.
|
||||
|
||||
You can save trial artifacts directly in the trainable, as shown below:
|
||||
|
||||
.. tip:: Make sure that any logging calls or objects stay within scope of the Trainable.
|
||||
You may see pickling or other serialization errors or inconsistent logs otherwise.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Function API
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import logging_library # ex: mlflow, wandb
|
||||
from ray import tune
|
||||
|
||||
def trainable(config):
|
||||
logging_library.init(
|
||||
name=trial_id,
|
||||
id=trial_id,
|
||||
resume=trial_id,
|
||||
reinit=True,
|
||||
allow_val_change=True)
|
||||
logging_library.set_log_path(os.getcwd())
|
||||
|
||||
for step in range(100):
|
||||
logging_library.log_model(...)
|
||||
logging_library.log(results, step=step)
|
||||
|
||||
# You can also just write to a file directly.
|
||||
# The working directory is set to the trial directory, so
|
||||
# you don't need to worry about multiple workers saving
|
||||
# to the same location.
|
||||
with open(f"./artifact_{step}.txt", "w") as f:
|
||||
f.write("Artifact Data")
|
||||
|
||||
tune.report(results)
|
||||
|
||||
|
||||
.. tab-item:: Class API
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import logging_library # ex: mlflow, wandb
|
||||
from ray import tune
|
||||
|
||||
class CustomLogging(tune.Trainable)
|
||||
def setup(self, config):
|
||||
trial_id = self.trial_id
|
||||
logging_library.init(
|
||||
name=trial_id,
|
||||
id=trial_id,
|
||||
resume=trial_id,
|
||||
reinit=True,
|
||||
allow_val_change=True
|
||||
)
|
||||
logging_library.set_log_path(os.getcwd())
|
||||
|
||||
def step(self):
|
||||
logging_library.log_model(...)
|
||||
|
||||
# You can also write to a file directly.
|
||||
# The working directory is set to the trial directory, so
|
||||
# you don't need to worry about multiple workers saving
|
||||
# to the same location.
|
||||
with open(f"./artifact_{self.iteration}.txt", "w") as f:
|
||||
f.write("Artifact Data")
|
||||
|
||||
def log_result(self, result):
|
||||
res_dict = {
|
||||
str(k): v
|
||||
for k, v in result.items()
|
||||
if (v and "config" not in k and not isinstance(v, str))
|
||||
}
|
||||
step = result["training_iteration"]
|
||||
logging_library.log(res_dict, step=step)
|
||||
|
||||
|
||||
In the code snippet above, ``logging_library`` refers to whatever 3rd party logging library you are using.
|
||||
Note that ``logging_library.set_log_path(os.getcwd())`` is an imaginary API that we are using
|
||||
for demonstration purposes, and it highlights that the third-party library
|
||||
should be configured to log to the Trainable's *working directory.* By default,
|
||||
the current working directory of both functional and class trainables is set to the
|
||||
corresponding trial directory once it's been launched as a remote Ray actor.
|
||||
|
||||
|
||||
How to Build Custom Tune Loggers?
|
||||
---------------------------------
|
||||
|
||||
You can create a custom logger by inheriting the LoggerCallback interface (:ref:`logger-interface`):
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from typing import Dict, List
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
from ray.tune.logger import LoggerCallback
|
||||
|
||||
|
||||
class CustomLoggerCallback(LoggerCallback):
|
||||
"""Custom logger interface"""
|
||||
|
||||
def __init__(self, filename: str = "log.txt"):
|
||||
self._trial_files = {}
|
||||
self._filename = filename
|
||||
|
||||
def log_trial_start(self, trial: "Trial"):
|
||||
trial_logfile = os.path.join(trial.logdir, self._filename)
|
||||
self._trial_files[trial] = open(trial_logfile, "at")
|
||||
|
||||
def log_trial_result(self, iteration: int, trial: "Trial", result: Dict):
|
||||
if trial in self._trial_files:
|
||||
self._trial_files[trial].write(json.dumps(result))
|
||||
|
||||
def on_trial_complete(self, iteration: int, trials: List["Trial"],
|
||||
trial: "Trial", **info):
|
||||
if trial in self._trial_files:
|
||||
self._trial_files[trial].close()
|
||||
del self._trial_files[trial]
|
||||
|
||||
|
||||
You can then pass in your own logger as follows:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from ray import tune
|
||||
|
||||
tuner = tune.Tuner(
|
||||
MyTrainableClass,
|
||||
run_config=tune.RunConfig(
|
||||
name="experiment_name", callbacks=[CustomLoggerCallback("log_test.txt")]
|
||||
)
|
||||
)
|
||||
results = tuner.fit()
|
||||
|
||||
|
||||
Per default, Ray Tune creates JSON, CSV and TensorBoardX logger callbacks if you don't pass them yourself.
|
||||
You can disable this behavior by setting the ``TUNE_DISABLE_AUTO_CALLBACK_LOGGERS`` environment variable to ``"1"``.
|
||||
|
||||
An example of creating a custom logger can be found in :doc:`/tune/examples/includes/logging_example`.
|
||||
@@ -0,0 +1,141 @@
|
||||
.. _tune-parallelism:
|
||||
|
||||
A Guide To Parallelism and Resources for Ray Tune
|
||||
-------------------------------------------------
|
||||
|
||||
Parallelism is determined by per trial resources (defaulting to 1 CPU, 0 GPU per trial)
|
||||
and the resources available to Tune (``ray.cluster_resources()``).
|
||||
|
||||
By default, Tune automatically runs `N` concurrent trials, where `N` is the number
|
||||
of CPUs (cores) on your machine.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# If you have 4 CPUs on your machine, this will run 4 concurrent trials at a time.
|
||||
tuner = tune.Tuner(
|
||||
trainable,
|
||||
tune_config=tune.TuneConfig(num_samples=10)
|
||||
)
|
||||
results = tuner.fit()
|
||||
|
||||
You can override this per trial resources with :func:`tune.with_resources <ray.tune.with_resources>`. Here you can
|
||||
specify your resource requests using either a dictionary or a
|
||||
:class:`PlacementGroupFactory <ray.tune.execution.placement_groups.PlacementGroupFactory>`
|
||||
object. In either case, Ray Tune will try to start a placement group for each trial.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# If you have 4 CPUs on your machine, this will run 2 concurrent trials at a time.
|
||||
trainable_with_resources = tune.with_resources(trainable, {"cpu": 2})
|
||||
tuner = tune.Tuner(
|
||||
trainable_with_resources,
|
||||
tune_config=tune.TuneConfig(num_samples=10)
|
||||
)
|
||||
results = tuner.fit()
|
||||
|
||||
# If you have 4 CPUs on your machine, this will run 1 trial at a time.
|
||||
trainable_with_resources = tune.with_resources(trainable, {"cpu": 4})
|
||||
tuner = tune.Tuner(
|
||||
trainable_with_resources,
|
||||
tune_config=tune.TuneConfig(num_samples=10)
|
||||
)
|
||||
results = tuner.fit()
|
||||
|
||||
# Fractional values are also supported, (i.e., {"cpu": 0.5}).
|
||||
# If you have 4 CPUs on your machine, this will run 8 concurrent trials at a time.
|
||||
trainable_with_resources = tune.with_resources(trainable, {"cpu": 0.5})
|
||||
tuner = tune.Tuner(
|
||||
trainable_with_resources,
|
||||
tune_config=tune.TuneConfig(num_samples=10)
|
||||
)
|
||||
results = tuner.fit()
|
||||
|
||||
# Custom resource allocation via lambda functions are also supported.
|
||||
# If you want to allocate gpu resources to trials based on a setting in your config
|
||||
trainable_with_resources = tune.with_resources(trainable,
|
||||
resources=lambda config: {"gpu": 1} if config["use_gpu"] else {"gpu": 0})
|
||||
tuner = tune.Tuner(
|
||||
trainable_with_resources,
|
||||
tune_config=tune.TuneConfig(num_samples=10)
|
||||
)
|
||||
results = tuner.fit()
|
||||
|
||||
|
||||
Tune will allocate the specified GPU and CPU as specified by ``tune.with_resources`` to each individual trial.
|
||||
Even if the trial cannot be scheduled right now, Ray Tune will still try to start the respective placement group. If not enough resources are available, this will trigger
|
||||
:ref:`autoscaling behavior <cluster-index>` if you're using the Ray cluster launcher.
|
||||
|
||||
It is also possible to specify memory (``"memory"``, in bytes) and custom resource requirements.
|
||||
|
||||
If your trainable function starts more remote workers, you will need to pass so-called placement group
|
||||
factory objects to request these resources.
|
||||
See the :class:`PlacementGroupFactory documentation <ray.tune.execution.placement_groups.PlacementGroupFactory>`
|
||||
for further information.
|
||||
This also applies if you are using other libraries making use of Ray, such as Modin.
|
||||
Failure to set resources correctly may result in a deadlock, "hanging" the cluster.
|
||||
|
||||
.. note::
|
||||
The resources specified this way will only be allocated for scheduling Tune trials.
|
||||
These resources will not be enforced on your objective function (Tune trainable) automatically.
|
||||
You will have to make sure your trainable has enough resources to run (e.g. by setting ``n_jobs`` for a
|
||||
scikit-learn model accordingly).
|
||||
|
||||
How to leverage GPUs in Tune?
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
To leverage GPUs, you must set ``gpu`` in ``tune.with_resources(trainable, resources_per_trial)``.
|
||||
This will automatically set ``CUDA_VISIBLE_DEVICES`` for each trial.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# If you have 8 GPUs, this will run 8 trials at once.
|
||||
trainable_with_gpu = tune.with_resources(trainable, {"gpu": 1})
|
||||
tuner = tune.Tuner(
|
||||
trainable_with_gpu,
|
||||
tune_config=tune.TuneConfig(num_samples=10)
|
||||
)
|
||||
results = tuner.fit()
|
||||
|
||||
# If you have 4 CPUs and 1 GPU on your machine, this will run 1 trial at a time.
|
||||
trainable_with_cpu_gpu = tune.with_resources(trainable, {"cpu": 2, "gpu": 1})
|
||||
tuner = tune.Tuner(
|
||||
trainable_with_cpu_gpu,
|
||||
tune_config=tune.TuneConfig(num_samples=10)
|
||||
)
|
||||
results = tuner.fit()
|
||||
|
||||
You can find an example of this in the :doc:`Keras MNIST example </tune/examples/tune_mnist_keras>`.
|
||||
|
||||
.. warning:: If ``gpu`` is not set, ``CUDA_VISIBLE_DEVICES`` environment variable will be set as empty, disallowing GPU access.
|
||||
|
||||
**Troubleshooting**: Occasionally, you may run into GPU memory issues when running a new trial. This may be
|
||||
due to the previous trial not cleaning up its GPU state fast enough. To avoid this,
|
||||
you can use :func:`tune.utils.wait_for_gpu <ray.tune.utils.wait_for_gpu>`.
|
||||
|
||||
.. _tune-dist-training:
|
||||
|
||||
How to run distributed training with Tune?
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
To tune distributed training jobs, you can use Ray Tune with Ray Train. Ray Tune will run multiple trials in parallel, with each trial running distributed training with Ray Train.
|
||||
|
||||
For more details, see :ref:`Ray Train Hyperparameter Optimization <train-tune>`.
|
||||
|
||||
How to limit concurrency in Tune?
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
To specifies the max number of trials to run concurrently, set `max_concurrent_trials` in :class:`TuneConfig <ray.tune.tune_config.TuneConfig>`.
|
||||
|
||||
Note that actual parallelism can be less than `max_concurrent_trials` and will be determined by how many trials
|
||||
can fit in the cluster at once (i.e., if you have a trial that requires 16 GPUs, your cluster has 32 GPUs,
|
||||
and `max_concurrent_trials=10`, the `Tuner` can only run 2 trials concurrently).
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from ray.tune import TuneConfig
|
||||
|
||||
config = TuneConfig(
|
||||
# ...
|
||||
num_samples=100,
|
||||
max_concurrent_trials=10,
|
||||
)
|
||||
@@ -0,0 +1,90 @@
|
||||
.. _tune-parallel-experiments-guide:
|
||||
|
||||
Running Basic Tune Experiments
|
||||
==============================
|
||||
|
||||
The most common way to use Tune is also the simplest: as a parallel experiment runner. If you can define experiment trials in a Python function, you can use Tune to run hundreds to thousands of independent trial instances in a cluster. Tune manages trial execution, status reporting, and fault tolerance.
|
||||
|
||||
Running Independent Tune Trials in Parallel
|
||||
-------------------------------------------
|
||||
|
||||
As a general example, let's consider executing ``N`` independent model training trials using Tune as a simple grid sweep. Each trial can execute different code depending on a passed-in config dictionary.
|
||||
|
||||
**Step 1:** First, we define the model training function that we want to run variations of. The function takes in a config dictionary as argument, and returns a simple dict output. Learn more about logging Tune results at :ref:`tune-logging`.
|
||||
|
||||
.. literalinclude:: ../doc_code/tune.py
|
||||
:language: python
|
||||
:start-after: __step1_begin__
|
||||
:end-before: __step1_end__
|
||||
|
||||
**Step 2:** Next, define the space of trials to run. Here, we define a simple grid sweep from ``0..NUM_MODELS``, which will generate the config dicts to be passed to each model function. Learn more about what features Tune offers for defining spaces at :ref:`tune-search-space-tutorial`.
|
||||
|
||||
.. literalinclude:: ../doc_code/tune.py
|
||||
:language: python
|
||||
:start-after: __step2_begin__
|
||||
:end-before: __step2_end__
|
||||
|
||||
**Step 3:** Optionally, configure the resources allocated per trial. Tune uses this resources allocation to control the parallelism. For example, if each trial was configured to use 4 CPUs, and the cluster had only 32 CPUs, then Tune will limit the number of concurrent trials to 8 to avoid overloading the cluster. For more information, see :ref:`tune-parallelism`.
|
||||
|
||||
.. literalinclude:: ../doc_code/tune.py
|
||||
:language: python
|
||||
:start-after: __step3_begin__
|
||||
:end-before: __step3_end__
|
||||
|
||||
**Step 4:** Run the trial with Tune. Tune will report on experiment status, and after the experiment finishes, you can inspect the results. Tune can retry failed trials automatically, as well as entire experiments; see :ref:`tune-stopping-guide`.
|
||||
|
||||
.. literalinclude:: ../doc_code/tune.py
|
||||
:language: python
|
||||
:start-after: __step4_begin__
|
||||
:end-before: __step4_end__
|
||||
|
||||
**Step 5:** Inspect results. They will look something like this. Tune periodically prints a status summary to stdout showing the ongoing experiment status, until it finishes:
|
||||
|
||||
.. code::
|
||||
|
||||
== Status ==
|
||||
Current time: 2022-09-21 10:19:34 (running for 00:00:04.54)
|
||||
Memory usage on this node: 6.9/31.1 GiB
|
||||
Using FIFO scheduling algorithm.
|
||||
Resources requested: 0/8 CPUs, 0/0 GPUs, 0.0/16.13 GiB heap, 0.0/8.06 GiB objects
|
||||
Result logdir: /home/ubuntu/ray_results/train_model_2022-09-21_10-19-26
|
||||
Number of trials: 100/100 (100 TERMINATED)
|
||||
+-------------------------+------------+----------------------+------------+--------+------------------+
|
||||
| Trial name | status | loc | model_id | iter | total time (s) |
|
||||
|-------------------------+------------+----------------------+------------+--------+------------------|
|
||||
| train_model_8d627_00000 | TERMINATED | 192.168.1.67:2381731 | model_0 | 1 | 8.46386e-05 |
|
||||
| train_model_8d627_00001 | TERMINATED | 192.168.1.67:2381761 | model_1 | 1 | 0.000126362 |
|
||||
| train_model_8d627_00002 | TERMINATED | 192.168.1.67:2381763 | model_2 | 1 | 0.000112772 |
|
||||
...
|
||||
| train_model_8d627_00097 | TERMINATED | 192.168.1.67:2381731 | model_97 | 1 | 5.57899e-05 |
|
||||
| train_model_8d627_00098 | TERMINATED | 192.168.1.67:2381767 | model_98 | 1 | 6.05583e-05 |
|
||||
| train_model_8d627_00099 | TERMINATED | 192.168.1.67:2381763 | model_99 | 1 | 6.69956e-05 |
|
||||
+-------------------------+------------+----------------------+------------+--------+------------------+
|
||||
|
||||
2022-09-21 10:19:35,159 INFO tune.py:762 -- Total run time: 5.06 seconds (4.46 seconds for the tuning loop).
|
||||
|
||||
The final result objects contain finished trial metadata:
|
||||
|
||||
.. code::
|
||||
|
||||
Result(metrics={'score': 'model_0', 'other_data': Ellipsis, 'done': True, 'trial_id': '8d627_00000', 'experiment_tag': '0_model_id=model_0'}, error=None, log_dir=PosixPath('/home/ubuntu/ray_results/train_model_2022-09-21_10-19-26/train_model_8d627_00000_0_model_id=model_0_2022-09-21_10-19-30'))
|
||||
Result(metrics={'score': 'model_1', 'other_data': Ellipsis, 'done': True, 'trial_id': '8d627_00001', 'experiment_tag': '1_model_id=model_1'}, error=None, log_dir=PosixPath('/home/ubuntu/ray_results/train_model_2022-09-21_10-19-26/train_model_8d627_00001_1_model_id=model_1_2022-09-21_10-19-31'))
|
||||
Result(metrics={'score': 'model_2', 'other_data': Ellipsis, 'done': True, 'trial_id': '8d627_00002', 'experiment_tag': '2_model_id=model_2'}, error=None, log_dir=PosixPath('/home/ubuntu/ray_results/train_model_2022-09-21_10-19-26/train_model_8d627_00002_2_model_id=model_2_2022-09-21_10-19-31'))
|
||||
|
||||
How does Tune compare to using Ray Core (``ray.remote``)?
|
||||
----------------------------------------------------------
|
||||
|
||||
You might be wondering how Tune differs from simply using :ref:`ray-remote-functions` for parallel trial execution. Indeed, the above example could be re-written similarly as:
|
||||
|
||||
.. literalinclude:: ../doc_code/tune.py
|
||||
:language: python
|
||||
:start-after: __tasks_begin__
|
||||
:end-before: __tasks_end__
|
||||
|
||||
Compared to using Ray tasks, Tune offers the following additional functionality:
|
||||
|
||||
* Status reporting and tracking, including integrations and callbacks to common monitoring tools.
|
||||
* Checkpointing of trials for fine-grained fault-tolerance.
|
||||
* Gang scheduling of multi-worker trials.
|
||||
|
||||
In short, consider using Tune if you need status tracking or support for more advanced ML workloads.
|
||||
@@ -0,0 +1,206 @@
|
||||
:orphan:
|
||||
|
||||
Scalability and Overhead Benchmarks for Ray Tune
|
||||
================================================
|
||||
|
||||
We conducted a series of micro-benchmarks where we evaluated the scalability of Ray Tune and analyzed the
|
||||
performance overhead we observed. The results from these benchmarks are reflected in the documentation,
|
||||
e.g. when we make suggestions on :ref:`how to remove performance bottlenecks <tune-bottlenecks>`.
|
||||
|
||||
This page gives an overview over the experiments we did. For each of these experiments, the goal was to
|
||||
examine the total runtime of the experiment and address issues when the observed overhead compared to the
|
||||
minimal theoretical time was too high (e.g. more than 20% overhead).
|
||||
|
||||
In some of the experiments we tweaked the default settings for maximum throughput, e.g. by disabling
|
||||
trial synchronization or result logging. If this is the case, this is stated in the respective benchmark
|
||||
description.
|
||||
|
||||
|
||||
.. list-table:: Ray Tune scalability benchmarks overview
|
||||
:header-rows: 1
|
||||
|
||||
* - Variable
|
||||
- # of trials
|
||||
- Results/second /trial
|
||||
- # of nodes
|
||||
- # CPUs/node
|
||||
- Trial length (s)
|
||||
- Observed runtime
|
||||
* - `Trial bookkeeping /scheduling overhead <https://github.com/ray-project/ray/blob/master/release/tune_tests/scalability_tests/workloads/test_bookkeeping_overhead.py>`_
|
||||
- 10,000
|
||||
- 1
|
||||
- 1
|
||||
- 16
|
||||
- 1
|
||||
- | 715.27
|
||||
| (625 minimum)
|
||||
* - `Result throughput (many trials) <https://github.com/ray-project/ray/blob/master/release/tune_tests/scalability_tests/workloads/test_result_throughput_cluster.py>`_
|
||||
- 1,000
|
||||
- 0.1
|
||||
- 16
|
||||
- 64
|
||||
- 100
|
||||
- 168.18
|
||||
* - `Result throughput (many results) <https://github.com/ray-project/ray/blob/master/release/tune_tests/scalability_tests/workloads/test_result_throughput_single_node.py>`_
|
||||
- 96
|
||||
- 10
|
||||
- 1
|
||||
- 96
|
||||
- 100
|
||||
- 168.94
|
||||
* - `Network communication overhead <https://github.com/ray-project/ray/blob/master/release/tune_tests/scalability_tests/workloads/test_network_overhead.py>`_
|
||||
- 200
|
||||
- 1
|
||||
- 200
|
||||
- 2
|
||||
- 300
|
||||
- 2280.82
|
||||
* - `Long running, 3.75 GB checkpoints <https://github.com/ray-project/ray/blob/master/release/tune_tests/scalability_tests/workloads/test_long_running_large_checkpoints.py>`_
|
||||
- 16
|
||||
- | Results: 1/60
|
||||
| Checkpoint: 1/900
|
||||
- 1
|
||||
- 16
|
||||
- 86,400
|
||||
- 88687.41
|
||||
* - `Durable trainable <https://github.com/ray-project/ray/blob/master/release/tune_tests/scalability_tests/workloads/test_durable_trainable.py>`_
|
||||
- 16
|
||||
- | 10/60
|
||||
| with 10MB CP
|
||||
- 16
|
||||
- 2
|
||||
- 300
|
||||
- 392.42
|
||||
|
||||
|
||||
Below we discuss some insights on results where we observed much overhead.
|
||||
|
||||
|
||||
Result throughput
|
||||
-----------------
|
||||
|
||||
Result throughput describes the number of results Ray Tune can process in a given timeframe (e.g.
|
||||
"results per second").
|
||||
The higher the throughput, the more concurrent results can be processed without major delays.
|
||||
|
||||
Result throughput is limited by the time it takes to process results. When a trial reports results, it only
|
||||
continues training once the trial executor re-triggered the remote training function. If many trials report
|
||||
results at the same time, each subsequent remote training call is only triggered after handling that trial's
|
||||
results.
|
||||
|
||||
To speed the process up, Ray Tune adaptively buffers results, so that trial training is continued earlier if
|
||||
many trials are running in parallel and report many results at the same time. Still, processing hundreds of
|
||||
results per trial for dozens or hundreds of trials can become a bottleneck.
|
||||
|
||||
**Main insight**: Ray Tune will throw a warning when trial processing becomes a bottleneck. If you notice
|
||||
that this becomes a problem, please follow our guidelines outlined :ref:`in the FAQ <tune-bottlenecks>`.
|
||||
Generally, it is advised to not report too many results at the same time. Consider increasing the report
|
||||
intervals by a factor of 5-10x.
|
||||
|
||||
Below we present more detailed results on the result throughput performance.
|
||||
|
||||
Benchmarking many concurrent Tune trials
|
||||
""""""""""""""""""""""""""""""""""""""""
|
||||
|
||||
In this setup, loggers (CSV, JSON, and TensorBoardX) and trial synchronization are disabled, except when
|
||||
explicitly noted.
|
||||
|
||||
In this experiment, we're running many concurrent trials (up to 1,000) on a cluster. We then adjust the
|
||||
reporting frequency (number of results per second) of the trials to measure the throughput limits.
|
||||
|
||||
It seems that around 500 total results/second seem to be the threshold for acceptable performance
|
||||
when logging and synchronization are disabled. With logging enabled, around 50-100 results per second
|
||||
can still be managed without too much overhead, but after that measures to decrease incoming results
|
||||
should be considered.
|
||||
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
| # of trials | Results / second / trial | # Nodes | # CPUs / Node | Length of trial. | Current |
|
||||
+=============+==========================+=========+===============+==================+=========+
|
||||
| 1,000 | 10 | 16 | 64 | 100s | 248.39 |
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
| 1,000 | 1 | 16 | 64 | 100s | 175.00 |
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
| 1,000 | 0.1 with logging | 16 | 64 | 100s | 168.18 |
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
| 384 | 10 | 16 | 64 | 100s | 125.17 |
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
| 256 | 50 | 16 | 64 | 100s | 307.02 |
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
| 256 | 20 | 16 | 64 | 100s | 146.20 |
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
| 256 | 10 | 16 | 64 | 100s | 113.40 |
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
| 256 | 10 with logging | 16 | 64 | 100s | 436.12 |
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
| 256 | 0.1 with logging | 16 | 64 | 100s | 106.75 |
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
|
||||
|
||||
Benchmarking many Tune results on a single node
|
||||
"""""""""""""""""""""""""""""""""""""""""""""""
|
||||
|
||||
In this setup, loggers (CSV, JSON, and TensorBoardX) are disabled, except when
|
||||
explicitly noted.
|
||||
|
||||
In this experiment, we're running 96 concurrent trials on a single node. We then adjust the
|
||||
reporting frequency (number of results per second) of the trials to find the throughput limits.
|
||||
Compared to the cluster experiment setup, we report much more often, as we're running less total trials in parallel.
|
||||
|
||||
On a single node, throughput seems to be a bit higher. With logging, handling 1000 results per second
|
||||
seems acceptable in terms of overhead, though you should probably still target for a lower number.
|
||||
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
| # of trials | Results / second / trial | # Nodes | # CPUs / Node | Length of trial. | Current |
|
||||
+=============+==========================+=========+===============+==================+=========+
|
||||
| 96 | 500 | 1 | 96 | 100s | 959.32 |
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
| 96 | 100 | 1 | 96 | 100s | 219.48 |
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
| 96 | 80 | 1 | 96 | 100s | 197.15 |
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
| 96 | 50 | 1 | 96 | 100s | 110.55 |
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
| 96 | 50 with logging | 1 | 96 | 100s | 702.64 |
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
| 96 | 10 | 1 | 96 | 100s | 103.51 |
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
| 96 | 10 with logging | 1 | 96 | 100s | 168.94 |
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
|
||||
|
||||
Network overhead in Ray Tune
|
||||
----------------------------
|
||||
|
||||
Running Ray Tune on a distributed setup leads to network communication overhead. This is mostly due to
|
||||
trial synchronization, where results and checkpoints are periodically synchronized and sent via the network.
|
||||
Per default this happens via SSH, where connection initialization can take between 1 and 2 seconds each time.
|
||||
Since this is a blocking operation that happens on a per-trial basis, running many concurrent trials
|
||||
quickly becomes bottlenecked by this synchronization.
|
||||
|
||||
In this experiment, we ran a number of trials on a cluster. Each trial was run on a separate node. We
|
||||
varied the number of concurrent trials (and nodes) to see how much network communication affects
|
||||
total runtime.
|
||||
|
||||
**Main insight**: When running many concurrent trials in a distributed setup, consider using
|
||||
:ref:`cloud checkpointing <tune-cloud-checkpointing>` for checkpoint synchronization instead. Another option would
|
||||
be to use a shared storage and disable syncing to driver. The best practices are described
|
||||
:ref:`here for Kubernetes setups <tune-kubernetes>` but is applicable for any kind of setup.
|
||||
|
||||
|
||||
In the table below we present more detailed results on the network communication overhead.
|
||||
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
| # of trials | Results / second / trial | # Nodes | # CPUs / Node | Length of trial | Current |
|
||||
+=============+==========================+=========+===============+==================+=========+
|
||||
| 200 | 1 | 200 | 2 | 300s | 2280.82 |
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
| 100 | 1 | 100 | 2 | 300s | 1470 |
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
| 100 | 0.01 | 100 | 2 | 300s | 473.41 |
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
| 50 | 1 | 50 | 2 | 300s | 474.30 |
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
| 50 | 0.1 | 50 | 2 | 300s | 441.54 |
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
| 10 | 1 | 10 | 2 | 300s | 334.37 |
|
||||
+-------------+--------------------------+---------+---------------+------------------+---------+
|
||||
@@ -0,0 +1,192 @@
|
||||
.. _tune-search-space-tutorial:
|
||||
|
||||
Working with Tune Search Spaces
|
||||
===============================
|
||||
|
||||
Tune has a native interface for specifying search spaces.
|
||||
You can specify the search space via ``Tuner(param_space=...)``.
|
||||
|
||||
Thereby, you can either use the ``tune.grid_search`` primitive to use grid search:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
tuner = tune.Tuner(
|
||||
trainable,
|
||||
param_space={"bar": tune.grid_search([True, False])})
|
||||
results = tuner.fit()
|
||||
|
||||
|
||||
Or you can use one of the random sampling primitives to specify distributions (:doc:`/tune/api/search_space`):
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
tuner = tune.Tuner(
|
||||
trainable,
|
||||
param_space={
|
||||
"param1": tune.choice([True, False]),
|
||||
"bar": tune.uniform(0, 10),
|
||||
"alpha": tune.sample_from(lambda _: np.random.uniform(100) ** 2),
|
||||
"const": "hello" # It is also ok to specify constant values.
|
||||
})
|
||||
results = tuner.fit()
|
||||
|
||||
.. caution:: If you use a SearchAlgorithm, you may not be able to specify lambdas or grid search with this
|
||||
interface, as some search algorithms may not be compatible.
|
||||
|
||||
|
||||
To sample multiple times/run multiple trials, specify ``tune.RunConfig(num_samples=N``.
|
||||
If ``grid_search`` is provided as an argument, the *same* grid will be repeated ``N`` times.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# 13 different configs.
|
||||
tuner = tune.Tuner(trainable, tune_config=tune.TuneConfig(num_samples=13), param_space={
|
||||
"x": tune.choice([0, 1, 2]),
|
||||
}
|
||||
)
|
||||
tuner.fit()
|
||||
|
||||
# 13 different configs.
|
||||
tuner = tune.Tuner(trainable, tune_config=tune.TuneConfig(num_samples=13), param_space={
|
||||
"x": tune.choice([0, 1, 2]),
|
||||
"y": tune.randn([0, 1, 2]),
|
||||
}
|
||||
)
|
||||
tuner.fit()
|
||||
|
||||
# 4 different configs.
|
||||
tuner = tune.Tuner(trainable, tune_config=tune.TuneConfig(num_samples=1), param_space={"x": tune.grid_search([1, 2, 3, 4])})
|
||||
tuner.fit()
|
||||
|
||||
# 3 different configs.
|
||||
tuner = tune.Tuner(trainable, tune_config=tune.TuneConfig(num_samples=1), param_space={"x": tune.grid_search([1, 2, 3])})
|
||||
tuner.fit()
|
||||
|
||||
# 6 different configs.
|
||||
tuner = tune.Tuner(trainable, tune_config=tune.TuneConfig(num_samples=2), param_space={"x": tune.grid_search([1, 2, 3])})
|
||||
tuner.fit()
|
||||
|
||||
# 9 different configs.
|
||||
tuner = tune.Tuner(trainable, tune_config=tune.TuneConfig(num_samples=1), param_space={
|
||||
"x": tune.grid_search([1, 2, 3]),
|
||||
"y": tune.grid_search([a, b, c])}
|
||||
)
|
||||
tuner.fit()
|
||||
|
||||
# 18 different configs.
|
||||
tuner = tune.Tuner(trainable, tune_config=tune.TuneConfig(num_samples=2), param_space={
|
||||
"x": tune.grid_search([1, 2, 3]),
|
||||
"y": tune.grid_search([a, b, c])}
|
||||
)
|
||||
tuner.fit()
|
||||
|
||||
# 45 different configs.
|
||||
tuner = tune.Tuner(trainable, tune_config=tune.TuneConfig(num_samples=5), param_space={
|
||||
"x": tune.grid_search([1, 2, 3]),
|
||||
"y": tune.grid_search([a, b, c])}
|
||||
)
|
||||
tuner.fit()
|
||||
|
||||
|
||||
|
||||
Note that grid search and random search primitives are inter-operable.
|
||||
Each can be used independently or in combination with each other.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# 6 different configs.
|
||||
tuner = tune.Tuner(trainable, tune_config=tune.TuneConfig(num_samples=2), param_space={
|
||||
"x": tune.sample_from(...),
|
||||
"y": tune.grid_search([a, b, c])
|
||||
}
|
||||
)
|
||||
tuner.fit()
|
||||
|
||||
In the below example, ``num_samples=10`` repeats the 3x3 grid search 10 times,
|
||||
for a total of 90 trials, each with randomly sampled values of ``alpha`` and ``beta``.
|
||||
|
||||
.. code-block:: python
|
||||
:emphasize-lines: 12
|
||||
|
||||
tuner = tune.Tuner(
|
||||
my_trainable,
|
||||
run_config=tune.RunConfig(name="my_trainable"),
|
||||
# num_samples will repeat the entire config 10 times.
|
||||
tune_config=tune.TuneConfig(num_samples=10),
|
||||
param_space={
|
||||
# ``sample_from`` creates a generator to call the lambda once per trial.
|
||||
"alpha": tune.sample_from(lambda _: np.random.uniform(100)),
|
||||
# ``sample_from`` also supports "conditional search spaces"
|
||||
"beta": tune.sample_from(lambda config: config["alpha"] * np.random.normal()),
|
||||
"nn_layers": [
|
||||
# tune.grid_search will make it so that all values are evaluated.
|
||||
tune.grid_search([16, 64, 256]),
|
||||
tune.grid_search([16, 64, 256]),
|
||||
],
|
||||
},
|
||||
)
|
||||
tuner.fit()
|
||||
|
||||
.. tip::
|
||||
|
||||
Avoid passing large objects as values in the search space, as that will incur a performance overhead.
|
||||
Use :func:`tune.with_parameters <ray.tune.with_parameters>` to pass large objects in or load them inside your trainable
|
||||
from disk (making sure that all nodes have access to the files) or cloud storage.
|
||||
See :ref:`tune-bottlenecks` for more information.
|
||||
|
||||
.. _tune_custom-search:
|
||||
|
||||
How to use Custom and Conditional Search Spaces in Tune?
|
||||
--------------------------------------------------------
|
||||
|
||||
You'll often run into awkward search spaces (i.e., when one hyperparameter depends on another).
|
||||
Use ``tune.sample_from(func)`` to provide a **custom** callable function for generating a search space.
|
||||
|
||||
The parameter ``func`` should take in a ``config`` dict, which contains the values
|
||||
already sampled for the trial, letting you access other hyperparameters.
|
||||
This is useful for conditional distributions:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
tuner = tune.Tuner(
|
||||
...,
|
||||
param_space={
|
||||
# A random function
|
||||
"alpha": tune.sample_from(lambda _: np.random.uniform(100)),
|
||||
# Use the `config` dict to access other hyperparameters
|
||||
"beta": tune.sample_from(lambda config: config["alpha"] * np.random.normal())
|
||||
}
|
||||
)
|
||||
tuner.fit()
|
||||
|
||||
Here's an example showing a grid search over two nested parameters combined with random sampling from
|
||||
two lambda functions, generating 9 different trials.
|
||||
Note that the value of ``beta`` depends on the value of ``alpha``,
|
||||
which is represented by referencing ``config["alpha"]`` in the lambda function.
|
||||
This lets you specify conditional parameter distributions.
|
||||
|
||||
.. code-block:: python
|
||||
:emphasize-lines: 4-11
|
||||
|
||||
tuner = tune.Tuner(
|
||||
my_trainable,
|
||||
run_config=RunConfig(name="my_trainable"),
|
||||
param_space={
|
||||
"alpha": tune.sample_from(lambda _: np.random.uniform(100)),
|
||||
"beta": tune.sample_from(lambda config: config["alpha"] * np.random.normal()),
|
||||
"nn_layers": [
|
||||
tune.grid_search([16, 64, 256]),
|
||||
tune.grid_search([16, 64, 256]),
|
||||
],
|
||||
}
|
||||
)
|
||||
|
||||
.. note::
|
||||
|
||||
This format is not supported by every SearchAlgorithm, and only some SearchAlgorithms, like :ref:`HyperOpt <tune-hyperopt>`
|
||||
and :ref:`Optuna <tune-optuna>`, handle conditional search spaces at all.
|
||||
|
||||
In order to use conditional search spaces with :ref:`HyperOpt <tune-hyperopt>`,
|
||||
a `Hyperopt search space <http://hyperopt.github.io/hyperopt/getting-started/search_spaces/>`_ isnecessary.
|
||||
:ref:`Optuna <tune-optuna>` supports conditional search spaces through its define-by-run
|
||||
interface (:doc:`/tune/examples/optuna_example`).
|
||||
@@ -0,0 +1,190 @@
|
||||
.. _tune-stopping-guide:
|
||||
.. _tune-stopping-ref:
|
||||
|
||||
How to Define Stopping Criteria for a Ray Tune Experiment
|
||||
=========================================================
|
||||
|
||||
When running a Tune experiment, it can be challenging to determine the ideal duration of training beforehand. Stopping criteria in Tune can be useful for terminating training based on specific conditions.
|
||||
|
||||
For instance, one may want to set up the experiment to stop under the following circumstances:
|
||||
|
||||
1. Set up an experiment to end after ``N`` epochs or when the reported evaluation score surpasses a particular threshold, whichever occurs first.
|
||||
2. Stop the experiment after ``T`` seconds.
|
||||
3. Terminate when trials encounter runtime errors.
|
||||
4. Stop underperforming trials early by utilizing Tune's early-stopping schedulers.
|
||||
|
||||
This user guide will illustrate how to achieve these types of stopping criteria in a Tune experiment.
|
||||
|
||||
For all the code examples, we use the following training function for demonstration:
|
||||
|
||||
.. literalinclude:: /tune/doc_code/stopping.py
|
||||
:language: python
|
||||
:start-after: __stopping_example_trainable_start__
|
||||
:end-before: __stopping_example_trainable_end__
|
||||
|
||||
Stop a Tune experiment manually
|
||||
-------------------------------
|
||||
|
||||
If you send a ``SIGINT`` signal to the process running :meth:`Tuner.fit() <ray.tune.Tuner.fit>`
|
||||
(which is usually what happens when you press ``Ctrl+C`` in the terminal), Ray Tune shuts
|
||||
down training gracefully and saves the final experiment state.
|
||||
|
||||
.. note::
|
||||
|
||||
Forcefully terminating a Tune experiment, for example, through multiple ``Ctrl+C``
|
||||
commands, will not give Tune the opportunity to snapshot the experiment state
|
||||
one last time. If you resume the experiment in the future, this could result
|
||||
in resuming with stale state.
|
||||
|
||||
Ray Tune also accepts the ``SIGUSR1`` signal to interrupt training gracefully. This
|
||||
should be used when running Ray Tune in a remote Ray task
|
||||
as Ray will filter out ``SIGINT`` and ``SIGTERM`` signals per default.
|
||||
|
||||
|
||||
Stop using metric-based criteria
|
||||
--------------------------------
|
||||
|
||||
In addition to manual stopping, Tune provides several ways to stop experiments programmatically. The simplest way is to use metric-based criteria. These are a fixed set of thresholds that determine when the experiment should stop.
|
||||
|
||||
You can implement the stopping criteria using either a dictionary, a function, or a custom :class:`Stopper <ray.tune.stopper.Stopper>`.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Dictionary
|
||||
|
||||
If a dictionary is passed in, the keys may be any field in the return result of ``tune.report`` in the
|
||||
Function API or ``step()`` in the Class API.
|
||||
|
||||
.. note::
|
||||
|
||||
This includes :ref:`auto-filled metrics <tune-autofilled-metrics>` such as ``training_iteration``.
|
||||
|
||||
In the example below, each trial will be stopped either when it completes ``10`` iterations or when it
|
||||
reaches a mean accuracy of ``0.8`` or more.
|
||||
|
||||
These metrics are assumed to be **increasing**, so the trial will stop once the reported metric has exceeded the threshold specified in the dictionary.
|
||||
|
||||
.. literalinclude:: /tune/doc_code/stopping.py
|
||||
:language: python
|
||||
:start-after: __stopping_dict_start__
|
||||
:end-before: __stopping_dict_end__
|
||||
|
||||
.. tab-item:: User-defined Function
|
||||
|
||||
For more flexibility, you can pass in a function instead.
|
||||
If a function is passed in, it must take ``(trial_id: str, result: dict)`` as arguments and return a boolean
|
||||
(``True`` if trial should be stopped and ``False`` otherwise).
|
||||
|
||||
In the example below, each trial will be stopped either when it completes ``10`` iterations or when it
|
||||
reaches a mean accuracy of ``0.8`` or more.
|
||||
|
||||
.. literalinclude:: /tune/doc_code/stopping.py
|
||||
:language: python
|
||||
:start-after: __stopping_fn_start__
|
||||
:end-before: __stopping_fn_end__
|
||||
|
||||
.. tab-item:: Custom Stopper Class
|
||||
|
||||
Finally, you can implement the :class:`~ray.tune.stopper.Stopper` interface for
|
||||
stopping individual trials or even entire experiments based on custom stopping
|
||||
criteria. For example, the following example stops all trials after the criteria
|
||||
is achieved by any individual trial and prevents new ones from starting:
|
||||
|
||||
.. literalinclude:: /tune/doc_code/stopping.py
|
||||
:language: python
|
||||
:start-after: __stopping_cls_start__
|
||||
:end-before: __stopping_cls_end__
|
||||
|
||||
In the example, once any trial reaches a ``mean_accuracy`` of 0.8 or more, all trials will stop.
|
||||
|
||||
.. note::
|
||||
|
||||
When returning ``True`` from ``stop_all``, currently running trials will not stop immediately.
|
||||
They will stop after finishing their ongoing training iteration (after ``tune.report`` or ``step``).
|
||||
|
||||
Ray Tune comes with a set of out-of-the-box stopper classes. See the :ref:`Stopper <tune-stoppers>` documentation.
|
||||
|
||||
|
||||
Stop trials after a certain amount of time
|
||||
------------------------------------------
|
||||
|
||||
There are two choices to stop a Tune experiment based on time: stopping trials individually
|
||||
after a specified timeout, or stopping the full experiment after a certain amount of time.
|
||||
|
||||
Stop trials individually with a timeout
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can use a dictionary stopping criteria as described above, using the ``time_total_s`` metric that is auto-filled by Tune.
|
||||
|
||||
.. literalinclude:: /tune/doc_code/stopping.py
|
||||
:language: python
|
||||
:start-after: __stopping_trials_by_time_start__
|
||||
:end-before: __stopping_trials_by_time_end__
|
||||
|
||||
.. note::
|
||||
|
||||
You need to include some intermediate reporting via :meth:`tune.report <ray.tune.report>`
|
||||
if using the :ref:`Function Trainable API <tune-function-api>`.
|
||||
Each report will automatically record the trial's ``time_total_s``, which allows Tune to stop based on time as a metric.
|
||||
|
||||
If the training loop hangs somewhere, Tune will not be able to intercept the training and stop the trial for you.
|
||||
In this case, you can explicitly implement timeout logic in the training loop.
|
||||
|
||||
|
||||
Stop the experiment with a timeout
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use the ``TuneConfig(time_budget_s)`` configuration to tell Tune to stop the experiment after ``time_budget_s`` seconds.
|
||||
|
||||
.. literalinclude:: /tune/doc_code/stopping.py
|
||||
:language: python
|
||||
:start-after: __stopping_experiment_by_time_start__
|
||||
:end-before: __stopping_experiment_by_time_end__
|
||||
|
||||
.. note::
|
||||
|
||||
You need to include some intermediate reporting via :meth:`tune.report <ray.tune.report>`
|
||||
if using the :ref:`Function Trainable API <tune-function-api>`, for the same reason as above.
|
||||
|
||||
|
||||
Stop on trial failures
|
||||
----------------------
|
||||
|
||||
In addition to stopping trials based on their performance, you can also stop the entire experiment if any trial encounters a runtime error. To do this, you can use the :class:`ray.tune.FailureConfig` class.
|
||||
|
||||
With this configuration, if any trial encounters an error, the entire experiment will stop immediately.
|
||||
|
||||
.. literalinclude:: /tune/doc_code/stopping.py
|
||||
:language: python
|
||||
:start-after: __stopping_on_trial_error_start__
|
||||
:end-before: __stopping_on_trial_error_end__
|
||||
|
||||
This is useful when you are debugging a Tune experiment with many trials.
|
||||
|
||||
|
||||
Early stopping with Tune schedulers
|
||||
-----------------------------------
|
||||
|
||||
Another way to stop Tune experiments is to use early stopping schedulers.
|
||||
These schedulers monitor the performance of trials and stop them early if they are not making sufficient progress.
|
||||
|
||||
:class:`~ray.tune.schedulers.AsyncHyperBandScheduler` and :class:`~ray.tune.schedulers.HyperBandForBOHB` are examples of early stopping schedulers built into Tune.
|
||||
See :ref:`the Tune scheduler API reference <tune-schedulers>` for a full list, as well as more realistic examples.
|
||||
|
||||
In the following example, we use both a dictionary stopping criteria along with an early-stopping criteria:
|
||||
|
||||
.. literalinclude:: /tune/doc_code/stopping.py
|
||||
:language: python
|
||||
:start-after: __early_stopping_start__
|
||||
:end-before: __early_stopping_end__
|
||||
|
||||
Summary
|
||||
-------
|
||||
|
||||
In this user guide, we learned how to stop Tune experiments using metrics, trial errors,
|
||||
and early stopping schedulers.
|
||||
|
||||
See the following resources for more information:
|
||||
|
||||
- :ref:`Tune Stopper API reference <tune-stoppers>`
|
||||
- For an experiment that was manually interrupted or the cluster dies unexpectedly while trials are still running, it's possible to resume the experiment. See :ref:`tune-fault-tolerance-ref`.
|
||||
@@ -0,0 +1,220 @@
|
||||
.. _tune-storage-options:
|
||||
|
||||
How to Configure Persistent Storage in Ray Tune
|
||||
===============================================
|
||||
|
||||
.. seealso::
|
||||
|
||||
Before diving into storage options, one can take a look at
|
||||
:ref:`the different types of data stored by Tune <tune-persisted-experiment-data>`.
|
||||
|
||||
Tune allows you to configure persistent storage options to enable following use cases in a distributed Ray cluster:
|
||||
|
||||
- **Trial-level fault tolerance**: When trials are restored (e.g. after a node failure or when the experiment was paused),
|
||||
they may be scheduled on different nodes, but still would need access to their latest checkpoint.
|
||||
- **Experiment-level fault tolerance**: For an entire experiment to be restored (e.g. if the cluster crashes unexpectedly),
|
||||
Tune needs to be able to access the latest experiment state, along with all trial
|
||||
checkpoints to start from where the experiment left off.
|
||||
- **Post-experiment analysis**: A consolidated location storing data from all trials is useful for post-experiment analysis
|
||||
such as accessing the best checkpoints and hyperparameter configs after the cluster has already been terminated.
|
||||
- **Bridge with downstream serving/batch inference tasks**: With a configured storage, you can easily access the models
|
||||
and artifacts generated by trials, share them with others or use them in downstream tasks.
|
||||
|
||||
|
||||
Storage Options in Tune
|
||||
-----------------------
|
||||
|
||||
Tune provides support for three scenarios:
|
||||
|
||||
1. When using cloud storage (e.g. AWS S3 or Google Cloud Storage) accessible by all machines in the cluster.
|
||||
2. When using a network filesystem (NFS) mounted to all machines in the cluster.
|
||||
3. When running Tune on a single node and using the local filesystem as the persistent storage location.
|
||||
|
||||
.. note::
|
||||
|
||||
A network filesystem or cloud storage can be configured for single-node
|
||||
experiments. This can be useful to persist your experiment results in external storage
|
||||
if, for example, the instance you run your experiment on clears its local storage
|
||||
after termination.
|
||||
|
||||
.. seealso::
|
||||
|
||||
See :class:`~ray.tune.SyncConfig` for the full set of configuration options as well as more details.
|
||||
|
||||
|
||||
.. _tune-cloud-checkpointing:
|
||||
|
||||
Configuring Tune with cloud storage (AWS S3, Google Cloud Storage)
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
If all nodes in a Ray cluster have access to cloud storage, e.g. AWS S3 or Google Cloud Storage (GCS),
|
||||
then all experiment outputs can be saved in a shared cloud bucket.
|
||||
|
||||
We can configure cloud storage by telling Ray Tune to **upload to a remote** ``storage_path``:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from ray import tune
|
||||
|
||||
tuner = tune.Tuner(
|
||||
trainable,
|
||||
run_config=tune.RunConfig(
|
||||
name="experiment_name",
|
||||
storage_path="s3://bucket-name/sub-path/",
|
||||
)
|
||||
)
|
||||
tuner.fit()
|
||||
|
||||
In this example, all experiment results can be found in the shared storage at ``s3://bucket-name/sub-path/experiment_name`` for further processing.
|
||||
|
||||
.. note::
|
||||
|
||||
The head node will not have access to all experiment results locally. If you want to process
|
||||
e.g. the best checkpoint further, you will first have to fetch it from the cloud storage.
|
||||
|
||||
Experiment restoration should also be done using the experiment directory at the cloud storage
|
||||
URI, rather than the local experiment directory on the head node. See :ref:`here for an example <tune-syncing-restore-from-uri>`.
|
||||
|
||||
|
||||
|
||||
Configuring Tune with a network filesystem (NFS)
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
If all Ray nodes have access to a network filesystem, e.g. AWS EFS or Google Cloud Filestore,
|
||||
they can all write experiment outputs to this directory.
|
||||
|
||||
All we need to do is **set the shared network filesystem as the path to save results**.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from ray import tune
|
||||
|
||||
tuner = tune.Tuner(
|
||||
trainable,
|
||||
run_config=tune.RunConfig(
|
||||
name="experiment_name",
|
||||
storage_path="/mnt/path/to/shared/storage/",
|
||||
)
|
||||
)
|
||||
tuner.fit()
|
||||
|
||||
In this example, all experiment results can be found in the shared storage at ``/path/to/shared/storage/experiment_name`` for further processing.
|
||||
|
||||
|
||||
.. _tune-default-syncing:
|
||||
|
||||
Configure Tune without external persistent storage
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
On a single-node cluster
|
||||
************************
|
||||
|
||||
If you're just running an experiment on a single node (e.g., on a laptop), Tune will use the
|
||||
local filesystem as the default storage location for checkpoints and other artifacts.
|
||||
Results are saved to ``~/ray_results`` in a sub-directory with a unique auto-generated name by default,
|
||||
unless you customize this with ``storage_path`` and ``name`` in :class:`~ray.tune.RunConfig`.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from ray import tune
|
||||
|
||||
tuner = tune.Tuner(
|
||||
trainable,
|
||||
run_config=tune.RunConfig(
|
||||
storage_path="/tmp/custom/storage/path",
|
||||
name="experiment_name",
|
||||
)
|
||||
)
|
||||
tuner.fit()
|
||||
|
||||
In this example, all experiment results can be found locally at ``/tmp/custom/storage/path/experiment_name`` for further processing.
|
||||
|
||||
|
||||
On a multi-node cluster (Deprecated)
|
||||
************************************
|
||||
|
||||
.. warning::
|
||||
|
||||
When running on multiple nodes, using the local filesystem of the head node as the persistent storage location is *deprecated*.
|
||||
If you save trial checkpoints and run on a multi-node cluster, Tune will raise an error by default, if NFS or cloud storage is not setup.
|
||||
See `this issue <https://github.com/ray-project/ray/issues/37177>`_ for more information.
|
||||
|
||||
|
||||
Examples
|
||||
--------
|
||||
|
||||
Let's show some examples of configuring storage location and synchronization options.
|
||||
We'll also show how to resume the experiment for each of the examples, in the case that your experiment gets interrupted.
|
||||
See :ref:`tune-fault-tolerance-ref` for more information on resuming experiments.
|
||||
|
||||
In each example, we'll give a practical explanation of how *trial checkpoints* are saved
|
||||
across the cluster and the external storage location (if one is provided).
|
||||
See :ref:`tune-persisted-experiment-data` for an overview of other experiment data that Tune needs to persist.
|
||||
|
||||
|
||||
Example: Running Tune with cloud storage
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Let's assume that you're running this example script from your Ray cluster's head node.
|
||||
|
||||
In the example below, ``my_trainable`` is a Tune :ref:`trainable <trainable-docs>`
|
||||
that implements saving and loading checkpoints.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import os
|
||||
import ray
|
||||
from ray import tune
|
||||
from your_module import my_trainable
|
||||
|
||||
|
||||
tuner = tune.Tuner(
|
||||
my_trainable,
|
||||
run_config=tune.RunConfig(
|
||||
# Name of your experiment
|
||||
name="my-tune-exp",
|
||||
# Configure how experiment data and checkpoints are persisted.
|
||||
# We recommend cloud storage checkpointing as it survives the cluster when
|
||||
# instances are terminated and has better performance.
|
||||
storage_path="s3://my-checkpoints-bucket/path/",
|
||||
checkpoint_config=tune.CheckpointConfig(
|
||||
# We'll keep the best five checkpoints at all times
|
||||
# (with the highest AUC scores, a metric reported by the trainable)
|
||||
checkpoint_score_attribute="max-auc",
|
||||
checkpoint_score_order="max",
|
||||
num_to_keep=5,
|
||||
),
|
||||
),
|
||||
)
|
||||
# This starts the run!
|
||||
results = tuner.fit()
|
||||
|
||||
In this example, trial checkpoints will be saved to: ``s3://my-checkpoints-bucket/path/my-tune-exp/<trial_name>/checkpoint_<step>``
|
||||
|
||||
.. _tune-syncing-restore-from-uri:
|
||||
|
||||
If this run stopped for any reason (ex: user CTRL+C, terminated due to out of memory issues),
|
||||
you can resume it any time starting from the experiment state saved in the cloud:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from ray import tune
|
||||
tuner = tune.Tuner.restore(
|
||||
"s3://my-checkpoints-bucket/path/my-tune-exp",
|
||||
trainable=my_trainable,
|
||||
resume_errored=True,
|
||||
)
|
||||
tuner.fit()
|
||||
|
||||
|
||||
There are a few options for restoring an experiment:
|
||||
``resume_unfinished``, ``resume_errored`` and ``restart_errored``.
|
||||
Please see the documentation of
|
||||
:meth:`~ray.tune.Tuner.restore` for more details.
|
||||
|
||||
|
||||
Advanced configuration
|
||||
----------------------
|
||||
|
||||
See :ref:`Ray Train's section on advanced storage configuration <train-storage-advanced>`.
|
||||
All of the configurations also apply to Ray Tune.
|
||||
@@ -0,0 +1,198 @@
|
||||
.. _tune-trial-checkpoint:
|
||||
|
||||
How to Save and Load Trial Checkpoints
|
||||
======================================
|
||||
|
||||
Trial checkpoints are one of :ref:`the three types of data stored by Tune <tune-persisted-experiment-data>`.
|
||||
These are user-defined and are meant to snapshot your training progress!
|
||||
|
||||
Trial-level checkpoints are saved via the :ref:`Tune Trainable <tune-60-seconds>` API: this is how you define your
|
||||
custom training logic, and it's also where you'll define which trial state to checkpoint.
|
||||
In this guide, we will show how to save and load checkpoints for Tune's Function Trainable and Class Trainable APIs,
|
||||
as well as walk you through configuration options.
|
||||
|
||||
.. _tune-function-trainable-checkpointing:
|
||||
|
||||
Function API Checkpointing
|
||||
--------------------------
|
||||
|
||||
If using Ray Tune's Function API, one can save and load checkpoints in the following manner.
|
||||
To create a checkpoint, use the :meth:`~ray.tune.Checkpoint.from_directory` APIs.
|
||||
|
||||
.. literalinclude:: /tune/doc_code/trial_checkpoint.py
|
||||
:language: python
|
||||
:start-after: __function_api_checkpointing_from_dir_start__
|
||||
:end-before: __function_api_checkpointing_from_dir_end__
|
||||
|
||||
In the above code snippet:
|
||||
|
||||
- We implement *checkpoint saving* with :meth:`tune.report(..., checkpoint=checkpoint) <ray.tune.report>`. Note that every checkpoint must be reported alongside a set of metrics -- this way, checkpoints can be ordered with respect to a specified metric.
|
||||
- The saved checkpoint during training iteration `epoch` is saved to the path ``<storage_path>/<exp_name>/<trial_name>/checkpoint_<epoch>`` on the node on which training happens and can be further synced to a consolidated storage location depending on the :ref:`storage configuration <tune-storage-options>`.
|
||||
- We implement *checkpoint loading* with :meth:`tune.get_checkpoint() <ray.tune.get_checkpoint>`. This will be populated with a trial's latest checkpoint whenever Tune restores a trial. This happens when (1) a trial is configured to retry after encountering a failure, (2) the experiment is being restored, and (3) the trial is being resumed after a pause (ex: :doc:`PBT </tune/examples/pbt_guide>`).
|
||||
|
||||
.. TODO: for (1), link to tune fault tolerance guide. For (2), link to tune restore guide.
|
||||
|
||||
.. note::
|
||||
``checkpoint_frequency`` and ``checkpoint_at_end`` will not work with Function API checkpointing.
|
||||
These are configured manually with Function Trainable. For example, if you want to checkpoint every three
|
||||
epochs, you can do so through:
|
||||
|
||||
.. literalinclude:: /tune/doc_code/trial_checkpoint.py
|
||||
:language: python
|
||||
:start-after: __function_api_checkpointing_periodic_start__
|
||||
:end-before: __function_api_checkpointing_periodic_end__
|
||||
|
||||
|
||||
See :class:`here for more information on creating checkpoints <ray.tune.Checkpoint>`.
|
||||
|
||||
|
||||
.. _tune-class-trainable-checkpointing:
|
||||
|
||||
Class API Checkpointing
|
||||
-----------------------
|
||||
|
||||
You can also implement checkpoint/restore using the Trainable Class API:
|
||||
|
||||
.. literalinclude:: /tune/doc_code/trial_checkpoint.py
|
||||
:language: python
|
||||
:start-after: __class_api_checkpointing_start__
|
||||
:end-before: __class_api_checkpointing_end__
|
||||
|
||||
You can checkpoint with three different mechanisms: manually, periodically, and at termination.
|
||||
|
||||
.. _tune-class-trainable-checkpointing_manual-checkpointing:
|
||||
|
||||
Manual Checkpointing by Trainable
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
A custom Trainable can manually trigger checkpointing by returning ``should_checkpoint: True``
|
||||
(or ``tune.result.SHOULD_CHECKPOINT: True``) in the result dictionary of `step`.
|
||||
This can be especially helpful in spot instances:
|
||||
|
||||
.. literalinclude:: /tune/doc_code/trial_checkpoint.py
|
||||
:language: python
|
||||
:start-after: __class_api_manual_checkpointing_start__
|
||||
:end-before: __class_api_manual_checkpointing_end__
|
||||
|
||||
In the above example, if ``detect_instance_preemption`` returns True, manual checkpointing can be triggered.
|
||||
|
||||
|
||||
.. _tune-callback-checkpointing:
|
||||
|
||||
Manual Checkpointing by Tuner Callback
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Similar to :ref:`tune-class-trainable-checkpointing_manual-checkpointing`,
|
||||
you can also trigger checkpointing through :class:`Tuner <ray.tune.Tuner>` :class:`Callback <ray.tune.callback.Callback>` methods
|
||||
by setting the ``result["should_checkpoint"] = True`` (or ``result[tune.result.SHOULD_CHECKPOINT] = True``) flag
|
||||
within the :meth:`on_trial_result() <ray.tune.Callback.on_trial_result>` method of your custom callback.
|
||||
In contrast to checkpointing within the Trainable Class API, this approach decouples checkpointing logic from
|
||||
the training logic, and provides access to all :class:`Trial <ray.tune.Trial>` instances allowing for more
|
||||
complex checkpointing strategies.
|
||||
|
||||
.. literalinclude:: /tune/doc_code/trial_checkpoint.py
|
||||
:language: python
|
||||
:start-after: __callback_api_checkpointing_start__
|
||||
:end-before: __callback_api_checkpointing_end__
|
||||
|
||||
|
||||
Periodic Checkpointing
|
||||
~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
This can be enabled by setting ``checkpoint_frequency=N`` to checkpoint trials every *N* iterations, e.g.:
|
||||
|
||||
.. literalinclude:: /tune/doc_code/trial_checkpoint.py
|
||||
:language: python
|
||||
:start-after: __class_api_periodic_checkpointing_start__
|
||||
:end-before: __class_api_periodic_checkpointing_end__
|
||||
|
||||
|
||||
Checkpointing at Termination
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The checkpoint_frequency may not coincide with the exact end of an experiment.
|
||||
If you want a checkpoint to be created at the end of a trial, you can additionally set the ``checkpoint_at_end=True``:
|
||||
|
||||
.. literalinclude:: /tune/doc_code/trial_checkpoint.py
|
||||
:language: python
|
||||
:start-after: __class_api_end_checkpointing_start__
|
||||
:end-before: __class_api_end_checkpointing_end__
|
||||
|
||||
|
||||
Configurations
|
||||
--------------
|
||||
Checkpointing can be configured through :class:`CheckpointConfig <ray.tune.CheckpointConfig>`.
|
||||
Some of the configurations do not apply to Function Trainable API, since checkpointing frequency
|
||||
is determined manually within the user-defined training loop. See the compatibility matrix below.
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
|
||||
* -
|
||||
- Class API
|
||||
- Function API
|
||||
* - ``num_to_keep``
|
||||
- ✅
|
||||
- ✅
|
||||
* - ``checkpoint_score_attribute``
|
||||
- ✅
|
||||
- ✅
|
||||
* - ``checkpoint_score_order``
|
||||
- ✅
|
||||
- ✅
|
||||
* - ``checkpoint_frequency``
|
||||
- ✅
|
||||
- ❌
|
||||
* - ``checkpoint_at_end``
|
||||
- ✅
|
||||
- ❌
|
||||
|
||||
|
||||
|
||||
Summary
|
||||
-------
|
||||
|
||||
In this user guide, we covered how to save and load trial checkpoints in Tune. Once checkpointing is enabled,
|
||||
move onto one of the following guides to find out how to:
|
||||
|
||||
- :ref:`Extract checkpoints from Tune experiment results <tune-analysis-guide>`
|
||||
- :ref:`Configure persistent storage options <tune-storage-options>` for a :ref:`distributed Tune experiment <tune-distributed-ref>`
|
||||
|
||||
.. _tune-persisted-experiment-data:
|
||||
|
||||
Appendix: Types of data stored by Tune
|
||||
--------------------------------------
|
||||
|
||||
Experiment Checkpoints
|
||||
~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Experiment-level checkpoints save the experiment state. This includes the state of the searcher,
|
||||
the list of trials and their statuses (e.g., PENDING, RUNNING, TERMINATED, ERROR), and
|
||||
metadata pertaining to each trial (e.g., hyperparameter configuration, some derived trial results
|
||||
(min, max, last), etc).
|
||||
|
||||
The experiment-level checkpoint is periodically saved by the driver on the head node.
|
||||
By default, the frequency at which it is saved is automatically
|
||||
adjusted so that at most 5% of the time is spent saving experiment checkpoints,
|
||||
and the remaining time is used for handling training results and scheduling.
|
||||
This time can also be adjusted with the
|
||||
:ref:`TUNE_GLOBAL_CHECKPOINT_S environment variable <tune-env-vars>`.
|
||||
|
||||
Trial Checkpoints
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
Trial-level checkpoints capture the per-trial state. This often includes the model and optimizer states.
|
||||
Following are a few uses of trial checkpoints:
|
||||
|
||||
- If the trial is interrupted for some reason (e.g., on spot instances), it can be resumed from the last state. No training time is lost.
|
||||
- Some searchers or schedulers pause trials to free up resources for other trials to train in the meantime. This only makes sense if the trials can then continue training from the latest state.
|
||||
- The checkpoint can be later used for other downstream tasks like batch inference.
|
||||
|
||||
Learn how to save and load trial checkpoints :ref:`here <tune-trial-checkpoint>`.
|
||||
|
||||
Trial Results
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
Metrics reported by trials are saved and logged to their respective trial directories.
|
||||
This is the data stored in CSV, JSON or Tensorboard (events.out.tfevents.*) formats.
|
||||
that can be inspected by Tensorboard and used for post-experiment analysis.
|
||||
@@ -0,0 +1,394 @@
|
||||
# Getting Data in and out of Tune
|
||||
|
||||
Often, you will find yourself needing to pass data into Tune [Trainables](tune_60_seconds_trainables) (datasets, models, other large parameters) and get data out of them (metrics, checkpoints, other artifacts). In this guide, we'll explore different ways of doing that and see in what circumstances they should be used.
|
||||
|
||||
```{contents}
|
||||
:local:
|
||||
:backlinks: none
|
||||
```
|
||||
|
||||
Let's start by defining a simple Trainable function. We'll be expanding this function with different functionality as we go.
|
||||
|
||||
```python
|
||||
import random
|
||||
import time
|
||||
import pandas as pd
|
||||
|
||||
|
||||
def training_function(config):
|
||||
# For now, we have nothing here.
|
||||
data = None
|
||||
model = {"hyperparameter_a": None, "hyperparameter_b": None}
|
||||
epochs = 0
|
||||
|
||||
# Simulate training & evaluation - we obtain back a "metric" and a "trained_model".
|
||||
for epoch in range(epochs):
|
||||
# Simulate doing something expensive.
|
||||
time.sleep(1)
|
||||
metric = (0.1 + model["hyperparameter_a"] * epoch / 100) ** (
|
||||
-1
|
||||
) + model["hyperparameter_b"] * 0.1 * data["A"].sum()
|
||||
trained_model = {"state": model, "epoch": epoch}
|
||||
```
|
||||
|
||||
Our `training_function` function requires a pandas DataFrame, a model with some hyperparameters and the number of epochs to train the model for as inputs. The hyperparameters of the model impact the metric returned, and in each epoch (iteration of training), the `trained_model` state is changed.
|
||||
|
||||
We will run hyperparameter optimization using the [Tuner API](tune-run-ref).
|
||||
|
||||
```python
|
||||
from ray.tune import Tuner
|
||||
from ray import tune
|
||||
|
||||
tuner = Tuner(training_function, tune_config=tune.TuneConfig(num_samples=4))
|
||||
```
|
||||
|
||||
## Getting data into Tune
|
||||
|
||||
First order of business is to provide the inputs for the Trainable. We can broadly separate them into two categories - variables and constants.
|
||||
|
||||
Variables are the parameters we want to tune. They will be different for every [Trial](tune_60_seconds_trials). For example, those may be the learning rate and batch size for a neural network, number of trees and the maximum depth for a random forest, or the data partition if you are using Tune as an execution engine for batch training.
|
||||
|
||||
Constants are the parameters that are the same for every Trial. Those can be the number of epochs, model hyperparameters we want to set but not tune, the dataset and so on. Often, the constants will be quite large (e.g. the dataset or the model).
|
||||
|
||||
```{warning}
|
||||
Objects from the outer scope of the `training_function` will also be automatically serialized and sent to Trial Actors, which may lead to unintended behavior. Examples include global locks not working (as each Actor operates on a copy) or general errors related to serialization. Best practice is to not refer to any objects from outer scope in the `training_function`.
|
||||
```
|
||||
|
||||
### Passing data into a Tune run through search spaces
|
||||
|
||||
```{note}
|
||||
TL;DR - use the `param_space` argument to specify small, serializable constants and variables.
|
||||
```
|
||||
|
||||
The first way of passing inputs into Trainables is the [*search space*](tune-key-concepts-search-spaces) (it may also be called *parameter space* or *config*). In the Trainable itself, it maps to the `config` dict passed in as an argument to the function. You define the search space using the `param_space` argument of the `Tuner`. The search space is a dict and may be composed of [*distributions*](<tune-search-space>), which will sample a different value for each Trial, or of constant values. The search space may be composed of nested dictionaries, and those in turn can have distributions as well.
|
||||
|
||||
```{warning}
|
||||
Each value in the search space will be saved directly in the Trial metadata. This means that every value in the search space **must** be serializable and take up a small amount of memory.
|
||||
```
|
||||
|
||||
For example, passing in a large pandas DataFrame or an unserializable model object as a value in the search space will lead to unwanted behavior. At best it will cause large slowdowns and disk space usage as Trial metadata saved to disk will also contain this data. At worst, an exception will be raised, as the data cannot be sent over to the Trial workers. For more details, see {ref}`tune-bottlenecks`.
|
||||
|
||||
Instead, use strings or other identifiers as your values, and initialize/load the objects inside your Trainable directly depending on those.
|
||||
|
||||
```{note}
|
||||
[Datasets](data_quickstart) can be used as values in the search space directly.
|
||||
```
|
||||
|
||||
In our example, we want to tune the two model hyperparameters. We also want to set the number of epochs, so that we can easily tweak it later. For the hyperparameters, we will use the `tune.uniform` distribution. We will also modify the `training_function` to obtain those values from the `config` dictionary.
|
||||
|
||||
```python
|
||||
def training_function(config):
|
||||
# For now, we have nothing here.
|
||||
data = None
|
||||
|
||||
model = {
|
||||
"hyperparameter_a": config["hyperparameter_a"],
|
||||
"hyperparameter_b": config["hyperparameter_b"],
|
||||
}
|
||||
epochs = config["epochs"]
|
||||
|
||||
# Simulate training & evaluation - we obtain back a "metric" and a "trained_model".
|
||||
for epoch in range(epochs):
|
||||
# Simulate doing something expensive.
|
||||
time.sleep(1)
|
||||
metric = (0.1 + model["hyperparameter_a"] * epoch / 100) ** (
|
||||
-1
|
||||
) + model["hyperparameter_b"] * 0.1 * data["A"].sum()
|
||||
trained_model = {"state": model, "epoch": epoch}
|
||||
|
||||
|
||||
tuner = Tuner(
|
||||
training_function,
|
||||
param_space={
|
||||
"hyperparameter_a": tune.uniform(0, 20),
|
||||
"hyperparameter_b": tune.uniform(-100, 100),
|
||||
"epochs": 10,
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
### Using `tune.with_parameters` access data in Tune runs
|
||||
|
||||
```{note}
|
||||
TL;DR - use the `tune.with_parameters` util function to specify large constant parameters.
|
||||
```
|
||||
|
||||
If we have large objects that are constant across Trials, we can use the {func}`tune.with_parameters <ray.tune.with_parameters>` utility to pass them into the Trainable directly. The objects will be stored in the [Ray object store](serialization-guide) so that each Trial worker may access them to obtain a local copy to use in its process.
|
||||
|
||||
```{tip}
|
||||
Objects put into the Ray object store must be serializable.
|
||||
```
|
||||
|
||||
Note that the serialization (once) and deserialization (for each Trial) of large objects may incur a performance overhead.
|
||||
|
||||
In our example, we will pass the `data` DataFrame using `tune.with_parameters`. In order to do that, we need to modify our function signature to include `data` as an argument.
|
||||
|
||||
```python
|
||||
def training_function(config, data):
|
||||
model = {
|
||||
"hyperparameter_a": config["hyperparameter_a"],
|
||||
"hyperparameter_b": config["hyperparameter_b"],
|
||||
}
|
||||
epochs = config["epochs"]
|
||||
|
||||
# Simulate training & evaluation - we obtain back a "metric" and a "trained_model".
|
||||
for epoch in range(epochs):
|
||||
# Simulate doing something expensive.
|
||||
time.sleep(1)
|
||||
metric = (0.1 + model["hyperparameter_a"] * epoch / 100) ** (
|
||||
-1
|
||||
) + model["hyperparameter_b"] * 0.1 * data["A"].sum()
|
||||
trained_model = {"state": model, "epoch": epoch}
|
||||
|
||||
|
||||
tuner = Tuner(
|
||||
training_function,
|
||||
param_space={
|
||||
"hyperparameter_a": tune.uniform(0, 20),
|
||||
"hyperparameter_b": tune.uniform(-100, 100),
|
||||
"epochs": 10,
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
Next step is to wrap the `training_function` using `tune.with_parameters` before passing it into the `Tuner`. Every keyword argument of the `tune.with_parameters` call will be mapped to the keyword arguments in the Trainable signature.
|
||||
|
||||
```python
|
||||
data = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
|
||||
|
||||
tuner = Tuner(
|
||||
tune.with_parameters(training_function, data=data),
|
||||
param_space={
|
||||
"hyperparameter_a": tune.uniform(0, 20),
|
||||
"hyperparameter_b": tune.uniform(-100, 100),
|
||||
"epochs": 10,
|
||||
},
|
||||
tune_config=tune.TuneConfig(num_samples=4),
|
||||
)
|
||||
```
|
||||
|
||||
### Loading data in a Tune Trainable
|
||||
|
||||
You can also load data directly in Trainable from e.g. cloud storage, shared file storage such as NFS, or from the local disk of the Trainable worker.
|
||||
|
||||
```{warning}
|
||||
When loading from disk, ensure that all nodes in your cluster have access to the file you are trying to load.
|
||||
```
|
||||
|
||||
A common use-case is to load the dataset from S3 or any other cloud storage with pandas, arrow or any other framework.
|
||||
|
||||
The working directory of the Trainable worker will be automatically changed to the corresponding Trial directory. For more details, see {ref}`tune-working-dir`.
|
||||
|
||||
Our tuning run can now be run, though we will not yet obtain any meaningful outputs back.
|
||||
|
||||
```python
|
||||
results = tuner.fit()
|
||||
```
|
||||
|
||||
## Getting data out of Ray Tune
|
||||
|
||||
We can now run our tuning run using the `training_function` Trainable. The next step is to report *metrics* to Tune that can be used to guide the optimization. We will also want to *checkpoint* our trained models so that we can resume the training after an interruption, and to use them for prediction later.
|
||||
|
||||
The `ray.tune.report` API is used to get data out of the Trainable workers. It can be called multiple times in the Trainable function. Each call corresponds to one iteration (epoch, step, tree) of training.
|
||||
|
||||
### Reporting metrics with Tune
|
||||
|
||||
*Metrics* are values passed through the `metrics` argument in a `tune.report` call. Metrics can be used by Tune [Search Algorithms](search-alg-ref) and [Schedulers](schedulers-ref) to direct the search. After the tuning run is complete, you can [analyze the results](tune-analysis-guide), which include the reported metrics.
|
||||
|
||||
```{note}
|
||||
Similarly to search space values, each value reported as a metric will be saved directly in the Trial metadata. This means that every value reported as a metric **must** be serializable and take up a small amount of memory.
|
||||
```
|
||||
|
||||
```{note}
|
||||
Tune will automatically include some metrics, such as the training iteration, timestamp and more. See [here](tune-autofilled-metrics) for the entire list.
|
||||
```
|
||||
|
||||
In our example, we want to maximize the `metric`. We will report it each epoch to Tune, and set the `metric` and `mode` arguments in `tune.TuneConfig` to let Tune know that it should use it as the optimization objective.
|
||||
|
||||
```python
|
||||
from ray import tune
|
||||
|
||||
|
||||
def training_function(config, data):
|
||||
model = {
|
||||
"hyperparameter_a": config["hyperparameter_a"],
|
||||
"hyperparameter_b": config["hyperparameter_b"],
|
||||
}
|
||||
epochs = config["epochs"]
|
||||
|
||||
# Simulate training & evaluation - we obtain back a "metric" and a "trained_model".
|
||||
for epoch in range(epochs):
|
||||
# Simulate doing something expensive.
|
||||
time.sleep(1)
|
||||
metric = (0.1 + model["hyperparameter_a"] * epoch / 100) ** (
|
||||
-1
|
||||
) + model["hyperparameter_b"] * 0.1 * data["A"].sum()
|
||||
trained_model = {"state": model, "epoch": epoch}
|
||||
tune.report(metrics={"metric": metric})
|
||||
|
||||
|
||||
tuner = Tuner(
|
||||
tune.with_parameters(training_function, data=data),
|
||||
param_space={
|
||||
"hyperparameter_a": tune.uniform(0, 20),
|
||||
"hyperparameter_b": tune.uniform(-100, 100),
|
||||
"epochs": 10,
|
||||
},
|
||||
tune_config=tune.TuneConfig(num_samples=4, metric="metric", mode="max"),
|
||||
)
|
||||
```
|
||||
|
||||
### Logging metrics with Tune callbacks
|
||||
|
||||
Every metric logged using `tune.report` can be accessed during the tuning run through Tune [Callbacks](tune-logging). Ray Tune provides [several built-in integrations](loggers-docstring) with popular frameworks, such as MLFlow, Weights & Biases, CometML and more. You can also use the [Callback API](tune-callbacks-docs) to create your own callbacks.
|
||||
|
||||
Callbacks are passed in the `callback` argument of the `Tuner`'s `RunConfig`.
|
||||
|
||||
In our example, we'll use the MLFlow callback to track the progress of our tuning run and the changing value of the `metric` (requires `mlflow` to be installed).
|
||||
|
||||
```python
|
||||
import ray.tune
|
||||
from ray.tune.logger.mlflow import MLflowLoggerCallback
|
||||
|
||||
|
||||
def training_function(config, data):
|
||||
model = {
|
||||
"hyperparameter_a": config["hyperparameter_a"],
|
||||
"hyperparameter_b": config["hyperparameter_b"],
|
||||
}
|
||||
epochs = config["epochs"]
|
||||
|
||||
# Simulate training & evaluation - we obtain back a "metric" and a "trained_model".
|
||||
for epoch in range(epochs):
|
||||
# Simulate doing something expensive.
|
||||
time.sleep(1)
|
||||
metric = (0.1 + model["hyperparameter_a"] * epoch / 100) ** (
|
||||
-1
|
||||
) + model["hyperparameter_b"] * 0.1 * data["A"].sum()
|
||||
trained_model = {"state": model, "epoch": epoch}
|
||||
tune.report(metrics={"metric": metric})
|
||||
|
||||
|
||||
tuner = tune.Tuner(
|
||||
tune.with_parameters(training_function, data=data),
|
||||
param_space={
|
||||
"hyperparameter_a": tune.uniform(0, 20),
|
||||
"hyperparameter_b": tune.uniform(-100, 100),
|
||||
"epochs": 10,
|
||||
},
|
||||
tune_config=tune.TuneConfig(num_samples=4, metric="metric", mode="max"),
|
||||
run_config=tune.RunConfig(
|
||||
callbacks=[MLflowLoggerCallback(experiment_name="example")]
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
### Getting data out of Tune using checkpoints & other artifacts
|
||||
|
||||
Aside from metrics, you may want to save the state of your trained model and any other artifacts to allow resumption from training failure and further inspection and usage. Those cannot be saved as metrics, as they are often far too large and may not be easily serializable. Finally, they should be persisted on disk or cloud storage to allow access after the Tune run is interrupted or terminated.
|
||||
|
||||
Ray Train provides a {class}`Checkpoint <ray.tune.Checkpoint>` API for that purpose. `Checkpoint` objects can be created from various sources (dictionaries, directories, cloud storage).
|
||||
|
||||
In Ray Tune, `Checkpoints` are created by the user in their Trainable functions and reported using the optional `checkpoint` argument of `tune.report`. `Checkpoints` can contain arbitrary data and can be freely passed around the Ray cluster. After a tuning run is over, `Checkpoints` can be [obtained from the results](tune-analysis-guide).
|
||||
|
||||
Ray Tune can be configured to [automatically sync checkpoints to cloud storage](tune-storage-options), keep only a certain number of checkpoints to save space (with {class}`ray.tune.CheckpointConfig`) and more.
|
||||
|
||||
```{note}
|
||||
The experiment state itself is checkpointed separately. See {ref}`tune-persisted-experiment-data` for more details.
|
||||
```
|
||||
|
||||
In our example, we want to be able to resume the training from the latest checkpoint, and to save the `trained_model` in a checkpoint every iteration. To accomplish this, we will use the `session` and `Checkpoint` APIs.
|
||||
|
||||
```python
|
||||
import os
|
||||
import pickle
|
||||
import tempfile
|
||||
|
||||
from ray import tune
|
||||
|
||||
def training_function(config, data):
|
||||
model = {
|
||||
"hyperparameter_a": config["hyperparameter_a"],
|
||||
"hyperparameter_b": config["hyperparameter_b"],
|
||||
}
|
||||
epochs = config["epochs"]
|
||||
|
||||
# Load the checkpoint, if there is any.
|
||||
checkpoint = tune.get_checkpoint()
|
||||
start_epoch = 0
|
||||
if checkpoint:
|
||||
with checkpoint.as_directory() as checkpoint_dir:
|
||||
with open(os.path.join(checkpoint_dir, "model.pkl"), "rb") as f:
|
||||
checkpoint_dict = pickle.load(f)
|
||||
start_epoch = checkpoint_dict["epoch"] + 1
|
||||
model = checkpoint_dict["state"]
|
||||
|
||||
# Simulate training & evaluation - we obtain back a "metric" and a "trained_model".
|
||||
for epoch in range(start_epoch, epochs):
|
||||
# Simulate doing something expensive.
|
||||
time.sleep(1)
|
||||
metric = (0.1 + model["hyperparameter_a"] * epoch / 100) ** (
|
||||
-1
|
||||
) + model["hyperparameter_b"] * 0.1 * data["A"].sum()
|
||||
|
||||
checkpoint_dict = {"state": model, "epoch": epoch}
|
||||
|
||||
# Create the checkpoint.
|
||||
with tempfile.TemporaryDirectory() as temp_checkpoint_dir:
|
||||
with open(os.path.join(temp_checkpoint_dir, "model.pkl"), "wb") as f:
|
||||
pickle.dump(checkpoint_dict, f)
|
||||
tune.report(
|
||||
{"metric": metric},
|
||||
checkpoint=tune.Checkpoint.from_directory(temp_checkpoint_dir),
|
||||
)
|
||||
|
||||
|
||||
tuner = tune.Tuner(
|
||||
tune.with_parameters(training_function, data=data),
|
||||
param_space={
|
||||
"hyperparameter_a": tune.uniform(0, 20),
|
||||
"hyperparameter_b": tune.uniform(-100, 100),
|
||||
"epochs": 10,
|
||||
},
|
||||
tune_config=tune.TuneConfig(num_samples=4, metric="metric", mode="max"),
|
||||
run_config=tune.RunConfig(
|
||||
callbacks=[MLflowLoggerCallback(experiment_name="example")]
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
With all of those changes implemented, we can now run our tuning and obtain meaningful metrics and artifacts.
|
||||
|
||||
```python
|
||||
results = tuner.fit()
|
||||
results.get_dataframe()
|
||||
```
|
||||
|
||||
2022-11-30 17:40:28,839 INFO tune.py:762 -- Total run time: 15.79 seconds (15.65 seconds for the tuning loop).
|
||||
|
||||
<div>
|
||||
<style scoped>
|
||||
.dataframe tbody tr th:only-of-type { vertical-align: middle; }
|
||||
|
||||
.dataframe tbody tr th { vertical-align: top; }
|
||||
|
||||
.dataframe thead th { text-align: right; }
|
||||
</style>
|
||||
<table border="1" class="dataframe">
|
||||
<thead>
|
||||
<tr style="text-align: right;"> <th></th> <th>metric</th> <th>time_this_iter_s</th> <th>should_checkpoint</th> <th>done</th> <th>timesteps_total</th> <th>episodes_total</th> <th>training_iteration</th> <th>trial_id</th> <th>experiment_id</th> <th>date</th> <th>...</th> <th>hostname</th> <th>node_ip</th> <th>time_since_restore</th> <th>timesteps_since_restore</th> <th>iterations_since_restore</th> <th>warmup_time</th> <th>config/epochs</th> <th>config/hyperparameter_a</th> <th>config/hyperparameter_b</th> <th>logdir</th> </tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr> <th>0</th> <td>-58.399962</td> <td>1.015951</td> <td>True</td> <td>False</td> <td>NaN</td> <td>NaN</td> <td>10</td> <td>0b239_00000</td> <td>acf38c19d59c4cf2ad7955807657b6ea</td> <td>2022-11-30_17-40-26</td> <td>...</td> <td>ip-172-31-43-110</td> <td>172.31.43.110</td> <td>10.282120</td> <td>0</td> <td>10</td> <td>0.003541</td> <td>10</td> <td>18.065981</td> <td>-98.298928</td> <td>/home/ubuntu/ray_results/training_function_202...</td> </tr> <tr> <th>1</th> <td>-24.461518</td> <td>1.030420</td> <td>True</td> <td>False</td> <td>NaN</td> <td>NaN</td> <td>10</td> <td>0b239_00001</td> <td>5ca9e03d7cca46a7852cd501bc3f7b38</td> <td>2022-11-30_17-40-28</td> <td>...</td> <td>ip-172-31-43-110</td> <td>172.31.43.110</td> <td>10.362581</td> <td>0</td> <td>10</td> <td>0.004031</td> <td>10</td> <td>1.544918</td> <td>-47.741455</td> <td>/home/ubuntu/ray_results/training_function_202...</td> </tr> <tr> <th>2</th> <td>18.510299</td> <td>1.034228</td> <td>True</td> <td>False</td> <td>NaN</td> <td>NaN</td> <td>10</td> <td>0b239_00002</td> <td>aa38dd786c714486a8d69fa5b372df48</td> <td>2022-11-30_17-40-28</td> <td>...</td> <td>ip-172-31-43-110</td> <td>172.31.43.110</td> <td>10.333781</td> <td>0</td> <td>10</td> <td>0.005286</td> <td>10</td> <td>8.129285</td> <td>28.846415</td> <td>/home/ubuntu/ray_results/training_function_202...</td> </tr> <tr> <th>3</th> <td>-16.138780</td> <td>1.020072</td> <td>True</td> <td>False</td> <td>NaN</td> <td>NaN</td> <td>10</td> <td>0b239_00003</td> <td>5b401e15ab614332b631d552603a8d77</td> <td>2022-11-30_17-40-28</td> <td>...</td> <td>ip-172-31-43-110</td> <td>172.31.43.110</td> <td>10.242707</td> <td>0</td> <td>10</td> <td>0.003809</td> <td>10</td> <td>17.982020</td> <td>-27.867871</td> <td>/home/ubuntu/ray_results/training_function_202...</td> </tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>4 rows × 23 columns</p>
|
||||
</div>
|
||||
|
||||
Checkpoints, metrics, and the log directory for each trial can be accessed through the `ResultGrid` output of a Tune experiment. For more information on how to interact with the returned `ResultGrid`, see {doc}`/tune/examples/tune_analyze_results`.
|
||||
|
||||
### How do I access Tune results after I am finished?
|
||||
|
||||
After you have finished running the Python session, you can still access the results and checkpoints. By default, Tune will save the experiment results to the `~/ray_results` local directory. You can configure Tune to persist results in the cloud as well. See {ref}`tune-storage-options` for more information on how to configure storage options for persisting experiment results.
|
||||
|
||||
You can restore the Tune experiment by calling {meth}`Tuner.restore(path_or_cloud_uri, trainable) <ray.tune.Tuner.restore>`, where `path_or_cloud_uri` points to a location either on the filesystem or cloud where the experiment was saved to. After the `Tuner` has been restored, you can access the results and checkpoints by calling `Tuner.get_results()` to receive the `ResultGrid` object, and then proceeding as outlined in the previous section.
|
||||
Reference in New Issue
Block a user