wehub/ray-project--ray
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:17:40 +08:00
commit f1825c8ceb
10096 changed files with 2364182 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/tune/api/api.rst
+30
View File
@@ -0,0 +1,30 @@
.. _tune-api-ref:
Ray Tune API
============
.. tip:: We'd love to hear your feedback on using Tune - `get in touch <https://forms.gle/PTRvGLbKRdUfuzQo9>`_!
This section contains a reference for the Tune API. If there is anything missing, please open an issue
on `GitHub`_.
.. _`GitHub`: https://github.com/ray-project/ray/issues
.. toctree::
:maxdepth: 2
execution.rst
result_grid.rst
trainable.rst
search_space.rst
suggestion.rst
schedulers.rst
stoppers.rst
reporters.rst
syncing.rst
logging.rst
callbacks.rst
env.rst
integration.rst
internals.rst
cli.rst
doc/source/tune/api/callbacks.rst
+63
View File
@@ -0,0 +1,63 @@
.. _tune-callbacks-docs:
Tune Callbacks (tune.Callback)
==============================
See :doc:`this user guide </tune/tutorials/tune-metrics>` for more details.
.. seealso::
:doc:`Tune's built-in loggers </tune/api/logging>` use the ``Callback`` interface.
Callback Interface
------------------
Callback Initialization and Setup
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. currentmodule:: ray.tune
.. autosummary::
:nosignatures:
:toctree: doc/
Callback
.. autosummary::
:nosignatures:
:toctree: doc/
Callback.setup
Callback Hooks
~~~~~~~~~~~~~~
.. autosummary::
:nosignatures:
:toctree: doc/
Callback.on_checkpoint
Callback.on_experiment_end
Callback.on_step_begin
Callback.on_step_end
Callback.on_trial_complete
Callback.on_trial_error
Callback.on_trial_restore
Callback.on_trial_result
Callback.on_trial_save
Callback.on_trial_start
Stateful Callbacks
~~~~~~~~~~~~~~~~~~
The following methods must be overridden for stateful callbacks to be saved/restored
properly by Tune.
.. autosummary::
:nosignatures:
:toctree: doc/
Callback.get_state
Callback.set_state
doc/source/tune/api/cli.rst
+39
View File
@@ -0,0 +1,39 @@
Tune CLI (Experimental)
=======================
``tune`` has an easy-to-use command line interface (CLI) to manage and monitor your experiments on Ray.
Here is an example command line call:
``tune list-trials``: List tabular information about trials within an experiment.
Empty columns will be dropped by default. Add the ``--sort`` flag to sort the output by specific columns.
Add the ``--filter`` flag to filter the output in the format ``"<column> <operator> <value>"``.
Add the ``--output`` flag to write the trial information to a specific file (CSV or Pickle).
Add the ``--columns`` and ``--result-columns`` flags to select specific columns to display.
.. code-block:: bash
$ tune list-trials [EXPERIMENT_DIR] --output note.csv
+------------------+-----------------------+------------+
| trainable_name | experiment_tag | trial_id |
|------------------+-----------------------+------------|
| MyTrainableClass | 0_height=40,width=37 | 87b54a1d |
| MyTrainableClass | 1_height=21,width=70 | 23b89036 |
| MyTrainableClass | 2_height=99,width=90 | 518dbe95 |
| MyTrainableClass | 3_height=54,width=21 | 7b99a28a |
| MyTrainableClass | 4_height=90,width=69 | ae4e02fb |
+------------------+-----------------------+------------+
Dropped columns: ['status', 'last_update_time']
Please increase your terminal size to view remaining columns.
Output saved at: note.csv
$ tune list-trials [EXPERIMENT_DIR] --filter "trial_id == 7b99a28a"
+------------------+-----------------------+------------+
| trainable_name | experiment_tag | trial_id |
|------------------+-----------------------+------------|
| MyTrainableClass | 3_height=54,width=21 | 7b99a28a |
+------------------+-----------------------+------------+
Dropped columns: ['status', 'last_update_time']
Please increase your terminal size to view remaining columns.
doc/source/tune/api/env.rst
+96
View File
@@ -0,0 +1,96 @@
.. _tune-env-vars:
Environment variables used by Ray Tune
--------------------------------------
Some of Ray Tune's behavior can be configured using environment variables.
These are the environment variables Ray Tune currently considers:
* **TUNE_DISABLE_AUTO_CALLBACK_LOGGERS**: Ray Tune automatically adds a CSV and
JSON logger callback if they haven't been passed. Setting this variable to
`1` disables this automatic creation. Please note that this will most likely
affect analyzing your results after the tuning run.
* **TUNE_DISABLE_AUTO_INIT**: Disable automatically calling ``ray.init()`` if
not attached to a Ray session.
* **TUNE_DISABLE_DATED_SUBDIR**: Ray Tune automatically adds a date string to experiment
directories when the name is not specified explicitly or the trainable isn't passed
as a string. Setting this environment variable to ``1`` disables adding these date strings.
* **TUNE_DISABLE_STRICT_METRIC_CHECKING**: When you report metrics to Tune via
``tune.report()`` and passed a ``metric`` parameter to ``Tuner()``, a scheduler,
or a search algorithm, Tune will error
if the metric was not reported in the result. Setting this environment variable
to ``1`` will disable this check.
* **TUNE_DISABLE_SIGINT_HANDLER**: Ray Tune catches SIGINT signals (e.g. sent by
Ctrl+C) to gracefully shutdown and do a final checkpoint. Setting this variable
to ``1`` will disable signal handling and stop execution right away. Defaults to
``0``.
* **TUNE_FORCE_TRIAL_CLEANUP_S**: By default, Ray Tune will gracefully terminate trials,
letting them finish the current training step and any user-defined cleanup.
Setting this variable to a non-zero, positive integer will cause trials to be forcefully
terminated after a grace period of that many seconds. Defaults to ``600`` (seconds).
* **TUNE_FUNCTION_THREAD_TIMEOUT_S**: Time in seconds the function API waits
for threads to finish after instructing them to complete. Defaults to ``2``.
* **TUNE_GLOBAL_CHECKPOINT_S**: Time in seconds that limits how often
experiment state is checkpointed. If not, set this will default to ``'auto'``.
``'auto'`` measures the time it takes to snapshot the experiment state
and adjusts the period so that ~5% of the driver's time is spent on snapshotting.
You should set this to a fixed value (ex: ``TUNE_GLOBAL_CHECKPOINT_S=60``)
to snapshot your experiment state every X seconds.
* **TUNE_MAX_LEN_IDENTIFIER**: Maximum length of trial subdirectory names (those
with the parameter values in them)
* **TUNE_MAX_PENDING_TRIALS_PG**: Maximum number of pending trials when placement groups are used. Defaults
to ``auto``, which will be updated to ``max(200, cluster_cpus * 1.1)`` for random/grid search and ``1``
for any other search algorithms.
* **TUNE_PLACEMENT_GROUP_PREFIX**: Prefix for placement groups created by Ray Tune. This prefix is used
e.g. to identify placement groups that should be cleaned up on start/stop of the tuning run. This is
initialized to a unique name at the start of the first run.
* **TUNE_PLACEMENT_GROUP_RECON_INTERVAL**: How often to reconcile placement groups. Reconcilation is
used to make sure that the number of requested placement groups and pending/running trials are in sync.
In normal circumstances these shouldn't differ anyway, but reconcilation makes sure to capture cases when
placement groups are manually destroyed. Reconcilation doesn't take much time, but it can add up when
running a large number of short trials. Defaults to every ``5`` (seconds).
* **TUNE_PRINT_ALL_TRIAL_ERRORS**: If ``1``, will print all trial errors as they come up. Otherwise, errors
will only be saved as text files to the trial directory and not printed. Defaults to ``1``.
* **TUNE_RESULT_BUFFER_LENGTH**: Ray Tune can buffer results from trainables before they are passed
to the driver. Enabling this might delay scheduling decisions, as trainables are speculatively
continued. Setting this to ``1`` disables result buffering. Cannot be used with ``checkpoint_at_end``.
Defaults to disabled.
* **TUNE_RESULT_DELIM**: Delimiter used for nested entries in
:class:`ExperimentAnalysis <ray.tune.ExperimentAnalysis>` dataframes. Defaults to ``.`` (but will be
changed to ``/`` in future versions of Ray).
* **TUNE_RESULT_BUFFER_MAX_TIME_S**: Similarly, Ray Tune buffers results up to ``number_of_trial/10`` seconds,
but never longer than this value. Defaults to 100 (seconds).
* **TUNE_RESULT_BUFFER_MIN_TIME_S**: Additionally, you can specify a minimum time to buffer results. Defaults to 0.
* **TUNE_WARN_THRESHOLD_S**: Threshold for logging if an Tune event loop operation takes too long. Defaults to 0.5 (seconds).
* **TUNE_WARN_INSUFFICIENT_RESOURCE_THRESHOLD_S**: Threshold for throwing a warning if no active trials are in ``RUNNING`` state
for this amount of seconds. If the Ray Tune job is stuck in this state (most likely due to insufficient resources),
the warning message is printed repeatedly every this amount of seconds. Defaults to 60 (seconds).
* **TUNE_WARN_INSUFFICIENT_RESOURCE_THRESHOLD_S_AUTOSCALER**: Threshold for throwing a warning when the autoscaler is enabled and
if no active trials are in ``RUNNING`` state for this amount of seconds.
If the Ray Tune job is stuck in this state (most likely due to insufficient resources), the warning message is printed
repeatedly every this amount of seconds. Defaults to 60 (seconds).
* **TUNE_WARN_SLOW_EXPERIMENT_CHECKPOINT_SYNC_THRESHOLD_S**: Threshold for logging a warning if the experiment state syncing
takes longer than this time in seconds. The experiment state files should be very lightweight, so this should not take longer than ~5 seconds.
Defaults to 5 (seconds).
* **TUNE_STATE_REFRESH_PERIOD**: Frequency of updating the resource tracking from Ray. Defaults to 10 (seconds).
* **TUNE_RESTORE_RETRY_NUM**: The number of retries that are done before a particular trial's restore is determined
unsuccessful. After that, the trial is not restored to its previous checkpoint but rather from scratch.
Default is ``0``. While this retry counter is taking effect, per trial failure number will not be incremented, which
is compared against ``max_failures``.
* **TUNE_ONLY_STORE_CHECKPOINT_SCORE_ATTRIBUTE**: If set to ``1``, only the metric defined by ``checkpoint_score_attribute``
will be stored with each ``Checkpoint``. As a result, ``Result.best_checkpoints`` will contain only this metric,
omitting others that would normally be included. This can significantly reduce memory usage, especially when many
checkpoints are stored or when metrics are large. Defaults to ``0`` (i.e., all metrics are stored).
* **RAY_AIR_FULL_TRACEBACKS**: If set to 1, will print full tracebacks for training functions,
including internal code paths. Otherwise, abbreviated tracebacks that only show user code
are printed. Defaults to 0 (disabled).
* **RAY_AIR_NEW_OUTPUT**: If set to 0, this disables
the `experimental new console output <https://github.com/ray-project/ray/issues/36949>`_.
There are some environment variables that are mostly relevant for integrated libraries:
* **WANDB_API_KEY**: Weights and Biases API key. You can also use ``wandb login``
instead.
doc/source/tune/api/execution.rst
+58
View File
@@ -0,0 +1,58 @@
Tune Execution (tune.Tuner)
===========================
.. _tune-run-ref:
Tuner
-----
.. currentmodule:: ray.tune
.. autosummary::
:nosignatures:
:toctree: doc/
Tuner
.. autosummary::
:nosignatures:
:toctree: doc/
Tuner.fit
Tuner.get_results
Tuner Configuration
~~~~~~~~~~~~~~~~~~~
.. autosummary::
:nosignatures:
:toctree: doc/
TuneConfig
RunConfig
CheckpointConfig
FailureConfig
Restoring a Tuner
~~~~~~~~~~~~~~~~~
.. autosummary::
:nosignatures:
:toctree: doc/
Tuner.restore
Tuner.can_restore
tune.run_experiments
--------------------
.. autosummary::
:nosignatures:
:toctree: doc/
run_experiments
run
Experiment
TuneError
doc/source/tune/api/integration.rst
+42
View File
@@ -0,0 +1,42 @@
.. _tune-integration:
External library integrations for Ray Tune
===========================================
.. currentmodule:: ray
.. _tune-integration-pytorch-lightning:
PyTorch Lightning (tune.integration.pytorch_lightning)
------------------------------------------------------
.. autosummary::
:nosignatures:
:toctree: doc/
~tune.integration.pytorch_lightning.TuneReportCheckpointCallback
.. _tune-integration-xgboost:
XGBoost (tune.integration.xgboost)
----------------------------------
.. autosummary::
:nosignatures:
:template: autosummary/class_without_autosummary.rst
:toctree: doc/
~tune.integration.xgboost.TuneReportCheckpointCallback
.. _tune-integration-lightgbm:
LightGBM (tune.integration.lightgbm)
------------------------------------
.. autosummary::
:nosignatures:
:template: autosummary/class_without_autosummary.rst
:toctree: doc/
~tune.integration.lightgbm.TuneReportCheckpointCallback
doc/source/tune/api/internals.rst
+43
View File
@@ -0,0 +1,43 @@
Tune Internals
==============
.. _raytrialexecutor-docstring:
TunerInternal
---------------
.. autoclass:: ray.tune.impl.tuner_internal.TunerInternal
:members:
.. _trial-docstring:
Trial
-----
.. autoclass:: ray.tune.experiment.trial.Trial
:members:
FunctionTrainable
-----------------
.. autoclass:: ray.tune.trainable.function_trainable.FunctionTrainable
.. autofunction:: ray.tune.trainable.function_trainable.wrap_function
Registry
--------
.. autofunction:: ray.tune.register_trainable
.. autofunction:: ray.tune.register_env
Output
------
.. autoclass:: ray.tune.experimental.output.ProgressReporter
.. autoclass:: ray.tune.experimental.output.TrainReporter
.. autoclass:: ray.tune.experimental.output.TuneReporterBase
.. autoclass:: ray.tune.experimental.output.TuneTerminalReporter
doc/source/tune/api/logging.rst
+126
View File
@@ -0,0 +1,126 @@
.. _loggers-docstring:
Tune Loggers (tune.logger)
==========================
Tune automatically uses loggers for TensorBoard, CSV, and JSON formats.
By default, Tune only logs the returned result dictionaries from the training function.
If you need to log something lower level like model weights or gradients,
see :ref:`Trainable Logging <trainable-logging>`.
.. note::
Tune's per-trial ``Logger`` classes have been deprecated. Use the ``LoggerCallback`` interface instead.
.. currentmodule:: ray
.. _logger-interface:
LoggerCallback Interface (tune.logger.LoggerCallback)
-----------------------------------------------------
.. autosummary::
:nosignatures:
:toctree: doc/
~tune.logger.LoggerCallback
.. autosummary::
:nosignatures:
:toctree: doc/
~tune.logger.LoggerCallback.log_trial_start
~tune.logger.LoggerCallback.log_trial_restore
~tune.logger.LoggerCallback.log_trial_save
~tune.logger.LoggerCallback.log_trial_result
~tune.logger.LoggerCallback.log_trial_end
Tune Built-in Loggers
---------------------
.. autosummary::
:nosignatures:
:toctree: doc/
tune.logger.JsonLoggerCallback
tune.logger.CSVLoggerCallback
tune.logger.TBXLoggerCallback
MLFlow Integration
------------------
Tune also provides a logger for `MLflow <https://mlflow.org>`_.
You can install MLflow via ``pip install mlflow``.
See the :doc:`tutorial here </tune/examples/tune-mlflow>`.
.. autosummary::
:nosignatures:
:toctree: doc/
~air.integrations.mlflow.MLflowLoggerCallback
~air.integrations.mlflow.setup_mlflow
Wandb Integration
-----------------
Tune also provides a logger for `Weights & Biases <https://www.wandb.ai/>`_.
You can install Wandb via ``pip install wandb``.
See the :doc:`tutorial here </tune/examples/tune-wandb>`.
.. autosummary::
:nosignatures:
:toctree: doc/
~air.integrations.wandb.WandbLoggerCallback
~air.integrations.wandb.setup_wandb
Comet Integration
------------------------------
Tune also provides a logger for `Comet <https://www.comet.com/>`_.
You can install Comet via ``pip install comet-ml``.
See the :doc:`tutorial here </tune/examples/tune-comet>`.
.. autosummary::
:nosignatures:
:toctree: doc/
~air.integrations.comet.CometLoggerCallback
Aim Integration
---------------
Tune also provides a logger for the `Aim <https://aimstack.io/>`_ experiment tracker.
You can install Aim via ``pip install aim``.
See the :doc:`tutorial here </tune/examples/tune-aim>`.
.. autosummary::
:nosignatures:
:toctree: doc/
~tune.logger.aim.AimLoggerCallback
Other Integrations
------------------
Viskit
~~~~~~
Tune automatically integrates with `Viskit <https://github.com/vitchyr/viskit>`_ via the ``CSVLoggerCallback`` outputs.
To use VisKit (you may have to install some dependencies), run:
.. code-block:: bash
$ git clone https://github.com/vitchyr/viskit.git
$ python viskit/viskit/frontend.py ~/ray_results/my_experiment
The non-relevant metrics (like timing stats) can be disabled on the left to show only the
relevant ones (like accuracy, loss, etc.).
.. image:: ../images/ray-tune-viskit.png
doc/source/tune/api/reporters.rst
+119
View File
@@ -0,0 +1,119 @@
.. _tune-reporter-doc:
Tune Console Output (Reporters)
===============================
By default, Tune reports experiment progress periodically to the command-line as follows.
.. code-block:: bash
== Status ==
Memory usage on this node: 11.4/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 4/12 CPUs, 0/0 GPUs, 0.0/3.17 GiB heap, 0.0/1.07 GiB objects
Result logdir: /Users/foo/ray_results/myexp
Number of trials: 4 (4 RUNNING)
+----------------------+----------+---------------------+-----------+--------+--------+--------+--------+------------------+-------+
| Trial name | status | loc | param1 | param2 | param3 | acc | loss | total time (s) | iter |
|----------------------+----------+---------------------+-----------+--------+--------+--------+--------+------------------+-------|
| MyTrainable_a826033a | RUNNING | 10.234.98.164:31115 | 0.303706 | 0.0761 | 0.4328 | 0.1289 | 1.8572 | 7.54952 | 15 |
| MyTrainable_a8263fc6 | RUNNING | 10.234.98.164:31117 | 0.929276 | 0.158 | 0.3417 | 0.4865 | 1.6307 | 7.0501 | 14 |
| MyTrainable_a8267914 | RUNNING | 10.234.98.164:31111 | 0.068426 | 0.0319 | 0.1147 | 0.9585 | 1.9603 | 7.0477 | 14 |
| MyTrainable_a826b7bc | RUNNING | 10.234.98.164:31112 | 0.729127 | 0.0748 | 0.1784 | 0.1797 | 1.7161 | 7.05715 | 14 |
+----------------------+----------+---------------------+-----------+--------+--------+--------+--------+------------------+-------+
Note that columns will be hidden if they are completely empty. The output can be configured in various ways by
instantiating a ``CLIReporter`` instance (or ``JupyterNotebookReporter`` if you're using jupyter notebook).
Here's an example:
.. TODO: test these snippets
.. code-block:: python
import ray.tune
from ray.tune import CLIReporter
# Limit the number of rows.
reporter = CLIReporter(max_progress_rows=10)
# Add a custom metric column, in addition to the default metrics.
# Note that this must be a metric that is returned in your training results.
reporter.add_metric_column("custom_metric")
tuner = tune.Tuner(my_trainable, run_config=ray.tune.RunConfig(progress_reporter=reporter))
results = tuner.fit()
Extending ``CLIReporter`` lets you control reporting frequency. For example:
.. code-block:: python
from ray.tune.experiment.trial import Trial
class ExperimentTerminationReporter(CLIReporter):
def should_report(self, trials, done=False):
"""Reports only on experiment termination."""
return done
tuner = tune.Tuner(my_trainable, run_config=ray.tune.RunConfig(progress_reporter=ExperimentTerminationReporter()))
results = tuner.fit()
class TrialTerminationReporter(CLIReporter):
def __init__(self):
super(TrialTerminationReporter, self).__init__()
self.num_terminated = 0
def should_report(self, trials, done=False):
"""Reports only on trial termination events."""
old_num_terminated = self.num_terminated
self.num_terminated = len([t for t in trials if t.status == Trial.TERMINATED])
return self.num_terminated > old_num_terminated
tuner = tune.Tuner(my_trainable, run_config=ray.tune.RunConfig(progress_reporter=TrialTerminationReporter()))
results = tuner.fit()
The default reporting style can also be overridden more broadly by extending the ``ProgressReporter`` interface directly. Note that you can print to any output stream, file etc.
.. code-block:: python
from ray.tune import ProgressReporter
class CustomReporter(ProgressReporter):
def should_report(self, trials, done=False):
return True
def report(self, trials, *sys_info):
print(*sys_info)
print("\n".join([str(trial) for trial in trials]))
tuner = tune.Tuner(my_trainable, run_config=ray.tune.RunConfig(progress_reporter=CustomReporter()))
results = tuner.fit()
.. currentmodule:: ray.tune
Reporter Interface (tune.ProgressReporter)
------------------------------------------
.. autosummary::
:nosignatures:
:toctree: doc/
ProgressReporter
.. autosummary::
:nosignatures:
:toctree: doc/
ProgressReporter.report
ProgressReporter.should_report
Tune Built-in Reporters
-----------------------
.. autosummary::
:nosignatures:
:toctree: doc/
CLIReporter
JupyterNotebookReporter
doc/source/tune/api/result_grid.rst
+55
View File
@@ -0,0 +1,55 @@
.. _air-results-ref:
.. _tune-analysis-docs:
.. _result-grid-docstring:
Tune Experiment Results (tune.ResultGrid)
=========================================
ResultGrid (tune.ResultGrid)
----------------------------
.. currentmodule:: ray
.. autosummary::
:nosignatures:
:toctree: doc/
~tune.ResultGrid
.. autosummary::
:nosignatures:
:toctree: doc/
~tune.ResultGrid.get_best_result
~tune.ResultGrid.get_dataframe
.. _result-docstring:
Result (tune.Result)
---------------------
.. autosummary::
:nosignatures:
:template: autosummary/class_without_autosummary.rst
:toctree: doc/
~tune.Result
.. _exp-analysis-docstring:
ExperimentAnalysis (tune.ExperimentAnalysis)
--------------------------------------------
.. note::
An `ExperimentAnalysis` is the output of the ``tune.run`` API.
It's now recommended to use :meth:`Tuner.fit <ray.tune.Tuner.fit>`,
which outputs a `ResultGrid` object.
.. autosummary::
:nosignatures:
:toctree: doc/
~tune.ExperimentAnalysis
doc/source/tune/api/schedulers.rst
+371
View File
@@ -0,0 +1,371 @@
.. _tune-schedulers:
Tune Trial Schedulers (tune.schedulers)
=======================================
In Tune, some hyperparameter optimization algorithms are written as "scheduling algorithms".
These Trial Schedulers can early terminate bad trials, pause trials, clone trials,
and alter hyperparameters of a running trial.
All Trial Schedulers take in a ``metric``, which is a value returned in the result dict of your
Trainable and is maximized or minimized according to ``mode``.
.. code-block:: python
from ray import tune
from ray.tune.schedulers import ASHAScheduler
def train_fn(config):
# This objective function is just for demonstration purposes
tune.report({"loss": config["param"]})
tuner = tune.Tuner(
train_fn,
tune_config=tune.TuneConfig(
scheduler=ASHAScheduler(),
metric="loss",
mode="min",
num_samples=10,
),
param_space={"param": tune.uniform(0, 1)},
)
results = tuner.fit()
.. currentmodule:: ray.tune.schedulers
.. _tune-scheduler-hyperband:
ASHA (tune.schedulers.ASHAScheduler)
------------------------------------
The `ASHA <https://openreview.net/forum?id=S1Y7OOlRZ>`__ scheduler can be used by
setting the ``scheduler`` parameter of ``tune.TuneConfig``, which is taken in by ``Tuner``, e.g.
.. code-block:: python
from ray import tune
from ray.tune.schedulers import ASHAScheduler
asha_scheduler = ASHAScheduler(
time_attr='training_iteration',
metric='loss',
mode='min',
max_t=100,
grace_period=10,
reduction_factor=3,
brackets=1,
)
tuner = tune.Tuner(
train_fn,
tune_config=tune.TuneConfig(scheduler=asha_scheduler),
)
results = tuner.fit()
Compared to the original version of HyperBand, this implementation provides better
parallelism and avoids straggler issues during eliminations.
**We recommend using this over the standard HyperBand scheduler.**
An example of this can be found here: :doc:`/tune/examples/includes/async_hyperband_example`.
Even though the original paper mentions a bracket count of 3, discussions with the authors concluded
that the value should be left to 1 bracket.
This is the default used if no value is provided for the ``brackets`` argument.
.. autosummary::
:nosignatures:
:toctree: doc/
:template: autosummary/class_without_autosummary.rst
AsyncHyperBandScheduler
ASHAScheduler
.. _tune-original-hyperband:
HyperBand (tune.schedulers.HyperBandScheduler)
----------------------------------------------
Tune implements the `standard version of HyperBand <https://arxiv.org/abs/1603.06560>`__.
**We recommend using the ASHA Scheduler over the standard HyperBand scheduler.**
.. autosummary::
:nosignatures:
:toctree: doc/
HyperBandScheduler
HyperBand Implementation Details
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Implementation details may deviate slightly from theory but are focused on increasing usability.
Note: ``R``, ``s_max``, and ``eta`` are parameters of HyperBand given by the paper.
See `this post <https://blog.ml.cmu.edu/2018/12/12/massively-parallel-hyperparameter-optimization/>`_ for context.
1. Both ``s_max`` (representing the ``number of brackets - 1``) and ``eta``, representing the downsampling rate, are fixed.
In many practical settings, ``R``, which represents some resource unit and often the number of training iterations,
can be set reasonably large, like ``R >= 200``.
For simplicity, assume ``eta = 3``. Varying ``R`` between ``R = 200`` and ``R = 1000``
creates a huge range of the number of trials needed to fill up all brackets.
.. image:: /images/hyperband_bracket.png
On the other hand, holding ``R`` constant at ``R = 300`` and varying ``eta`` also leads to
HyperBand configurations that are not very intuitive:
.. image:: /images/hyperband_eta.png
The implementation takes the same configuration as the example given in the paper
and exposes ``max_t``, which is not a parameter in the paper.
2. The example in the `post <https://blog.ml.cmu.edu/2018/12/12/massively-parallel-hyperparameter-optimization/>`_ to calculate ``n_0``
is actually a little different than the algorithm given in the paper.
In this implementation, we implement ``n_0`` according to the paper (which is `n` in the below example):
.. image:: /images/hyperband_allocation.png
3. There are also implementation specific details like how trials are placed into brackets which are not covered in the paper.
This implementation places trials within brackets according to smaller bracket first - meaning
that with low number of trials, there will be less early stopping.
.. _tune-scheduler-msr:
Median Stopping Rule (tune.schedulers.MedianStoppingRule)
---------------------------------------------------------
The Median Stopping Rule implements the simple strategy of stopping a trial if its performance falls
below the median of other trials at similar points in time.
.. autosummary::
:nosignatures:
:toctree: doc/
MedianStoppingRule
.. _tune-scheduler-pbt:
Population Based Training (tune.schedulers.PopulationBasedTraining)
-------------------------------------------------------------------
Tune includes a distributed implementation of `Population Based Training (PBT) <https://www.deepmind.com/blog/population-based-training-of-neural-networks>`__.
This can be enabled by setting the ``scheduler`` parameter of ``tune.TuneConfig``, which is taken in by ``Tuner``, e.g.
.. code-block:: python
from ray import tune
from ray.tune.schedulers import PopulationBasedTraining
pbt_scheduler = PopulationBasedTraining(
time_attr='training_iteration',
metric='loss',
mode='min',
perturbation_interval=1,
hyperparam_mutations={
"lr": [1e-3, 5e-4, 1e-4, 5e-5, 1e-5],
"alpha": tune.uniform(0.0, 1.0),
}
)
tuner = tune.Tuner(
train_fn,
tune_config=tune.TuneConfig(
num_samples=4,
scheduler=pbt_scheduler,
),
)
tuner.fit()
When the PBT scheduler is enabled, each trial variant is treated as a member of the population.
Periodically, **top-performing trials are checkpointed**
(this requires your Trainable to support :ref:`save and restore <tune-trial-checkpoint>`).
**Low-performing trials clone the hyperparameter configurations of top performers and
perturb them** slightly in the hopes of discovering even better hyperparameter settings.
**Low-performing trials also resume from the checkpoints of the top performers**, allowing
the trials to explore the new hyperparameter configuration starting from a partially
trained model (e.g. by copying model weights from one of the top-performing trials).
Take a look at :doc:`/tune/examples/pbt_visualization/pbt_visualization` to get an idea
of how PBT operates. :doc:`/tune/examples/pbt_guide` gives more examples
of PBT usage.
.. autosummary::
:nosignatures:
:toctree: doc/
PopulationBasedTraining
.. _tune-scheduler-pbt-replay:
Population Based Training Replay (tune.schedulers.PopulationBasedTrainingReplay)
--------------------------------------------------------------------------------
Tune includes a utility to replay hyperparameter schedules of Population Based Training runs.
You just specify an existing experiment directory and the ID of the trial you would
like to replay. The scheduler accepts only one trial, and it will update its
config according to the obtained schedule.
.. code-block:: python
from ray import tune
from ray.tune.schedulers import PopulationBasedTrainingReplay
replay = PopulationBasedTrainingReplay(
experiment_dir="~/ray_results/pbt_experiment/",
trial_id="XXXXX_00001"
)
tuner = tune.Tuner(
train_fn,
tune_config=tune.TuneConfig(scheduler=replay)
)
results = tuner.fit()
See :ref:`here for an example <tune-advanced-tutorial-pbt-replay>` on how to use the
replay utility in practice.
.. autosummary::
:nosignatures:
:toctree: doc/
PopulationBasedTrainingReplay
.. _tune-scheduler-pb2:
Population Based Bandits (PB2) (tune.schedulers.pb2.PB2)
--------------------------------------------------------
Tune includes a distributed implementation of `Population Based Bandits (PB2) <https://arxiv.org/abs/2002.02518>`__.
This algorithm builds upon PBT, with the main difference being that instead of using random perturbations,
PB2 selects new hyperparameter configurations using a Gaussian Process model.
The Tune implementation of PB2 requires scikit-learn to be installed:
.. code-block:: bash
pip install scikit-learn
PB2 can be enabled by setting the ``scheduler`` parameter of ``tune.TuneConfig`` which is taken in by ``Tuner``, e.g.:
.. code-block:: python
from ray.tune.schedulers.pb2 import PB2
pb2_scheduler = PB2(
time_attr='time_total_s',
metric='mean_accuracy',
mode='max',
perturbation_interval=600.0,
hyperparam_bounds={
"lr": [1e-3, 1e-5],
"alpha": [0.0, 1.0],
...
}
)
tuner = tune.Tuner( ... , tune_config=tune.TuneConfig(scheduler=pb2_scheduler))
results = tuner.fit()
When the PB2 scheduler is enabled, each trial variant is treated as a member of the population.
Periodically, top-performing trials are checkpointed (this requires your Trainable to
support :ref:`save and restore <tune-trial-checkpoint>`).
Low-performing trials clone the checkpoints of top performers and perturb the configurations
in the hope of discovering an even better variation.
The primary motivation for PB2 is the ability to find promising hyperparameters with only a small population size.
With that in mind, you can run this :doc:`PB2 PPO example </tune/examples/includes/pb2_ppo_example>` to compare PB2 vs. PBT,
with a population size of ``4`` (as in the paper).
The example uses the ``BipedalWalker`` environment so does not require any additional licenses.
.. autosummary::
:nosignatures:
:toctree: doc/
pb2.PB2
.. _tune-scheduler-bohb:
BOHB (tune.schedulers.HyperBandForBOHB)
---------------------------------------
This class is a variant of HyperBand that enables the `BOHB Algorithm <https://arxiv.org/abs/1807.01774>`_.
This implementation is true to the original HyperBand implementation and does not implement pipelining nor
straggler mitigation.
This is to be used in conjunction with the Tune BOHB search algorithm.
See :ref:`TuneBOHB <suggest-TuneBOHB>` for package requirements, examples, and details.
An example of this in use can be found here: :doc:`/tune/examples/includes/bohb_example`.
.. autosummary::
:nosignatures:
:toctree: doc/
HyperBandForBOHB
.. _tune-resource-changing-scheduler:
ResourceChangingScheduler
-------------------------
This class is a utility scheduler, allowing for trial resource requirements to be changed during tuning.
It wraps around another scheduler and uses its decisions.
* If you are using the Trainable (class) API for tuning, your Trainable must implement ``Trainable.update_resources``,
which will let your model know about the new resources assigned. You can also obtain the current trial resources
by calling ``Trainable.trial_resources``.
* If you are using the functional API for tuning, get the current trial resources obtained by calling
`tune.get_trial_resources()` inside the training function.
The function should be able to :ref:`load and save checkpoints <tune-function-trainable-checkpointing>`
(the latter preferably every iteration).
An example of this in use can be found here: :doc:`/tune/examples/includes/xgboost_dynamic_resources_example`.
.. autosummary::
:nosignatures:
:toctree: doc/
ResourceChangingScheduler
resource_changing_scheduler.DistributeResources
resource_changing_scheduler.DistributeResourcesToTopJob
FIFOScheduler (Default Scheduler)
---------------------------------
.. autosummary::
:nosignatures:
:toctree: doc/
FIFOScheduler
TrialScheduler Interface
------------------------
.. autosummary::
:nosignatures:
:toctree: doc/
TrialScheduler
.. autosummary::
:nosignatures:
:toctree: doc/
TrialScheduler.choose_trial_to_run
TrialScheduler.on_trial_result
TrialScheduler.on_trial_complete
Shim Instantiation (tune.create_scheduler)
------------------------------------------
There is also a shim function that constructs the scheduler based on the provided string.
This can be useful if the scheduler you want to use changes often (e.g., specifying the scheduler
via a CLI option or config file).
.. autosummary::
:nosignatures:
:toctree: doc/
create_scheduler
doc/source/tune/api/search_space.rst
+116
View File
@@ -0,0 +1,116 @@
.. _tune-search-space:
Tune Search Space API
=====================
This section covers the functions you can use to define your search spaces.
.. caution::
Not all Search Algorithms support all distributions. In particular,
``tune.sample_from`` and ``tune.grid_search`` are often unsupported.
The default :ref:`tune-basicvariant` supports all distributions.
.. tip::
Avoid passing large objects as values in the search space, as that will incur a performance overhead.
Use :func:`tune.with_parameters <ray.tune.with_parameters>` to pass large objects in or load them inside your trainable
from disk (making sure that all nodes have access to the files) or cloud storage.
See :ref:`tune-bottlenecks` for more information.
For a high-level overview, see this example:
.. TODO: test this
.. code-block :: python
config = {
# Sample a float uniformly between -5.0 and -1.0
"uniform": tune.uniform(-5, -1),
# Sample a float uniformly between 3.2 and 5.4,
# rounding to multiples of 0.2
"quniform": tune.quniform(3.2, 5.4, 0.2),
# Sample a float uniformly between 0.0001 and 0.01, while
# sampling in log space
"loguniform": tune.loguniform(1e-4, 1e-2),
# Sample a float uniformly between 0.0001 and 0.1, while
# sampling in log space and rounding to multiples of 0.00005
"qloguniform": tune.qloguniform(1e-4, 1e-1, 5e-5),
# Sample a random float from a normal distribution with
# mean=10 and sd=2
"randn": tune.randn(10, 2),
# Sample a random float from a normal distribution with
# mean=10 and sd=2, rounding to multiples of 0.2
"qrandn": tune.qrandn(10, 2, 0.2),
# Sample a integer uniformly between -9 (inclusive) and 15 (exclusive)
"randint": tune.randint(-9, 15),
# Sample a random uniformly between -21 (inclusive) and 12 (inclusive (!))
# rounding to multiples of 3 (includes 12)
# if q is 1, then randint is called instead with the upper bound exclusive
"qrandint": tune.qrandint(-21, 12, 3),
# Sample a integer uniformly between 1 (inclusive) and 10 (exclusive),
# while sampling in log space
"lograndint": tune.lograndint(1, 10),
# Sample a integer uniformly between 1 (inclusive) and 10 (inclusive (!)),
# while sampling in log space and rounding to multiples of 2
# if q is 1, then lograndint is called instead with the upper bound exclusive
"qlograndint": tune.qlograndint(1, 10, 2),
# Sample an option uniformly from the specified choices
"choice": tune.choice(["a", "b", "c"]),
# Sample from a random function, in this case one that
# depends on another value from the search space
"func": tune.sample_from(lambda config: config["uniform"] * 0.01),
# Do a grid search over these values. Every value will be sampled
# ``num_samples`` times (``num_samples`` is the parameter you pass to ``tune.TuneConfig``,
# which is taken in by ``Tuner``)
"grid": tune.grid_search([32, 64, 128])
}
.. currentmodule:: ray
Random Distributions API
------------------------
.. autosummary::
:nosignatures:
:toctree: doc/
tune.uniform
tune.quniform
tune.loguniform
tune.qloguniform
tune.randn
tune.qrandn
tune.randint
tune.qrandint
tune.lograndint
tune.qlograndint
tune.choice
Grid Search and Custom Function APIs
------------------------------------
.. autosummary::
:nosignatures:
:toctree: doc/
tune.grid_search
tune.sample_from
References
----------
See also :ref:`tune-basicvariant`.
doc/source/tune/api/stoppers.rst
+50
View File
@@ -0,0 +1,50 @@
.. _tune-stoppers:
Tune Stopping Mechanisms (tune.stopper)
=======================================
In addition to Trial Schedulers like :ref:`ASHA <tune-scheduler-hyperband>`, where a number of
trials are stopped if they perform subpar, Ray Tune also supports custom stopping mechanisms to stop trials early. They can also stop the entire experiment after a condition is met.
For instance, stopping mechanisms can specify to stop trials when they reached a plateau and the metric
doesn't change anymore.
Ray Tune comes with several stopping mechanisms out of the box. For custom stopping behavior, you can
inherit from the :class:`Stopper <ray.tune.Stopper>` class.
Other stopping behaviors are described :ref:`in the user guide <tune-stopping-ref>`.
.. _tune-stop-ref:
Stopper Interface (tune.Stopper)
--------------------------------
.. currentmodule:: ray.tune.stopper
.. autosummary::
:nosignatures:
:toctree: doc/
Stopper
.. autosummary::
:nosignatures:
:toctree: doc/
Stopper.__call__
Stopper.stop_all
Tune Built-in Stoppers
----------------------
.. autosummary::
:nosignatures:
:toctree: doc/
MaximumIterationStopper
ExperimentPlateauStopper
TrialPlateauStopper
TimeoutStopper
CombinedStopper
~function_stopper.FunctionStopper
~noop.NoopStopper
doc/source/tune/api/suggestion.rst
+295
View File
@@ -0,0 +1,295 @@
.. _tune-search-alg:
Tune Search Algorithms (tune.search)
====================================
Tune's Search Algorithms are wrappers around open-source optimization libraries for efficient hyperparameter selection.
Each library has a specific way of defining the search space - please refer to their documentation for more details.
Tune will automatically convert search spaces passed to ``Tuner`` to the library format in most cases.
You can utilize these search algorithms as follows:
.. code-block:: python
from ray import tune
from ray.tune.search.optuna import OptunaSearch
def train_fn(config):
# This objective function is just for demonstration purposes
tune.report({"loss": config["param"]})
tuner = tune.Tuner(
train_fn,
tune_config=tune.TuneConfig(
search_alg=OptunaSearch(),
num_samples=100,
metric="loss",
mode="min",
),
param_space={"param": tune.uniform(0, 1)},
)
results = tuner.fit()
Saving and Restoring Tune Search Algorithms
-------------------------------------------
.. TODO: what to do about this section? It doesn't really belong here and is not worth its own guide.
.. TODO: at least check that this pseudo-code runs.
Certain search algorithms have ``save/restore`` implemented,
allowing reuse of searchers that are fitted on the results of multiple tuning runs.
.. code-block:: python
search_alg = HyperOptSearch()
tuner_1 = tune.Tuner(
train_fn,
tune_config=tune.TuneConfig(search_alg=search_alg)
)
results_1 = tuner_1.fit()
search_alg.save("./my-checkpoint.pkl")
# Restore the saved state onto another search algorithm,
# in a new tuning script
search_alg2 = HyperOptSearch()
search_alg2.restore("./my-checkpoint.pkl")
tuner_2 = tune.Tuner(
train_fn,
tune_config=tune.TuneConfig(search_alg=search_alg2)
)
results_2 = tuner_2.fit()
Tune automatically saves searcher state inside the current experiment folder during tuning.
See ``Result logdir: ...`` in the output logs for this location.
Note that if you have two Tune runs with the same experiment folder,
the previous state checkpoint will be overwritten. You can
avoid this by making sure ``RunConfig(name=...)`` is set to a unique
identifier:
.. code-block:: python
search_alg = HyperOptSearch()
tuner_1 = tune.Tuner(
train_fn,
tune_config=tune.TuneConfig(
num_samples=5,
search_alg=search_alg,
),
run_config=tune.RunConfig(
name="my-experiment-1",
storage_path="~/my_results",
)
)
results = tuner_1.fit()
search_alg2 = HyperOptSearch()
search_alg2.restore_from_dir(
os.path.join("~/my_results", "my-experiment-1")
)
.. _tune-basicvariant:
Random search and grid search (tune.search.basic_variant.BasicVariantGenerator)
-------------------------------------------------------------------------------
The default and most basic way to do hyperparameter search is via random and grid search.
Ray Tune does this through the :class:`BasicVariantGenerator <ray.tune.search.basic_variant.BasicVariantGenerator>`
class that generates trial variants given a search space definition.
The :class:`BasicVariantGenerator <ray.tune.search.basic_variant.BasicVariantGenerator>` is used per
default if no search algorithm is passed to
:func:`Tuner <ray.tune.Tuner>`.
.. currentmodule:: ray.tune.search
.. autosummary::
:nosignatures:
:toctree: doc/
basic_variant.BasicVariantGenerator
.. _tune-ax:
Ax (tune.search.ax.AxSearch)
----------------------------
.. autosummary::
:nosignatures:
:toctree: doc/
ax.AxSearch
.. _bayesopt:
Bayesian Optimization (tune.search.bayesopt.BayesOptSearch)
-----------------------------------------------------------
.. autosummary::
:nosignatures:
:toctree: doc/
bayesopt.BayesOptSearch
.. _suggest-TuneBOHB:
BOHB (tune.search.bohb.TuneBOHB)
--------------------------------
BOHB (Bayesian Optimization HyperBand) is an algorithm that both terminates bad trials
and also uses Bayesian Optimization to improve the hyperparameter search.
It is available from the `HpBandSter library <https://github.com/automl/HpBandSter>`_.
Importantly, BOHB is intended to be paired with a specific scheduler class: :ref:`HyperBandForBOHB <tune-scheduler-bohb>`.
In order to use this search algorithm, you will need to install ``HpBandSter`` and ``ConfigSpace``:
.. code-block:: bash
$ pip install hpbandster ConfigSpace
See the `BOHB paper <https://arxiv.org/abs/1807.01774>`_ for more details.
.. autosummary::
:nosignatures:
:toctree: doc/
bohb.TuneBOHB
.. _tune-hebo:
HEBO (tune.search.hebo.HEBOSearch)
----------------------------------
.. autosummary::
:nosignatures:
:toctree: doc/
hebo.HEBOSearch
.. _tune-hyperopt:
HyperOpt (tune.search.hyperopt.HyperOptSearch)
----------------------------------------------
.. autosummary::
:nosignatures:
:toctree: doc/
hyperopt.HyperOptSearch
.. _nevergrad:
Nevergrad (tune.search.nevergrad.NevergradSearch)
-------------------------------------------------
.. autosummary::
:nosignatures:
:toctree: doc/
nevergrad.NevergradSearch
.. _tune-optuna:
Optuna (tune.search.optuna.OptunaSearch)
----------------------------------------
.. autosummary::
:nosignatures:
:toctree: doc/
optuna.OptunaSearch
.. _zoopt:
ZOOpt (tune.search.zoopt.ZOOptSearch)
-------------------------------------
.. autosummary::
:nosignatures:
:toctree: doc/
zoopt.ZOOptSearch
.. _repeater:
Repeated Evaluations (tune.search.Repeater)
-------------------------------------------
Use ``ray.tune.search.Repeater`` to average over multiple evaluations of the same
hyperparameter configurations. This is useful in cases where the evaluated
training procedure has high variance (i.e., in reinforcement learning).
By default, ``Repeater`` will take in a ``repeat`` parameter and a ``search_alg``.
The ``search_alg`` will suggest new configurations to try, and the ``Repeater``
will run ``repeat`` trials of the configuration. It will then average the
``search_alg.metric`` from the final results of each repeated trial.
.. warning:: It is recommended to not use ``Repeater`` with a TrialScheduler.
Early termination can negatively affect the average reported metric.
.. autosummary::
:nosignatures:
:toctree: doc/
Repeater
.. _limiter:
ConcurrencyLimiter (tune.search.ConcurrencyLimiter)
---------------------------------------------------
Use ``ray.tune.search.ConcurrencyLimiter`` to limit the amount of concurrency when using a search algorithm.
This is useful when a given optimization algorithm does not parallelize very well (like a naive Bayesian Optimization).
.. autosummary::
:nosignatures:
:toctree: doc/
ConcurrencyLimiter
.. _byo-algo:
Custom Search Algorithms (tune.search.Searcher)
-----------------------------------------------
If you are interested in implementing or contributing a new Search Algorithm, provide the following interface:
.. autosummary::
:nosignatures:
:toctree: doc/
Searcher
.. autosummary::
:nosignatures:
:toctree: doc/
Searcher.suggest
Searcher.save
Searcher.restore
Searcher.on_trial_result
Searcher.on_trial_complete
If contributing, make sure to add test cases and an entry in the function described below.
.. _shim:
Shim Instantiation (tune.create_searcher)
-----------------------------------------
There is also a shim function that constructs the search algorithm based on the provided string.
This can be useful if the search algorithm you want to use changes often
(e.g., specifying the search algorithm via a CLI option or config file).
.. autosummary::
:nosignatures:
:toctree: doc/
create_searcher
doc/source/tune/api/syncing.rst
+18
View File
@@ -0,0 +1,18 @@
Syncing in Tune
===============
.. seealso::
See :doc:`this user guide </tune/tutorials/tune-storage>` for more details and examples.
.. _tune-sync-config:
Tune Syncing Configuration
--------------------------
.. autosummary::
:nosignatures:
:toctree: doc/
~ray.tune.SyncConfig
doc/source/tune/api/trainable.rst
+311
View File
@@ -0,0 +1,311 @@
.. _trainable-docs:
.. TODO: these "basic" sections before the actual API docs start don't really belong here. Then again, the function
API does not really have a signature to just describe.
.. TODO: Reusing actors and advanced resources allocation seem ill-placed.
Training in Tune (tune.Trainable, tune.report)
=================================================
Training can be done with either a **Function API** (:func:`tune.report() <ray.tune.report>`) or
**Class API** (:ref:`tune.Trainable <tune-trainable-docstring>`).
For the sake of example, let's maximize this objective function:
.. literalinclude:: /tune/doc_code/trainable.py
:language: python
:start-after: __example_objective_start__
:end-before: __example_objective_end__
.. _tune-function-api:
Function Trainable API
----------------------
Use the Function API to define a custom training function that Tune runs in Ray actor processes. Each trial is placed
into a Ray actor process and runs in parallel.
The ``config`` argument in the function is a dictionary populated automatically by Ray Tune and corresponding to
the hyperparameters selected for the trial from the :ref:`search space <tune-key-concepts-search-spaces>`.
With the Function API, you can report intermediate metrics by simply calling :func:`tune.report() <ray.tune.report>` within the function.
.. literalinclude:: /tune/doc_code/trainable.py
:language: python
:start-after: __function_api_report_intermediate_metrics_start__
:end-before: __function_api_report_intermediate_metrics_end__
.. tip:: Do not use :func:`tune.report() <ray.tune.report>` within a ``Trainable`` class.
In the previous example, we reported on every step, but this metric reporting frequency
is configurable. For example, we could also report only a single time at the end with the final score:
.. literalinclude:: /tune/doc_code/trainable.py
:language: python
:start-after: __function_api_report_final_metrics_start__
:end-before: __function_api_report_final_metrics_end__
It's also possible to return a final set of metrics to Tune by returning them from your function:
.. literalinclude:: /tune/doc_code/trainable.py
:language: python
:start-after: __function_api_return_final_metrics_start__
:end-before: __function_api_return_final_metrics_end__
Note that Ray Tune outputs extra values in addition to the user reported metrics,
such as ``iterations_since_restore``. See :ref:`tune-autofilled-metrics` for an explanation of these values.
See how to configure checkpointing for a function trainable :ref:`here <tune-function-trainable-checkpointing>`.
.. _tune-class-api:
Class Trainable API
--------------------------
.. caution:: Do not use :func:`tune.report() <ray.tune.report>` within a ``Trainable`` class.
The Trainable **class API** will require users to subclass ``ray.tune.Trainable``. Here's a naive example of this API:
.. literalinclude:: /tune/doc_code/trainable.py
:language: python
:start-after: __class_api_example_start__
:end-before: __class_api_example_end__
As a subclass of ``tune.Trainable``, Tune will create a ``Trainable`` object on a
separate process (using the :ref:`Ray Actor API <actor-guide>`).
1. ``setup`` function is invoked once training starts.
2. ``step`` is invoked **multiple times**.
Each time, the Trainable object executes one logical iteration of training in the tuning process,
which may include one or more iterations of actual training.
3. ``cleanup`` is invoked when training is finished.
The ``config`` argument in the ``setup`` method is a dictionary populated automatically by Tune and corresponding to
the hyperparameters selected for the trial from the :ref:`search space <tune-key-concepts-search-spaces>`.
.. tip:: As a rule of thumb, the execution time of ``step`` should be large enough to avoid overheads
(i.e. more than a few seconds), but short enough to report progress periodically (i.e. at most a few minutes).
You'll notice that Ray Tune will output extra values in addition to the user reported metrics,
such as ``iterations_since_restore``.
See :ref:`tune-autofilled-metrics` for an explanation/glossary of these values.
See how to configure checkpoint for class trainable :ref:`here <tune-class-trainable-checkpointing>`.
Advanced: Reusing Actors in Tune
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. note:: This feature is only for the Trainable Class API.
Your Trainable can often take a long time to start.
To avoid this, you can do ``tune.TuneConfig(reuse_actors=True)`` (which is taken in by ``Tuner``) to reuse the same Trainable Python process and
object for multiple hyperparameters.
This requires you to implement ``Trainable.reset_config``, which provides a new set of hyperparameters.
It is up to the user to correctly update the hyperparameters of your trainable.
.. code-block:: python
from time import sleep
import ray
from ray import tune
from ray.tune.tuner import Tuner
def expensive_setup():
print("EXPENSIVE SETUP")
sleep(1)
class QuadraticTrainable(tune.Trainable):
def setup(self, config):
self.config = config
expensive_setup() # use reuse_actors=True to only run this once
self.max_steps = 5
self.step_count = 0
def step(self):
# Extract hyperparameters from the config
h1 = self.config["hparam1"]
h2 = self.config["hparam2"]
# Compute a simple quadratic objective where the optimum is at hparam1=3 and hparam2=5
loss = (h1 - 3) ** 2 + (h2 - 5) ** 2
metrics = {"loss": loss}
self.step_count += 1
if self.step_count > self.max_steps:
metrics["done"] = True
# Return the computed loss as the metric
return metrics
def reset_config(self, new_config):
# Update the configuration for a new trial while reusing the actor
self.config = new_config
return True
ray.init()
tuner_with_reuse = Tuner(
QuadraticTrainable,
param_space={
"hparam1": tune.uniform(-10, 10),
"hparam2": tune.uniform(-10, 10),
},
tune_config=tune.TuneConfig(
num_samples=10,
max_concurrent_trials=1,
reuse_actors=True, # Enable actor reuse and avoid expensive setup
),
run_config=ray.tune.RunConfig(
verbose=0,
checkpoint_config=ray.tune.CheckpointConfig(checkpoint_at_end=False),
),
)
tuner_with_reuse.fit()
Comparing Tune's Function API and Class API
-------------------------------------------
Here are a few key concepts and what they look like for the Function and Class API's.
======================= =============================================== ==============================================
Concept Function API Class API
======================= =============================================== ==============================================
Training Iteration Increments on each `tune.report` call Increments on each `Trainable.step` call
Report metrics `tune.report(metrics)` Return metrics from `Trainable.step`
Saving a checkpoint `tune.report(..., checkpoint=checkpoint)` `Trainable.save_checkpoint`
Loading a checkpoint `tune.get_checkpoint()` `Trainable.load_checkpoint`
Accessing config Passed as an argument `def train_func(config):` Passed through `Trainable.setup`
======================= =============================================== ==============================================
Advanced Resource Allocation
----------------------------
Trainables can themselves be distributed. If your trainable function / class creates further Ray actors or tasks
that also consume CPU / GPU resources, you will want to add more bundles to the :class:`PlacementGroupFactory`
to reserve extra resource slots.
For example, if a trainable class requires 1 GPU itself, but also launches 4 actors, each using another GPU,
then you should use :func:`tune.with_resources <ray.tune.with_resources>` like this:
.. code-block:: python
:emphasize-lines: 4-10
tuner = tune.Tuner(
tune.with_resources(my_trainable, tune.PlacementGroupFactory([
{"CPU": 1, "GPU": 1},
{"GPU": 1},
{"GPU": 1},
{"GPU": 1},
{"GPU": 1}
])),
run_config=RunConfig(name="my_trainable")
)
The ``Trainable`` also provides the ``default_resource_requests`` interface to automatically
declare the resources per trial based on the given configuration.
It is also possible to specify memory (``"memory"``, in bytes) and custom resource requirements.
.. currentmodule:: ray
Function API
------------
For reporting results and checkpoints with the function API,
see the :ref:`Ray Train utilities <train-loop-api>` documentation.
**Classes**
.. autosummary::
:nosignatures:
:toctree: doc/
~tune.Checkpoint
~tune.TuneContext
**Functions**
.. autosummary::
:nosignatures:
:toctree: doc/
~tune.get_checkpoint
~tune.get_context
~tune.report
.. _tune-trainable-docstring:
Trainable (Class API)
---------------------
Constructor
~~~~~~~~~~~
.. autosummary::
:nosignatures:
:toctree: doc/
~tune.Trainable
Trainable Methods to Implement
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autosummary::
:nosignatures:
:toctree: doc/
~tune.Trainable.setup
~tune.Trainable.save_checkpoint
~tune.Trainable.load_checkpoint
~tune.Trainable.step
~tune.Trainable.reset_config
~tune.Trainable.cleanup
~tune.Trainable.default_resource_request
.. _tune-util-ref:
Tune Trainable Utilities
-------------------------
Tune Data Ingestion Utilities
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autosummary::
:nosignatures:
:toctree: doc/
tune.with_parameters
Tune Resource Assignment Utilities
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autosummary::
:nosignatures:
:toctree: doc/
tune.with_resources
~tune.execution.placement_groups.PlacementGroupFactory
tune.utils.wait_for_gpu
Tune Trainable Debugging Utilities
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autosummary::
:nosignatures:
:toctree: doc/
tune.utils.diagnose_serialization
tune.utils.validate_save_restore
tune.utils.util.validate_warmstart
doc/source/tune/doc_code/faq.py
+352
View File
@@ -0,0 +1,352 @@
# flake8: noqa
# __reproducible_start__
import numpy as np
from ray import tune
def train_func(config):
# Set seed for trainable random result.
# If you remove this line, you will get different results
# each time you run the trial, even if the configuration
# is the same.
np.random.seed(config["seed"])
random_result = np.random.uniform(0, 100, size=1).item()
tune.report({"result": random_result})
# Set seed for Ray Tune's random search.
# If you remove this line, you will get different configurations
# each time you run the script.
np.random.seed(1234)
tuner = tune.Tuner(
train_func,
tune_config=tune.TuneConfig(
num_samples=10,
search_alg=tune.search.BasicVariantGenerator(),
),
param_space={"seed": tune.randint(0, 1000)},
)
tuner.fit()
# __reproducible_end__
# __basic_config_start__
config = {"a": {"x": tune.uniform(0, 10)}, "b": tune.choice([1, 2, 3])}
# __basic_config_end__
# __conditional_spaces_start__
config = {
"a": tune.randint(5, 10),
"b": tune.sample_from(lambda config: np.random.randint(0, config["a"])),
}
# __conditional_spaces_end__
# __iter_start__
def _iter():
for a in range(5, 10):
for b in range(a):
yield a, b
config = {
"ab": tune.grid_search(list(_iter())),
}
# __iter_end__
def train_func(config):
random_result = np.random.uniform(0, 100, size=1).item()
tune.report({"result": random_result})
train_fn = train_func
MOCK = True
# Note we put this check here to make sure at least the syntax of
# the code is correct. Some of these snippets simply can't be run on the nose.
if not MOCK:
# __resources_start__
tuner = tune.Tuner(
tune.with_resources(
train_fn, resources={"cpu": 2, "gpu": 0.5, "custom_resources": {"hdd": 80}}
),
)
tuner.fit()
# __resources_end__
# __resources_pgf_start__
tuner = tune.Tuner(
tune.with_resources(
train_fn,
resources=tune.PlacementGroupFactory(
[
{"CPU": 2, "GPU": 0.5, "hdd": 80},
{"CPU": 1},
{"CPU": 1},
],
strategy="PACK",
),
)
)
tuner.fit()
# __resources_pgf_end__
# __resources_lambda_start__
tuner = tune.Tuner(
tune.with_resources(
train_fn,
resources=lambda config: {"GPU": 1} if config["use_gpu"] else {"GPU": 0},
),
param_space={
"use_gpu": True,
},
)
tuner.fit()
# __resources_lambda_end__
metric = None
# __modin_start__
def train_fn(config):
# some Modin operations here
# import modin.pandas as pd
tune.report({"metric": metric})
tuner = tune.Tuner(
tune.with_resources(
train_fn,
resources=tune.PlacementGroupFactory(
[
{"CPU": 1}, # this bundle will be used by the trainable itself
{"CPU": 1}, # this bundle will be used by Modin
],
strategy="PACK",
),
)
)
tuner.fit()
# __modin_end__
# __huge_data_start__
from ray import tune
import numpy as np
def train_func(config, num_epochs=5, data=None):
for i in range(num_epochs):
for sample in data:
# ... train on sample
pass
# Some huge dataset
data = np.random.random(size=100000000)
tuner = tune.Tuner(tune.with_parameters(train_func, num_epochs=5, data=data))
tuner.fit()
# __huge_data_end__
# __seeded_1_start__
import random
random.seed(1234)
output = [random.randint(0, 100) for _ in range(10)]
# The output will always be the same.
assert output == [99, 56, 14, 0, 11, 74, 4, 85, 88, 10]
# __seeded_1_end__
# __seeded_2_start__
# This should suffice to initialize the RNGs for most Python-based libraries
import random
import numpy as np
random.seed(1234)
np.random.seed(5678)
# __seeded_2_end__
# __torch_tf_seeds_start__
import torch
torch.manual_seed(0)
import tensorflow as tf
tf.random.set_seed(0)
# __torch_tf_seeds_end__
# __torch_seed_example_start__
import random
import numpy as np
from ray import tune
def trainable(config):
# config["seed"] is set deterministically, but differs between training runs
random.seed(config["seed"])
np.random.seed(config["seed"])
# torch.manual_seed(config["seed"])
# ... training code
config = {
"seed": tune.randint(0, 10000),
# ...
}
if __name__ == "__main__":
# Set seed for the search algorithms/schedulers
random.seed(1234)
np.random.seed(1234)
# Don't forget to check if the search alg has a `seed` parameter
tuner = tune.Tuner(trainable, param_space=config)
tuner.fit()
# __torch_seed_example_end__
# __large_data_start__
from ray import tune
import numpy as np
def f(config, data=None):
pass
# use data
data = np.random.random(size=100000000)
tuner = tune.Tuner(tune.with_parameters(f, data=data))
tuner.fit()
# __large_data_end__
import ray
ray.shutdown()
# __grid_search_start__
parameters = {
"qux": tune.sample_from(lambda spec: 2 + 2),
"bar": tune.grid_search([True, False]),
"foo": tune.grid_search([1, 2, 3]),
"baz": "asd", # a constant value
}
tuner = tune.Tuner(train_fn, param_space=parameters)
tuner.fit()
# __grid_search_end__
# __grid_search_2_start__
# num_samples=10 repeats the 3x3 grid search 10 times, for a total of 90 trials
tuner = tune.Tuner(
train_fn,
run_config=tune.RunConfig(name="my_trainable"),
param_space={
"alpha": tune.uniform(100, 200),
"beta": tune.sample_from(lambda config: config["alpha"] * np.random.normal()),
"nn_layers": [
tune.grid_search([16, 64, 256]),
tune.grid_search([16, 64, 256]),
],
},
tune_config=tune.TuneConfig(num_samples=10),
)
# __grid_search_2_end__
if not MOCK:
import os
from pathlib import Path
# __no_chdir_start__
def train_func(config):
# Read from relative paths
print(open("./read.txt").read())
# The working directory shouldn't have changed from the original
# NOTE: The `TUNE_ORIG_WORKING_DIR` environment variable is deprecated.
assert os.getcwd() == os.environ["TUNE_ORIG_WORKING_DIR"]
# Write to the Tune trial directory, not the shared working dir
tune_trial_dir = Path(ray.tune.get_context().get_trial_dir())
with open(tune_trial_dir / "write.txt", "w") as f:
f.write("trial saved artifact")
os.environ["RAY_CHDIR_TO_TRIAL_DIR"] = "0"
tuner = tune.Tuner(train_func)
tuner.fit()
# __no_chdir_end__
# __iter_experimentation_initial_start__
import os
import tempfile
import torch
from ray import tune
from ray.tune import Checkpoint
import random
def trainable(config):
for epoch in range(1, config["num_epochs"]):
# Do some training...
with tempfile.TemporaryDirectory() as tempdir:
torch.save(
{"model_state_dict": {"x": 1}}, os.path.join(tempdir, "model.pt")
)
tune.report(
{"score": random.random()},
checkpoint=Checkpoint.from_directory(tempdir),
)
tuner = tune.Tuner(
trainable,
param_space={"num_epochs": 10, "hyperparam": tune.grid_search([1, 2, 3])},
tune_config=tune.TuneConfig(metric="score", mode="max"),
)
result_grid = tuner.fit()
best_result = result_grid.get_best_result()
best_checkpoint = best_result.checkpoint
# __iter_experimentation_initial_end__
# __iter_experimentation_resume_start__
import ray
def trainable(config):
# Add logic to handle the initial checkpoint.
checkpoint: Checkpoint = config["start_from_checkpoint"]
with checkpoint.as_directory() as checkpoint_dir:
model_state_dict = torch.load(os.path.join(checkpoint_dir, "model.pt"))
# Initialize a model from the checkpoint...
# model = ...
# model.load_state_dict(model_state_dict)
for epoch in range(1, config["num_epochs"]):
# Do some more training...
...
tune.report({"score": random.random()})
new_tuner = tune.Tuner(
trainable,
param_space={
"num_epochs": 10,
"hyperparam": tune.grid_search([4, 5, 6]),
"start_from_checkpoint": best_checkpoint,
},
tune_config=tune.TuneConfig(metric="score", mode="max"),
)
result_grid = new_tuner.fit()
# __iter_experimentation_resume_end__
doc/source/tune/doc_code/fault_tolerance.py
+162
View File
@@ -0,0 +1,162 @@
# flake8: noqa
# __ft_initial_run_start__
import json
import os
import tempfile
from ray import tune
def trainable(config):
# Checkpoint loading
checkpoint = tune.get_checkpoint()
start = 1
if checkpoint:
with checkpoint.as_directory() as checkpoint_dir:
with open(os.path.join(checkpoint_dir, "checkpoint.json"), "r") as f:
state = json.load(f)
start = state["epoch"] + 1
for epoch in range(start, config["num_epochs"]):
# Do some training...
# Checkpoint saving
with tempfile.TemporaryDirectory() as temp_checkpoint_dir:
with open(os.path.join(temp_checkpoint_dir, "checkpoint.json"), "w") as f:
json.dump({"epoch": epoch}, f)
tune.report(
{"epoch": epoch},
checkpoint=tune.Checkpoint.from_directory(temp_checkpoint_dir),
)
tuner = tune.Tuner(
trainable,
param_space={"num_epochs": 10},
run_config=tune.RunConfig(
storage_path=os.path.expanduser("~/ray_results"),
name="tune_fault_tolerance_guide",
),
)
result_grid = tuner.fit()
# __ft_initial_run_end__
assert not result_grid.errors
# __ft_restored_run_start__
tuner = tune.Tuner.restore(
os.path.expanduser("~/ray_results/tune_fault_tolerance_guide"),
trainable=trainable,
resume_errored=True,
)
tuner.fit()
# __ft_restored_run_end__
# __ft_restore_options_start__
tuner = tune.Tuner.restore(
os.path.expanduser("~/ray_results/tune_fault_tolerance_guide"),
trainable=trainable,
resume_errored=True,
restart_errored=False,
resume_unfinished=True,
)
# __ft_restore_options_end__
# __ft_restore_multiplexing_start__
import os
from ray import tune
storage_path = os.path.expanduser("~/ray_results")
exp_name = "tune_fault_tolerance_guide"
path = os.path.join(storage_path, exp_name)
if tune.Tuner.can_restore(path):
tuner = tune.Tuner.restore(path, trainable=trainable, resume_errored=True)
else:
tuner = tune.Tuner(
trainable,
param_space={"num_epochs": 10},
run_config=tune.RunConfig(storage_path=storage_path, name=exp_name),
)
tuner.fit()
# __ft_restore_multiplexing_end__
# Run the multiplexed logic again to make sure it goes through the restore branch.
if tune.Tuner.can_restore(path):
tuner = tune.Tuner.restore(path, trainable=trainable, resume_errored=True)
else:
tuner = tune.Tuner(
trainable,
param_space={"num_epochs": 10},
run_config=tune.RunConfig(storage_path=storage_path, name=exp_name),
)
assert tuner.get_results()
# __ft_restore_objrefs_initial_start__
import ray
from ray import tune
class LargeModel:
def __init__(self, model_id):
self.model_id = model_id
# Load weights based on the `model_id`...
def train_fn(config):
# Retrieve the model from the object store.
model = ray.get(config["model_ref"])
print(model.model_id)
# These models may be large, so `ray.put` them in the Ray Object Store
# to share the models between trials.
model_refs = [ray.put(LargeModel(1)), ray.put(LargeModel(2))]
tuner = tune.Tuner(
train_fn,
# Tune over the object references!
param_space={"model_ref": tune.grid_search(model_refs)},
run_config=tune.RunConfig(
storage_path=os.path.expanduser("~/ray_results"), name="restore_object_refs"
),
)
tuner.fit()
# __ft_restore_objrefs_initial_end__
if ray.is_initialized():
ray.shutdown()
# __ft_restore_objrefs_restored_start__
# Re-create the objects and put them in the object store.
param_space = {
"model_ref": tune.grid_search([ray.put(LargeModel(1)), ray.put(LargeModel(2))])
}
tuner = tune.Tuner.restore(
os.path.expanduser("~/ray_results/restore_object_refs"),
trainable=train_fn,
# Re-specify the `param_space` to update the object references.
param_space=param_space,
resume_errored=True,
)
tuner.fit()
# __ft_restore_objrefs_restored_end__
# __ft_trial_failure_start__
from ray import tune
tuner = tune.Tuner(
trainable,
param_space={"num_epochs": 10},
run_config=tune.RunConfig(
storage_path=os.path.expanduser("~/ray_results"),
name="trial_fault_tolerance",
failure_config=tune.FailureConfig(max_failures=3),
),
)
tuner.fit()
# __ft_trial_failure_end__
doc/source/tune/doc_code/keras_hyperopt.py
+35
View File
@@ -0,0 +1,35 @@
# flake8: noqa
accuracy = 42
# __keras_hyperopt_start__
from ray import tune
from ray.tune.search.hyperopt import HyperOptSearch
import keras
def objective(config): # <1>
model = keras.models.Sequential()
model.add(keras.layers.Dense(784, activation=config["activation"]))
model.add(keras.layers.Dense(10, activation="softmax"))
model.compile(loss="binary_crossentropy", optimizer="adam", metrics=["accuracy"])
# model.fit(...)
# loss, accuracy = model.evaluate(...)
return {"accuracy": accuracy}
search_space = {"activation": tune.choice(["relu", "tanh"])} # <2>
algo = HyperOptSearch()
tuner = tune.Tuner( # <3>
objective,
tune_config=tune.TuneConfig(
metric="accuracy",
mode="max",
search_alg=algo,
),
param_space=search_space,
)
results = tuner.fit()
# __keras_hyperopt_end__
doc/source/tune/doc_code/key_concepts.py
+162
View File
@@ -0,0 +1,162 @@
# flake8: noqa
# __function_api_start__
from ray import tune
def objective(x, a, b): # Define an objective function.
return a * (x**2) + b
def trainable(config): # Pass a "config" dictionary into your trainable.
for x in range(20): # "Train" for 20 iterations and compute intermediate scores.
score = objective(x, config["a"], config["b"])
tune.report({"score": score}) # Send the score to Tune.
# __function_api_end__
# __class_api_start__
from ray import tune
def objective(x, a, b):
return a * (x**2) + b
class Trainable(tune.Trainable):
def setup(self, config):
# config (dict): A dict of hyperparameters
self.x = 0
self.a = config["a"]
self.b = config["b"]
def step(self): # This is called iteratively.
score = objective(self.x, self.a, self.b)
self.x += 1
return {"score": score}
# __class_api_end__
# TODO: this example does not work as advertised. Errors out.
def save_checkpoint(self, checkpoint_dir):
pass
def load_checkpoint(self, checkpoint_dir):
pass
# __run_tunable_start__
# Pass in a Trainable class or function, along with a search space "config".
tuner = tune.Tuner(trainable, param_space={"a": 2, "b": 4})
tuner.fit()
# __run_tunable_end__
# __run_tunable_samples_start__
tuner = tune.Tuner(
trainable, param_space={"a": 2, "b": 4}, tune_config=tune.TuneConfig(num_samples=10)
)
tuner.fit()
# __run_tunable_samples_end__
# __search_space_start__
space = {"a": tune.uniform(0, 1), "b": tune.uniform(0, 1)}
tuner = tune.Tuner(
trainable, param_space=space, tune_config=tune.TuneConfig(num_samples=10)
)
tuner.fit()
# __search_space_end__
# __config_start__
config = {
"uniform": tune.uniform(-5, -1), # Uniform float between -5 and -1
"quniform": tune.quniform(3.2, 5.4, 0.2), # Round to multiples of 0.2
"loguniform": tune.loguniform(1e-4, 1e-1), # Uniform float in log space
"qloguniform": tune.qloguniform(1e-4, 1e-1, 5e-5), # Round to multiples of 0.00005
"randn": tune.randn(10, 2), # Normal distribution with mean 10 and sd 2
"qrandn": tune.qrandn(10, 2, 0.2), # Round to multiples of 0.2
"randint": tune.randint(-9, 15), # Random integer between -9 and 15
"qrandint": tune.qrandint(-21, 12, 3), # Round to multiples of 3 (includes 12)
"lograndint": tune.lograndint(1, 10), # Random integer in log space
"qlograndint": tune.qlograndint(1, 10, 2), # Round to multiples of 2
"choice": tune.choice(["a", "b", "c"]), # Choose one of these options uniformly
"func": tune.sample_from(
lambda config: config["uniform"] * 0.01
), # Depends on other value
"grid": tune.grid_search([32, 64, 128]), # Search over all these values
}
# __config_end__
# __bayes_start__
from ray.tune.search.bayesopt import BayesOptSearch
# Define the search space
search_space = {"a": tune.uniform(0, 1), "b": tune.uniform(0, 20)}
algo = BayesOptSearch(random_search_steps=4)
tuner = tune.Tuner(
trainable,
tune_config=tune.TuneConfig(
metric="score",
mode="min",
search_alg=algo,
),
run_config=tune.RunConfig(stop={"training_iteration": 20}),
param_space=search_space,
)
tuner.fit()
# __bayes_end__
# __hyperband_start__
from ray.tune.schedulers import HyperBandScheduler
# Create HyperBand scheduler and minimize the score
hyperband = HyperBandScheduler(metric="score", mode="max")
config = {"a": tune.uniform(0, 1), "b": tune.uniform(0, 1)}
tuner = tune.Tuner(
trainable,
tune_config=tune.TuneConfig(
num_samples=20,
scheduler=hyperband,
),
param_space=config,
)
tuner.fit()
# __hyperband_end__
# __analysis_start__
tuner = tune.Tuner(
trainable,
tune_config=tune.TuneConfig(
metric="score",
mode="min",
search_alg=BayesOptSearch(random_search_steps=4),
),
run_config=tune.RunConfig(
stop={"training_iteration": 20},
),
param_space=config,
)
results = tuner.fit()
best_result = results.get_best_result() # Get best result object
best_config = best_result.config # Get best trial's hyperparameters
best_logdir = best_result.path # Get best trial's result directory
best_checkpoint = best_result.checkpoint # Get best trial's best checkpoint
best_metrics = best_result.metrics # Get best trial's last results
best_result_df = best_result.metrics_dataframe # Get best result as pandas dataframe
# __analysis_end__
# __results_start__
# Get a dataframe with the last results for each trial
df_results = results.get_dataframe()
# Get a dataframe of results for a specific score or mode
df = results.get_dataframe(filter_metric="score", filter_mode="max")
# __results_end__
doc/source/tune/doc_code/pytorch_optuna.py
+116
View File
@@ -0,0 +1,116 @@
# flake8: noqa
import os
from filelock import FileLock
import torch.nn as nn
import torch.nn.functional as F
from torchvision import datasets, transforms
EPOCH_SIZE = 512
TEST_SIZE = 256
def train_epoch(model, optimizer, train_loader, device=None):
device = device or torch.device("cpu")
model.train()
for batch_idx, (data, target) in enumerate(train_loader):
if batch_idx * len(data) > EPOCH_SIZE:
return
data, target = data.to(device), target.to(device)
optimizer.zero_grad()
output = model(data)
loss = F.nll_loss(output, target)
loss.backward()
optimizer.step()
def test(model, data_loader, device=None):
device = device or torch.device("cpu")
model.eval()
correct = 0
total = 0
with torch.no_grad():
for batch_idx, (data, target) in enumerate(data_loader):
if batch_idx * len(data) > TEST_SIZE:
break
data, target = data.to(device), target.to(device)
outputs = model(data)
_, predicted = torch.max(outputs.data, 1)
total += target.size(0)
correct += (predicted == target).sum().item()
return correct / total
def load_data():
mnist_transforms = transforms.Compose(
[transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]
)
with FileLock(os.path.expanduser("~/data.lock")):
train_loader = torch.utils.data.DataLoader(
datasets.MNIST(
"~/data", train=True, download=True, transform=mnist_transforms
),
batch_size=64,
shuffle=True,
)
test_loader = torch.utils.data.DataLoader(
datasets.MNIST(
"~/data", train=False, download=True, transform=mnist_transforms
),
batch_size=64,
shuffle=True,
)
return train_loader, test_loader
class ConvNet(nn.Module):
def __init__(self):
super(ConvNet, self).__init__()
self.conv1 = nn.Conv2d(1, 3, kernel_size=3)
self.fc = nn.Linear(192, 10)
def forward(self, x):
x = F.relu(F.max_pool2d(self.conv1(x), 3))
x = x.view(-1, 192)
x = self.fc(x)
return F.log_softmax(x, dim=1)
# __pytorch_optuna_start__
import torch
from ray import tune
from ray.tune.search.optuna import OptunaSearch
def objective(config): # <1>
train_loader, test_loader = load_data() # Load some data
model = ConvNet().to("cpu") # Create a PyTorch conv net
optimizer = torch.optim.SGD( # Tune the optimizer
model.parameters(), lr=config["lr"], momentum=config["momentum"]
)
while True:
train_epoch(model, optimizer, train_loader) # Train the model
acc = test(model, test_loader) # Compute test accuracy
tune.report({"mean_accuracy": acc}) # Report to Tune
search_space = {"lr": tune.loguniform(1e-4, 1e-2), "momentum": tune.uniform(0.1, 0.9)}
algo = OptunaSearch() # <2>
tuner = tune.Tuner( # <3>
objective,
tune_config=tune.TuneConfig(
metric="mean_accuracy",
mode="max",
search_alg=algo,
),
run_config=tune.RunConfig(
stop={"training_iteration": 5},
),
param_space=search_space,
)
results = tuner.fit()
print("Best config is:", results.get_best_result().config)
# __pytorch_optuna_end__
doc/source/tune/doc_code/stopping.py
+169
View File
@@ -0,0 +1,169 @@
# flake8: noqa
# fmt: off
# __stopping_example_trainable_start__
from ray import tune
import time
def my_trainable(config):
i = 1
while True:
# Do some training...
time.sleep(1)
# Report some metrics for demonstration...
tune.report({"mean_accuracy": min(i / 10, 1.0)})
i += 1
# __stopping_example_trainable_end__
# fmt: on
def my_trainable(config):
# NOTE: This re-defines the training loop with the sleep removed for faster testing.
i = 1
# Training won't finish unless one of the stopping criteria is met!
while True:
# Do some training, and report some metrics for demonstration...
tune.report({"mean_accuracy": min(i / 10, 1.0)})
i += 1
# __stopping_dict_start__
from ray import tune
tuner = tune.Tuner(
my_trainable,
run_config=tune.RunConfig(stop={"training_iteration": 10, "mean_accuracy": 0.8}),
)
result_grid = tuner.fit()
# __stopping_dict_end__
final_iter = result_grid[0].metrics["training_iteration"]
assert final_iter == 8, final_iter
# __stopping_fn_start__
from ray import tune
def stop_fn(trial_id: str, result: dict) -> bool:
return result["mean_accuracy"] >= 0.8 or result["training_iteration"] >= 10
tuner = tune.Tuner(my_trainable, run_config=tune.RunConfig(stop=stop_fn))
result_grid = tuner.fit()
# __stopping_fn_end__
final_iter = result_grid[0].metrics["training_iteration"]
assert final_iter == 8, final_iter
# __stopping_cls_start__
from ray import tune
from ray.tune import Stopper
class CustomStopper(Stopper):
def __init__(self):
self.should_stop = False
def __call__(self, trial_id: str, result: dict) -> bool:
if not self.should_stop and result["mean_accuracy"] >= 0.8:
self.should_stop = True
return self.should_stop
def stop_all(self) -> bool:
"""Returns whether to stop trials and prevent new ones from starting."""
return self.should_stop
stopper = CustomStopper()
tuner = tune.Tuner(
my_trainable,
run_config=tune.RunConfig(stop=stopper),
tune_config=tune.TuneConfig(num_samples=2),
)
result_grid = tuner.fit()
# __stopping_cls_end__
for result in result_grid:
final_iter = result.metrics.get("training_iteration", 0)
assert final_iter <= 8, final_iter
# __stopping_on_trial_error_start__
from ray import tune
import time
def my_failing_trainable(config):
if config["should_fail"]:
raise RuntimeError("Failing (on purpose)!")
# Do some training...
time.sleep(10)
tune.report({"mean_accuracy": 0.9})
tuner = tune.Tuner(
my_failing_trainable,
param_space={"should_fail": tune.grid_search([True, False])},
run_config=tune.RunConfig(failure_config=tune.FailureConfig(fail_fast=True)),
)
result_grid = tuner.fit()
# __stopping_on_trial_error_end__
for result in result_grid:
# Should never get to report
final_iter = result.metrics.get("training_iteration")
assert not final_iter, final_iter
# __early_stopping_start__
from ray import tune
from ray.tune.schedulers import AsyncHyperBandScheduler
scheduler = AsyncHyperBandScheduler(time_attr="training_iteration")
tuner = tune.Tuner(
my_trainable,
run_config=tune.RunConfig(stop={"training_iteration": 10}),
tune_config=tune.TuneConfig(
scheduler=scheduler, num_samples=2, metric="mean_accuracy", mode="max"
),
)
result_grid = tuner.fit()
# __early_stopping_end__
def my_trainable(config):
# NOTE: Introduce the sleep again for the time-based unit-tests.
i = 1
while True:
time.sleep(1)
# Do some training, and report some metrics for demonstration...
tune.report({"mean_accuracy": min(i / 10, 1.0)})
i += 1
# __stopping_trials_by_time_start__
from ray import tune
tuner = tune.Tuner(
my_trainable,
# Stop a trial after it's run for more than 5 seconds.
run_config=tune.RunConfig(stop={"time_total_s": 5}),
)
result_grid = tuner.fit()
# __stopping_trials_by_time_end__
# Should only get ~5 reports
assert result_grid[0].metrics["training_iteration"] < 8
# __stopping_experiment_by_time_start__
from ray import tune
# Stop the entire experiment after ANY trial has run for more than 5 seconds.
tuner = tune.Tuner(my_trainable, tune_config=tune.TuneConfig(time_budget_s=5.0))
result_grid = tuner.fit()
# __stopping_experiment_by_time_end__
# Should only get ~5 reports
assert result_grid[0].metrics["training_iteration"] < 8
doc/source/tune/doc_code/trainable.py
+82
View File
@@ -0,0 +1,82 @@
# flake8: noqa
# fmt: off
# __example_objective_start__
def objective(x, a, b):
return a * (x ** 0.5) + b
# __example_objective_end__
# fmt: on
# __function_api_report_intermediate_metrics_start__
from ray import tune
def trainable(config: dict):
intermediate_score = 0
for x in range(20):
intermediate_score = objective(x, config["a"], config["b"])
tune.report({"score": intermediate_score}) # This sends the score to Tune.
tuner = tune.Tuner(trainable, param_space={"a": 2, "b": 4})
results = tuner.fit()
# __function_api_report_intermediate_metrics_end__
# __function_api_report_final_metrics_start__
from ray import tune
def trainable(config: dict):
final_score = 0
for x in range(20):
final_score = objective(x, config["a"], config["b"])
tune.report({"score": final_score}) # This sends the score to Tune.
tuner = tune.Tuner(trainable, param_space={"a": 2, "b": 4})
results = tuner.fit()
# __function_api_report_final_metrics_end__
# fmt: off
# __function_api_return_final_metrics_start__
def trainable(config: dict):
final_score = 0
for x in range(20):
final_score = objective(x, config["a"], config["b"])
return {"score": final_score} # This sends the score to Tune.
# __function_api_return_final_metrics_end__
# fmt: on
# __class_api_example_start__
from ray import tune
class Trainable(tune.Trainable):
def setup(self, config: dict):
# config (dict): A dict of hyperparameters
self.x = 0
self.a = config["a"]
self.b = config["b"]
def step(self): # This is called iteratively.
score = objective(self.x, self.a, self.b)
self.x += 1
return {"score": score}
tuner = tune.Tuner(
Trainable,
run_config=tune.RunConfig(
# Train for 20 steps
stop={"training_iteration": 20},
checkpoint_config=tune.CheckpointConfig(
# We haven't implemented checkpointing yet. See below!
checkpoint_at_end=False
),
),
param_space={"a": 2, "b": 4},
)
results = tuner.fit()
# __class_api_example_end__
doc/source/tune/doc_code/trial_checkpoint.py
+188
View File
@@ -0,0 +1,188 @@
# flake8: noqa
# __class_api_checkpointing_start__
import os
import torch
from torch import nn
from ray import tune
class MyTrainableClass(tune.Trainable):
def setup(self, config):
self.model = nn.Sequential(
nn.Linear(config.get("input_size", 32), 32), nn.ReLU(), nn.Linear(32, 10)
)
def step(self):
return {}
def save_checkpoint(self, tmp_checkpoint_dir):
checkpoint_path = os.path.join(tmp_checkpoint_dir, "model.pth")
torch.save(self.model.state_dict(), checkpoint_path)
return tmp_checkpoint_dir
def load_checkpoint(self, tmp_checkpoint_dir):
checkpoint_path = os.path.join(tmp_checkpoint_dir, "model.pth")
self.model.load_state_dict(torch.load(checkpoint_path))
tuner = tune.Tuner(
MyTrainableClass,
param_space={"input_size": 64},
run_config=tune.RunConfig(
stop={"training_iteration": 2},
checkpoint_config=tune.CheckpointConfig(checkpoint_frequency=2),
),
)
tuner.fit()
# __class_api_checkpointing_end__
# __class_api_manual_checkpointing_start__
import random
# to be implemented by user.
def detect_instance_preemption():
choice = random.randint(1, 100)
# simulating a 1% chance of preemption.
return choice <= 1
def train_func(self):
# training code
result = {"mean_accuracy": "my_accuracy"}
if detect_instance_preemption():
result.update(should_checkpoint=True)
return result
# __class_api_manual_checkpointing_end__
# __class_api_periodic_checkpointing_start__
tuner = tune.Tuner(
MyTrainableClass,
run_config=tune.RunConfig(
stop={"training_iteration": 2},
checkpoint_config=tune.CheckpointConfig(checkpoint_frequency=10),
),
)
tuner.fit()
# __class_api_periodic_checkpointing_end__
# __class_api_end_checkpointing_start__
tuner = tune.Tuner(
MyTrainableClass,
run_config=tune.RunConfig(
stop={"training_iteration": 2},
checkpoint_config=tune.CheckpointConfig(
checkpoint_frequency=10, checkpoint_at_end=True
),
),
)
tuner.fit()
# __class_api_end_checkpointing_end__
class MyModel:
def state_dict(self) -> dict:
return {}
def load_state_dict(self, state_dict):
pass
# __function_api_checkpointing_from_dir_start__
import os
import tempfile
from ray import tune
from ray.tune import Checkpoint
def train_func(config):
start = 1
my_model = MyModel()
checkpoint = tune.get_checkpoint()
if checkpoint:
with checkpoint.as_directory() as checkpoint_dir:
checkpoint_dict = torch.load(os.path.join(checkpoint_dir, "checkpoint.pt"))
start = checkpoint_dict["epoch"] + 1
my_model.load_state_dict(checkpoint_dict["model_state"])
for epoch in range(start, config["epochs"] + 1):
# Model training here
# ...
metrics = {"metric": 1}
with tempfile.TemporaryDirectory() as tempdir:
torch.save(
{"epoch": epoch, "model_state": my_model.state_dict()},
os.path.join(tempdir, "checkpoint.pt"),
)
tune.report(metrics=metrics, checkpoint=Checkpoint.from_directory(tempdir))
tuner = tune.Tuner(train_func, param_space={"epochs": 5})
result_grid = tuner.fit()
# __function_api_checkpointing_from_dir_end__
assert not result_grid.errors
# __function_api_checkpointing_periodic_start__
NUM_EPOCHS = 12
# checkpoint every three epochs.
CHECKPOINT_FREQ = 3
def train_func(config):
for epoch in range(1, config["epochs"] + 1):
# Model training here
# ...
# Report metrics and save a checkpoint
metrics = {"metric": "my_metric"}
if epoch % CHECKPOINT_FREQ == 0:
with tempfile.TemporaryDirectory() as tempdir:
# Save a checkpoint in tempdir.
tune.report(metrics, checkpoint=Checkpoint.from_directory(tempdir))
else:
tune.report(metrics)
tuner = tune.Tuner(train_func, param_space={"epochs": NUM_EPOCHS})
result_grid = tuner.fit()
# __function_api_checkpointing_periodic_end__
assert not result_grid.errors
assert len(result_grid[0].best_checkpoints) == NUM_EPOCHS // CHECKPOINT_FREQ
# __callback_api_checkpointing_start__
from ray import tune
from ray.tune.experiment import Trial
from ray.tune.result import SHOULD_CHECKPOINT, TRAINING_ITERATION
class CheckpointByStepsTaken(tune.Callback):
def __init__(self, iterations_per_checkpoint: int):
self.steps_per_checkpoint = iterations_per_checkpoint
self._trials_last_checkpoint = {}
def on_trial_result(
self, iteration: int, trials: list[Trial], trial: Trial, result: dict, **info
):
current_iteration = result[TRAINING_ITERATION]
if (
current_iteration - self._trials_last_checkpoint.get(trial, -1)
>= self.steps_per_checkpoint
):
result[SHOULD_CHECKPOINT] = True
self._trials_last_checkpoint[trial] = current_iteration
# __callback_api_checkpointing_end__
doc/source/tune/doc_code/tune.py
+60
View File
@@ -0,0 +1,60 @@
# flake8: noqa
# fmt: off
# __step1_begin__
from ray import tune
import ray
import os
NUM_MODELS = 100
def train_model(config):
score = config["model_id"]
# Import model libraries, etc...
# Load data and train model code here...
# Return final stats. You can also return intermediate progress
# using ray.tune.report() if needed.
# To return your model, you could write it to storage and return its
# URI in this dict, or return it as a Tune Checkpoint:
# https://docs.ray.io/en/latest/tune/tutorials/tune-checkpoints.html
return {"score": score, "other_data": ...}
# __step1_end__
# __step2_begin__
# Define trial parameters as a single grid sweep.
trial_space = {
# This is an example parameter. You could replace it with filesystem paths,
# model types, or even full nested Python dicts of model configurations, etc.,
# that enumerate the set of trials to run.
"model_id": tune.grid_search([
"model_{}".format(i)
for i in range(NUM_MODELS)
])
}
# __step2_end__
# __step3_begin__
# Can customize resources per trial, here we set 1 CPU each.
train_model = tune.with_resources(train_model, {"cpu": 1})
# __step3_end__
# __step4_begin__
# Start a Tune run and print the best result.
tuner = tune.Tuner(train_model, param_space=trial_space)
results = tuner.fit()
# Access individual results.
print(results[0])
print(results[1])
print(results[2])
# __step4_end__
# __tasks_begin__
remote_train = ray.remote(train_model)
futures = [remote_train.remote({"model_id": i}) for i in range(NUM_MODELS)]
print("Submitting tasks...")
results = ray.get(futures)
print("Trial results", results)
# __tasks_end__
doc/source/tune/examples/BUILD.bazel
+55
View File
@@ -0,0 +1,55 @@
load("//bazel:python.bzl", "py_test_run_all_notebooks")
filegroup(
name = "tune_examples",
srcs = glob(["*.ipynb"]),
visibility = ["//doc:__subpackages__"],
)
# --------------------------------------------------------------------
# Test all doc/source/tune/examples notebooks.
# --------------------------------------------------------------------
# pbt_ppo_example.ipynb is not tested right now due to large resource
# requirements
py_test_run_all_notebooks(
size = "medium",
include = ["*.ipynb"],
data = ["//doc/source/tune/examples:tune_examples"],
exclude = [
"pbt_ppo_example.ipynb",
"tune-xgboost.ipynb",
"pbt_transformers.ipynb", # Transformers uses legacy Tune APIs.
"tune-aim.ipynb", # CI does not have `aim`
"bohb_example.ipynb", # CI does not have bohb requirements
],
tags = [
"exclusive",
"team:ml",
],
env = {"RAY_TRAIN_V2_ENABLED": "1"},
)
# GPU tests
py_test_run_all_notebooks(
size = "large",
include = ["tune-xgboost.ipynb"],
data = ["//doc/source/tune/examples:tune_examples"],
exclude = [],
tags = [
"exclusive",
"gpu",
"team:ml",
],
env = {"RAY_TRAIN_V2_ENABLED": "1"},
)
filegroup(
name = "tune_examples_ci_configs",
srcs = glob([
"**/ci/aws.yaml",
"**/ci/gce.yaml",
]),
visibility = ["//doc:__pkg__"],
)
doc/source/tune/examples/ax_example.ipynb
+928
View File
@@ -0,0 +1,928 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "47de02e1",
"metadata": {},
"source": [
"# Running Tune experiments with AxSearch\n",
"\n",
"<a id=\"try-anyscale-quickstart-ray-tune-ax_example\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=ray-tune-ax_example\">\n",
" <img src=\"../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
"</a>\n",
"<br></br>\n",
"\n",
"In this tutorial we introduce Ax, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with Ax and, as a result, allow you to seamlessly scale up a Ax optimization process - without sacrificing performance.\n",
"\n",
"Ax is a platform for optimizing any kind of experiment, including machine learning experiments, A/B tests, and simulations. Ax can optimize discrete configurations (e.g., variants of an A/B test) using multi-armed bandit optimization, and continuous/ordered configurations (e.g. float/int parameters) using Bayesian optimization. Results of A/B tests and simulations with reinforcement learning agents often exhibit high amounts of noise. Ax supports state-of-the-art algorithms which work better than traditional Bayesian optimization in high-noise settings. Ax also supports multi-objective and constrained optimization which are common to real-world problems (e.g. improving load time without increasing data use). Ax belongs to the domain of \"derivative-free\" and \"black-box\" optimization.\n",
"\n",
"In this example we minimize a simple objective to briefly demonstrate the usage of AxSearch with Ray Tune via `AxSearch`. It's useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume `ax-platform==0.2.4` library is installed withe python version >= 3.7. To learn more, please refer to the [Ax website](https://ax.dev/)."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "297d8b18",
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: ax-platform==0.2.4 in ~/.pyenv/versions/3.7.7/lib/python3.7/site-packages (0.2.4)\n",
"Requirement already satisfied: botorch==0.6.2 in ~/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (0.6.2)\n",
"Requirement already satisfied: jinja2 in ~/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (3.0.3)\n",
"Requirement already satisfied: pandas in ~/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (1.3.5)\n",
"Requirement already satisfied: scipy in ~/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (1.4.1)\n",
"Requirement already satisfied: plotly in ~/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (5.6.0)\n",
"Requirement already satisfied: scikit-learn in ~/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (0.24.2)\n",
"Requirement already satisfied: typeguard in ~/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from ax-platform==0.2.4) (2.13.3)\n",
"Requirement already satisfied: gpytorch>=1.6 in ~/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from botorch==0.6.2->ax-platform==0.2.4) (1.6.0)\n",
"Requirement already satisfied: torch>=1.9 in ~/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from botorch==0.6.2->ax-platform==0.2.4) (1.9.0)\n",
"Requirement already satisfied: multipledispatch in ~/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from botorch==0.6.2->ax-platform==0.2.4) (0.6.0)\n",
"Requirement already satisfied: MarkupSafe>=2.0 in ~/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from jinja2->ax-platform==0.2.4) (2.0.1)\n",
"Requirement already satisfied: pytz>=2017.3 in ~/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from pandas->ax-platform==0.2.4) (2022.1)\n",
"Requirement already satisfied: numpy>=1.17.3 in ~/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from pandas->ax-platform==0.2.4) (1.21.6)\n",
"Requirement already satisfied: python-dateutil>=2.7.3 in ~/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from pandas->ax-platform==0.2.4) (2.8.2)\n",
"Requirement already satisfied: tenacity>=6.2.0 in ~/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from plotly->ax-platform==0.2.4) (8.0.1)\n",
"Requirement already satisfied: six in ~/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from plotly->ax-platform==0.2.4) (1.16.0)\n",
"Requirement already satisfied: joblib>=0.11 in ~/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-learn->ax-platform==0.2.4) (1.1.0)\n",
"Requirement already satisfied: threadpoolctl>=2.0.0 in ~/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-learn->ax-platform==0.2.4) (3.0.0)\n",
"Requirement already satisfied: typing-extensions in ~/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from torch>=1.9->botorch==0.6.2->ax-platform==0.2.4) (4.1.1)\n",
"\u001b[33mWARNING: There was an error checking the latest version of pip.\u001b[0m\u001b[33m\n",
"\u001b[0m"
]
}
],
"source": [
"# !pip install ray[tune]\n",
"!pip install ax-platform==0.2.4"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "59b1e0d1",
"metadata": {},
"source": [
"Click below to see all the imports we need for this example."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "cbae6dbe",
"metadata": {
"tags": [
"hide-input"
]
},
"outputs": [],
"source": [
"import numpy as np\n",
"import time\n",
"\n",
"import ray\n",
"from ray import tune\n",
"from ray.tune.search.ax import AxSearch"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "7b2b6af7",
"metadata": {},
"source": [
"Let's start by defining a classic benchmark for global optimization.\n",
"The form here is explicit for demonstration, yet it is typically a black-box.\n",
"We artificially sleep for a bit (`0.02` seconds) to simulate a long-running ML experiment.\n",
"This setup assumes that we're running multiple `step`s of an experiment and try to tune 6-dimensions of the `x` hyperparameter."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "0f7fbe0f",
"metadata": {},
"outputs": [],
"source": [
"def landscape(x):\n",
" \"\"\"\n",
" Hartmann 6D function containing 6 local minima.\n",
" It is a classic benchmark for developing global optimization algorithms.\n",
" \"\"\"\n",
" alpha = np.array([1.0, 1.2, 3.0, 3.2])\n",
" A = np.array(\n",
" [\n",
" [10, 3, 17, 3.5, 1.7, 8],\n",
" [0.05, 10, 17, 0.1, 8, 14],\n",
" [3, 3.5, 1.7, 10, 17, 8],\n",
" [17, 8, 0.05, 10, 0.1, 14],\n",
" ]\n",
" )\n",
" P = 10 ** (-4) * np.array(\n",
" [\n",
" [1312, 1696, 5569, 124, 8283, 5886],\n",
" [2329, 4135, 8307, 3736, 1004, 9991],\n",
" [2348, 1451, 3522, 2883, 3047, 6650],\n",
" [4047, 8828, 8732, 5743, 1091, 381],\n",
" ]\n",
" )\n",
" y = 0.0\n",
" for j, alpha_j in enumerate(alpha):\n",
" t = 0\n",
" for k in range(6):\n",
" t += A[j, k] * ((x[k] - P[j, k]) ** 2)\n",
" y -= alpha_j * np.exp(-t)\n",
" return y"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "0b1ae9df",
"metadata": {},
"source": [
"Next, our `objective` function takes a Tune `config`, evaluates the `landscape` of our experiment in a training loop,\n",
"and uses `tune.report` to report the `landscape` back to Tune."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "8c3f252e",
"metadata": {},
"outputs": [],
"source": [
"def objective(config):\n",
" for i in range(config[\"iterations\"]):\n",
" x = np.array([config.get(\"x{}\".format(i + 1)) for i in range(6)])\n",
" tune.report(\n",
" {\"timesteps_total\": i, \"landscape\": landscape(x), \"l2norm\": np.sqrt((x ** 2).sum())}\n",
" )\n",
" time.sleep(0.02)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "d9982d95",
"metadata": {},
"source": [
"Next we define a search space. The critical assumption is that the optimal hyperparameters live within this space. Yet, if the space is very large, then those hyperparameters may be difficult to find in a short amount of time."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "30f75f5a",
"metadata": {},
"outputs": [],
"source": [
"search_space = {\n",
" \"iterations\":100,\n",
" \"x1\": tune.uniform(0.0, 1.0),\n",
" \"x2\": tune.uniform(0.0, 1.0),\n",
" \"x3\": tune.uniform(0.0, 1.0),\n",
" \"x4\": tune.uniform(0.0, 1.0),\n",
" \"x5\": tune.uniform(0.0, 1.0),\n",
" \"x6\": tune.uniform(0.0, 1.0)\n",
"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "106d8578",
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [],
"source": [
"ray.init(configure_logging=False)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "932f74e6",
"metadata": {},
"source": [
"Now we define the search algorithm from `AxSearch`. If you want to constrain your parameters or even the space of outcomes, that can be easily done by passing the argumentsas below."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "34dd5c95",
"metadata": {},
"outputs": [],
"source": [
"algo = AxSearch(\n",
" parameter_constraints=[\"x1 + x2 <= 2.0\"],\n",
" outcome_constraints=[\"l2norm <= 1.25\"],\n",
")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "f6d18a99",
"metadata": {},
"source": [
"We also use `ConcurrencyLimiter` to constrain to 4 concurrent trials. "
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "dcd905ef",
"metadata": {},
"outputs": [],
"source": [
"algo = tune.search.ConcurrencyLimiter(algo, max_concurrent=4)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "10fd5427",
"metadata": {},
"source": [
"The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to `1000` samples.\n",
"You can decrease this if it takes too long on your machine, or you can set a time limit easily through `stop` argument in the `RunConfig()` as we will show here."
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "c53349a5",
"metadata": {},
"outputs": [],
"source": [
"num_samples = 100\n",
"stop_timesteps = 200"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "6c661045",
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [],
"source": [
"# Reducing samples for smoke tests\n",
"num_samples = 10"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "91076c5a",
"metadata": {},
"source": [
"Finally, we run the experiment to find the global minimum of the provided landscape (which contains 5 false minima). The argument to metric, `\"landscape\"`, is provided via the `objective` function's `tune.report`. The experiment `\"min\"`imizes the \"mean_loss\" of the `landscape` by searching within `search_space` via `algo`, `num_samples` times or when `\"timesteps_total\": stop_timesteps`. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute `tuner.fit()`."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "2f519d63",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 07-22 15:04:18] ax.service.ax_client: Starting optimization with verbose logging. To disable logging, set the `verbose_logging` argument to `False`. Note that float values in the logs are rounded to 6 decimal points.\n",
"[INFO 07-22 15:04:18] ax.service.utils.instantiation: Created search space: SearchSpace(parameters=[FixedParameter(name='iterations', parameter_type=INT, value=100), RangeParameter(name='x1', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x2', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x3', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x4', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x5', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x6', parameter_type=FLOAT, range=[0.0, 1.0])], parameter_constraints=[ParameterConstraint(1.0*x1 + 1.0*x2 <= 2.0)]).\n",
"[INFO 07-22 15:04:18] ax.modelbridge.dispatch_utils: Using Bayesian optimization since there are more ordered parameters than there are categories for the unordered categorical parameters.\n",
"[INFO 07-22 15:04:18] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 12 trials, GPEI for subsequent trials]). Iterations after 12 will take longer to generate due to model-fitting.\n",
"Detected sequential enforcement. Be sure to use a ConcurrencyLimiter.\n"
]
},
{
"data": {
"text/html": [
"== Status ==<br>Current time: 2022-07-22 15:04:35 (running for 00:00:16.56)<br>Memory usage on this node: 9.9/16.0 GiB<br>Using FIFO scheduling algorithm.<br>Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.13 GiB heap, 0.0/2.0 GiB objects<br>Current best trial: 34b7abda with landscape=-1.6624439263544026 and parameters={'iterations': 100, 'x1': 0.26526361983269453, 'x2': 0.9248840995132923, 'x3': 0.15171580761671066, 'x4': 0.43602637108415365, 'x5': 0.8573104059323668, 'x6': 0.08981018699705601}<br>Result logdir: ~/ray_results/ax<br>Number of trials: 10/10 (10 TERMINATED)<br><table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>status </th><th>loc </th><th style=\"text-align: right;\"> iterations</th><th style=\"text-align: right;\"> x1</th><th style=\"text-align: right;\"> x2</th><th style=\"text-align: right;\"> x3</th><th style=\"text-align: right;\"> x4</th><th style=\"text-align: right;\"> x5</th><th style=\"text-align: right;\"> x6</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> ts</th><th style=\"text-align: right;\"> landscape</th><th style=\"text-align: right;\"> l2norm</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>objective_2dfbe86a</td><td>TERMINATED</td><td>127.0.0.1:44721</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">0.0558336</td><td style=\"text-align: right;\">0.0896192</td><td style=\"text-align: right;\">0.958956 </td><td style=\"text-align: right;\">0.234474 </td><td style=\"text-align: right;\">0.174516 </td><td style=\"text-align: right;\">0.970311 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 2.57372</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">-0.805233 </td><td style=\"text-align: right;\"> 1.39917</td></tr>\n",
"<tr><td>objective_2fa776c0</td><td>TERMINATED</td><td>127.0.0.1:44726</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">0.744772 </td><td style=\"text-align: right;\">0.754537 </td><td style=\"text-align: right;\">0.0950125</td><td style=\"text-align: right;\">0.273877 </td><td style=\"text-align: right;\">0.0966829</td><td style=\"text-align: right;\">0.368943 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 2.6361 </td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">-0.11286 </td><td style=\"text-align: right;\"> 1.16341</td></tr>\n",
"<tr><td>objective_2fabaa1a</td><td>TERMINATED</td><td>127.0.0.1:44727</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">0.405704 </td><td style=\"text-align: right;\">0.374626 </td><td style=\"text-align: right;\">0.935628 </td><td style=\"text-align: right;\">0.222185 </td><td style=\"text-align: right;\">0.787212 </td><td style=\"text-align: right;\">0.00812439</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 2.62393</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">-0.11348 </td><td style=\"text-align: right;\"> 1.35995</td></tr>\n",
"<tr><td>objective_2faee7c0</td><td>TERMINATED</td><td>127.0.0.1:44728</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">0.664728 </td><td style=\"text-align: right;\">0.207519 </td><td style=\"text-align: right;\">0.359514 </td><td style=\"text-align: right;\">0.704578 </td><td style=\"text-align: right;\">0.755882 </td><td style=\"text-align: right;\">0.812402 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 2.62069</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">-0.0119837 </td><td style=\"text-align: right;\"> 1.53035</td></tr>\n",
"<tr><td>objective_313d3d3a</td><td>TERMINATED</td><td>127.0.0.1:44747</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">0.0418746</td><td style=\"text-align: right;\">0.992783 </td><td style=\"text-align: right;\">0.906027 </td><td style=\"text-align: right;\">0.594429 </td><td style=\"text-align: right;\">0.825393 </td><td style=\"text-align: right;\">0.646362 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 3.16233</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">-0.00677976</td><td style=\"text-align: right;\"> 1.80573</td></tr>\n",
"<tr><td>objective_32c9acd8</td><td>TERMINATED</td><td>127.0.0.1:44726</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">0.126064 </td><td style=\"text-align: right;\">0.703408 </td><td style=\"text-align: right;\">0.344681 </td><td style=\"text-align: right;\">0.337363 </td><td style=\"text-align: right;\">0.401396 </td><td style=\"text-align: right;\">0.679202 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 3.12119</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">-0.904622 </td><td style=\"text-align: right;\"> 1.16864</td></tr>\n",
"<tr><td>objective_32cf8ca2</td><td>TERMINATED</td><td>127.0.0.1:44756</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">0.0910936</td><td style=\"text-align: right;\">0.304138 </td><td style=\"text-align: right;\">0.869848 </td><td style=\"text-align: right;\">0.405435 </td><td style=\"text-align: right;\">0.567922 </td><td style=\"text-align: right;\">0.228608 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 2.70791</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">-0.146532 </td><td style=\"text-align: right;\"> 1.18178</td></tr>\n",
"<tr><td>objective_32d8dd20</td><td>TERMINATED</td><td>127.0.0.1:44758</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">0.603178 </td><td style=\"text-align: right;\">0.409057 </td><td style=\"text-align: right;\">0.729056 </td><td style=\"text-align: right;\">0.0825984</td><td style=\"text-align: right;\">0.572948 </td><td style=\"text-align: right;\">0.508304 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 2.64158</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">-0.247223 </td><td style=\"text-align: right;\"> 1.28691</td></tr>\n",
"<tr><td>objective_34adf04a</td><td>TERMINATED</td><td>127.0.0.1:44768</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">0.454189 </td><td style=\"text-align: right;\">0.271772 </td><td style=\"text-align: right;\">0.530871 </td><td style=\"text-align: right;\">0.991841 </td><td style=\"text-align: right;\">0.691843 </td><td style=\"text-align: right;\">0.472366 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 2.70327</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">-0.0132915 </td><td style=\"text-align: right;\"> 1.49917</td></tr>\n",
"<tr><td>objective_34b7abda</td><td>TERMINATED</td><td>127.0.0.1:44771</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">0.265264 </td><td style=\"text-align: right;\">0.924884 </td><td style=\"text-align: right;\">0.151716 </td><td style=\"text-align: right;\">0.436026 </td><td style=\"text-align: right;\">0.85731 </td><td style=\"text-align: right;\">0.0898102 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 2.68521</td><td style=\"text-align: right;\"> 99</td><td style=\"text-align: right;\">-1.66244 </td><td style=\"text-align: right;\"> 1.37185</td></tr>\n",
"</tbody>\n",
"</table><br><br>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 07-22 15:04:19] ax.service.ax_client: Generated new trial 0 with parameters {'x1': 0.055834, 'x2': 0.089619, 'x3': 0.958956, 'x4': 0.234474, 'x5': 0.174516, 'x6': 0.970311, 'iterations': 100}.\n",
"[INFO 07-22 15:04:22] ax.service.ax_client: Generated new trial 1 with parameters {'x1': 0.744772, 'x2': 0.754537, 'x3': 0.095012, 'x4': 0.273877, 'x5': 0.096683, 'x6': 0.368943, 'iterations': 100}.\n",
"[INFO 07-22 15:04:22] ax.service.ax_client: Generated new trial 2 with parameters {'x1': 0.405704, 'x2': 0.374626, 'x3': 0.935628, 'x4': 0.222185, 'x5': 0.787212, 'x6': 0.008124, 'iterations': 100}.\n",
"[INFO 07-22 15:04:22] ax.service.ax_client: Generated new trial 3 with parameters {'x1': 0.664728, 'x2': 0.207519, 'x3': 0.359514, 'x4': 0.704578, 'x5': 0.755882, 'x6': 0.812402, 'iterations': 100}.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_2dfbe86a:\n",
" date: 2022-07-22_15-04-22\n",
" done: false\n",
" experiment_id: 4ef8a12ac94a4f4fa483ec18e347967f\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" l2norm: 1.3991721132671366\n",
" landscape: -0.8052333562869153\n",
" node_ip: 127.0.0.1\n",
" pid: 44721\n",
" time_since_restore: 0.00022912025451660156\n",
" time_this_iter_s: 0.00022912025451660156\n",
" time_total_s: 0.00022912025451660156\n",
" timestamp: 1658498662\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 0\n",
" training_iteration: 1\n",
" trial_id: 2dfbe86a\n",
" warmup_time: 0.0035619735717773438\n",
" \n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 07-22 15:04:24] ax.service.ax_client: Completed trial 0 with data: {'landscape': (-0.805233, None), 'l2norm': (1.399172, None)}.\n",
"[INFO 07-22 15:04:24] ax.service.ax_client: Generated new trial 4 with parameters {'x1': 0.041875, 'x2': 0.992783, 'x3': 0.906027, 'x4': 0.594429, 'x5': 0.825393, 'x6': 0.646362, 'iterations': 100}.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_2faee7c0:\n",
" date: 2022-07-22_15-04-24\n",
" done: false\n",
" experiment_id: 3699644e85ac439cb7c1a36ed0976307\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" l2norm: 1.530347488145437\n",
" landscape: -0.011983676977099367\n",
" node_ip: 127.0.0.1\n",
" pid: 44728\n",
" time_since_restore: 0.00022292137145996094\n",
" time_this_iter_s: 0.00022292137145996094\n",
" time_total_s: 0.00022292137145996094\n",
" timestamp: 1658498664\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 0\n",
" training_iteration: 1\n",
" trial_id: 2faee7c0\n",
" warmup_time: 0.0027179718017578125\n",
" \n",
"Result for objective_2fa776c0:\n",
" date: 2022-07-22_15-04-24\n",
" done: false\n",
" experiment_id: c555bfed13ac43e5b8c8e9f6d4b9b2f7\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" l2norm: 1.1634068454629019\n",
" landscape: -0.11285961764770336\n",
" node_ip: 127.0.0.1\n",
" pid: 44726\n",
" time_since_restore: 0.000225067138671875\n",
" time_this_iter_s: 0.000225067138671875\n",
" time_total_s: 0.000225067138671875\n",
" timestamp: 1658498664\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 0\n",
" training_iteration: 1\n",
" trial_id: 2fa776c0\n",
" warmup_time: 0.0026290416717529297\n",
" \n",
"Result for objective_2dfbe86a:\n",
" date: 2022-07-22_15-04-24\n",
" done: true\n",
" experiment_id: 4ef8a12ac94a4f4fa483ec18e347967f\n",
" experiment_tag: 1_iterations=100,x1=0.0558,x2=0.0896,x3=0.9590,x4=0.2345,x5=0.1745,x6=0.9703\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 100\n",
" l2norm: 1.3991721132671366\n",
" landscape: -0.8052333562869153\n",
" node_ip: 127.0.0.1\n",
" pid: 44721\n",
" time_since_restore: 2.573719024658203\n",
" time_this_iter_s: 0.0251619815826416\n",
" time_total_s: 2.573719024658203\n",
" timestamp: 1658498664\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 99\n",
" training_iteration: 100\n",
" trial_id: 2dfbe86a\n",
" warmup_time: 0.0035619735717773438\n",
" \n",
"Result for objective_2fabaa1a:\n",
" date: 2022-07-22_15-04-24\n",
" done: false\n",
" experiment_id: eb9287e4fe5f44c7868dc943e2642312\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" l2norm: 1.3599537840291782\n",
" landscape: -0.11348012497414121\n",
" node_ip: 127.0.0.1\n",
" pid: 44727\n",
" time_since_restore: 0.00022077560424804688\n",
" time_this_iter_s: 0.00022077560424804688\n",
" time_total_s: 0.00022077560424804688\n",
" timestamp: 1658498664\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 0\n",
" training_iteration: 1\n",
" trial_id: 2fabaa1a\n",
" warmup_time: 0.0025510787963867188\n",
" \n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 07-22 15:04:27] ax.service.ax_client: Completed trial 3 with data: {'landscape': (-0.011984, None), 'l2norm': (1.530347, None)}.\n",
"[INFO 07-22 15:04:27] ax.service.ax_client: Generated new trial 5 with parameters {'x1': 0.126064, 'x2': 0.703408, 'x3': 0.344681, 'x4': 0.337363, 'x5': 0.401396, 'x6': 0.679202, 'iterations': 100}.\n",
"[INFO 07-22 15:04:27] ax.service.ax_client: Completed trial 1 with data: {'landscape': (-0.11286, None), 'l2norm': (1.163407, None)}.\n",
"[INFO 07-22 15:04:27] ax.service.ax_client: Generated new trial 6 with parameters {'x1': 0.091094, 'x2': 0.304138, 'x3': 0.869848, 'x4': 0.405435, 'x5': 0.567922, 'x6': 0.228608, 'iterations': 100}.\n",
"[INFO 07-22 15:04:27] ax.service.ax_client: Completed trial 2 with data: {'landscape': (-0.11348, None), 'l2norm': (1.359954, None)}.\n",
"[INFO 07-22 15:04:27] ax.service.ax_client: Generated new trial 7 with parameters {'x1': 0.603178, 'x2': 0.409057, 'x3': 0.729056, 'x4': 0.082598, 'x5': 0.572948, 'x6': 0.508304, 'iterations': 100}.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_313d3d3a:\n",
" date: 2022-07-22_15-04-27\n",
" done: false\n",
" experiment_id: fa7afd557e154fbebe4f54d8eedb3573\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" l2norm: 1.805729990121368\n",
" landscape: -0.006779757704679272\n",
" node_ip: 127.0.0.1\n",
" pid: 44747\n",
" time_since_restore: 0.00021076202392578125\n",
" time_this_iter_s: 0.00021076202392578125\n",
" time_total_s: 0.00021076202392578125\n",
" timestamp: 1658498667\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 0\n",
" training_iteration: 1\n",
" trial_id: 313d3d3a\n",
" warmup_time: 0.0029790401458740234\n",
" \n",
"Result for objective_2faee7c0:\n",
" date: 2022-07-22_15-04-27\n",
" done: true\n",
" experiment_id: 3699644e85ac439cb7c1a36ed0976307\n",
" experiment_tag: 4_iterations=100,x1=0.6647,x2=0.2075,x3=0.3595,x4=0.7046,x5=0.7559,x6=0.8124\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 100\n",
" l2norm: 1.530347488145437\n",
" landscape: -0.011983676977099367\n",
" node_ip: 127.0.0.1\n",
" pid: 44728\n",
" time_since_restore: 2.6206929683685303\n",
" time_this_iter_s: 0.027359962463378906\n",
" time_total_s: 2.6206929683685303\n",
" timestamp: 1658498667\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 99\n",
" training_iteration: 100\n",
" trial_id: 2faee7c0\n",
" warmup_time: 0.0027179718017578125\n",
" \n",
"Result for objective_2fa776c0:\n",
" date: 2022-07-22_15-04-27\n",
" done: true\n",
" experiment_id: c555bfed13ac43e5b8c8e9f6d4b9b2f7\n",
" experiment_tag: 2_iterations=100,x1=0.7448,x2=0.7545,x3=0.0950,x4=0.2739,x5=0.0967,x6=0.3689\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 100\n",
" l2norm: 1.1634068454629019\n",
" landscape: -0.11285961764770336\n",
" node_ip: 127.0.0.1\n",
" pid: 44726\n",
" time_since_restore: 2.6361019611358643\n",
" time_this_iter_s: 0.0264589786529541\n",
" time_total_s: 2.6361019611358643\n",
" timestamp: 1658498667\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 99\n",
" training_iteration: 100\n",
" trial_id: 2fa776c0\n",
" warmup_time: 0.0026290416717529297\n",
" \n",
"Result for objective_32c9acd8:\n",
" date: 2022-07-22_15-04-27\n",
" done: false\n",
" experiment_id: c555bfed13ac43e5b8c8e9f6d4b9b2f7\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" l2norm: 1.1686440476629836\n",
" landscape: -0.9046216637367911\n",
" node_ip: 127.0.0.1\n",
" pid: 44726\n",
" time_since_restore: 0.00020194053649902344\n",
" time_this_iter_s: 0.00020194053649902344\n",
" time_total_s: 0.00020194053649902344\n",
" timestamp: 1658498667\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 0\n",
" training_iteration: 1\n",
" trial_id: 32c9acd8\n",
" warmup_time: 0.0026290416717529297\n",
" \n",
"Result for objective_2fabaa1a:\n",
" date: 2022-07-22_15-04-27\n",
" done: true\n",
" experiment_id: eb9287e4fe5f44c7868dc943e2642312\n",
" experiment_tag: 3_iterations=100,x1=0.4057,x2=0.3746,x3=0.9356,x4=0.2222,x5=0.7872,x6=0.0081\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 100\n",
" l2norm: 1.3599537840291782\n",
" landscape: -0.11348012497414121\n",
" node_ip: 127.0.0.1\n",
" pid: 44727\n",
" time_since_restore: 2.623929977416992\n",
" time_this_iter_s: 0.032716989517211914\n",
" time_total_s: 2.623929977416992\n",
" timestamp: 1658498667\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 99\n",
" training_iteration: 100\n",
" trial_id: 2fabaa1a\n",
" warmup_time: 0.0025510787963867188\n",
" \n",
"Result for objective_32d8dd20:\n",
" date: 2022-07-22_15-04-30\n",
" done: false\n",
" experiment_id: 171527593b0f4cbf941c0a03faaf0953\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" l2norm: 1.2869105702896437\n",
" landscape: -0.24722262157458608\n",
" node_ip: 127.0.0.1\n",
" pid: 44758\n",
" time_since_restore: 0.00021886825561523438\n",
" time_this_iter_s: 0.00021886825561523438\n",
" time_total_s: 0.00021886825561523438\n",
" timestamp: 1658498670\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 0\n",
" training_iteration: 1\n",
" trial_id: 32d8dd20\n",
" warmup_time: 0.002732992172241211\n",
" \n",
"Result for objective_32cf8ca2:\n",
" date: 2022-07-22_15-04-29\n",
" done: false\n",
" experiment_id: 37610500f6df493aae4e7e46bb21bf09\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" l2norm: 1.1817810425508524\n",
" landscape: -0.14653248187442922\n",
" node_ip: 127.0.0.1\n",
" pid: 44756\n",
" time_since_restore: 0.00025081634521484375\n",
" time_this_iter_s: 0.00025081634521484375\n",
" time_total_s: 0.00025081634521484375\n",
" timestamp: 1658498669\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 0\n",
" training_iteration: 1\n",
" trial_id: 32cf8ca2\n",
" warmup_time: 0.0032138824462890625\n",
" \n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 07-22 15:04:30] ax.service.ax_client: Completed trial 4 with data: {'landscape': (-0.00678, None), 'l2norm': (1.80573, None)}.\n",
"[INFO 07-22 15:04:30] ax.service.ax_client: Generated new trial 8 with parameters {'x1': 0.454189, 'x2': 0.271772, 'x3': 0.530871, 'x4': 0.991841, 'x5': 0.691843, 'x6': 0.472366, 'iterations': 100}.\n",
"[INFO 07-22 15:04:30] ax.service.ax_client: Completed trial 5 with data: {'landscape': (-0.904622, None), 'l2norm': (1.168644, None)}.\n",
"[INFO 07-22 15:04:30] ax.service.ax_client: Generated new trial 9 with parameters {'x1': 0.265264, 'x2': 0.924884, 'x3': 0.151716, 'x4': 0.436026, 'x5': 0.85731, 'x6': 0.08981, 'iterations': 100}.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_313d3d3a:\n",
" date: 2022-07-22_15-04-30\n",
" done: true\n",
" experiment_id: fa7afd557e154fbebe4f54d8eedb3573\n",
" experiment_tag: 5_iterations=100,x1=0.0419,x2=0.9928,x3=0.9060,x4=0.5944,x5=0.8254,x6=0.6464\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 100\n",
" l2norm: 1.805729990121368\n",
" landscape: -0.006779757704679272\n",
" node_ip: 127.0.0.1\n",
" pid: 44747\n",
" time_since_restore: 3.1623308658599854\n",
" time_this_iter_s: 0.02911996841430664\n",
" time_total_s: 3.1623308658599854\n",
" timestamp: 1658498670\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 99\n",
" training_iteration: 100\n",
" trial_id: 313d3d3a\n",
" warmup_time: 0.0029790401458740234\n",
" \n",
"Result for objective_32c9acd8:\n",
" date: 2022-07-22_15-04-30\n",
" done: true\n",
" experiment_id: c555bfed13ac43e5b8c8e9f6d4b9b2f7\n",
" experiment_tag: 6_iterations=100,x1=0.1261,x2=0.7034,x3=0.3447,x4=0.3374,x5=0.4014,x6=0.6792\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 100\n",
" l2norm: 1.1686440476629836\n",
" landscape: -0.9046216637367911\n",
" node_ip: 127.0.0.1\n",
" pid: 44726\n",
" time_since_restore: 3.1211891174316406\n",
" time_this_iter_s: 0.02954697608947754\n",
" time_total_s: 3.1211891174316406\n",
" timestamp: 1658498670\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 99\n",
" training_iteration: 100\n",
" trial_id: 32c9acd8\n",
" warmup_time: 0.0026290416717529297\n",
" \n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 07-22 15:04:32] ax.service.ax_client: Completed trial 7 with data: {'landscape': (-0.247223, None), 'l2norm': (1.286911, None)}.\n",
"[INFO 07-22 15:04:32] ax.service.ax_client: Completed trial 6 with data: {'landscape': (-0.146532, None), 'l2norm': (1.181781, None)}.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_32d8dd20:\n",
" date: 2022-07-22_15-04-32\n",
" done: true\n",
" experiment_id: 171527593b0f4cbf941c0a03faaf0953\n",
" experiment_tag: 8_iterations=100,x1=0.6032,x2=0.4091,x3=0.7291,x4=0.0826,x5=0.5729,x6=0.5083\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 100\n",
" l2norm: 1.2869105702896437\n",
" landscape: -0.24722262157458608\n",
" node_ip: 127.0.0.1\n",
" pid: 44758\n",
" time_since_restore: 2.6415798664093018\n",
" time_this_iter_s: 0.026781082153320312\n",
" time_total_s: 2.6415798664093018\n",
" timestamp: 1658498672\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 99\n",
" training_iteration: 100\n",
" trial_id: 32d8dd20\n",
" warmup_time: 0.002732992172241211\n",
" \n",
"Result for objective_32cf8ca2:\n",
" date: 2022-07-22_15-04-32\n",
" done: true\n",
" experiment_id: 37610500f6df493aae4e7e46bb21bf09\n",
" experiment_tag: 7_iterations=100,x1=0.0911,x2=0.3041,x3=0.8698,x4=0.4054,x5=0.5679,x6=0.2286\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 100\n",
" l2norm: 1.1817810425508524\n",
" landscape: -0.14653248187442922\n",
" node_ip: 127.0.0.1\n",
" pid: 44756\n",
" time_since_restore: 2.707913875579834\n",
" time_this_iter_s: 0.027456998825073242\n",
" time_total_s: 2.707913875579834\n",
" timestamp: 1658498672\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 99\n",
" training_iteration: 100\n",
" trial_id: 32cf8ca2\n",
" warmup_time: 0.0032138824462890625\n",
" \n",
"Result for objective_34adf04a:\n",
" date: 2022-07-22_15-04-33\n",
" done: false\n",
" experiment_id: 4f65c5b68f5c49d98fda388e37c83deb\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" l2norm: 1.4991655675380078\n",
" landscape: -0.01329150870283869\n",
" node_ip: 127.0.0.1\n",
" pid: 44768\n",
" time_since_restore: 0.00021600723266601562\n",
" time_this_iter_s: 0.00021600723266601562\n",
" time_total_s: 0.00021600723266601562\n",
" timestamp: 1658498673\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 0\n",
" training_iteration: 1\n",
" trial_id: 34adf04a\n",
" warmup_time: 0.0027239322662353516\n",
" \n",
"Result for objective_34b7abda:\n",
" date: 2022-07-22_15-04-33\n",
" done: false\n",
" experiment_id: f135a2c40f5644ba9d2ae096a9dd10e0\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" l2norm: 1.3718451333547932\n",
" landscape: -1.6624439263544026\n",
" node_ip: 127.0.0.1\n",
" pid: 44771\n",
" time_since_restore: 0.0002338886260986328\n",
" time_this_iter_s: 0.0002338886260986328\n",
" time_total_s: 0.0002338886260986328\n",
" timestamp: 1658498673\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 0\n",
" training_iteration: 1\n",
" trial_id: 34b7abda\n",
" warmup_time: 0.002721071243286133\n",
" \n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"[INFO 07-22 15:04:35] ax.service.ax_client: Completed trial 8 with data: {'landscape': (-0.013292, None), 'l2norm': (1.499166, None)}.\n",
"[INFO 07-22 15:04:35] ax.service.ax_client: Completed trial 9 with data: {'landscape': (-1.662444, None), 'l2norm': (1.371845, None)}.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_34adf04a:\n",
" date: 2022-07-22_15-04-35\n",
" done: true\n",
" experiment_id: 4f65c5b68f5c49d98fda388e37c83deb\n",
" experiment_tag: 9_iterations=100,x1=0.4542,x2=0.2718,x3=0.5309,x4=0.9918,x5=0.6918,x6=0.4724\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 100\n",
" l2norm: 1.4991655675380078\n",
" landscape: -0.01329150870283869\n",
" node_ip: 127.0.0.1\n",
" pid: 44768\n",
" time_since_restore: 2.7032668590545654\n",
" time_this_iter_s: 0.029300928115844727\n",
" time_total_s: 2.7032668590545654\n",
" timestamp: 1658498675\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 99\n",
" training_iteration: 100\n",
" trial_id: 34adf04a\n",
" warmup_time: 0.0027239322662353516\n",
" \n",
"Result for objective_34b7abda:\n",
" date: 2022-07-22_15-04-35\n",
" done: true\n",
" experiment_id: f135a2c40f5644ba9d2ae096a9dd10e0\n",
" experiment_tag: 10_iterations=100,x1=0.2653,x2=0.9249,x3=0.1517,x4=0.4360,x5=0.8573,x6=0.0898\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 100\n",
" l2norm: 1.3718451333547932\n",
" landscape: -1.6624439263544026\n",
" node_ip: 127.0.0.1\n",
" pid: 44771\n",
" time_since_restore: 2.6852078437805176\n",
" time_this_iter_s: 0.029579877853393555\n",
" time_total_s: 2.6852078437805176\n",
" timestamp: 1658498675\n",
" timesteps_since_restore: 0\n",
" timesteps_total: 99\n",
" training_iteration: 100\n",
" trial_id: 34b7abda\n",
" warmup_time: 0.002721071243286133\n",
" \n"
]
}
],
"source": [
"tuner = tune.Tuner(\n",
" objective,\n",
" tune_config=tune.TuneConfig(\n",
" metric=\"landscape\",\n",
" mode=\"min\",\n",
" search_alg=algo,\n",
" num_samples=num_samples,\n",
" ),\n",
" run_config=tune.RunConfig(\n",
" name=\"ax\",\n",
" stop={\"timesteps_total\": stop_timesteps}\n",
" ),\n",
" param_space=search_space,\n",
")\n",
"results = tuner.fit()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "860b53b0",
"metadata": {},
"source": [
"And now we have the hyperparameters found to minimize the mean loss."
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "12906421",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best hyperparameters found were: {'iterations': 100, 'x1': 0.26526361983269453, 'x2': 0.9248840995132923, 'x3': 0.15171580761671066, 'x4': 0.43602637108415365, 'x5': 0.8573104059323668, 'x6': 0.08981018699705601}\n"
]
}
],
"source": [
"print(\"Best hyperparameters found were: \", results.get_best_result().config)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "68872424",
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [],
"source": [
"ray.shutdown()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.13"
},
"orphan": true
},
"nbformat": 4,
"nbformat_minor": 5
}
doc/source/tune/examples/batch_tuning.ipynb
+1216
View File
File diff suppressed because it is too large Load Diff
doc/source/tune/examples/bayesopt_example.ipynb
+1158
View File
File diff suppressed because it is too large Load Diff
doc/source/tune/examples/bohb_example.ipynb
+1870
View File
File diff suppressed because it is too large Load Diff
doc/source/tune/examples/hyperopt_example.ipynb
+689
View File
@@ -0,0 +1,689 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "58fc50bc",
"metadata": {},
"source": [
"# Running Tune experiments with HyperOpt\n",
"\n",
"<a id=\"try-anyscale-quickstart-ray-tune-hyperopt_example\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=ray-tune-hyperopt_example\">\n",
" <img src=\"../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
"</a>\n",
"<br></br>\n",
"\n",
"In this tutorial we introduce HyperOpt, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with HyperOpt and, as a result, allow you to seamlessly scale up a Hyperopt optimization process - without sacrificing performance.\n",
"\n",
"HyperOpt provides gradient/derivative-free optimization able to handle noise over the objective landscape, including evolutionary, bandit, and Bayesian optimization algorithms. HyperOpt internally supports search spaces which are continuous, discrete or a mixture of thereof. It also provides a library of functions on which to test the optimization algorithms and compare with other benchmarks.\n",
"\n",
"In this example we minimize a simple objective to briefly demonstrate the usage of HyperOpt with Ray Tune via `HyperOptSearch`. It's useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume `hyperopt==0.2.5` library is installed. To learn more, please refer to [HyperOpt website](http://hyperopt.github.io/hyperopt).\n",
"\n",
"We include a important example on conditional search spaces (stringing together relationships among hyperparameters)."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "e4586d28",
"metadata": {},
"source": [
"Background information:\n",
"- [HyperOpt website](http://hyperopt.github.io/hyperopt)\n",
"\n",
"Necessary requirements:\n",
"- `pip install \"ray[tune]\" hyperopt==0.2.5`"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "6567f2dc",
"metadata": {
"tags": [
"hide-cell"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: hyperopt==0.2.5 in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (0.2.5)\n",
"Requirement already satisfied: numpy in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (from hyperopt==0.2.5) (2.2.3)\n",
"Requirement already satisfied: scipy in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (from hyperopt==0.2.5) (1.15.2)\n",
"Requirement already satisfied: six in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (from hyperopt==0.2.5) (1.17.0)\n",
"Requirement already satisfied: networkx>=2.2 in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (from hyperopt==0.2.5) (3.4.2)\n",
"Requirement already satisfied: future in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (from hyperopt==0.2.5) (1.0.0)\n",
"Requirement already satisfied: tqdm in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (from hyperopt==0.2.5) (4.67.1)\n",
"Requirement already satisfied: cloudpickle in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (from hyperopt==0.2.5) (3.1.1)\n"
]
}
],
"source": [
"# install in a hidden cell\n",
"# !pip install \"ray[tune]\"\n",
"!pip install hyperopt==0.2.5"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "b8e9e0cd",
"metadata": {},
"source": [
"Click below to see all the imports we need for this example."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "6592315e",
"metadata": {
"tags": [
"hide-input",
"hide-output"
]
},
"outputs": [],
"source": [
"import time\n",
"\n",
"import ray\n",
"from ray import tune\n",
"from ray.tune.search import ConcurrencyLimiter\n",
"from ray.tune.search.hyperopt import HyperOptSearch\n",
"from hyperopt import hp"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "d4b6d1d5",
"metadata": {},
"source": [
"Let's start by defining a simple evaluation function.\n",
"We artificially sleep for a bit (`0.1` seconds) to simulate a long-running ML experiment.\n",
"This setup assumes that we're running multiple `step`s of an experiment and try to tune two hyperparameters,\n",
"namely `width` and `height`."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "12d4efc8",
"metadata": {},
"outputs": [],
"source": [
"def evaluate(step, width, height):\n",
" time.sleep(0.1)\n",
" return (0.1 + width * step / 100) ** (-1) + height * 0.1"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "4f4f5aa2",
"metadata": {},
"source": [
"Next, our ``objective`` function takes a Tune ``config``, evaluates the `score` of your experiment in a training loop,\n",
"and uses `tune.report` to report the `score` back to Tune."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "c9818009",
"metadata": {},
"outputs": [],
"source": [
"def objective(config):\n",
" for step in range(config[\"steps\"]):\n",
" score = evaluate(step, config[\"width\"], config[\"height\"])\n",
" tune.report({\"iterations\": step, \"mean_loss\": score})"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "33eddcb9",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "59f8090dd37c473cb24b97b6b28d109b",
"version_major": 2,
"version_minor": 0
},
"text/html": [
"<div class=\"lm-Widget p-Widget lm-Panel p-Panel jp-Cell-outputWrapper\">\n",
" <div style=\"margin-left: 50px;display: flex;flex-direction: row;align-items: center\">\n",
" <div class=\"jp-RenderedHTMLCommon\" style=\"display: flex; flex-direction: row;\">\n",
" <svg viewBox=\"0 0 567 224\" fill=\"none\" xmlns=\"http://www.w3.org/2000/svg\" style=\"height: 3em;\">\n",
" <g clip-path=\"url(#clip0_4338_178347)\">\n",
" <path d=\"M341.29 165.561H355.29L330.13 129.051C345.63 123.991 354.21 112.051 354.21 94.2307C354.21 71.3707 338.72 58.1807 311.88 58.1807H271V165.561H283.27V131.661H311.8C314.25 131.661 316.71 131.501 319.01 131.351L341.25 165.561H341.29ZM283.29 119.851V70.0007H311.82C331.3 70.0007 342.34 78.2907 342.34 94.5507C342.34 111.271 331.34 119.861 311.82 119.861L283.29 119.851ZM451.4 138.411L463.4 165.561H476.74L428.74 58.1807H416L367.83 165.561H380.83L392.83 138.411H451.4ZM446.19 126.601H398L422 72.1407L446.24 126.601H446.19ZM526.11 128.741L566.91 58.1807H554.35L519.99 114.181L485.17 58.1807H472.44L514.01 129.181V165.541H526.13V128.741H526.11Z\" fill=\"var(--jp-ui-font-color0)\"/>\n",
" <path d=\"M82.35 104.44C84.0187 97.8827 87.8248 92.0678 93.1671 87.9146C98.5094 83.7614 105.083 81.5067 111.85 81.5067C118.617 81.5067 125.191 83.7614 130.533 87.9146C135.875 92.0678 139.681 97.8827 141.35 104.44H163.75C164.476 101.562 165.622 98.8057 167.15 96.2605L127.45 56.5605C121.071 60.3522 113.526 61.6823 106.235 60.3005C98.9443 58.9187 92.4094 54.9203 87.8602 49.0574C83.3109 43.1946 81.0609 35.8714 81.5332 28.4656C82.0056 21.0599 85.1679 14.0819 90.4252 8.8446C95.6824 3.60726 102.672 0.471508 110.08 0.0272655C117.487 -0.416977 124.802 1.86091 130.647 6.4324C136.493 11.0039 140.467 17.5539 141.821 24.8501C143.175 32.1463 141.816 39.6859 138 46.0505L177.69 85.7505C182.31 82.9877 187.58 81.4995 192.962 81.4375C198.345 81.3755 203.648 82.742 208.33 85.3976C213.012 88.0532 216.907 91.9029 219.616 96.5544C222.326 101.206 223.753 106.492 223.753 111.875C223.753 117.258 222.326 122.545 219.616 127.197C216.907 131.848 213.012 135.698 208.33 138.353C203.648 141.009 198.345 142.375 192.962 142.313C187.58 142.251 182.31 140.763 177.69 138L138 177.7C141.808 184.071 143.155 191.614 141.79 198.91C140.424 206.205 136.44 212.75 130.585 217.313C124.731 221.875 117.412 224.141 110.004 223.683C102.596 223.226 95.6103 220.077 90.3621 214.828C85.1139 209.58 81.9647 202.595 81.5072 195.187C81.0497 187.779 83.3154 180.459 87.878 174.605C92.4405 168.751 98.9853 164.766 106.281 163.401C113.576 162.035 121.119 163.383 127.49 167.19L167.19 127.49C165.664 124.941 164.518 122.182 163.79 119.3H141.39C139.721 125.858 135.915 131.673 130.573 135.826C125.231 139.98 118.657 142.234 111.89 142.234C105.123 142.234 98.5494 139.98 93.2071 135.826C87.8648 131.673 84.0587 125.858 82.39 119.3H60C58.1878 126.495 53.8086 132.78 47.6863 136.971C41.5641 141.163 34.1211 142.972 26.7579 142.059C19.3947 141.146 12.6191 137.574 7.70605 132.014C2.79302 126.454 0.0813599 119.29 0.0813599 111.87C0.0813599 104.451 2.79302 97.2871 7.70605 91.7272C12.6191 86.1673 19.3947 82.5947 26.7579 81.6817C34.1211 80.7686 41.5641 82.5781 47.6863 86.7696C53.8086 90.9611 58.1878 97.2456 60 104.44H82.35ZM100.86 204.32C103.407 206.868 106.759 208.453 110.345 208.806C113.93 209.159 117.527 208.258 120.522 206.256C123.517 204.254 125.725 201.276 126.771 197.828C127.816 194.38 127.633 190.677 126.253 187.349C124.874 184.021 122.383 181.274 119.205 179.577C116.027 177.88 112.359 177.337 108.826 178.042C105.293 178.746 102.113 180.654 99.8291 183.44C97.5451 186.226 96.2979 189.718 96.3 193.32C96.2985 195.364 96.7006 197.388 97.4831 199.275C98.2656 201.163 99.4132 202.877 100.86 204.32ZM204.32 122.88C206.868 120.333 208.453 116.981 208.806 113.396C209.159 109.811 208.258 106.214 206.256 103.219C204.254 100.223 201.275 98.0151 197.827 96.97C194.38 95.9249 190.676 96.1077 187.348 97.4873C184.02 98.8669 181.274 101.358 179.577 104.536C177.879 107.714 177.337 111.382 178.041 114.915C178.746 118.448 180.653 121.627 183.439 123.911C186.226 126.195 189.717 127.443 193.32 127.44C195.364 127.443 197.388 127.042 199.275 126.259C201.163 125.476 202.878 124.328 204.32 122.88ZM122.88 19.4205C120.333 16.8729 116.981 15.2876 113.395 14.9347C109.81 14.5817 106.213 15.483 103.218 17.4849C100.223 19.4868 98.0146 22.4654 96.9696 25.9131C95.9245 29.3608 96.1073 33.0642 97.4869 36.3922C98.8665 39.7202 101.358 42.4668 104.535 44.1639C107.713 45.861 111.381 46.4036 114.914 45.6992C118.447 44.9949 121.627 43.0871 123.911 40.301C126.195 37.515 127.442 34.0231 127.44 30.4205C127.44 28.3772 127.038 26.3539 126.255 24.4664C125.473 22.5788 124.326 20.8642 122.88 19.4205ZM19.42 100.86C16.8725 103.408 15.2872 106.76 14.9342 110.345C14.5813 113.93 15.4826 117.527 17.4844 120.522C19.4863 123.518 22.4649 125.726 25.9127 126.771C29.3604 127.816 33.0638 127.633 36.3918 126.254C39.7198 124.874 42.4664 122.383 44.1635 119.205C45.8606 116.027 46.4032 112.359 45.6988 108.826C44.9944 105.293 43.0866 102.114 40.3006 99.8296C37.5145 97.5455 34.0227 96.2983 30.42 96.3005C26.2938 96.3018 22.337 97.9421 19.42 100.86ZM100.86 100.86C98.3125 103.408 96.7272 106.76 96.3742 110.345C96.0213 113.93 96.9226 117.527 98.9244 120.522C100.926 123.518 103.905 125.726 107.353 126.771C110.8 127.816 114.504 127.633 117.832 126.254C121.16 124.874 123.906 122.383 125.604 119.205C127.301 116.027 127.843 112.359 127.139 108.826C126.434 105.293 124.527 102.114 121.741 99.8296C118.955 97.5455 115.463 96.2983 111.86 96.3005C109.817 96.299 107.793 96.701 105.905 97.4835C104.018 98.2661 102.303 99.4136 100.86 100.86Z\" fill=\"#00AEEF\"/>\n",
" </g>\n",
" <defs>\n",
" <clipPath id=\"clip0_4338_178347\">\n",
" <rect width=\"566.93\" height=\"223.75\" fill=\"white\"/>\n",
" </clipPath>\n",
" </defs>\n",
" </svg>\n",
"</div>\n",
"\n",
" <table class=\"jp-RenderedHTMLCommon\" style=\"border-collapse: collapse;color: var(--jp-ui-font-color1);font-size: var(--jp-ui-font-size1);\">\n",
" <tr>\n",
" <td style=\"text-align: left\"><b>Python version:</b></td>\n",
" <td style=\"text-align: left\"><b>3.11.11</b></td>\n",
" </tr>\n",
" <tr>\n",
" <td style=\"text-align: left\"><b>Ray version:</b></td>\n",
" <td style=\"text-align: left\"><b>2.42.1</b></td>\n",
" </tr>\n",
" \n",
"</table>\n",
"\n",
" </div>\n",
"</div>\n"
],
"text/plain": [
"RayContext(dashboard_url='', python_version='3.11.11', ray_version='2.42.1', ray_commit='c2e38f7b75be223c0c033986472daada8622d64f')"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ray.init(configure_logging=False)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "5be35d5e",
"metadata": {},
"source": [
"While defining the search algorithm, we may choose to provide an initial set of hyperparameters that we believe are especially promising or informative, and\n",
"pass this information as a helpful starting point for the `HyperOptSearch` object.\n",
"\n",
"We also set the maximum concurrent trials to `4` with a `ConcurrencyLimiter`."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "d4615bed",
"metadata": {
"lines_to_next_cell": 0
},
"outputs": [],
"source": [
"initial_params = [\n",
" {\"width\": 1, \"height\": 2, \"activation\": \"relu\"},\n",
" {\"width\": 4, \"height\": 2, \"activation\": \"tanh\"},\n",
"]\n",
"algo = HyperOptSearch(points_to_evaluate=initial_params)\n",
"algo = ConcurrencyLimiter(algo, max_concurrent=4)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "2a51e7c1",
"metadata": {},
"source": [
"The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to `1000` samples.\n",
"(you can decrease this if it takes too long on your machine)."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "2dbb2be0",
"metadata": {},
"outputs": [],
"source": [
"num_samples = 1000"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "950558ed",
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [],
"source": [
"# If 1000 samples take too long, you can reduce this number.\n",
"# We override this number here for our smoke tests.\n",
"num_samples = 10"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "6e3629cb",
"metadata": {},
"source": [
"Next we define a search space. The critical assumption is that the optimal hyperparameters live within this space. Yet, if the space is very large, then those hyperparameters may be difficult to find in a short amount of time."
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "65189946",
"metadata": {},
"outputs": [],
"source": [
"search_config = {\n",
" \"steps\": 100,\n",
" \"width\": tune.uniform(0, 20),\n",
" \"height\": tune.uniform(-100, 100),\n",
" \"activation\": tune.choice([\"relu\", \"tanh\"])\n",
"}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "1b94c93b",
"metadata": {},
"source": [
"Finally, we run the experiment to `\"min\"`imize the \"mean_loss\" of the `objective` by searching `search_config` via `algo`, `num_samples` times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute `tuner.fit()`."
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "9a99a3a7",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div class=\"tuneStatus\">\n",
" <div style=\"display: flex;flex-direction: row\">\n",
" <div style=\"display: flex;flex-direction: column;\">\n",
" <h3>Tune Status</h3>\n",
" <table>\n",
"<tbody>\n",
"<tr><td>Current time:</td><td>2025-02-18 13:14:59</td></tr>\n",
"<tr><td>Running for: </td><td>00:00:36.03 </td></tr>\n",
"<tr><td>Memory: </td><td>22.1/36.0 GiB </td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
" <div class=\"vDivider\"></div>\n",
" <div class=\"systemInfo\">\n",
" <h3>System Info</h3>\n",
" Using FIFO scheduling algorithm.<br>Logical resource usage: 1.0/12 CPUs, 0/0 GPUs\n",
" </div>\n",
" \n",
" </div>\n",
" <div class=\"hDivider\"></div>\n",
" <div class=\"trialStatus\">\n",
" <h3>Trial Status</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>status </th><th>loc </th><th>activation </th><th style=\"text-align: right;\"> height</th><th style=\"text-align: right;\"> steps</th><th style=\"text-align: right;\"> width</th><th style=\"text-align: right;\"> loss</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> iterations</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>objective_5b05c00a</td><td>TERMINATED</td><td>127.0.0.1:50205</td><td>relu </td><td style=\"text-align: right;\"> 2 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 1 </td><td style=\"text-align: right;\"> 1.11743 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.335 </td><td style=\"text-align: right;\"> 99</td></tr>\n",
"<tr><td>objective_b813f49d</td><td>TERMINATED</td><td>127.0.0.1:50207</td><td>tanh </td><td style=\"text-align: right;\"> 2 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 4 </td><td style=\"text-align: right;\"> 0.446305</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.3299</td><td style=\"text-align: right;\"> 99</td></tr>\n",
"<tr><td>objective_6dadd2bd</td><td>TERMINATED</td><td>127.0.0.1:50212</td><td>tanh </td><td style=\"text-align: right;\">-40.9318</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 6.20615 </td><td style=\"text-align: right;\">-3.93303 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.3213</td><td style=\"text-align: right;\"> 99</td></tr>\n",
"<tr><td>objective_9faffc0f</td><td>TERMINATED</td><td>127.0.0.1:50217</td><td>tanh </td><td style=\"text-align: right;\"> 91.9688</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 9.25147 </td><td style=\"text-align: right;\"> 9.30488 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.353 </td><td style=\"text-align: right;\"> 99</td></tr>\n",
"<tr><td>objective_7834e74c</td><td>TERMINATED</td><td>127.0.0.1:50266</td><td>tanh </td><td style=\"text-align: right;\">-17.9521</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">11.436 </td><td style=\"text-align: right;\">-1.70766 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.3753</td><td style=\"text-align: right;\"> 99</td></tr>\n",
"<tr><td>objective_741253c7</td><td>TERMINATED</td><td>127.0.0.1:50271</td><td>tanh </td><td style=\"text-align: right;\"> 58.1279</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 0.737879</td><td style=\"text-align: right;\"> 7.01689 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.3565</td><td style=\"text-align: right;\"> 99</td></tr>\n",
"<tr><td>objective_39682bcf</td><td>TERMINATED</td><td>127.0.0.1:50272</td><td>tanh </td><td style=\"text-align: right;\">-31.2589</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 4.89265 </td><td style=\"text-align: right;\">-2.92361 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.3225</td><td style=\"text-align: right;\"> 99</td></tr>\n",
"<tr><td>objective_bfc7e150</td><td>TERMINATED</td><td>127.0.0.1:50274</td><td>tanh </td><td style=\"text-align: right;\">-14.7877</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 7.36477 </td><td style=\"text-align: right;\">-1.34347 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.3744</td><td style=\"text-align: right;\"> 99</td></tr>\n",
"<tr><td>objective_e1f1a193</td><td>TERMINATED</td><td>127.0.0.1:50314</td><td>tanh </td><td style=\"text-align: right;\"> 50.9579</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">17.4499 </td><td style=\"text-align: right;\"> 5.15334 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.3675</td><td style=\"text-align: right;\"> 99</td></tr>\n",
"<tr><td>objective_1192dd4e</td><td>TERMINATED</td><td>127.0.0.1:50316</td><td>tanh </td><td style=\"text-align: right;\"> 66.5306</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">16.9549 </td><td style=\"text-align: right;\"> 6.71228 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.3478</td><td style=\"text-align: right;\"> 99</td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
"</div>\n",
"<style>\n",
".tuneStatus {\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".tuneStatus .systemInfo {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus td {\n",
" white-space: nowrap;\n",
"}\n",
".tuneStatus .trialStatus {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus h3 {\n",
" font-weight: bold;\n",
"}\n",
".tuneStatus .hDivider {\n",
" border-bottom-width: var(--jp-border-width);\n",
" border-bottom-color: var(--jp-border-color0);\n",
" border-bottom-style: solid;\n",
"}\n",
".tuneStatus .vDivider {\n",
" border-left-width: var(--jp-border-width);\n",
" border-left-color: var(--jp-border-color0);\n",
" border-left-style: solid;\n",
" margin: 0.5em 1em 0.5em 1em;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"tuner = tune.Tuner(\n",
" objective,\n",
" tune_config=tune.TuneConfig(\n",
" metric=\"mean_loss\",\n",
" mode=\"min\",\n",
" search_alg=algo,\n",
" num_samples=num_samples,\n",
" ),\n",
" param_space=search_config,\n",
")\n",
"results = tuner.fit()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "49be6f01",
"metadata": {},
"source": [
"Here are the hyperparameters found to minimize the mean loss of the defined objective."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "7036798c",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best hyperparameters found were: {'steps': 100, 'width': 6.206149011253133, 'height': -40.93182668460948, 'activation': 'tanh'}\n"
]
}
],
"source": [
"print(\"Best hyperparameters found were: \", results.get_best_result().config)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "504e9d2a",
"metadata": {},
"source": [
"## Conditional search spaces\n",
"\n",
"Sometimes we may want to build a more complicated search space that has conditional dependencies on other hyperparameters. In this case, we pass a nested dictionary to `objective_two`, which has been slightly adjusted from `objective` to deal with the conditional search space."
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "2f7b5449",
"metadata": {},
"outputs": [],
"source": [
"def evaluation_fn(step, width, height, mult=1):\n",
" return (0.1 + width * step / 100) ** (-1) + height * 0.1 * mult"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "4b83b81c",
"metadata": {},
"outputs": [],
"source": [
"def objective_two(config):\n",
" width, height = config[\"width\"], config[\"height\"]\n",
" sub_dict = config[\"activation\"]\n",
" mult = sub_dict.get(\"mult\", 1)\n",
" \n",
" for step in range(config[\"steps\"]):\n",
" intermediate_score = evaluation_fn(step, width, height, mult)\n",
" tune.report({\"iterations\": step, \"mean_loss\": intermediate_score})\n",
" time.sleep(0.1)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "75cea99e",
"metadata": {},
"outputs": [],
"source": [
"conditional_space = {\n",
" \"activation\": hp.choice(\n",
" \"activation\",\n",
" [\n",
" {\"activation\": \"relu\", \"mult\": hp.uniform(\"mult\", 1, 2)},\n",
" {\"activation\": \"tanh\"},\n",
" ],\n",
" ),\n",
" \"width\": hp.uniform(\"width\", 0, 20),\n",
" \"height\": hp.uniform(\"height\", -100, 100),\n",
" \"steps\": 100,\n",
"}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "7df282c1",
"metadata": {},
"source": [
"Now we the define the search algorithm built from `HyperOptSearch` constrained by `ConcurrencyLimiter`. When the hyperparameter search space is conditional, we pass it (`conditional_space`) into `HyperOptSearch`."
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "ea2c71a6",
"metadata": {},
"outputs": [],
"source": [
"algo = HyperOptSearch(space=conditional_space, metric=\"mean_loss\", mode=\"min\")\n",
"algo = ConcurrencyLimiter(algo, max_concurrent=4)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "630f84ab",
"metadata": {},
"source": [
"Now we run the experiment, this time with an empty `config` because we instead provided `space` to the `HyperOptSearch` `search_alg`."
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "14111e9e",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div class=\"tuneStatus\">\n",
" <div style=\"display: flex;flex-direction: row\">\n",
" <div style=\"display: flex;flex-direction: column;\">\n",
" <h3>Tune Status</h3>\n",
" <table>\n",
"<tbody>\n",
"<tr><td>Current time:</td><td>2025-02-18 13:15:34</td></tr>\n",
"<tr><td>Running for: </td><td>00:00:34.71 </td></tr>\n",
"<tr><td>Memory: </td><td>22.9/36.0 GiB </td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
" <div class=\"vDivider\"></div>\n",
" <div class=\"systemInfo\">\n",
" <h3>System Info</h3>\n",
" Using FIFO scheduling algorithm.<br>Logical resource usage: 1.0/12 CPUs, 0/0 GPUs\n",
" </div>\n",
" \n",
" </div>\n",
" <div class=\"hDivider\"></div>\n",
" <div class=\"trialStatus\">\n",
" <h3>Trial Status</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>status </th><th>loc </th><th>activation/activatio\n",
"n </th><th style=\"text-align: right;\"> activation/mult</th><th style=\"text-align: right;\"> height</th><th style=\"text-align: right;\"> steps</th><th style=\"text-align: right;\"> width</th><th style=\"text-align: right;\"> loss</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> iterations</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>objective_two_13de5867</td><td>TERMINATED</td><td>127.0.0.1:50350</td><td>tanh</td><td style=\"text-align: right;\"> </td><td style=\"text-align: right;\">-35.0329 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">13.2254 </td><td style=\"text-align: right;\"> -3.42749 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.2102</td><td style=\"text-align: right;\"> 99</td></tr>\n",
"<tr><td>objective_two_3100f5ee</td><td>TERMINATED</td><td>127.0.0.1:50355</td><td>relu</td><td style=\"text-align: right;\"> 1.44584</td><td style=\"text-align: right;\"> 76.2581 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 0.123165</td><td style=\"text-align: right;\"> 15.5316 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.2683</td><td style=\"text-align: right;\"> 99</td></tr>\n",
"<tr><td>objective_two_6b4044aa</td><td>TERMINATED</td><td>127.0.0.1:50356</td><td>relu</td><td style=\"text-align: right;\"> 1.67475</td><td style=\"text-align: right;\"> 57.9612 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">19.4794 </td><td style=\"text-align: right;\"> 9.75866 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.2724</td><td style=\"text-align: right;\"> 99</td></tr>\n",
"<tr><td>objective_two_4aa1269b</td><td>TERMINATED</td><td>127.0.0.1:50357</td><td>tanh</td><td style=\"text-align: right;\"> </td><td style=\"text-align: right;\"> -9.95686</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">12.9749 </td><td style=\"text-align: right;\"> -0.918437</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.2373</td><td style=\"text-align: right;\"> 99</td></tr>\n",
"<tr><td>objective_two_d8c42c9f</td><td>TERMINATED</td><td>127.0.0.1:50402</td><td>tanh</td><td style=\"text-align: right;\"> </td><td style=\"text-align: right;\">-96.6184 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 8.03869 </td><td style=\"text-align: right;\"> -9.53774 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.2407</td><td style=\"text-align: right;\"> 99</td></tr>\n",
"<tr><td>objective_two_de956d10</td><td>TERMINATED</td><td>127.0.0.1:50404</td><td>relu</td><td style=\"text-align: right;\"> 1.06986</td><td style=\"text-align: right;\"> 9.64996</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 0.672962</td><td style=\"text-align: right;\"> 2.3375 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.2427</td><td style=\"text-align: right;\"> 99</td></tr>\n",
"<tr><td>objective_two_0e2b2751</td><td>TERMINATED</td><td>127.0.0.1:50413</td><td>tanh</td><td style=\"text-align: right;\"> </td><td style=\"text-align: right;\">-82.9292 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">17.4889 </td><td style=\"text-align: right;\"> -8.2355 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.293 </td><td style=\"text-align: right;\"> 99</td></tr>\n",
"<tr><td>objective_two_dad93a03</td><td>TERMINATED</td><td>127.0.0.1:50415</td><td>relu</td><td style=\"text-align: right;\"> 1.85364</td><td style=\"text-align: right;\">-63.6309 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\">16.6414 </td><td style=\"text-align: right;\">-11.7345 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.285 </td><td style=\"text-align: right;\"> 99</td></tr>\n",
"<tr><td>objective_two_e472727a</td><td>TERMINATED</td><td>127.0.0.1:50442</td><td>relu</td><td style=\"text-align: right;\"> 1.2359 </td><td style=\"text-align: right;\">-75.2253 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 7.49782 </td><td style=\"text-align: right;\"> -9.16415 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.2506</td><td style=\"text-align: right;\"> 99</td></tr>\n",
"<tr><td>objective_two_5d1eacff</td><td>TERMINATED</td><td>127.0.0.1:50449</td><td>tanh</td><td style=\"text-align: right;\"> </td><td style=\"text-align: right;\">-20.158 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 6.18643 </td><td style=\"text-align: right;\"> -1.85514 </td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 10.2566</td><td style=\"text-align: right;\"> 99</td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
"</div>\n",
"<style>\n",
".tuneStatus {\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".tuneStatus .systemInfo {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus td {\n",
" white-space: nowrap;\n",
"}\n",
".tuneStatus .trialStatus {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus h3 {\n",
" font-weight: bold;\n",
"}\n",
".tuneStatus .hDivider {\n",
" border-bottom-width: var(--jp-border-width);\n",
" border-bottom-color: var(--jp-border-color0);\n",
" border-bottom-style: solid;\n",
"}\n",
".tuneStatus .vDivider {\n",
" border-left-width: var(--jp-border-width);\n",
" border-left-color: var(--jp-border-color0);\n",
" border-left-style: solid;\n",
" margin: 0.5em 1em 0.5em 1em;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"tuner = tune.Tuner(\n",
" objective_two,\n",
" tune_config=tune.TuneConfig(\n",
" metric=\"mean_loss\",\n",
" mode=\"min\",\n",
" search_alg=algo,\n",
" num_samples=num_samples,\n",
" ),\n",
")\n",
"results = tuner.fit()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "e6172afa",
"metadata": {},
"source": [
"Finally, we again show the hyperparameters that minimize the mean loss defined by the score of the objective function above. "
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "03c3fc49",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best hyperparameters found were: {'activation': {'activation': 'relu', 'mult': 1.8536380640438768}, 'height': -63.630920754630125, 'steps': 100, 'width': 16.641403933591928}\n"
]
}
],
"source": [
"print(\"Best hyperparameters found were: \", results.get_best_result().config)"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "2f7b72d3",
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [],
"source": [
"ray.shutdown()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "hyperopt_example",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.11"
},
"orphan": true
},
"nbformat": 4,
"nbformat_minor": 5
}
doc/source/tune/examples/includes/async_hyperband_example.rst
+19
View File
@@ -0,0 +1,19 @@
:orphan:
Asynchronous HyperBand Example
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This example demonstrates how to use Ray Tune's Asynchronous Successive Halving Algorithm (ASHA) scheduler
to efficiently optimize hyperparameters for a machine learning model. ASHA is particularly useful for
large-scale hyperparameter optimization as it can adaptively allocate resources and end
poorly performing trials early.
Requirements: `pip install "ray[tune]"`
.. literalinclude:: /../../python/ray/tune/examples/async_hyperband_example.py
See Also
--------
- `ASHA Paper <https://arxiv.org/abs/1810.05934>`_
doc/source/tune/examples/includes/ax_example.rst
+6
View File
@@ -0,0 +1,6 @@
:orphan:
AX Example
~~~~~~~~~~
.. literalinclude:: /../../python/ray/tune/examples/ax_example.py
doc/source/tune/examples/includes/bayesopt_example.rst
+6
View File
@@ -0,0 +1,6 @@
:orphan:
BayesOpt Example
~~~~~~~~~~~~~~~~
.. literalinclude:: /../../python/ray/tune/examples/bayesopt_example.py
doc/source/tune/examples/includes/bohb_example.rst
+6
View File
@@ -0,0 +1,6 @@
:orphan:
BOHB Example
~~~~~~~~~~~~
.. literalinclude:: /../../python/ray/tune/examples/bohb_example.py
doc/source/tune/examples/includes/custom_func_checkpointing.rst
+6
View File
@@ -0,0 +1,6 @@
:orphan:
Custom Checkpointing Example
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. literalinclude:: /../../python/ray/tune/examples/custom_func_checkpointing.py
doc/source/tune/examples/includes/hyperband_example.rst
+6
View File
@@ -0,0 +1,6 @@
:orphan:
HyperBand Example
=================
.. literalinclude:: /../../python/ray/tune/examples/hyperband_example.py
doc/source/tune/examples/includes/hyperband_function_example.rst
+6
View File
@@ -0,0 +1,6 @@
:orphan:
HyperBand Function Example
~~~~~~~~~~~~~~~~~~~~~~~~~~
.. literalinclude:: /../../python/ray/tune/examples/hyperband_function_example.py
doc/source/tune/examples/includes/hyperopt_conditional_search_space_example.rst
+6
View File
@@ -0,0 +1,6 @@
:orphan:
Hyperopt Conditional Search Space Example
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. literalinclude:: /../../python/ray/tune/examples/hyperopt_conditional_search_space_example.py
doc/source/tune/examples/includes/images/pbt_function_results.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 97 KiB

doc/source/tune/examples/includes/logging_example.rst
+6
View File
@@ -0,0 +1,6 @@
:orphan:
Logging Example
~~~~~~~~~~~~~~~
.. literalinclude:: /../../python/ray/tune/examples/logging_example.py
doc/source/tune/examples/includes/mlflow_ptl_example.rst
+6
View File
@@ -0,0 +1,6 @@
:orphan:
MLflow PyTorch Lightning Example
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. literalinclude:: /../../python/ray/tune/examples/mlflow_ptl.py
doc/source/tune/examples/includes/mnist_ptl_mini.rst
+6
View File
@@ -0,0 +1,6 @@
:orphan:
MNIST PyTorch Lightning Example
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. literalinclude:: /../../python/ray/tune/examples/mnist_ptl_mini.py
doc/source/tune/examples/includes/mnist_pytorch.rst
+10
View File
@@ -0,0 +1,10 @@
:orphan:
MNIST PyTorch Example
~~~~~~~~~~~~~~~~~~~~~
.. literalinclude:: /../../python/ray/tune/examples/mnist_pytorch.py
If you consider switching to PyTorch Lightning to get rid of some of your boilerplate
training code, please know that we also have a walkthrough on :doc:`how to use Tune with
PyTorch Lightning models </tune/examples/tune-pytorch-lightning>`.
doc/source/tune/examples/includes/mnist_pytorch_trainable.rst
+6
View File
@@ -0,0 +1,6 @@
:orphan:
MNIST PyTorch Trainable Example
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. literalinclude:: /../../python/ray/tune/examples/mnist_pytorch_trainable.py
doc/source/tune/examples/includes/nevergrad_example.rst
+6
View File
@@ -0,0 +1,6 @@
:orphan:
Nevergrad Example
~~~~~~~~~~~~~~~~~
.. literalinclude:: /../../python/ray/tune/examples/nevergrad_example.py
doc/source/tune/examples/includes/pb2_example.rst
+6
View File
@@ -0,0 +1,6 @@
:orphan:
PB2 Example
~~~~~~~~~~~
.. literalinclude:: /../../python/ray/tune/examples/pb2_example.py
doc/source/tune/examples/includes/pb2_ppo_example.rst
+6
View File
@@ -0,0 +1,6 @@
:orphan:
PB2 PPO Example
~~~~~~~~~~~~~~~
.. literalinclude:: /../../python/ray/tune/examples/pb2_ppo_example.py
doc/source/tune/examples/includes/pbt_convnet_function_example.rst
+6
View File
@@ -0,0 +1,6 @@
:orphan:
PBT ConvNet Example
~~~~~~~~~~~~~~~~~~~
.. literalinclude:: /../../python/ray/tune/examples/pbt_convnet_function_example.py
doc/source/tune/examples/includes/pbt_example.rst
+6
View File
@@ -0,0 +1,6 @@
:orphan:
PBT Example
~~~~~~~~~~~
.. literalinclude:: /../../python/ray/tune/examples/pbt_example.py
doc/source/tune/examples/includes/pbt_function.rst
+11
View File
@@ -0,0 +1,11 @@
:orphan:
PBT Function Example
~~~~~~~~~~~~~~~~~~~~
The following script produces the following results. For a population of 8 trials,
the PBT learning rate schedule roughly matches the optimal learning rate schedule.
.. image:: images/pbt_function_results.png
.. literalinclude:: /../../python/ray/tune/examples/pbt_function.py
doc/source/tune/examples/includes/pbt_memnn_example.rst
+6
View File
@@ -0,0 +1,6 @@
:orphan:
Memory NN Example
~~~~~~~~~~~~~~~~~
.. literalinclude:: /../../python/ray/tune/examples/pbt_memnn_example.py
doc/source/tune/examples/includes/pbt_tune_cifar10_with_keras.rst
+6
View File
@@ -0,0 +1,6 @@
:orphan:
Keras Cifar10 Example
~~~~~~~~~~~~~~~~~~~~~
.. literalinclude:: /../../python/ray/tune/examples/pbt_tune_cifar10_with_keras.py
doc/source/tune/examples/includes/tf_mnist_example.rst
+6
View File
@@ -0,0 +1,6 @@
:orphan:
TensorFlow MNIST Example
~~~~~~~~~~~~~~~~~~~~~~~~
.. literalinclude:: /../../python/ray/tune/examples/tf_mnist_example.py
doc/source/tune/examples/includes/tune_basic_example.rst
+6
View File
@@ -0,0 +1,6 @@
:orphan:
tune_basic_example
~~~~~~~~~~~~~~~~~~
.. literalinclude:: /../../python/ray/tune/examples/tune_basic_example.py
doc/source/tune/examples/includes/xgboost_dynamic_resources_example.rst
+6
View File
@@ -0,0 +1,6 @@
:orphan:
XGBoost Dynamic Resources Example
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. literalinclude:: /../../python/ray/tune/examples/xgboost_dynamic_resources_example.py
doc/source/tune/examples/index.rst
+158
View File
@@ -0,0 +1,158 @@
.. _tune-examples-ref:
.. _tune-recipes:
=================
Ray Tune Examples
=================
.. tip::
See :ref:`tune-main` to learn more about Tune features.
Below are examples for using Ray Tune for a variety of use cases and sorted by categories:
* `ML frameworks`_
* `Experiment tracking tools`_
* `Hyperparameter optimization frameworks`_
* `Others`_
* `Exercises`_
.. _ml-frameworks:
ML frameworks
-------------
.. toctree::
:hidden:
PyTorch Example <tune-pytorch-cifar>
PyTorch Lightning Example <tune-pytorch-lightning>
XGBoost Example <tune-xgboost>
LightGBM Example <lightgbm_example>
Hugging Face Transformers Example <pbt_transformers>
Ray RLlib Example <pbt_ppo_example>
Keras Example <tune_mnist_keras>
PyTorch with ASHA </_collections/tune/examples/tune_pytorch_asha/README>
Ray Tune integrates with many popular machine learning frameworks. Here you find a few practical examples showing you how to tune your models. At the end of these guides you will often find links to even more examples.
.. list-table::
* - :doc:`How to use Tune with Keras and TensorFlow models <tune_mnist_keras>`
* - :doc:`How to use Tune with PyTorch models <tune-pytorch-cifar>`
* - :doc:`How to tune PyTorch Lightning models <tune-pytorch-lightning>`
* - :doc:`Tuning RL experiments with Ray Tune and Ray Serve <pbt_ppo_example>`
* - :doc:`Tuning XGBoost parameters with Tune <tune-xgboost>`
* - :doc:`Tuning LightGBM parameters with Tune <lightgbm_example>`
* - :doc:`Tuning Hugging Face Transformers with Tune <pbt_transformers>`
* - :doc:`Hyperparameter tuning with PyTorch and ASHA </_collections/tune/examples/tune_pytorch_asha/README>`
.. _experiment-tracking-tools:
Experiment tracking tools
-------------------------
.. toctree::
:hidden:
Weights & Biases Example <tune-wandb>
MLflow Example <tune-mlflow>
Aim Example <tune-aim>
Comet Example <tune-comet>
Ray Tune integrates with some popular Experiment tracking and management tools,
such as CometML, or Weights & Biases. For how
to use Ray Tune with Tensorboard, see
:ref:`Guide to logging and outputs <tune-logging>`.
.. list-table::
* - :doc:`Using Aim with Ray Tune for experiment management <tune-aim>`
* - :doc:`Using Comet with Ray Tune for experiment management <tune-comet>`
* - :doc:`Tracking your experiment process Weights & Biases <tune-wandb>`
* - :doc:`Using MLflow tracking and auto logging with Tune <tune-mlflow>`
.. _hyperparameter-optimization-frameworks:
Hyperparameter optimization frameworks
--------------------------------------
.. toctree::
:hidden:
Ax Example <ax_example>
HyperOpt Example <hyperopt_example>
Bayesopt Example <bayesopt_example>
BOHB Example <bohb_example>
Nevergrad Example <nevergrad_example>
Optuna Example <optuna_example>
Tune integrates with a wide variety of hyperparameter optimization frameworks
and their respective search algorithms. See the following detailed examples
for each integration:
.. list-table::
* - :doc:`ax_example`
* - :doc:`hyperopt_example`
* - :doc:`bayesopt_example`
* - :doc:`bohb_example`
* - :doc:`nevergrad_example`
* - :doc:`optuna_example`
.. _tune-examples-others:
Others
------
.. list-table::
* - :doc:`Simple example for doing a basic random and grid search <includes/tune_basic_example>`
* - :doc:`Example of using a simple tuning function with AsyncHyperBandScheduler <includes/async_hyperband_example>`
* - :doc:`Example of using a trainable function with HyperBandScheduler and the AsyncHyperBandScheduler <includes/hyperband_function_example>`
* - :doc:`Configuring and running (synchronous) PBT and understanding the underlying algorithm behavior with a simple example <pbt_visualization/pbt_visualization>`
* - :doc:`includes/pbt_function`
* - :doc:`includes/pb2_example`
* - :doc:`includes/logging_example`
.. _tune-examples-exercises:
Exercises
---------
Learn how to use Tune in your browser with the following Colab-based exercises.
.. list-table::
:widths: 50 30 20
:header-rows: 1
* - Description
- Library
- Colab link
* - Basics of using Tune
- PyTorch
- .. image:: https://colab.research.google.com/assets/colab-badge.svg
:target: https://colab.research.google.com/github/ray-project/tutorial/blob/master/tune_exercises/exercise_1_basics.ipynb
:alt: Open in Colab
* - Using search algorithms and trial schedulers to optimize your model
- PyTorch
- .. image:: https://colab.research.google.com/assets/colab-badge.svg
:target: https://colab.research.google.com/github/ray-project/tutorial/blob/master/tune_exercises/exercise_2_optimize.ipynb
:alt: Open in Colab
* - Using Population-Based Training (PBT)
- PyTorch
- .. image:: https://colab.research.google.com/assets/colab-badge.svg
:target: https://colab.research.google.com/github/ray-project/tutorial/blob/master/tune_exercises/exercise_3_pbt.ipynb" target="_parent
:alt: Open in Colab
* - Fine-tuning Hugging Face Transformers with PBT
- Hugging Face Transformers and PyTorch
- .. image:: https://colab.research.google.com/assets/colab-badge.svg
:target: https://colab.research.google.com/drive/1tQgAKgcKQzheoh503OzhS4N9NtfFgmjF?usp=sharing
:alt: Open in Colab
* - Logging Tune runs to Comet ML
- Comet
- .. image:: https://colab.research.google.com/assets/colab-badge.svg
:target: https://colab.research.google.com/drive/1dp3VwVoAH1acn_kG7RuT62mICnOqxU1z?usp=sharing
:alt: Open in Colab
Tutorial source files are on `GitHub <https://github.com/ray-project/tutorial>`_.
doc/source/tune/examples/lightgbm_example.ipynb
+259
View File
@@ -0,0 +1,259 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "12ada6c3",
"metadata": {},
"source": [
"(tune-lightgbm-example)=\n",
"\n",
"# Using LightGBM with Tune\n",
"\n",
"<a id=\"try-anyscale-quickstart-ray-tune-lightgbm_example\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=ray-tune-lightgbm_example\">\n",
" <img src=\"../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
"</a>\n",
"<br></br>\n",
"\n",
"```{image} /images/lightgbm_logo.png\n",
":align: center\n",
":alt: LightGBM Logo\n",
":height: 120px\n",
":target: https://lightgbm.readthedocs.io\n",
"```\n",
"\n",
"```{contents}\n",
":backlinks: none\n",
":local: true\n",
"```\n",
"\n",
"This tutorial shows how to use Ray Tune to optimize hyperparameters for a LightGBM model. We'll use the breast cancer classification dataset from scikit-learn to demonstrate how to:\n",
"\n",
"1. Set up a LightGBM training function with Ray Tune\n",
"2. Configure hyperparameter search spaces\n",
"3. Use the ASHA scheduler for efficient hyperparameter tuning\n",
"4. Report and checkpoint training progress\n",
"\n",
"## Installation\n",
"\n",
"First, let's install the required dependencies:\n",
"\n",
"```bash\n",
"pip install \"ray[tune]\" lightgbm scikit-learn numpy\n",
"```\n",
"\n",
"## Training script\n",
"\n",
"The script below defines a `train_breast_cancer` training function, configures the search space, and runs the tuner with the {class}`~ray.tune.schedulers.ASHAScheduler`."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "b4c3f1e1",
"metadata": {
"pycharm": {
"name": "#%%\n"
},
"tags": [
"hide-output"
]
},
"outputs": [
{
"data": {
"text/html": [
"<div class=\"tuneStatus\">\n",
" <div style=\"display: flex;flex-direction: row\">\n",
" <div style=\"display: flex;flex-direction: column;\">\n",
" <h3>Tune Status</h3>\n",
" <table>\n",
"<tbody>\n",
"<tr><td>Current time:</td><td>2025-02-18 17:33:55</td></tr>\n",
"<tr><td>Running for: </td><td>00:00:01.27 </td></tr>\n",
"<tr><td>Memory: </td><td>25.8/36.0 GiB </td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
" <div class=\"vDivider\"></div>\n",
" <div class=\"systemInfo\">\n",
" <h3>System Info</h3>\n",
" Using AsyncHyperBand: num_stopped=4<br>Bracket: Iter 64.000: -0.1048951048951049 | Iter 16.000: -0.3076923076923077 | Iter 4.000: -0.3076923076923077 | Iter 1.000: -0.32342657342657344<br>Logical resource usage: 1.0/12 CPUs, 0/0 GPUs\n",
" </div>\n",
" \n",
" </div>\n",
" <div class=\"hDivider\"></div>\n",
" <div class=\"trialStatus\">\n",
" <h3>Trial Status</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>status </th><th>loc </th><th>boosting_type </th><th style=\"text-align: right;\"> learning_rate</th><th style=\"text-align: right;\"> num_leaves</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> binary_error</th><th style=\"text-align: right;\"> binary_logloss</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>train_breast_cancer_945ea_00000</td><td>TERMINATED</td><td>127.0.0.1:26189</td><td>gbdt </td><td style=\"text-align: right;\"> 0.00372129 </td><td style=\"text-align: right;\"> 622</td><td style=\"text-align: right;\"> 100</td><td style=\"text-align: right;\"> 0.0507247 </td><td style=\"text-align: right;\"> 0.104895</td><td style=\"text-align: right;\"> 0.45487 </td></tr>\n",
"<tr><td>train_breast_cancer_945ea_00001</td><td>TERMINATED</td><td>127.0.0.1:26191</td><td>dart </td><td style=\"text-align: right;\"> 0.0065691 </td><td style=\"text-align: right;\"> 998</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\"> 0.013751 </td><td style=\"text-align: right;\"> 0.391608</td><td style=\"text-align: right;\"> 0.665636</td></tr>\n",
"<tr><td>train_breast_cancer_945ea_00002</td><td>TERMINATED</td><td>127.0.0.1:26190</td><td>gbdt </td><td style=\"text-align: right;\"> 1.17012e-07</td><td style=\"text-align: right;\"> 995</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\"> 0.0146749 </td><td style=\"text-align: right;\"> 0.412587</td><td style=\"text-align: right;\"> 0.68387 </td></tr>\n",
"<tr><td>train_breast_cancer_945ea_00003</td><td>TERMINATED</td><td>127.0.0.1:26192</td><td>dart </td><td style=\"text-align: right;\"> 0.000194983</td><td style=\"text-align: right;\"> 53</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\"> 0.00605583</td><td style=\"text-align: right;\"> 0.328671</td><td style=\"text-align: right;\"> 0.6405 </td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
"</div>\n",
"<style>\n",
".tuneStatus {\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".tuneStatus .systemInfo {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus td {\n",
" white-space: nowrap;\n",
"}\n",
".tuneStatus .trialStatus {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus h3 {\n",
" font-weight: bold;\n",
"}\n",
".tuneStatus .hDivider {\n",
" border-bottom-width: var(--jp-border-width);\n",
" border-bottom-color: var(--jp-border-color0);\n",
" border-bottom-style: solid;\n",
"}\n",
".tuneStatus .vDivider {\n",
" border-left-width: var(--jp-border-width);\n",
" border-left-color: var(--jp-border-color0);\n",
" border-left-style: solid;\n",
" margin: 0.5em 1em 0.5em 1em;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2025-02-18 17:33:55,300\tINFO tune.py:1009 -- Wrote the latest version of all result files and experiment state to '~/ray_results/train_breast_cancer_2025-02-18_17-33-54' in 0.0035s.\n",
"2025-02-18 17:33:55,302\tINFO tune.py:1041 -- Total run time: 1.28 seconds (1.27 seconds for the tuning loop).\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best hyperparameters found were: {'objective': 'binary', 'metric': ['binary_error', 'binary_logloss'], 'verbose': -1, 'boosting_type': 'gbdt', 'num_leaves': 622, 'learning_rate': 0.003721286118355498}\n"
]
}
],
"source": [
"import lightgbm as lgb\n",
"import numpy as np\n",
"import sklearn.datasets\n",
"import sklearn.metrics\n",
"from sklearn.model_selection import train_test_split\n",
"\n",
"from ray import tune\n",
"from ray.tune.schedulers import ASHAScheduler\n",
"from ray.tune.integration.lightgbm import TuneReportCheckpointCallback\n",
"\n",
"\n",
"def train_breast_cancer(config):\n",
"\n",
" data, target = sklearn.datasets.load_breast_cancer(return_X_y=True)\n",
" train_x, test_x, train_y, test_y = train_test_split(data, target, test_size=0.25)\n",
" train_set = lgb.Dataset(train_x, label=train_y)\n",
" test_set = lgb.Dataset(test_x, label=test_y)\n",
" gbm = lgb.train(\n",
" config,\n",
" train_set,\n",
" valid_sets=[test_set],\n",
" valid_names=[\"eval\"],\n",
" callbacks=[\n",
" TuneReportCheckpointCallback(\n",
" {\n",
" \"binary_error\": \"eval-binary_error\",\n",
" \"binary_logloss\": \"eval-binary_logloss\",\n",
" }\n",
" )\n",
" ],\n",
" )\n",
" preds = gbm.predict(test_x)\n",
" pred_labels = np.rint(preds)\n",
" tune.report(\n",
" {\n",
" \"mean_accuracy\": sklearn.metrics.accuracy_score(test_y, pred_labels),\n",
" \"done\": True,\n",
" }\n",
" )\n",
"\n",
"\n",
"if __name__ == \"__main__\":\n",
" config = {\n",
" \"objective\": \"binary\",\n",
" \"metric\": [\"binary_error\", \"binary_logloss\"],\n",
" \"verbose\": -1,\n",
" \"boosting_type\": tune.grid_search([\"gbdt\", \"dart\"]),\n",
" \"num_leaves\": tune.randint(10, 1000),\n",
" \"learning_rate\": tune.loguniform(1e-8, 1e-1),\n",
" }\n",
"\n",
" tuner = tune.Tuner(\n",
" train_breast_cancer,\n",
" tune_config=tune.TuneConfig(\n",
" metric=\"binary_error\",\n",
" mode=\"min\",\n",
" scheduler=ASHAScheduler(),\n",
" num_samples=2,\n",
" ),\n",
" param_space=config,\n",
" )\n",
" results = tuner.fit()\n",
"\n",
" print(f\"Best hyperparameters found were: {results.get_best_result().config}\")"
]
},
{
"cell_type": "markdown",
"id": "01d74c39",
"metadata": {},
"source": [
"## Expected output\n",
"\n",
"This should give an output like:\n",
"\n",
"```python\n",
"Best hyperparameters found were: {'objective': 'binary', 'metric': ['binary_error', 'binary_logloss'], 'verbose': -1, 'boosting_type': 'gbdt', 'num_leaves': 622, 'learning_rate': 0.003721286118355498}\n",
"```"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "lightgbm_example",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.11"
},
"orphan": true
},
"nbformat": 4,
"nbformat_minor": 5
}
doc/source/tune/examples/nevergrad_example.ipynb
+2190
View File
File diff suppressed because it is too large Load Diff
doc/source/tune/examples/optuna_example.ipynb
+1064
View File
File diff suppressed because it is too large Load Diff
doc/source/tune/examples/pbt_guide.ipynb
+824
View File
File diff suppressed because one or more lines are too long
doc/source/tune/examples/pbt_ppo_example.ipynb
+241
View File
@@ -0,0 +1,241 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "aa1c2614",
"metadata": {},
"source": [
"(tune-rllib-example)=\n",
"\n",
"# Using RLlib with Tune\n",
"\n",
"<a id=\"try-anyscale-quickstart-ray-tune-pbt_ppo_example\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=ray-tune-pbt_ppo_example\">\n",
" <img src=\"../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
"</a>\n",
"<br></br>\n",
"\n",
"```{image} /rllib/images/rllib-logo.png\n",
":align: center\n",
":alt: RLlib Logo\n",
":height: 120px\n",
":target: https://docs.ray.io\n",
"```\n",
"\n",
"```{contents}\n",
":backlinks: none\n",
":local: true\n",
"```\n",
"\n",
"## Example\n",
"\n",
"Example of using a Tune scheduler ([Population Based Training](tune-scheduler-pbt)) with RLlib.\n",
"\n",
"This example specifies `num_workers=4`, `num_cpus=1`, and `num_gpus=0`, which means that each\n",
"PPO trial will use 5 CPUs: 1 (for training) + 4 (for sample collection).\n",
"This example runs 2 trials, so at least 10 CPUs must be available in the cluster resources\n",
"in order to run both trials concurrently. Otherwise, the PBT scheduler will round-robin\n",
"between training each trial, which is less efficient.\n",
"\n",
"If you want to run this example with GPUs, you can set `num_gpus` accordingly."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f4621a1a",
"metadata": {},
"outputs": [],
"source": [
"import random\n",
"\n",
"from ray import tune\n",
"from ray.rllib.algorithms.ppo import PPOConfig\n",
"from ray.rllib.core.rl_module.default_model_config import DefaultModelConfig\n",
"from ray.tune.schedulers import PopulationBasedTraining\n",
"\n",
"if __name__ == \"__main__\":\n",
" import argparse\n",
"\n",
" parser = argparse.ArgumentParser()\n",
" parser.add_argument(\n",
" \"--smoke-test\", action=\"store_true\", help=\"Finish quickly for testing\"\n",
" )\n",
" args, _ = parser.parse_known_args()\n",
"\n",
" # Postprocess the perturbed config to ensure it's still valid\n",
" def explore(config):\n",
" # ensure we collect enough timesteps to do sgd\n",
" if config[\"train_batch_size\"] < config[\"sgd_minibatch_size\"] * 2:\n",
" config[\"train_batch_size\"] = config[\"sgd_minibatch_size\"] * 2\n",
" # ensure we run at least one sgd iter\n",
" if config[\"num_sgd_iter\"] < 1:\n",
" config[\"num_sgd_iter\"] = 1\n",
" return config\n",
"\n",
" hyperparam_mutations = {\n",
" \"clip_param\": lambda: random.uniform(0.01, 0.5),\n",
" \"lr\": [1e-3, 5e-4, 1e-4, 5e-5, 1e-5],\n",
" \"num_epochs\": lambda: random.randint(1, 30),\n",
" \"minibatch_size\": lambda: random.randint(128, 16384),\n",
" \"train_batch_size_per_learner\": lambda: random.randint(2000, 160000),\n",
" }\n",
"\n",
" pbt = PopulationBasedTraining(\n",
" time_attr=\"time_total_s\",\n",
" perturbation_interval=120,\n",
" resample_probability=0.25,\n",
" # Specifies the mutations of these hyperparams\n",
" hyperparam_mutations=hyperparam_mutations,\n",
" custom_explore_fn=explore,\n",
" )\n",
"\n",
" # Stop when we've either reached 100 training iterations or reward=300\n",
" stopping_criteria = {\"training_iteration\": 100, \"episode_reward_mean\": 300}\n",
"\n",
" config = (\n",
" PPOConfig()\n",
" .environment(\"Humanoid-v2\")\n",
" .env_runners(num_env_runners=4)\n",
" .training(\n",
" # These params are tuned from a fixed starting value.\n",
" kl_coeff=1.0,\n",
" lambda_=0.95,\n",
" clip_param=0.2,\n",
" lr=1e-4,\n",
" # These params start off randomly drawn from a set.\n",
" num_epochs=tune.choice([10, 20, 30]),\n",
" minibatch_size=tune.choice([128, 512, 2048]),\n",
" train_batch_size_per_learner=tune.choice([10000, 20000, 40000]),\n",
" )\n",
" .rl_module(\n",
" model_config=DefaultModelConfig(free_log_std=True),\n",
" )\n",
" )\n",
"\n",
" tuner = tune.Tuner(\n",
" \"PPO\",\n",
" tune_config=tune.TuneConfig(\n",
" metric=\"env_runners/episode_return_mean\",\n",
" mode=\"max\",\n",
" scheduler=pbt,\n",
" num_samples=1 if args.smoke_test else 2,\n",
" ),\n",
" param_space=config,\n",
" run_config=tune.RunConfig(stop=stopping_criteria),\n",
" )\n",
" results = tuner.fit()\n"
]
},
{
"cell_type": "code",
"execution_count": 35,
"id": "8cd3cc70",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best performing trial's final set of hyperparameters:\n",
"\n",
"{'clip_param': 0.2,\n",
" 'lambda': 0.95,\n",
" 'lr': 0.0001,\n",
" 'num_sgd_iter': 30,\n",
" 'sgd_minibatch_size': 2048,\n",
" 'train_batch_size': 20000}\n",
"\n",
"Best performing trial's final reported metrics:\n",
"\n",
"{'episode_len_mean': 61.09146341463415,\n",
" 'episode_reward_max': 567.4424113245353,\n",
" 'episode_reward_mean': 310.36948184391935,\n",
" 'episode_reward_min': 87.74736189944105}\n"
]
}
],
"source": [
"import pprint\n",
"\n",
"best_result = results.get_best_result()\n",
"\n",
"print(\"Best performing trial's final set of hyperparameters:\\n\")\n",
"pprint.pprint(\n",
" {k: v for k, v in best_result.config.items() if k in hyperparam_mutations}\n",
")\n",
"\n",
"print(\"\\nBest performing trial's final reported metrics:\\n\")\n",
"\n",
"metrics_to_print = [\n",
" \"episode_reward_mean\",\n",
" \"episode_reward_max\",\n",
" \"episode_reward_min\",\n",
" \"episode_len_mean\",\n",
"]\n",
"pprint.pprint({k: v for k, v in best_result.metrics.items() if k in metrics_to_print})\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e4cc4685",
"metadata": {},
"outputs": [],
"source": [
"from ray.rllib.algorithms.algorithm import Algorithm\n",
"\n",
"loaded_ppo = Algorithm.from_checkpoint(best_result.checkpoint)\n",
"loaded_policy = loaded_ppo.get_policy()\n",
"\n",
"# See your trained policy in action\n",
"# loaded_policy.compute_single_action(...)\n"
]
},
{
"cell_type": "markdown",
"id": "db534c4e",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"## More RLlib Examples\n",
"\n",
"- {doc}`/tune/examples/includes/pb2_ppo_example`:\n",
" Example of optimizing a distributed RLlib algorithm (PPO) with the PB2 scheduler.\n",
" Uses a small population size of 4, so can train on a laptop."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a3d4fb61",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.13"
},
"orphan": true
},
"nbformat": 4,
"nbformat_minor": 5
}
doc/source/tune/examples/pbt_transformers.ipynb
+307
View File
@@ -0,0 +1,307 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "3b05af3b",
"metadata": {},
"source": [
"(tune-huggingface-example)=\n",
"\n",
"# Using Huggingface Transformers with Tune\n",
"\n",
"<a id=\"try-anyscale-quickstart-ray-tune-pbt_transformers\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=ray-tune-pbt_transformers\">\n",
" <img src=\"../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
"</a>\n",
"<br></br>\n",
"\n",
"```{image} /images/hugging.png\n",
":align: center\n",
":alt: Huggingface Logo\n",
":height: 120px\n",
":target: https://huggingface.com\n",
"```\n",
"\n",
"```{contents}\n",
":backlinks: none\n",
":local: true\n",
"```\n",
"\n",
"This example tunes a Hugging Face Transformers model with Ray Tune's Population Based Training (PBT) scheduler, using Hugging Face's official `hyperparameter_search` API.\n",
"\n",
"The example is split into two parts:\n",
"\n",
"- A **training script** that defines the `tune_transformer` entry point and wires together the model, dataset, scheduler, and reporter.\n",
"- A **utility module** (`ray.tune.examples.pbt_transformers.utils`) that provides the data download and metrics helpers imported by the training script.\n",
"\n",
"## Training script"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "19e3c389",
"metadata": {},
"outputs": [],
"source": [
"\"\"\"\n",
"This example is uses the official\n",
"huggingface transformers `hyperparameter_search` API.\n",
"\"\"\"\n",
"import os\n",
"\n",
"import ray\n",
"from ray import tune\n",
"from ray.tune import CLIReporter\n",
"from ray.tune.examples.pbt_transformers.utils import (\n",
" download_data,\n",
" build_compute_metrics_fn,\n",
")\n",
"from ray.tune.schedulers import PopulationBasedTraining\n",
"from transformers import (\n",
" glue_tasks_num_labels,\n",
" AutoConfig,\n",
" AutoModelForSequenceClassification,\n",
" AutoTokenizer,\n",
" Trainer,\n",
" GlueDataset,\n",
" GlueDataTrainingArguments,\n",
" TrainingArguments,\n",
")\n",
"\n",
"\n",
"def tune_transformer(num_samples=8, gpus_per_trial=0, smoke_test=False):\n",
" data_dir_name = \"./data\" if not smoke_test else \"./test_data\"\n",
" data_dir = os.path.abspath(os.path.join(os.getcwd(), data_dir_name))\n",
" if not os.path.exists(data_dir):\n",
" os.mkdir(data_dir, 0o755)\n",
"\n",
" # Change these as needed.\n",
" model_name = (\n",
" \"bert-base-uncased\" if not smoke_test else \"sshleifer/tiny-distilroberta-base\"\n",
" )\n",
" task_name = \"rte\"\n",
"\n",
" task_data_dir = os.path.join(data_dir, task_name.upper())\n",
"\n",
" num_labels = glue_tasks_num_labels[task_name]\n",
"\n",
" config = AutoConfig.from_pretrained(\n",
" model_name, num_labels=num_labels, finetuning_task=task_name\n",
" )\n",
"\n",
" # Download and cache tokenizer, model, and features\n",
" print(\"Downloading and caching Tokenizer\")\n",
" tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
"\n",
" # Triggers tokenizer download to cache\n",
" print(\"Downloading and caching pre-trained model\")\n",
" AutoModelForSequenceClassification.from_pretrained(\n",
" model_name,\n",
" config=config,\n",
" )\n",
"\n",
" def get_model():\n",
" return AutoModelForSequenceClassification.from_pretrained(\n",
" model_name,\n",
" config=config,\n",
" )\n",
"\n",
" # Download data.\n",
" download_data(task_name, data_dir)\n",
"\n",
" data_args = GlueDataTrainingArguments(task_name=task_name, data_dir=task_data_dir)\n",
"\n",
" train_dataset = GlueDataset(\n",
" data_args, tokenizer=tokenizer, mode=\"train\", cache_dir=task_data_dir\n",
" )\n",
" eval_dataset = GlueDataset(\n",
" data_args, tokenizer=tokenizer, mode=\"dev\", cache_dir=task_data_dir\n",
" )\n",
"\n",
" training_args = TrainingArguments(\n",
" output_dir=\".\",\n",
" learning_rate=1e-5, # config\n",
" do_train=True,\n",
" do_eval=True,\n",
" use_cpu=gpus_per_trial <= 0,\n",
" eval_strategy=\"epoch\",\n",
" save_strategy=\"epoch\",\n",
" load_best_model_at_end=True,\n",
" num_train_epochs=2, # config\n",
" max_steps=-1,\n",
" per_device_train_batch_size=16, # config\n",
" per_device_eval_batch_size=16, # config\n",
" warmup_steps=0,\n",
" weight_decay=0.1, # config\n",
" logging_dir=\"./logs\",\n",
" skip_memory_metrics=True,\n",
" report_to=\"none\",\n",
" )\n",
"\n",
" trainer = Trainer(\n",
" model_init=get_model,\n",
" args=training_args,\n",
" train_dataset=train_dataset,\n",
" eval_dataset=eval_dataset,\n",
" compute_metrics=build_compute_metrics_fn(task_name),\n",
" )\n",
"\n",
" tune_config = {\n",
" \"per_device_train_batch_size\": 32,\n",
" \"per_device_eval_batch_size\": 32,\n",
" \"num_train_epochs\": tune.choice([2, 3, 4, 5]),\n",
" \"max_steps\": 1 if smoke_test else -1, # Used for smoke test.\n",
" }\n",
"\n",
" scheduler = PopulationBasedTraining(\n",
" time_attr=\"training_iteration\",\n",
" metric=\"eval_acc\",\n",
" mode=\"max\",\n",
" perturbation_interval=1,\n",
" hyperparam_mutations={\n",
" \"weight_decay\": tune.uniform(0.0, 0.3),\n",
" \"learning_rate\": tune.uniform(1e-5, 5e-5),\n",
" \"per_device_train_batch_size\": [16, 32, 64],\n",
" },\n",
" )\n",
"\n",
" reporter = CLIReporter(\n",
" parameter_columns={\n",
" \"weight_decay\": \"w_decay\",\n",
" \"learning_rate\": \"lr\",\n",
" \"per_device_train_batch_size\": \"train_bs/gpu\",\n",
" \"num_train_epochs\": \"num_epochs\",\n",
" },\n",
" metric_columns=[\"eval_acc\", \"eval_loss\", \"epoch\", \"training_iteration\"],\n",
" )\n",
"\n",
" trainer.hyperparameter_search(\n",
" hp_space=lambda _: tune_config,\n",
" backend=\"ray\",\n",
" n_trials=num_samples,\n",
" resources_per_trial={\"cpu\": 1, \"gpu\": gpus_per_trial},\n",
" scheduler=scheduler,\n",
" keep_checkpoints_num=1,\n",
" checkpoint_score_attr=\"training_iteration\",\n",
" stop={\"training_iteration\": 1} if smoke_test else None,\n",
" progress_reporter=reporter,\n",
" local_dir=\"~/ray_results/\",\n",
" name=\"tune_transformer_pbt\",\n",
" log_to_file=True,\n",
" )\n",
"\n",
"\n",
"if __name__ == \"__main__\":\n",
" import argparse\n",
"\n",
" parser = argparse.ArgumentParser()\n",
" parser.add_argument(\n",
" \"--smoke-test\",\n",
" default=True,\n",
" action=\"store_true\",\n",
" help=\"Finish quickly for testing\",\n",
" )\n",
" args, _ = parser.parse_known_args()\n",
"\n",
" ray.init()\n",
"\n",
" if args.smoke_test:\n",
" tune_transformer(num_samples=1, gpus_per_trial=0, smoke_test=True)\n",
" else:\n",
" # You can change the number of GPUs here:\n",
" tune_transformer(num_samples=8, gpus_per_trial=1)\n"
]
},
{
"cell_type": "markdown",
"id": "42b27553",
"metadata": {},
"source": [
"## Utility module\n",
"\n",
"The training script above imports the following helpers from `ray.tune.examples.pbt_transformers.utils`. They handle GLUE data download and define the `compute_metrics` function passed to the Hugging Face `Trainer`."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b5639782",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"\"\"\"Utilities to load and cache data.\"\"\"\n",
"\n",
"import os\n",
"from typing import Callable, Dict\n",
"import numpy as np\n",
"from transformers import EvalPrediction\n",
"from transformers import glue_compute_metrics, glue_output_modes\n",
"\n",
"\n",
"def build_compute_metrics_fn(task_name: str) -> Callable[[EvalPrediction], Dict]:\n",
" \"\"\"Function from transformers/examples/text-classification/run_glue.py\"\"\"\n",
" output_mode = glue_output_modes[task_name]\n",
"\n",
" def compute_metrics_fn(p: EvalPrediction):\n",
" if output_mode == \"classification\":\n",
" preds = np.argmax(p.predictions, axis=1)\n",
" elif output_mode == \"regression\":\n",
" preds = np.squeeze(p.predictions)\n",
" metrics = glue_compute_metrics(task_name, preds, p.label_ids)\n",
" return metrics\n",
"\n",
" return compute_metrics_fn\n",
"\n",
"\n",
"def download_data(task_name, data_dir=\"./data\"):\n",
" # Download RTE training data\n",
" print(\"Downloading dataset.\")\n",
" import urllib\n",
" import zipfile\n",
"\n",
" if task_name == \"rte\":\n",
" url = \"https://dl.fbaipublicfiles.com/glue/data/RTE.zip\"\n",
" else:\n",
" raise ValueError(\"Unknown task: {}\".format(task_name))\n",
" data_file = os.path.join(data_dir, \"{}.zip\".format(task_name))\n",
" if not os.path.exists(data_file):\n",
" urllib.request.urlretrieve(url, data_file)\n",
" with zipfile.ZipFile(data_file) as zip_ref:\n",
" zip_ref.extractall(data_dir)\n",
" print(\"Downloaded data for task {} to {}\".format(task_name, data_dir))\n",
" else:\n",
" print(\n",
" \"Data already exists. Using downloaded data for task {} from {}\".format(\n",
" task_name, data_dir\n",
" )\n",
" )\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.13"
},
"orphan": true
},
"nbformat": 4,
"nbformat_minor": 5
}
doc/source/tune/examples/pbt_visualization/__init__.py
View File
doc/source/tune/examples/pbt_visualization/figure_from_paper.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 780 KiB

doc/source/tune/examples/pbt_visualization/pbt.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 319 KiB

doc/source/tune/examples/pbt_visualization/pbt4.gif
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 458 KiB

doc/source/tune/examples/pbt_visualization/pbt_visualization.ipynb
+1964
View File
File diff suppressed because one or more lines are too long
doc/source/tune/examples/pbt_visualization/pbt_visualization_utils.py
+173
View File
@@ -0,0 +1,173 @@
import matplotlib.pyplot as plt
from matplotlib.animation import FuncAnimation, PillowWriter
import numpy as np
def get_init_theta():
return np.array([0.9, 0.9])
def Q_batch(theta):
"""Returns the true function value for a batch of parameters with size (B, 2)"""
return 1.2 - (3 / 4 * theta[:, 0] ** 2 + theta[:, 1] ** 2)
def get_arrows(theta_history, perturbation_interval):
"""
Computes the start points and deltas for arrows showing parameter perturbations.
Args:
theta_history: History of parameter values of shape (iterations, 2)
perturbation_interval: Number of iterations between perturbations
Returns:
Tuple[np.ndarray, np.ndarray]: Arrow start points and deltas
"""
theta_history = theta_history[1:, :]
arrow_start = theta_history[
np.arange(perturbation_interval - 1, len(theta_history), perturbation_interval)
]
arrow_end = theta_history[
np.arange(perturbation_interval, len(theta_history), perturbation_interval)
]
if len(arrow_end) > len(arrow_start):
arrow_end = arrow_end[: len(arrow_start)]
else:
arrow_start = arrow_start[: len(arrow_end)]
deltas = arrow_end - arrow_start
return arrow_start, deltas
def plot_parameter_history(
results,
colors,
labels,
perturbation_interval=None,
fig=None,
ax=None,
plot_until_iter=None,
include_colorbar=True,
):
"""
Plot parameter history overlaid on the true reward contour.
Args:
results: List of result objects containing metrics dataframes
colors: List of colors for each result
labels: List of labels for each result
perturbation_interval: Interval at which parameter perturbations occur
fig: Existing figure to plot on (creates new if None)
ax: Existing axes to plot on (creates new if None)
plot_until_iter: Maximum iteration to plot (plots all if None)
include_colorbar: Whether to include a colorbar for the contour plot
Returns:
List of scatter plot objects
"""
if fig is None or ax is None:
fig, ax = plt.subplots()
theta_0 = get_init_theta()
x = np.linspace(-0.2, 1.0, 50)
y = np.linspace(-0.2, 1.0, 50)
xx, yy = np.meshgrid(x, y)
xys = np.transpose(np.stack((xx, yy)).reshape(2, -1))
contour = ax.contourf(xx, yy, Q_batch(xys).reshape(xx.shape), 20)
ax.set_xlabel("theta0")
ax.set_ylabel("theta1")
ax.set_title("Parameter History and True Reward Q(theta) Contour")
scatters = []
for i in range(len(results)):
df = results[i].metrics_dataframe
# Append the initial theta values to the history
theta0_history = np.concatenate([[theta_0[0]], df["theta0"].to_numpy()])
theta1_history = np.concatenate([[theta_0[1]], df["theta1"].to_numpy()])
training_iters = np.concatenate([[0], df["training_iteration"].to_numpy()])
if plot_until_iter is None:
plot_until_iter = len(training_iters)
scatter = ax.scatter(
theta0_history[:plot_until_iter],
theta1_history[:plot_until_iter],
# Size of scatter point decreases as training iteration increases
s=100 / ((training_iters[:plot_until_iter] + 1) ** 1 / 3),
alpha=0.5,
c=colors[i],
label=labels[i],
)
scatters.append(scatter)
for i, theta0, theta1 in zip(training_iters, theta0_history, theta1_history):
if i % (perturbation_interval or 1) == 0 and i < plot_until_iter:
ax.annotate(i, (theta0, theta1))
if perturbation_interval is not None:
theta_history = np.hstack(
(theta0_history.reshape(-1, 1), theta1_history.reshape(-1, 1))
)[:plot_until_iter, :]
arrow_starts, deltas = get_arrows(theta_history, perturbation_interval)
for arrow_start, delta in zip(arrow_starts, deltas):
ax.arrow(
arrow_start[0],
arrow_start[1],
delta[0],
delta[1],
head_width=0.01,
length_includes_head=True,
alpha=0.25,
)
ax.legend(loc="upper left", title="Trial Initial Parameters")
if include_colorbar:
cbar = fig.colorbar(contour, ax=ax, orientation="vertical")
cbar.set_label("Reward Q(theta)")
return scatters
def plot_Q_history(results, colors, labels, ax=None):
"""
Plot the history of true reward values over training iterations.
Args:
results: List of result objects containing metrics dataframes
colors: List of colors for each result
labels: List of labels for each result
ax: Existing axes to plot on (creates new if None)
"""
if ax is None:
fig, ax = plt.subplots()
ax.set_title("True Reward (Q) Value Over Training Iterations")
ax.set_xlabel("Training Iteration")
ax.set_ylabel("Reward Q(theta)")
for i in range(len(results)):
df = results[i].metrics_dataframe
ax.plot(df["Q"], label=labels[i], color=colors[i])
ax.legend(title="Trial Initial Parameters")
def make_animation(
results, colors, labels, perturbation_interval=None, filename="pbt.gif", fps=5
):
fig, ax = plt.subplots(figsize=(8, 8))
def animate(i):
ax.clear()
ax.set_title("Parameter Evolution Over Iterations")
return plot_parameter_history(
results,
colors,
labels,
perturbation_interval=perturbation_interval,
fig=fig,
ax=ax,
plot_until_iter=i,
include_colorbar=False,
)
ani = FuncAnimation(
fig, animate, interval=1000 // fps, blit=True, repeat=True, frames=range(1, 101)
)
ani.save(filename, writer=PillowWriter(fps=fps))
plt.close()
doc/source/tune/examples/pbt_visualization/pbt_visualization_utils.rst
+8
View File
@@ -0,0 +1,8 @@
:orphan:
PBT Visualization Helper File
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Used in :doc:`/tune/examples/pbt_visualization/pbt_visualization`.
.. literalinclude:: ./pbt_visualization_utils.py
doc/source/tune/examples/tune-aim.ipynb
+412
View File
@@ -0,0 +1,412 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "ecad719c",
"metadata": {},
"source": [
"(tune-aim-ref)=\n",
"\n",
"# Using Aim with Tune\n",
"\n",
"<a id=\"try-anyscale-quickstart-tune-aim\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=tune-aim\">\n",
" <img src=\"../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
"</a>\n",
"<br></br>\n",
"\n",
"[Aim](https://aimstack.io) is an easy-to-use and supercharged open-source experiment tracker.\n",
"Aim logs your training runs, enables a well-designed UI to compare them, and provides an API to query them programmatically.\n",
"\n",
"```{image} /images/aim_logo_full.png\n",
":align: center\n",
":alt: Aim\n",
":width: 100%\n",
":target: https://aimstack.io\n",
"```\n",
"\n",
"Ray Tune currently offers built-in integration with Aim.\n",
"The {ref}`AimLoggerCallback <tune-aim-logger>` automatically logs metrics that are reported to Tune by using the Aim API.\n",
"\n",
"\n",
"```{contents}\n",
":backlinks: none\n",
":local: true\n",
"```\n",
"\n",
"## Logging Tune Hyperparameter Configurations and Results to Aim\n",
"\n",
"The following example demonstrates how the `AimLoggerCallback` can be used in a Tune experiment.\n",
"Begin by installing and importing the necessary modules:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1290b5b5",
"metadata": {},
"outputs": [],
"source": [
"%pip install aim\n",
"%pip install ray[tune]"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "100bcf8a",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"import ray\n",
"from ray import tune\n",
"from ray.tune.logger.aim import AimLoggerCallback"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "9346c0f6",
"metadata": {},
"source": [
"Next, define a simple `train_function`, which is a [`Trainable`](trainable-docs) that reports a loss to Tune.\n",
"The objective function itself is not important for this example, as our main focus is on the integration with Aim."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "e8b4fc4d",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"def train_function(config):\n",
" for _ in range(50):\n",
" loss = config[\"mean\"] + config[\"sd\"] * np.random.randn()\n",
" tune.report({\"loss\": loss})"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "831eed42",
"metadata": {},
"source": [
"Here is an example of how you can use the `AimLoggerCallback` with simple grid-search Tune experiment.\n",
"The logger will log each of the 9 grid-search trials as separate Aim runs."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "52988599",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-02-07 00:04:11,228\tINFO worker.py:1544 -- Started a local Ray instance. View the dashboard at \u001b[1m\u001b[32mhttp://127.0.0.1:8265 \u001b[39m\u001b[22m\n"
]
},
{
"data": {
"text/html": [
"<div class=\"tuneStatus\">\n",
" <div style=\"display: flex;flex-direction: row\">\n",
" <div style=\"display: flex;flex-direction: column;\">\n",
" <h3>Tune Status</h3>\n",
" <table>\n",
"<tbody>\n",
"<tr><td>Current time:</td><td>2023-02-07 00:04:19</td></tr>\n",
"<tr><td>Running for: </td><td>00:00:06.86 </td></tr>\n",
"<tr><td>Memory: </td><td>32.8/64.0 GiB </td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
" <div class=\"vDivider\"></div>\n",
" <div class=\"systemInfo\">\n",
" <h3>System Info</h3>\n",
" Using FIFO scheduling algorithm.<br>Resources requested: 0/10 CPUs, 0/0 GPUs, 0.0/26.93 GiB heap, 0.0/2.0 GiB objects\n",
" </div>\n",
" \n",
" </div>\n",
" <div class=\"hDivider\"></div>\n",
" <div class=\"trialStatus\">\n",
" <h3>Trial Status</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>status </th><th>loc </th><th style=\"text-align: right;\"> mean</th><th style=\"text-align: right;\"> sd</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> loss</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>train_function_01a3b_00000</td><td>TERMINATED</td><td>127.0.0.1:10277</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\">0.385428</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\"> 4.48031</td><td style=\"text-align: right;\">1.01928</td></tr>\n",
"<tr><td>train_function_01a3b_00001</td><td>TERMINATED</td><td>127.0.0.1:10296</td><td style=\"text-align: right;\"> 2</td><td style=\"text-align: right;\">0.819716</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\"> 2.97272</td><td style=\"text-align: right;\">3.01491</td></tr>\n",
"<tr><td>train_function_01a3b_00002</td><td>TERMINATED</td><td>127.0.0.1:10301</td><td style=\"text-align: right;\"> 3</td><td style=\"text-align: right;\">0.769197</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\"> 2.39572</td><td style=\"text-align: right;\">3.87155</td></tr>\n",
"<tr><td>train_function_01a3b_00003</td><td>TERMINATED</td><td>127.0.0.1:10307</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\">0.29466 </td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\"> 2.41568</td><td style=\"text-align: right;\">4.1507 </td></tr>\n",
"<tr><td>train_function_01a3b_00004</td><td>TERMINATED</td><td>127.0.0.1:10313</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\">0.152208</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\"> 1.68383</td><td style=\"text-align: right;\">5.10225</td></tr>\n",
"<tr><td>train_function_01a3b_00005</td><td>TERMINATED</td><td>127.0.0.1:10321</td><td style=\"text-align: right;\"> 6</td><td style=\"text-align: right;\">0.879814</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\"> 1.54015</td><td style=\"text-align: right;\">6.20238</td></tr>\n",
"<tr><td>train_function_01a3b_00006</td><td>TERMINATED</td><td>127.0.0.1:10329</td><td style=\"text-align: right;\"> 7</td><td style=\"text-align: right;\">0.487499</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\"> 1.44706</td><td style=\"text-align: right;\">7.79551</td></tr>\n",
"<tr><td>train_function_01a3b_00007</td><td>TERMINATED</td><td>127.0.0.1:10333</td><td style=\"text-align: right;\"> 8</td><td style=\"text-align: right;\">0.639783</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\"> 1.4261 </td><td style=\"text-align: right;\">7.94189</td></tr>\n",
"<tr><td>train_function_01a3b_00008</td><td>TERMINATED</td><td>127.0.0.1:10341</td><td style=\"text-align: right;\"> 9</td><td style=\"text-align: right;\">0.12285 </td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\"> 1.07701</td><td style=\"text-align: right;\">8.82304</td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
"</div>\n",
"<style>\n",
".tuneStatus {\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".tuneStatus .systemInfo {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus td {\n",
" white-space: nowrap;\n",
"}\n",
".tuneStatus .trialStatus {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus h3 {\n",
" font-weight: bold;\n",
"}\n",
".tuneStatus .hDivider {\n",
" border-bottom-width: var(--jp-border-width);\n",
" border-bottom-color: var(--jp-border-color0);\n",
" border-bottom-style: solid;\n",
"}\n",
".tuneStatus .vDivider {\n",
" border-left-width: var(--jp-border-width);\n",
" border-left-color: var(--jp-border-color0);\n",
" border-left-style: solid;\n",
" margin: 0.5em 1em 0.5em 1em;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<div class=\"trialProgress\">\n",
" <h3>Trial Progress</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>date </th><th>done </th><th>episodes_total </th><th>experiment_id </th><th>experiment_tag </th><th>hostname </th><th style=\"text-align: right;\"> iterations_since_restore</th><th style=\"text-align: right;\"> loss</th><th>node_ip </th><th style=\"text-align: right;\"> pid</th><th style=\"text-align: right;\"> time_since_restore</th><th style=\"text-align: right;\"> time_this_iter_s</th><th style=\"text-align: right;\"> time_total_s</th><th style=\"text-align: right;\"> timestamp</th><th style=\"text-align: right;\"> timesteps_since_restore</th><th>timesteps_total </th><th style=\"text-align: right;\"> training_iteration</th><th>trial_id </th><th style=\"text-align: right;\"> warmup_time</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>train_function_01a3b_00000</td><td>2023-02-07_00-04-18</td><td>True </td><td> </td><td>c8447fdceea6436c9edd6f030a5b1d82</td><td>0_mean=1,sd=0.3854</td><td>Justins-MacBook-Pro-16</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\">1.01928</td><td>127.0.0.1</td><td style=\"text-align: right;\">10277</td><td style=\"text-align: right;\"> 4.48031</td><td style=\"text-align: right;\"> 0.013865 </td><td style=\"text-align: right;\"> 4.48031</td><td style=\"text-align: right;\"> 1675757058</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 50</td><td>01a3b_00000</td><td style=\"text-align: right;\"> 0.00264072</td></tr>\n",
"<tr><td>train_function_01a3b_00001</td><td>2023-02-07_00-04-18</td><td>True </td><td> </td><td>7dd6d3ee24244a0885b354c285064728</td><td>1_mean=2,sd=0.8197</td><td>Justins-MacBook-Pro-16</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\">3.01491</td><td>127.0.0.1</td><td style=\"text-align: right;\">10296</td><td style=\"text-align: right;\"> 2.97272</td><td style=\"text-align: right;\"> 0.0584073 </td><td style=\"text-align: right;\"> 2.97272</td><td style=\"text-align: right;\"> 1675757058</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 50</td><td>01a3b_00001</td><td style=\"text-align: right;\"> 0.0316792 </td></tr>\n",
"<tr><td>train_function_01a3b_00002</td><td>2023-02-07_00-04-18</td><td>True </td><td> </td><td>e3da49ebad034c4b8fdaf0aa87927b1a</td><td>2_mean=3,sd=0.7692</td><td>Justins-MacBook-Pro-16</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\">3.87155</td><td>127.0.0.1</td><td style=\"text-align: right;\">10301</td><td style=\"text-align: right;\"> 2.39572</td><td style=\"text-align: right;\"> 0.0695491 </td><td style=\"text-align: right;\"> 2.39572</td><td style=\"text-align: right;\"> 1675757058</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 50</td><td>01a3b_00002</td><td style=\"text-align: right;\"> 0.0315411 </td></tr>\n",
"<tr><td>train_function_01a3b_00003</td><td>2023-02-07_00-04-18</td><td>True </td><td> </td><td>95c60c4f67c4481ebccff25b0a49e75d</td><td>3_mean=4,sd=0.2947</td><td>Justins-MacBook-Pro-16</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\">4.1507 </td><td>127.0.0.1</td><td style=\"text-align: right;\">10307</td><td style=\"text-align: right;\"> 2.41568</td><td style=\"text-align: right;\"> 0.0175381 </td><td style=\"text-align: right;\"> 2.41568</td><td style=\"text-align: right;\"> 1675757058</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 50</td><td>01a3b_00003</td><td style=\"text-align: right;\"> 0.0310779 </td></tr>\n",
"<tr><td>train_function_01a3b_00004</td><td>2023-02-07_00-04-18</td><td>True </td><td> </td><td>a216253cb41e47caa229e65488deb019</td><td>4_mean=5,sd=0.1522</td><td>Justins-MacBook-Pro-16</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\">5.10225</td><td>127.0.0.1</td><td style=\"text-align: right;\">10313</td><td style=\"text-align: right;\"> 1.68383</td><td style=\"text-align: right;\"> 0.064441 </td><td style=\"text-align: right;\"> 1.68383</td><td style=\"text-align: right;\"> 1675757058</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 50</td><td>01a3b_00004</td><td style=\"text-align: right;\"> 0.00450182</td></tr>\n",
"<tr><td>train_function_01a3b_00005</td><td>2023-02-07_00-04-18</td><td>True </td><td> </td><td>23834104277f476cb99d9c696281fceb</td><td>5_mean=6,sd=0.8798</td><td>Justins-MacBook-Pro-16</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\">6.20238</td><td>127.0.0.1</td><td style=\"text-align: right;\">10321</td><td style=\"text-align: right;\"> 1.54015</td><td style=\"text-align: right;\"> 0.00910306</td><td style=\"text-align: right;\"> 1.54015</td><td style=\"text-align: right;\"> 1675757058</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 50</td><td>01a3b_00005</td><td style=\"text-align: right;\"> 0.0480251 </td></tr>\n",
"<tr><td>train_function_01a3b_00006</td><td>2023-02-07_00-04-18</td><td>True </td><td> </td><td>15f650121df747c3bd2720481d47b265</td><td>6_mean=7,sd=0.4875</td><td>Justins-MacBook-Pro-16</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\">7.79551</td><td>127.0.0.1</td><td style=\"text-align: right;\">10329</td><td style=\"text-align: right;\"> 1.44706</td><td style=\"text-align: right;\"> 0.00600386</td><td style=\"text-align: right;\"> 1.44706</td><td style=\"text-align: right;\"> 1675757058</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 50</td><td>01a3b_00006</td><td style=\"text-align: right;\"> 0.00202489</td></tr>\n",
"<tr><td>train_function_01a3b_00007</td><td>2023-02-07_00-04-19</td><td>True </td><td> </td><td>78b1673cf2034ed99135b80a0cb31e0e</td><td>7_mean=8,sd=0.6398</td><td>Justins-MacBook-Pro-16</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\">7.94189</td><td>127.0.0.1</td><td style=\"text-align: right;\">10333</td><td style=\"text-align: right;\"> 1.4261 </td><td style=\"text-align: right;\"> 0.00225306</td><td style=\"text-align: right;\"> 1.4261 </td><td style=\"text-align: right;\"> 1675757059</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 50</td><td>01a3b_00007</td><td style=\"text-align: right;\"> 0.00209713</td></tr>\n",
"<tr><td>train_function_01a3b_00008</td><td>2023-02-07_00-04-19</td><td>True </td><td> </td><td>c7f5d86154cb46b6aa27bef523edcd6f</td><td>8_mean=9,sd=0.1228</td><td>Justins-MacBook-Pro-16</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\">8.82304</td><td>127.0.0.1</td><td style=\"text-align: right;\">10341</td><td style=\"text-align: right;\"> 1.07701</td><td style=\"text-align: right;\"> 0.00291467</td><td style=\"text-align: right;\"> 1.07701</td><td style=\"text-align: right;\"> 1675757059</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 50</td><td>01a3b_00008</td><td style=\"text-align: right;\"> 0.00240111</td></tr>\n",
"</tbody>\n",
"</table>\n",
"</div>\n",
"<style>\n",
".trialProgress {\n",
" display: flex;\n",
" flex-direction: column;\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".trialProgress h3 {\n",
" font-weight: bold;\n",
"}\n",
".trialProgress td {\n",
" white-space: nowrap;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2023-02-07 00:04:19,366\tINFO tune.py:798 -- Total run time: 7.38 seconds (6.85 seconds for the tuning loop).\n"
]
},
{
"data": {
"text/plain": [
"<ray.tune.result_grid.ResultGrid at 0x137de07c0>"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tuner = tune.Tuner(\n",
" train_function,\n",
" run_config=tune.RunConfig(\n",
" callbacks=[AimLoggerCallback()],\n",
" storage_path=\"/tmp/ray_results\",\n",
" name=\"aim_example\",\n",
" ),\n",
" param_space={\n",
" \"mean\": tune.grid_search([1, 2, 3, 4, 5, 6, 7, 8, 9]),\n",
" \"sd\": tune.uniform(0.1, 0.9),\n",
" },\n",
" tune_config=tune.TuneConfig(\n",
" metric=\"loss\",\n",
" mode=\"min\",\n",
" ),\n",
")\n",
"tuner.fit()\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "941f25f2",
"metadata": {},
"source": [
"When the script executes, a grid-search is carried out and the results are saved to the Aim repo,\n",
"stored at the default location -- the experiment log directory (in this case, it's at `/tmp/ray_results/aim_example`).\n",
"\n",
"### More Configuration Options for Aim\n",
"\n",
"In the example above, we used the default configuration for the `AimLoggerCallback`.\n",
"There are a few options that can be configured as arguments to the callback. For example,\n",
"setting `AimLoggerCallback(repo=\"/path/to/repo\")` will log results to the Aim repo at that\n",
"filepath, which could be useful if you have a central location where the results of multiple\n",
"Tune experiments are stored. Relative paths to the working directory where Tune script is\n",
"launched can be used as well. By default, the repo will be set to the experiment log\n",
"directory. See [the API reference](tune-aim-logger) for more configurations.\n",
"\n",
"## Launching the Aim UI\n",
"\n",
"Now that we have logged our results to the Aim repository, we can view it in Aim's web UI.\n",
"To do this, we first find the directory where the Aim repository lives, then we use\n",
"the Aim CLI to launch the web interface."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "880f55aa",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"--------------------------------------------------------------------------\n",
" Aim UI collects anonymous usage analytics. \n",
" Read how to opt-out here: \n",
" https://aimstack.readthedocs.io/en/latest/community/telemetry.html \n",
"--------------------------------------------------------------------------\n",
"\u001b[33mRunning Aim UI on repo `<Repo#-5734997863388805469 path=/tmp/ray_results/aim_example/.aim read_only=None>`\u001b[0m\n",
"Open http://127.0.0.1:43800\n",
"Press Ctrl+C to exit\n",
"^C\n"
]
}
],
"source": [
"# Uncomment the following line to launch the Aim UI!\n",
"#!aim up --repo=/tmp/ray_results/aim_example"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "adbe661a",
"metadata": {},
"source": [
"After launching the Aim UI, we can open the web interface at `localhost:43800`."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "7bb97157",
"metadata": {},
"source": [
"```{image} /images/aim_example_metrics_page.png\n",
":align: center\n",
":alt: Aim Metrics Explorer\n",
":target: https://aimstack.readthedocs.io/en/latest/ui/pages/explorers.html#metrics-explorer\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "2f6e9138",
"metadata": {},
"source": [
"The next sections contain more in-depth information on the API of the Tune-Aim integration.\n",
"\n",
"## Tune Aim Logger API\n",
"\n",
"(tune-aim-logger)=\n",
"\n",
"```{eval-rst}\n",
".. autoclass:: ray.tune.logger.aim.AimLoggerCallback\n",
" :noindex:\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "0ebd1904",
"metadata": {},
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "ray_dev_py38",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.13"
},
"orphan": true,
"vscode": {
"interpreter": {
"hash": "265d195fda5292fe8f69c6e37c435a5634a1ed3b6799724e66a975f68fa21517"
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
doc/source/tune/examples/tune-comet.ipynb
+177
View File
@@ -0,0 +1,177 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "3b05af3b",
"metadata": {},
"source": [
"(tune-comet-ref)=\n",
"\n",
"# Using Comet with Tune\n",
"\n",
"<a id=\"try-anyscale-quickstart-tune-comet\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=tune-comet\">\n",
" <img src=\"../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
"</a>\n",
"<br></br>\n",
"\n",
"[Comet](https://www.comet.ml/site/) is a tool to manage and optimize the\n",
"entire ML lifecycle, from experiment tracking, model optimization and dataset\n",
"versioning to model production monitoring.\n",
"\n",
"```{image} /images/comet_logo_full.png\n",
":align: center\n",
":alt: Comet\n",
":height: 120px\n",
":target: https://www.comet.ml/site/\n",
"```\n",
"\n",
"```{contents}\n",
":backlinks: none\n",
":local: true\n",
"```\n",
"\n",
"## Example\n",
"\n",
"To illustrate logging your trial results to Comet, we'll define a simple training function\n",
"that simulates a `loss` metric:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "19e3c389",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"from ray import tune\n",
"\n",
"\n",
"def train_function(config):\n",
" for i in range(30):\n",
" loss = config[\"mean\"] + config[\"sd\"] * np.random.randn()\n",
" tune.report({\"loss\": loss})"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "6fb69a24",
"metadata": {},
"source": [
"Now, given that you provide your Comet API key and your project name like so:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "993d5be6",
"metadata": {},
"outputs": [],
"source": [
"api_key = \"YOUR_COMET_API_KEY\"\n",
"project_name = \"YOUR_COMET_PROJECT_NAME\""
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "e9ce0d76",
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [],
"source": [
"# This cell is hidden from the rendered notebook. It makes the \n",
"from unittest.mock import MagicMock\n",
"from ray.air.integrations.comet import CometLoggerCallback\n",
"\n",
"CometLoggerCallback._logger_process_cls = MagicMock\n",
"api_key = \"abc\"\n",
"project_name = \"test\""
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "d792a1b0",
"metadata": {},
"source": [
"You can add a Comet logger by specifying the `callbacks` argument in your `RunConfig()` accordingly:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dbb761e7",
"metadata": {},
"outputs": [],
"source": [
"from ray.air.integrations.comet import CometLoggerCallback\n",
"\n",
"tuner = tune.Tuner(\n",
" train_function,\n",
" tune_config=tune.TuneConfig(\n",
" metric=\"loss\",\n",
" mode=\"min\",\n",
" ),\n",
" run_config=tune.RunConfig(\n",
" callbacks=[\n",
" CometLoggerCallback(\n",
" api_key=api_key, project_name=project_name, tags=[\"comet_example\"]\n",
" )\n",
" ],\n",
" ),\n",
" param_space={\"mean\": tune.grid_search([1, 2, 3]), \"sd\": tune.uniform(0.2, 0.8)},\n",
")\n",
"results = tuner.fit()\n",
"\n",
"print(results.get_best_result().config)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "d7e46189",
"metadata": {},
"source": [
"## Tune Comet Logger\n",
"\n",
"Ray Tune offers an integration with Comet through the `CometLoggerCallback`,\n",
"which automatically logs metrics and parameters reported to Tune to the Comet UI.\n",
"\n",
"Click on the following dropdown to see this callback API in detail:\n",
"\n",
"```{eval-rst}\n",
".. autoclass:: ray.air.integrations.comet.CometLoggerCallback\n",
" :noindex:\n",
"```"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
},
"orphan": true
},
"nbformat": 4,
"nbformat_minor": 5
}
doc/source/tune/examples/tune-mlflow.ipynb
+600
View File
@@ -0,0 +1,600 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "6df76a1f",
"metadata": {},
"source": [
"# Using MLflow with Tune\n",
"\n",
"<a id=\"try-anyscale-quickstart-tune-mlflow\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=tune-mlflow\">\n",
" <img src=\"../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
"</a>\n",
"<br></br>\n",
"\n",
"(tune-mlflow-ref)=\n",
"\n",
"[MLflow](https://mlflow.org/) is an open source platform to manage the ML lifecycle, including experimentation,\n",
"reproducibility, deployment, and a central model registry. It currently offers four components, including\n",
"MLflow Tracking to record and query experiments, including code, data, config, and results.\n",
"\n",
"```{image} /images/mlflow.png\n",
":align: center\n",
":alt: MLflow\n",
":height: 80px\n",
":target: https://www.mlflow.org/\n",
"```\n",
"\n",
"Ray Tune currently offers two lightweight integrations for MLflow Tracking.\n",
"One is the {ref}`MLflowLoggerCallback <tune-mlflow-logger>`, which automatically logs\n",
"metrics reported to Tune to the MLflow Tracking API.\n",
"\n",
"The other one is the {ref}`setup_mlflow <tune-mlflow-setup>` function, which can be\n",
"used with the function API. It automatically\n",
"initializes the MLflow API with Tune's training information and creates a run for each Tune trial.\n",
"Then within your training function, you can just use the\n",
"MLflow like you would normally do, e.g. using `mlflow.log_metrics()` or even `mlflow.autolog()`\n",
"to log to your training process.\n",
"\n",
"```{contents}\n",
":backlinks: none\n",
":local: true\n",
"```\n",
"\n",
"## Running an MLflow Example\n",
"\n",
"In the following example we're going to use both of the above methods, namely the `MLflowLoggerCallback` and\n",
"the `setup_mlflow` function to log metrics.\n",
"Let's start with a few crucial imports:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "b0e47339",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import tempfile\n",
"import time\n",
"\n",
"import mlflow\n",
"\n",
"from ray import tune\n",
"from ray.air.integrations.mlflow import MLflowLoggerCallback, setup_mlflow\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "618b6935",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"Next, let's define an easy training function (a Tune `Trainable`) that iteratively computes steps and evaluates\n",
"intermediate scores that we report to Tune."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "f449538e",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"def evaluation_fn(step, width, height):\n",
" return (0.1 + width * step / 100) ** (-1) + height * 0.1\n",
"\n",
"\n",
"def train_function(config):\n",
" width, height = config[\"width\"], config[\"height\"]\n",
"\n",
" for step in range(config.get(\"steps\", 100)):\n",
" # Iterative training function - can be any arbitrary training procedure\n",
" intermediate_score = evaluation_fn(step, width, height)\n",
" # Feed the score back to Tune.\n",
" tune.report({\"iterations\": step, \"mean_loss\": intermediate_score})\n",
" time.sleep(0.1)\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "722e5d2f",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"Given an MLFlow tracking URI, you can now simply use the `MLflowLoggerCallback` as a `callback` argument to\n",
"your `RunConfig()`:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "8e0b9ab7",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"def tune_with_callback(mlflow_tracking_uri, finish_fast=False):\n",
" tuner = tune.Tuner(\n",
" train_function,\n",
" tune_config=tune.TuneConfig(num_samples=5),\n",
" run_config=tune.RunConfig(\n",
" name=\"mlflow\",\n",
" callbacks=[\n",
" MLflowLoggerCallback(\n",
" tracking_uri=mlflow_tracking_uri,\n",
" experiment_name=\"mlflow_callback_example\",\n",
" save_artifact=True,\n",
" )\n",
" ],\n",
" ),\n",
" param_space={\n",
" \"width\": tune.randint(10, 100),\n",
" \"height\": tune.randint(0, 100),\n",
" \"steps\": 5 if finish_fast else 100,\n",
" },\n",
" )\n",
" results = tuner.fit()\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "e086f110",
"metadata": {},
"source": [
"To use the `setup_mlflow` utility, you simply call this function in your training function.\n",
"Note that we also use `mlflow.log_metrics(...)` to log metrics to MLflow.\n",
"Otherwise, this version of our training function is identical to its original."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "144b8f39",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"def train_function_mlflow(config):\n",
" tracking_uri = config.pop(\"tracking_uri\", None)\n",
" setup_mlflow(\n",
" config,\n",
" experiment_name=\"setup_mlflow_example\",\n",
" tracking_uri=tracking_uri,\n",
" )\n",
"\n",
" # Hyperparameters\n",
" width, height = config[\"width\"], config[\"height\"]\n",
"\n",
" for step in range(config.get(\"steps\", 100)):\n",
" # Iterative training function - can be any arbitrary training procedure\n",
" intermediate_score = evaluation_fn(step, width, height)\n",
" # Log the metrics to mlflow\n",
" mlflow.log_metrics(dict(mean_loss=intermediate_score), step=step)\n",
" # Feed the score back to Tune.\n",
" tune.report({\"iterations\": step, \"mean_loss\": intermediate_score})\n",
" time.sleep(0.1)\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "dc480366",
"metadata": {},
"source": [
"With this new objective function ready, you can now create a Tune run with it as follows:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "4b9fe6be",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"def tune_with_setup(mlflow_tracking_uri, finish_fast=False):\n",
" # Set the experiment, or create a new one if does not exist yet.\n",
" mlflow.set_tracking_uri(mlflow_tracking_uri)\n",
" mlflow.set_experiment(experiment_name=\"setup_mlflow_example\")\n",
"\n",
" tuner = tune.Tuner(\n",
" train_function_mlflow,\n",
" tune_config=tune.TuneConfig(num_samples=5),\n",
" run_config=tune.RunConfig(\n",
" name=\"mlflow\",\n",
" ),\n",
" param_space={\n",
" \"width\": tune.randint(10, 100),\n",
" \"height\": tune.randint(0, 100),\n",
" \"steps\": 5 if finish_fast else 100,\n",
" \"tracking_uri\": mlflow.get_tracking_uri(),\n",
" },\n",
" )\n",
" results = tuner.fit()\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "915dfd30",
"metadata": {},
"source": [
"If you hapen to have an MLFlow tracking URI, you can set it below in the `mlflow_tracking_uri` variable and set\n",
"`smoke_test=False`.\n",
"Otherwise, you can just run a quick test of the `tune_function` and `tune_decorated` functions without using MLflow."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "05d11774",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2022-12-22 10:37:53,580\tINFO worker.py:1542 -- Started a local Ray instance. View the dashboard at \u001b[1m\u001b[32mhttp://127.0.0.1:8265 \u001b[39m\u001b[22m\n"
]
},
{
"data": {
"text/html": [
"<div class=\"tuneStatus\">\n",
" <div style=\"display: flex;flex-direction: row\">\n",
" <div style=\"display: flex;flex-direction: column;\">\n",
" <h3>Tune Status</h3>\n",
" <table>\n",
"<tbody>\n",
"<tr><td>Current time:</td><td>2022-12-22 10:38:04</td></tr>\n",
"<tr><td>Running for: </td><td>00:00:06.73 </td></tr>\n",
"<tr><td>Memory: </td><td>10.4/16.0 GiB </td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
" <div class=\"vDivider\"></div>\n",
" <div class=\"systemInfo\">\n",
" <h3>System Info</h3>\n",
" Using FIFO scheduling algorithm.<br>Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/4.03 GiB heap, 0.0/2.0 GiB objects\n",
" </div>\n",
" \n",
" </div>\n",
" <div class=\"hDivider\"></div>\n",
" <div class=\"trialStatus\">\n",
" <h3>Trial Status</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>status </th><th>loc </th><th style=\"text-align: right;\"> height</th><th style=\"text-align: right;\"> width</th><th style=\"text-align: right;\"> loss</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> iterations</th><th style=\"text-align: right;\"> neg_mean_loss</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>train_function_b275b_00000</td><td>TERMINATED</td><td>127.0.0.1:801</td><td style=\"text-align: right;\"> 66</td><td style=\"text-align: right;\"> 36</td><td style=\"text-align: right;\">7.24935</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 0.587302</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> -7.24935</td></tr>\n",
"<tr><td>train_function_b275b_00001</td><td>TERMINATED</td><td>127.0.0.1:813</td><td style=\"text-align: right;\"> 33</td><td style=\"text-align: right;\"> 35</td><td style=\"text-align: right;\">3.96667</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 0.507423</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> -3.96667</td></tr>\n",
"<tr><td>train_function_b275b_00002</td><td>TERMINATED</td><td>127.0.0.1:814</td><td style=\"text-align: right;\"> 75</td><td style=\"text-align: right;\"> 29</td><td style=\"text-align: right;\">8.29365</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 0.518995</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> -8.29365</td></tr>\n",
"<tr><td>train_function_b275b_00003</td><td>TERMINATED</td><td>127.0.0.1:815</td><td style=\"text-align: right;\"> 28</td><td style=\"text-align: right;\"> 63</td><td style=\"text-align: right;\">3.18168</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 0.567739</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> -3.18168</td></tr>\n",
"<tr><td>train_function_b275b_00004</td><td>TERMINATED</td><td>127.0.0.1:816</td><td style=\"text-align: right;\"> 20</td><td style=\"text-align: right;\"> 18</td><td style=\"text-align: right;\">3.21951</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 0.526536</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> -3.21951</td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
"</div>\n",
"<style>\n",
".tuneStatus {\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".tuneStatus .systemInfo {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus td {\n",
" white-space: nowrap;\n",
"}\n",
".tuneStatus .trialStatus {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus h3 {\n",
" font-weight: bold;\n",
"}\n",
".tuneStatus .hDivider {\n",
" border-bottom-width: var(--jp-border-width);\n",
" border-bottom-color: var(--jp-border-color0);\n",
" border-bottom-style: solid;\n",
"}\n",
".tuneStatus .vDivider {\n",
" border-left-width: var(--jp-border-width);\n",
" border-left-color: var(--jp-border-color0);\n",
" border-left-style: solid;\n",
" margin: 0.5em 1em 0.5em 1em;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<div class=\"trialProgress\">\n",
" <h3>Trial Progress</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>date </th><th>done </th><th>episodes_total </th><th>experiment_id </th><th>experiment_tag </th><th>hostname </th><th style=\"text-align: right;\"> iterations</th><th style=\"text-align: right;\"> iterations_since_restore</th><th style=\"text-align: right;\"> mean_loss</th><th style=\"text-align: right;\"> neg_mean_loss</th><th>node_ip </th><th style=\"text-align: right;\"> pid</th><th style=\"text-align: right;\"> time_since_restore</th><th style=\"text-align: right;\"> time_this_iter_s</th><th style=\"text-align: right;\"> time_total_s</th><th style=\"text-align: right;\"> timestamp</th><th style=\"text-align: right;\"> timesteps_since_restore</th><th>timesteps_total </th><th style=\"text-align: right;\"> training_iteration</th><th>trial_id </th><th style=\"text-align: right;\"> warmup_time</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>train_function_b275b_00000</td><td>2022-12-22_10-38-01</td><td>True </td><td> </td><td>28feaa4dd8ab4edab810e8109e77502e</td><td>0_height=66,width=36</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 7.24935</td><td style=\"text-align: right;\"> -7.24935</td><td>127.0.0.1</td><td style=\"text-align: right;\"> 801</td><td style=\"text-align: right;\"> 0.587302</td><td style=\"text-align: right;\"> 0.126818</td><td style=\"text-align: right;\"> 0.587302</td><td style=\"text-align: right;\"> 1671705481</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 5</td><td>b275b_00000</td><td style=\"text-align: right;\"> 0.00293493</td></tr>\n",
"<tr><td>train_function_b275b_00001</td><td>2022-12-22_10-38-04</td><td>True </td><td> </td><td>245010d0c3d0439ebfb664764ae9db3c</td><td>1_height=33,width=35</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 3.96667</td><td style=\"text-align: right;\"> -3.96667</td><td>127.0.0.1</td><td style=\"text-align: right;\"> 813</td><td style=\"text-align: right;\"> 0.507423</td><td style=\"text-align: right;\"> 0.122086</td><td style=\"text-align: right;\"> 0.507423</td><td style=\"text-align: right;\"> 1671705484</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 5</td><td>b275b_00001</td><td style=\"text-align: right;\"> 0.00553799</td></tr>\n",
"<tr><td>train_function_b275b_00002</td><td>2022-12-22_10-38-04</td><td>True </td><td> </td><td>898afbf9b906448c980f399c72a2324c</td><td>2_height=75,width=29</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 8.29365</td><td style=\"text-align: right;\"> -8.29365</td><td>127.0.0.1</td><td style=\"text-align: right;\"> 814</td><td style=\"text-align: right;\"> 0.518995</td><td style=\"text-align: right;\"> 0.123554</td><td style=\"text-align: right;\"> 0.518995</td><td style=\"text-align: right;\"> 1671705484</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 5</td><td>b275b_00002</td><td style=\"text-align: right;\"> 0.0040431 </td></tr>\n",
"<tr><td>train_function_b275b_00003</td><td>2022-12-22_10-38-04</td><td>True </td><td> </td><td>03a4476f82734642b6ab0a5040ca58f8</td><td>3_height=28,width=63</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 3.18168</td><td style=\"text-align: right;\"> -3.18168</td><td>127.0.0.1</td><td style=\"text-align: right;\"> 815</td><td style=\"text-align: right;\"> 0.567739</td><td style=\"text-align: right;\"> 0.125471</td><td style=\"text-align: right;\"> 0.567739</td><td style=\"text-align: right;\"> 1671705484</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 5</td><td>b275b_00003</td><td style=\"text-align: right;\"> 0.00406194</td></tr>\n",
"<tr><td>train_function_b275b_00004</td><td>2022-12-22_10-38-04</td><td>True </td><td> </td><td>ff8c7c55ce6e404f9b0552c17f7a0c40</td><td>4_height=20,width=18</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 3.21951</td><td style=\"text-align: right;\"> -3.21951</td><td>127.0.0.1</td><td style=\"text-align: right;\"> 816</td><td style=\"text-align: right;\"> 0.526536</td><td style=\"text-align: right;\"> 0.123327</td><td style=\"text-align: right;\"> 0.526536</td><td style=\"text-align: right;\"> 1671705484</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 5</td><td>b275b_00004</td><td style=\"text-align: right;\"> 0.00332022</td></tr>\n",
"</tbody>\n",
"</table>\n",
"</div>\n",
"<style>\n",
".trialProgress {\n",
" display: flex;\n",
" flex-direction: column;\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".trialProgress h3 {\n",
" font-weight: bold;\n",
"}\n",
".trialProgress td {\n",
" white-space: nowrap;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2022-12-22 10:38:04,477\tINFO tune.py:772 -- Total run time: 7.99 seconds (6.71 seconds for the tuning loop).\n"
]
},
{
"data": {
"text/html": [
"<div class=\"tuneStatus\">\n",
" <div style=\"display: flex;flex-direction: row\">\n",
" <div style=\"display: flex;flex-direction: column;\">\n",
" <h3>Tune Status</h3>\n",
" <table>\n",
"<tbody>\n",
"<tr><td>Current time:</td><td>2022-12-22 10:38:11</td></tr>\n",
"<tr><td>Running for: </td><td>00:00:07.00 </td></tr>\n",
"<tr><td>Memory: </td><td>10.7/16.0 GiB </td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
" <div class=\"vDivider\"></div>\n",
" <div class=\"systemInfo\">\n",
" <h3>System Info</h3>\n",
" Using FIFO scheduling algorithm.<br>Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/4.03 GiB heap, 0.0/2.0 GiB objects\n",
" </div>\n",
" \n",
" </div>\n",
" <div class=\"hDivider\"></div>\n",
" <div class=\"trialStatus\">\n",
" <h3>Trial Status</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>status </th><th>loc </th><th style=\"text-align: right;\"> height</th><th style=\"text-align: right;\"> width</th><th style=\"text-align: right;\"> loss</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> iterations</th><th style=\"text-align: right;\"> neg_mean_loss</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>train_function_mlflow_b73bd_00000</td><td>TERMINATED</td><td>127.0.0.1:842</td><td style=\"text-align: right;\"> 37</td><td style=\"text-align: right;\"> 68</td><td style=\"text-align: right;\">4.05461</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 0.750435</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> -4.05461</td></tr>\n",
"<tr><td>train_function_mlflow_b73bd_00001</td><td>TERMINATED</td><td>127.0.0.1:853</td><td style=\"text-align: right;\"> 50</td><td style=\"text-align: right;\"> 20</td><td style=\"text-align: right;\">6.11111</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 0.652748</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> -6.11111</td></tr>\n",
"<tr><td>train_function_mlflow_b73bd_00002</td><td>TERMINATED</td><td>127.0.0.1:854</td><td style=\"text-align: right;\"> 38</td><td style=\"text-align: right;\"> 83</td><td style=\"text-align: right;\">4.0924 </td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 0.6513 </td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> -4.0924 </td></tr>\n",
"<tr><td>train_function_mlflow_b73bd_00003</td><td>TERMINATED</td><td>127.0.0.1:855</td><td style=\"text-align: right;\"> 15</td><td style=\"text-align: right;\"> 93</td><td style=\"text-align: right;\">1.76178</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 0.650586</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> -1.76178</td></tr>\n",
"<tr><td>train_function_mlflow_b73bd_00004</td><td>TERMINATED</td><td>127.0.0.1:856</td><td style=\"text-align: right;\"> 75</td><td style=\"text-align: right;\"> 43</td><td style=\"text-align: right;\">8.04945</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 0.656046</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> -8.04945</td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
"</div>\n",
"<style>\n",
".tuneStatus {\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".tuneStatus .systemInfo {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus td {\n",
" white-space: nowrap;\n",
"}\n",
".tuneStatus .trialStatus {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus h3 {\n",
" font-weight: bold;\n",
"}\n",
".tuneStatus .hDivider {\n",
" border-bottom-width: var(--jp-border-width);\n",
" border-bottom-color: var(--jp-border-color0);\n",
" border-bottom-style: solid;\n",
"}\n",
".tuneStatus .vDivider {\n",
" border-left-width: var(--jp-border-width);\n",
" border-left-color: var(--jp-border-color0);\n",
" border-left-style: solid;\n",
" margin: 0.5em 1em 0.5em 1em;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<div class=\"trialProgress\">\n",
" <h3>Trial Progress</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>date </th><th>done </th><th>episodes_total </th><th>experiment_id </th><th>experiment_tag </th><th>hostname </th><th style=\"text-align: right;\"> iterations</th><th style=\"text-align: right;\"> iterations_since_restore</th><th style=\"text-align: right;\"> mean_loss</th><th style=\"text-align: right;\"> neg_mean_loss</th><th>node_ip </th><th style=\"text-align: right;\"> pid</th><th style=\"text-align: right;\"> time_since_restore</th><th style=\"text-align: right;\"> time_this_iter_s</th><th style=\"text-align: right;\"> time_total_s</th><th style=\"text-align: right;\"> timestamp</th><th style=\"text-align: right;\"> timesteps_since_restore</th><th>timesteps_total </th><th style=\"text-align: right;\"> training_iteration</th><th>trial_id </th><th style=\"text-align: right;\"> warmup_time</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>train_function_mlflow_b73bd_00000</td><td>2022-12-22_10-38-08</td><td>True </td><td> </td><td>62703cfe82e54d74972377fbb525b000</td><td>0_height=37,width=68</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 4.05461</td><td style=\"text-align: right;\"> -4.05461</td><td>127.0.0.1</td><td style=\"text-align: right;\"> 842</td><td style=\"text-align: right;\"> 0.750435</td><td style=\"text-align: right;\"> 0.108625</td><td style=\"text-align: right;\"> 0.750435</td><td style=\"text-align: right;\"> 1671705488</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 5</td><td>b73bd_00000</td><td style=\"text-align: right;\"> 0.0030272 </td></tr>\n",
"<tr><td>train_function_mlflow_b73bd_00001</td><td>2022-12-22_10-38-11</td><td>True </td><td> </td><td>03ea89852115465392ed318db8021614</td><td>1_height=50,width=20</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 6.11111</td><td style=\"text-align: right;\"> -6.11111</td><td>127.0.0.1</td><td style=\"text-align: right;\"> 853</td><td style=\"text-align: right;\"> 0.652748</td><td style=\"text-align: right;\"> 0.110796</td><td style=\"text-align: right;\"> 0.652748</td><td style=\"text-align: right;\"> 1671705491</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 5</td><td>b73bd_00001</td><td style=\"text-align: right;\"> 0.00303078</td></tr>\n",
"<tr><td>train_function_mlflow_b73bd_00002</td><td>2022-12-22_10-38-11</td><td>True </td><td> </td><td>3731fc2966f9453ba58c650d89035ab4</td><td>2_height=38,width=83</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 4.0924 </td><td style=\"text-align: right;\"> -4.0924 </td><td>127.0.0.1</td><td style=\"text-align: right;\"> 854</td><td style=\"text-align: right;\"> 0.6513 </td><td style=\"text-align: right;\"> 0.108578</td><td style=\"text-align: right;\"> 0.6513 </td><td style=\"text-align: right;\"> 1671705491</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 5</td><td>b73bd_00002</td><td style=\"text-align: right;\"> 0.00310016</td></tr>\n",
"<tr><td>train_function_mlflow_b73bd_00003</td><td>2022-12-22_10-38-11</td><td>True </td><td> </td><td>fb35841742b348b9912d10203c730f1e</td><td>3_height=15,width=93</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 1.76178</td><td style=\"text-align: right;\"> -1.76178</td><td>127.0.0.1</td><td style=\"text-align: right;\"> 855</td><td style=\"text-align: right;\"> 0.650586</td><td style=\"text-align: right;\"> 0.109097</td><td style=\"text-align: right;\"> 0.650586</td><td style=\"text-align: right;\"> 1671705491</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 5</td><td>b73bd_00003</td><td style=\"text-align: right;\"> 0.0576491 </td></tr>\n",
"<tr><td>train_function_mlflow_b73bd_00004</td><td>2022-12-22_10-38-11</td><td>True </td><td> </td><td>6d3cbf9ecc3446369e607ff78c67bc29</td><td>4_height=75,width=43</td><td>kais-macbook-pro.anyscale.com.beta.tailscale.net</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 8.04945</td><td style=\"text-align: right;\"> -8.04945</td><td>127.0.0.1</td><td style=\"text-align: right;\"> 856</td><td style=\"text-align: right;\"> 0.656046</td><td style=\"text-align: right;\"> 0.109869</td><td style=\"text-align: right;\"> 0.656046</td><td style=\"text-align: right;\"> 1671705491</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 5</td><td>b73bd_00004</td><td style=\"text-align: right;\"> 0.00265694</td></tr>\n",
"</tbody>\n",
"</table>\n",
"</div>\n",
"<style>\n",
".trialProgress {\n",
" display: flex;\n",
" flex-direction: column;\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".trialProgress h3 {\n",
" font-weight: bold;\n",
"}\n",
".trialProgress td {\n",
" white-space: nowrap;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2022-12-22 10:38:11,514\tINFO tune.py:772 -- Total run time: 7.01 seconds (6.98 seconds for the tuning loop).\n"
]
}
],
"source": [
"smoke_test = True\n",
"\n",
"if smoke_test:\n",
" mlflow_tracking_uri = os.path.join(tempfile.gettempdir(), \"mlruns\")\n",
"else:\n",
" mlflow_tracking_uri = \"<MLFLOW_TRACKING_URI>\"\n",
"\n",
"tune_with_callback(mlflow_tracking_uri, finish_fast=smoke_test)\n",
"if not smoke_test:\n",
" df = mlflow.search_runs(\n",
" [mlflow.get_experiment_by_name(\"mlflow_callback_example\").experiment_id]\n",
" )\n",
" print(df)\n",
"\n",
"tune_with_setup(mlflow_tracking_uri, finish_fast=smoke_test)\n",
"if not smoke_test:\n",
" df = mlflow.search_runs(\n",
" [mlflow.get_experiment_by_name(\"setup_mlflow_example\").experiment_id]\n",
" )\n",
" print(df)\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "f0df0817",
"metadata": {},
"source": [
"This completes our Tune and MLflow walk-through.\n",
"In the following sections you can find more details on the API of the Tune-MLflow integration.\n",
"\n",
"## MLflow AutoLogging\n",
"\n",
"You can also check out {doc}`here </tune/examples/includes/mlflow_ptl_example>` for an example on how you can\n",
"leverage MLflow auto-logging, in this case with Pytorch Lightning\n",
"\n",
"## MLflow Logger API\n",
"\n",
"(tune-mlflow-logger)=\n",
"\n",
"```{eval-rst}\n",
".. autoclass:: ray.air.integrations.mlflow.MLflowLoggerCallback\n",
" :noindex:\n",
"```\n",
"\n",
"## MLflow setup API\n",
"\n",
"(tune-mlflow-setup)=\n",
"\n",
"```{eval-rst}\n",
".. autofunction:: ray.air.integrations.mlflow.setup_mlflow\n",
" :noindex:\n",
"```\n",
"\n",
"## More MLflow Examples\n",
"\n",
"- {doc}`/tune/examples/includes/mlflow_ptl_example`: Example for using [MLflow](https://github.com/mlflow/mlflow/)\n",
" and [Pytorch Lightning](https://github.com/PyTorchLightning/pytorch-lightning) with Ray Tune."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
},
"orphan": true
},
"nbformat": 4,
"nbformat_minor": 5
}
doc/source/tune/examples/tune-pytorch-cifar.ipynb
+555
View File
@@ -0,0 +1,555 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "586737af",
"metadata": {},
"source": [
"# How to use Tune with PyTorch\n",
"\n",
"<a id=\"try-anyscale-quickstart-tune-pytorch-cifar\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=tune-pytorch-cifar\">\n",
" <img src=\"../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
"</a>\n",
"<br></br>\n",
"\n",
"(tune-pytorch-cifar-ref)=\n",
"\n",
"In this walkthrough, we will show you how to integrate Tune into your PyTorch\n",
"training workflow. We will follow [this tutorial from the PyTorch documentation](https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html)\n",
"for training a CIFAR10 image classifier.\n",
"\n",
"```{image} /images/pytorch_logo.png\n",
":align: center\n",
"```\n",
"\n",
"Hyperparameter tuning can make the difference between an average model and a highly\n",
"accurate one. Often simple things like choosing a different learning rate or changing\n",
"a network layer size can have a dramatic impact on your model performance. Fortunately,\n",
"Tune makes exploring these optimal parameter combinations easy - and works nicely\n",
"together with PyTorch.\n",
"\n",
"As you will see, we only need to add some slight modifications. In particular, we\n",
"need to\n",
"\n",
"1. wrap data loading and training in functions,\n",
"2. make some network parameters configurable,\n",
"3. add checkpointing (optional),\n",
"4. and define the search space for the model tuning\n",
"\n",
"```{contents}\n",
":backlinks: none\n",
":local: true\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "7e8650d1",
"metadata": {},
"source": [
"## Setup / Imports\n",
"\n",
"First, the requirements (uncomment to install):"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "89728f69",
"metadata": {},
"outputs": [],
"source": [
"#!pip install -Uq \"ray[tune]\" torch torchvision pandas ipywidgets"
]
},
{
"cell_type": "markdown",
"id": "0cda8ce7",
"metadata": {},
"source": [
"Next, the imports:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "55529285",
"metadata": {
"tags": [
"hide-output"
]
},
"outputs": [],
"source": [
"import os\n",
"import tempfile\n",
"\n",
"import numpy as np\n",
"import torch\n",
"import torch.nn as nn\n",
"import torch.nn.functional as F\n",
"import torch.optim as optim\n",
"import torchvision\n",
"import torchvision.transforms as transforms\n",
"from filelock import FileLock\n",
"from torch.utils.data import random_split\n",
"\n",
"from ray import tune\n",
"from ray.tune.schedulers import ASHAScheduler"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "f59e551d",
"metadata": {},
"source": [
"Most of the imports are needed for building the PyTorch model. Only the last two\n",
"imports are for Ray Tune.\n",
"\n",
"## Data loaders\n",
"\n",
"We wrap the data loaders in their own function and pass a global data directory.\n",
"This way we can share a data directory between different trials."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "01471556",
"metadata": {},
"outputs": [],
"source": [
"def load_data(data_dir=\"./data\"):\n",
" \"\"\"Create dataloaders for normalized CIFAR10 training/test subsets.\"\"\"\n",
" transform = transforms.Compose([\n",
" transforms.ToTensor(),\n",
" transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),\n",
" ])\n",
"\n",
" # We add FileLock here because multiple workers will want to\n",
" # download data, and this may cause overwrites since\n",
" # DataLoader is not threadsafe.\n",
" with FileLock(os.path.expanduser(\"~/.data.lock\")):\n",
" trainset = torchvision.datasets.CIFAR10(\n",
" root=data_dir, train=True, download=True, transform=transform)\n",
"\n",
" testset = torchvision.datasets.CIFAR10(\n",
" root=data_dir, train=False, download=True, transform=transform)\n",
"\n",
" return trainset, testset\n",
"\n",
"def create_dataloaders(trainset, batch_size, num_workers=8):\n",
" \"\"\"Create train/val splits and dataloaders.\"\"\"\n",
" train_size = int(len(trainset) * 0.8)\n",
" train_subset, val_subset = random_split(\n",
" trainset, [train_size, len(trainset) - train_size])\n",
"\n",
" train_loader = torch.utils.data.DataLoader(\n",
" train_subset,\n",
" batch_size=batch_size, \n",
" shuffle=True,\n",
" num_workers=num_workers\n",
" )\n",
" val_loader = torch.utils.data.DataLoader(\n",
" val_subset,\n",
" batch_size=batch_size,\n",
" shuffle=False, \n",
" num_workers=num_workers\n",
" )\n",
" return train_loader, val_loader"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"def load_test_data():\n",
" # Load fake data for running a quick smoke-test.\n",
" trainset = torchvision.datasets.FakeData(\n",
" 128, (3, 32, 32), num_classes=10, transform=transforms.ToTensor()\n",
" )\n",
" testset = torchvision.datasets.FakeData(\n",
" 16, (3, 32, 32), num_classes=10, transform=transforms.ToTensor()\n",
" )\n",
" return trainset, testset"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "80958cf3",
"metadata": {},
"source": [
"## Configurable neural network\n",
"\n",
"We can only tune those parameters that are configurable. In this example, we can specify\n",
"the layer sizes of the fully connected layers:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "fff6bd0d",
"metadata": {},
"outputs": [],
"source": [
"class Net(nn.Module):\n",
" def __init__(self, l1=120, l2=84):\n",
" super(Net, self).__init__()\n",
" self.conv1 = nn.Conv2d(3, 6, 5)\n",
" self.pool = nn.MaxPool2d(2, 2)\n",
" self.conv2 = nn.Conv2d(6, 16, 5)\n",
" self.fc1 = nn.Linear(16 * 5 * 5, l1)\n",
" self.fc2 = nn.Linear(l1, l2)\n",
" self.fc3 = nn.Linear(l2, 10)\n",
"\n",
" def forward(self, x):\n",
" x = self.pool(F.relu(self.conv1(x)))\n",
" x = self.pool(F.relu(self.conv2(x)))\n",
" x = x.view(-1, 16 * 5 * 5)\n",
" x = F.relu(self.fc1(x))\n",
" x = F.relu(self.fc2(x))\n",
" x = self.fc3(x)\n",
" return x"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "fb619875",
"metadata": {},
"source": [
"## The train function\n",
"\n",
"Now it gets interesting, because we introduce some changes to the example [from the PyTorch\n",
"documentation](https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html).\n",
"\n",
"(communicating-with-ray-tune)=\n",
"\n",
"The full code example looks like this:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "fa0bdae0",
"metadata": {},
"outputs": [],
"source": [
"def train_cifar(config):\n",
" net = Net(config[\"l1\"], config[\"l2\"])\n",
" device = config[\"device\"]\n",
" if device == \"cuda\":\n",
" net = nn.DataParallel(net)\n",
" net.to(device)\n",
"\n",
" criterion = nn.CrossEntropyLoss()\n",
" optimizer = optim.SGD(net.parameters(), lr=config[\"lr\"], momentum=0.9, weight_decay=5e-5)\n",
"\n",
" # Load existing checkpoint through `get_checkpoint()` API.\n",
" if tune.get_checkpoint():\n",
" loaded_checkpoint = tune.get_checkpoint()\n",
" with loaded_checkpoint.as_directory() as loaded_checkpoint_dir:\n",
" model_state, optimizer_state = torch.load(\n",
" os.path.join(loaded_checkpoint_dir, \"checkpoint.pt\")\n",
" )\n",
" net.load_state_dict(model_state)\n",
" optimizer.load_state_dict(optimizer_state)\n",
"\n",
" # Data setup\n",
" if config[\"smoke_test\"]:\n",
" trainset, _ = load_test_data()\n",
" else:\n",
" trainset, _ = load_data()\n",
" train_loader, val_loader = create_dataloaders(\n",
" trainset, \n",
" config[\"batch_size\"],\n",
" num_workers=0 if config[\"smoke_test\"] else 8\n",
" )\n",
"\n",
" for epoch in range(config[\"max_num_epochs\"]): # loop over the dataset multiple times\n",
" net.train()\n",
" running_loss = 0.0\n",
" for inputs, labels in train_loader:\n",
" inputs, labels = inputs.to(device), labels.to(device)\n",
"\n",
" # forward + backward + optimize\n",
" optimizer.zero_grad() # reset gradients\n",
" outputs = net(inputs)\n",
" loss = criterion(outputs, labels)\n",
" loss.backward()\n",
" optimizer.step()\n",
" \n",
" running_loss += loss.item()\n",
"\n",
" # Validation\n",
" net.eval()\n",
" val_loss = 0.0\n",
" correct = total = 0\n",
" with torch.no_grad():\n",
" for inputs, labels in val_loader:\n",
" inputs, labels = inputs.to(device), labels.to(device)\n",
" outputs = net(inputs)\n",
" val_loss += criterion(outputs, labels).item()\n",
" _, predicted = outputs.max(1)\n",
" total += labels.size(0)\n",
" correct += predicted.eq(labels).sum().item()\n",
"\n",
" # Report metrics\n",
" metrics = {\n",
" \"loss\": val_loss / len(val_loader),\n",
" \"accuracy\": correct / total,\n",
" }\n",
"\n",
" # Here we save a checkpoint. It is automatically registered with\n",
" # Ray Tune and will potentially be accessed through in ``get_checkpoint()``\n",
" # in future iterations.\n",
" # Note to save a file-like checkpoint, you still need to put it under a directory\n",
" # to construct a checkpoint.\n",
" with tempfile.TemporaryDirectory() as temp_checkpoint_dir:\n",
" path = os.path.join(temp_checkpoint_dir, \"checkpoint.pt\")\n",
" torch.save(\n",
" (net.state_dict(), optimizer.state_dict()), path\n",
" )\n",
" checkpoint = tune.Checkpoint.from_directory(temp_checkpoint_dir)\n",
" tune.report(metrics, checkpoint=checkpoint)\n",
" print(\"Finished Training!\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "918d8baf",
"metadata": {},
"source": [
"As you can see, most of the code is adapted directly from the example.\n",
"\n",
"## Test set accuracy\n",
"\n",
"Commonly the performance of a machine learning model is tested on a hold-out test\n",
"set with data that has not been used for training the model. We also wrap this in a\n",
"function:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "93b5b4af",
"metadata": {},
"outputs": [],
"source": [
"def test_best_model(best_result, smoke_test=False):\n",
" best_trained_model = Net(best_result.config[\"l1\"], best_result.config[\"l2\"])\n",
" device = best_result.config[\"device\"]\n",
" if device == \"cuda\":\n",
" best_trained_model = nn.DataParallel(best_trained_model)\n",
" best_trained_model.to(device)\n",
"\n",
" checkpoint_path = os.path.join(best_result.checkpoint.to_directory(), \"checkpoint.pt\")\n",
"\n",
" model_state, _optimizer_state = torch.load(checkpoint_path)\n",
" best_trained_model.load_state_dict(model_state)\n",
"\n",
" if smoke_test:\n",
" _trainset, testset = load_test_data()\n",
" else:\n",
" _trainset, testset = load_data()\n",
"\n",
" testloader = torch.utils.data.DataLoader(\n",
" testset, batch_size=4, shuffle=False, num_workers=2\n",
" )\n",
"\n",
" correct = 0\n",
" total = 0\n",
" with torch.no_grad():\n",
" for data in testloader:\n",
" images, labels = data\n",
" images, labels = images.to(device), labels.to(device)\n",
" outputs = best_trained_model(images)\n",
" _, predicted = outputs.max(1)\n",
" total += labels.size(0)\n",
" correct += predicted.eq(labels).sum().item()\n",
"\n",
" print(f\"Best trial test set accuracy: {correct / total}\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "85f8230e",
"metadata": {},
"source": [
"As you can see, the function also expects a `device` parameter, so we can do the\n",
"test set validation on a GPU."
]
},
{
"cell_type": "markdown",
"id": "980720e3",
"metadata": {},
"source": [
"\n",
"## Configuring the search space\n",
"\n",
"Lastly, we need to define Tune's search space. Here is an example:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "564eab65",
"metadata": {},
"outputs": [],
"source": [
"# Set this to True for a smoke test that runs with a small synthetic dataset.\n",
"SMOKE_TEST = False"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "18d94d85",
"metadata": {
"tags": [
"hide-cell"
]
},
"outputs": [],
"source": [
"# For CI testing:\n",
"SMOKE_TEST = True"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "5416cece",
"metadata": {},
"outputs": [],
"source": [
"config = {\n",
" \"l1\": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),\n",
" \"l2\": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),\n",
" \"lr\": tune.loguniform(1e-4, 1e-1),\n",
" \"batch_size\": tune.choice([2, 4, 8, 16]),\n",
" \"smoke_test\": SMOKE_TEST,\n",
" \"num_trials\": 10 if not SMOKE_TEST else 2,\n",
" \"max_num_epochs\": 10 if not SMOKE_TEST else 2,\n",
" \"device\": \"cuda\" if torch.cuda.is_available() else \"cpu\",\n",
"}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "20af95cc",
"metadata": {},
"source": [
"The `tune.sample_from()` function makes it possible to define your own sample\n",
"methods to obtain hyperparameters. In this example, the layer sizes `l1` and `l2` \n",
"should be powers of 2 between 4 and 256, so either 4, 8, 16, 32, 64, 128, or 256.\n",
"The `lr` (learning rate) should be uniformly sampled between 0.0001 and 0.1. Lastly,\n",
"the batch size is a choice between 2, 4, 8, and 16.\n",
"\n",
"At each trial, Tune will now randomly sample a combination of parameters from these\n",
"search spaces. It will then train a number of models in parallel and find the best\n",
"performing one among these. We also use the `ASHAScheduler` which will terminate badly\n",
"performing trials early.\n",
"\n",
"You can specify the number of CPUs, which are then available e.g.\n",
"to increase the `num_workers` of the PyTorch `DataLoader` instances. The selected\n",
"number of GPUs are made visible to PyTorch in each trial. Trials do not have access to\n",
"GPUs that haven't been requested for them - so you don't have to care about two trials\n",
"using the same set of resources.\n",
"\n",
"Here we can also specify fractional GPUs, so something like `gpus_per_trial=0.5` is\n",
"completely valid. The trials will then share GPUs among each other.\n",
"You just have to make sure that the models still fit in the GPU memory.\n",
"\n",
"After training the models, we will find the best performing one and load the trained\n",
"network from the checkpoint file. We then obtain the test set accuracy and report\n",
"everything by printing.\n",
"\n",
"The full main function looks like this:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "91d83380",
"metadata": {
"tags": [
"hide-output"
]
},
"outputs": [],
"source": [
"def main(config, gpus_per_trial=1):\n",
" scheduler = ASHAScheduler(\n",
" time_attr=\"training_iteration\",\n",
" max_t=config[\"max_num_epochs\"],\n",
" grace_period=1,\n",
" reduction_factor=2)\n",
" \n",
" tuner = tune.Tuner(\n",
" tune.with_resources(\n",
" tune.with_parameters(train_cifar),\n",
" resources={\"cpu\": 2, \"gpu\": gpus_per_trial}\n",
" ),\n",
" tune_config=tune.TuneConfig(\n",
" metric=\"loss\",\n",
" mode=\"min\",\n",
" scheduler=scheduler,\n",
" num_samples=config[\"num_trials\"],\n",
" ),\n",
" param_space=config,\n",
" )\n",
" results = tuner.fit()\n",
" \n",
" best_result = results.get_best_result(\"loss\", \"min\")\n",
"\n",
" print(f\"Best trial config: {best_result.config}\")\n",
" print(f\"Best trial final validation loss: {best_result.metrics['loss']}\")\n",
" print(f\"Best trial final validation accuracy: {best_result.metrics['accuracy']}\")\n",
"\n",
" test_best_model(best_result, smoke_test=config[\"smoke_test\"])\n",
"\n",
"main(config, gpus_per_trial=1 if torch.cuda.is_available() else 0)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "b702b4ce",
"metadata": {},
"source": "If you run the code, an example output could look like this:\n\n```{code-block} bash\n:emphasize-lines: 7\n\n Number of trials: 10 (10 TERMINATED)\n +-------------------------+------------+-------+------+------+-------------+--------------+---------+------------+----------------------+\n | Trial name | status | loc | l1 | l2 | lr | batch_size | loss | accuracy | training_iteration |\n |-------------------------+------------+-------+------+------+-------------+--------------+---------+------------+----------------------|\n | train_cifar_87d1f_00000 | TERMINATED | | 64 | 4 | 0.00011629 | 2 | 1.87273 | 0.244 | 2 |\n | train_cifar_87d1f_00001 | TERMINATED | | 32 | 64 | 0.000339763 | 8 | 1.23603 | 0.567 | 8 |\n | train_cifar_87d1f_00002 | TERMINATED | | 8 | 16 | 0.00276249 | 16 | 1.1815 | 0.5836 | 10 |\n | train_cifar_87d1f_00003 | TERMINATED | | 4 | 64 | 0.000648721 | 4 | 1.31131 | 0.5224 | 8 |\n | train_cifar_87d1f_00004 | TERMINATED | | 32 | 16 | 0.000340753 | 8 | 1.26454 | 0.5444 | 8 |\n | train_cifar_87d1f_00005 | TERMINATED | | 8 | 4 | 0.000699775 | 8 | 1.99594 | 0.1983 | 2 |\n | train_cifar_87d1f_00006 | TERMINATED | | 256 | 8 | 0.0839654 | 16 | 2.3119 | 0.0993 | 1 |\n | train_cifar_87d1f_00007 | TERMINATED | | 16 | 128 | 0.0758154 | 16 | 2.33575 | 0.1327 | 1 |\n | train_cifar_87d1f_00008 | TERMINATED | | 16 | 8 | 0.0763312 | 16 | 2.31129 | 0.1042 | 4 |\n | train_cifar_87d1f_00009 | TERMINATED | | 128 | 16 | 0.000124903 | 4 | 2.26917 | 0.1945 | 1 |\n +-------------------------+------------+-------+------+------+-------------+--------------+---------+------------+----------------------+\n\n\n Best trial config: {'l1': 8, 'l2': 16, 'lr': 0.0027624906698231976, 'batch_size': 16, 'data_dir': '...'}\n Best trial final validation loss: 1.1815014744281769\n Best trial final validation accuracy: 0.5836\n Best trial test set accuracy: 0.5806\n```\n\nAs you can see, most trials have been stopped early in order to avoid wasting resources.\nThe best performing trial achieved a validation accuracy of about 58%, which could\nbe confirmed on the test set.\n\nSo that's it! You can now tune the parameters of your PyTorch models.\n\n## See More PyTorch Examples\n\n- {doc}`/tune/examples/includes/mnist_pytorch`: Converts the PyTorch MNIST example to use Tune with the function-based API.\n Also shows how to easily convert something relying on argparse to use Tune.\n- {doc}`/tune/examples/includes/pbt_convnet_function_example`: Example training a ConvNet with checkpointing in function API.\n- {doc}`/tune/examples/includes/mnist_pytorch_trainable`: Converts the PyTorch MNIST example to use Tune with Trainable API.\n Also uses the HyperBandScheduler and checkpoints the model at the end.\n- {doc}`/_collections/tune/examples/tune_pytorch_asha/README`: End-to-end tutorial with PyTorch covering search spaces, ASHA scheduling, checkpoint/resume logic, and best model evaluation on CIFAR-10."
}
],
"metadata": {
"kernelspec": {
"display_name": "tune-pytorch-cifar",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.11"
},
"orphan": true
},
"nbformat": 4,
"nbformat_minor": 5
}
doc/source/tune/examples/tune-pytorch-lightning.ipynb
+506
View File
@@ -0,0 +1,506 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"# Using PyTorch Lightning with Tune\n",
"\n",
"<a id=\"try-anyscale-quickstart-tune-pytorch-lightning\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=tune-pytorch-lightning\">\n",
" <img src=\"../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
"</a>\n",
"<br></br>\n",
"\n",
"(tune-pytorch-lightning-ref)=\n",
"\n",
"PyTorch Lightning is a framework which brings structure into training PyTorch models. It aims to avoid boilerplate code, so you don't have to write the same training loops all over again when building a new model.\n",
"\n",
"```{image} /images/pytorch_lightning_full.png\n",
":align: center\n",
"```\n",
"\n",
"The main abstraction of PyTorch Lightning is the `LightningModule` class, which should be extended by your application. There is [a great post on how to transfer your models from vanilla PyTorch to Lightning](https://towardsdatascience.com/from-pytorch-to-pytorch-lightning-a-gentle-introduction-b371b7caaf09).\n",
"\n",
"The class structure of PyTorch Lightning makes it very easy to define and tune model parameters. This tutorial will show you how to use Tune with PyTorch Lightning. Notably, the `LightningModule` does not have to be altered at all for this - so you can use it plug and play for your existing models, assuming their parameters are configurable!\n",
"\n",
":::{note}\n",
"To run this example, you will need to install the following:\n",
"\n",
"```bash\n",
"$ pip install -q \"ray[tune]\" torch torchvision lightning\n",
"```\n",
":::\n",
"\n",
"```{contents}\n",
":backlinks: none\n",
":local: true\n",
"```\n",
"\n",
"## PyTorch Lightning classifier for MNIST\n",
"\n",
"Let's first start with the basic PyTorch Lightning implementation of an MNIST classifier. This classifier does not include any tuning code at this point.\n",
"\n",
"First, we run some imports:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/home/ray/anaconda3/lib/python3.11/site-packages/lightning_utilities/core/imports.py:14: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.\n",
" import pkg_resources\n",
"/home/ray/anaconda3/lib/python3.11/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
" _torch_pytree._register_pytree_node(\n",
"/home/ray/anaconda3/lib/python3.11/site-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
" _torch_pytree._register_pytree_node(\n"
]
}
],
"source": [
"import os\n",
"import torch\n",
"import tempfile\n",
"\n",
"try:\n",
" import lightning.pytorch as pl\n",
"except ModuleNotFoundError:\n",
" import pytorch_lightning as pl\n",
"\n",
"import torch.nn.functional as F\n",
"from filelock import FileLock\n",
"from torchmetrics import Accuracy\n",
"from torch.utils.data import DataLoader, random_split\n",
"from torchvision.datasets import MNIST\n",
"from torchvision import transforms"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [],
"source": [
"# If you want to run full test, please set SMOKE_TEST to False\n",
"SMOKE_TEST = True"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Our example builds on the MNIST example from the [blog post](https://towardsdatascience.com/from-pytorch-to-pytorch-lightning-a-gentle-introduction-b371b7caaf09) we mentioned before. We adapted the original model and dataset definitions into `MNISTClassifier` and `MNISTDataModule`. "
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"class MNISTClassifier(pl.LightningModule):\n",
" def __init__(self, config):\n",
" super(MNISTClassifier, self).__init__()\n",
" self.accuracy = Accuracy(task=\"multiclass\", num_classes=10, top_k=1)\n",
" self.layer_1_size = config[\"layer_1_size\"]\n",
" self.layer_2_size = config[\"layer_2_size\"]\n",
" self.lr = config[\"lr\"]\n",
"\n",
" # mnist images are (1, 28, 28) (channels, width, height)\n",
" self.layer_1 = torch.nn.Linear(28 * 28, self.layer_1_size)\n",
" self.layer_2 = torch.nn.Linear(self.layer_1_size, self.layer_2_size)\n",
" self.layer_3 = torch.nn.Linear(self.layer_2_size, 10)\n",
" self.eval_loss = []\n",
" self.eval_accuracy = []\n",
"\n",
" def cross_entropy_loss(self, logits, labels):\n",
" return F.nll_loss(logits, labels)\n",
"\n",
" def forward(self, x):\n",
" batch_size, channels, width, height = x.size()\n",
" x = x.view(batch_size, -1)\n",
"\n",
" x = self.layer_1(x)\n",
" x = torch.relu(x)\n",
"\n",
" x = self.layer_2(x)\n",
" x = torch.relu(x)\n",
"\n",
" x = self.layer_3(x)\n",
" x = torch.log_softmax(x, dim=1)\n",
"\n",
" return x\n",
"\n",
" def training_step(self, train_batch, batch_idx):\n",
" x, y = train_batch\n",
" logits = self.forward(x)\n",
" loss = self.cross_entropy_loss(logits, y)\n",
" accuracy = self.accuracy(logits, y)\n",
"\n",
" self.log(\"ptl/train_loss\", loss)\n",
" self.log(\"ptl/train_accuracy\", accuracy)\n",
" return loss\n",
"\n",
" def validation_step(self, val_batch, batch_idx):\n",
" x, y = val_batch\n",
" logits = self.forward(x)\n",
" loss = self.cross_entropy_loss(logits, y)\n",
" accuracy = self.accuracy(logits, y)\n",
" self.eval_loss.append(loss)\n",
" self.eval_accuracy.append(accuracy)\n",
" return {\"val_loss\": loss, \"val_accuracy\": accuracy}\n",
"\n",
" def on_validation_epoch_end(self):\n",
" avg_loss = torch.stack(self.eval_loss).mean()\n",
" avg_acc = torch.stack(self.eval_accuracy).mean()\n",
" self.log(\"ptl/val_loss\", avg_loss, sync_dist=True)\n",
" self.log(\"ptl/val_accuracy\", avg_acc, sync_dist=True)\n",
" self.eval_loss.clear()\n",
" self.eval_accuracy.clear()\n",
"\n",
" def configure_optimizers(self):\n",
" optimizer = torch.optim.Adam(self.parameters(), lr=self.lr)\n",
" return optimizer\n",
"\n",
"\n",
"class MNISTDataModule(pl.LightningDataModule):\n",
" def __init__(self, batch_size=128):\n",
" super().__init__()\n",
" self.data_dir = tempfile.mkdtemp()\n",
" self.batch_size = batch_size\n",
" self.transform = transforms.Compose(\n",
" [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]\n",
" )\n",
"\n",
" def setup(self, stage=None):\n",
" with FileLock(f\"{self.data_dir}.lock\"):\n",
" mnist = MNIST(\n",
" self.data_dir, train=True, download=True, transform=self.transform\n",
" )\n",
" self.mnist_train, self.mnist_val = random_split(mnist, [55000, 5000])\n",
"\n",
" self.mnist_test = MNIST(\n",
" self.data_dir, train=False, download=True, transform=self.transform\n",
" )\n",
"\n",
" def train_dataloader(self):\n",
" return DataLoader(self.mnist_train, batch_size=self.batch_size, num_workers=4)\n",
"\n",
" def val_dataloader(self):\n",
" return DataLoader(self.mnist_val, batch_size=self.batch_size, num_workers=4)\n",
"\n",
" def test_dataloader(self):\n",
" return DataLoader(self.mnist_test, batch_size=self.batch_size, num_workers=4)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"default_config = {\n",
" \"layer_1_size\": 128,\n",
" \"layer_2_size\": 256,\n",
" \"lr\": 1e-3,\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Define a training function that creates model, `DataModule`, and the PyTorch Lightning `Trainer`."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"from ray.tune.integration.pytorch_lightning import TuneReportCheckpointCallback\n",
"\n",
"def train_func(config):\n",
" dm = MNISTDataModule(batch_size=config[\"batch_size\"])\n",
" model = MNISTClassifier(config)\n",
"\n",
" trainer = pl.Trainer(\n",
" devices=\"auto\",\n",
" accelerator=\"auto\",\n",
" callbacks=[TuneReportCheckpointCallback()],\n",
" enable_progress_bar=False,\n",
" )\n",
" trainer.fit(model, datamodule=dm)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Tuning the model parameters\n",
"\n",
"The parameters above should give you a good accuracy of over 90% already. However, we might improve on this simply by changing some of the hyperparameters. For instance, maybe we get an even higher accuracy if we used a smaller learning rate and larger middle layer size.\n",
"\n",
"Instead of manually loop through all the parameter combinitions, let's use Tune to systematically try out parameter combinations and find the best performing set.\n",
"\n",
"First, we need some additional imports:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"from ray import tune\n",
"from ray.tune.schedulers import ASHAScheduler"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Configuring the search space\n",
"\n",
"Now we configure the parameter search space. We would like to choose between different layer dimensions, learning rate, and batch sizes. The learning rate should be sampled uniformly between `0.0001` and `0.1`. The `tune.loguniform()` function is syntactic sugar to make sampling between these different orders of magnitude easier, specifically we are able to also sample small values. Similarly for `tune.choice()`, which samples from all the provided options."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"search_space = {\n",
" \"layer_1_size\": tune.choice([32, 64, 128]),\n",
" \"layer_2_size\": tune.choice([64, 128, 256]),\n",
" \"lr\": tune.loguniform(1e-4, 1e-1),\n",
" \"batch_size\": tune.choice([32, 64]),\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Selecting a scheduler\n",
"\n",
"In this example, we use an [Asynchronous Hyperband](https://blog.ml.cmu.edu/2018/12/12/massively-parallel-hyperparameter-optimization/)\n",
"scheduler. This scheduler decides at each iteration which trials are likely to perform\n",
"badly, and stops these trials. This way we don't waste any resources on bad hyperparameter\n",
"configurations."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"# The maximum training epochs\n",
"num_epochs = 5\n",
"\n",
"# Number of samples from parameter space\n",
"num_samples = 10"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"If you have more resources available, you can modify the above parameters accordingly. e.g. more epochs, more parameter samples."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [],
"source": [
"if SMOKE_TEST:\n",
" num_epochs = 1\n",
" num_samples = 3"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"scheduler = ASHAScheduler(max_t=num_epochs, grace_period=1, reduction_factor=2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Training with GPUs\n",
"\n",
"We can specify the number of resources, including GPUs, that Tune should request for each trial."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"train_fn_with_resources = tune.with_resources(train_func, resources={\"CPU\": 1, \"GPU\": 1})"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [],
"source": [
"if SMOKE_TEST:\n",
" train_fn_with_resources = tune.with_resources(train_func, resources={\"CPU\": 1})\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Putting it together\n",
"\n",
"Lastly, we need to create a `Tuner()` object and start Ray Tune with `tuner.fit()`. The full code looks like this:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": [
"hide-output"
]
},
"outputs": [],
"source": [
"def tune_mnist_asha(num_samples=10):\n",
" scheduler = ASHAScheduler(max_t=num_epochs, grace_period=1, reduction_factor=2)\n",
"\n",
" tuner = tune.Tuner(\n",
" train_fn_with_resources,\n",
" param_space=search_space,\n",
" tune_config=tune.TuneConfig(\n",
" metric=\"ptl/val_accuracy\",\n",
" mode=\"max\",\n",
" num_samples=num_samples,\n",
" scheduler=scheduler,\n",
" ),\n",
" run_config=tune.RunConfig(\n",
" checkpoint_config=tune.CheckpointConfig(\n",
" num_to_keep=2,\n",
" checkpoint_score_attribute=\"ptl/val_accuracy\",\n",
" checkpoint_score_order=\"max\",\n",
" ),\n",
" ),\n",
" )\n",
" return tuner.fit()\n",
"\n",
"\n",
"results = tune_mnist_asha(num_samples=num_samples)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"text/plain": [
"Result(\n",
" metrics={'ptl/train_loss': 0.001267582061700523, 'ptl/train_accuracy': 1.0, 'ptl/val_loss': 0.1036270260810852, 'ptl/val_accuracy': 0.9721123576164246},\n",
" path='/home/ray/ray_results/train_func_2025-09-23_13-37-55/train_func_2f534_00006_6_batch_size=64,layer_1_size=64,layer_2_size=64,lr=0.0020_2025-09-23_13-37-55',\n",
" filesystem='local',\n",
" checkpoint=Checkpoint(filesystem=local, path=/home/ray/ray_results/train_func_2025-09-23_13-37-55/train_func_2f534_00006_6_batch_size=64,layer_1_size=64,layer_2_size=64,lr=0.0020_2025-09-23_13-37-55/checkpoint_000004)\n",
")"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"results.get_best_result(metric=\"ptl/val_accuracy\", mode=\"max\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the example above, Tune runs 10 trials with different hyperparameter configurations.\n",
"\n",
"As you can see in the `training_iteration` column, trials with a high loss (and low accuracy) have been terminated early. The best performing trial used\n",
"`batch_size=64`, `layer_1_size=128`, `layer_2_size=256`, and `lr=0.0037`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## More PyTorch Lightning Examples\n",
"\n",
"- For running distributed PyTorch Lightning training with Ray Train, see the {ref}`quickstart <train-pytorch-lightning>`.\n",
"- {doc}`[Basic] Train a PyTorch Lightning Image Classifier with Ray Train <../../train/examples/lightning/lightning_mnist_example>`.\n",
"- {doc}`[Intermediate] Fine-tune a BERT Text Classifier with PyTorch Lightning and Ray Train <../../train/examples/lightning/lightning_cola_advanced>`\n",
"- {doc}`[Advanced] Fine-tune dolly-v2-7b with PyTorch Lightning and FSDP <../../train/examples/lightning/dolly_lightning_fsdp_finetuning>`\n",
"- {doc}`/tune/examples/includes/mlflow_ptl_example`: Example for using [MLflow](https://github.com/mlflow/mlflow/)\n",
" and [Pytorch Lightning](https://github.com/PyTorchLightning/pytorch-lightning) with Ray Tune.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.11"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
doc/source/tune/examples/tune-vanilla-pytorch-lightning.ipynb
+845
View File
@@ -0,0 +1,845 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "aa6af4d3",
"metadata": {},
"source": [
"# Using PyTorch Lightning with Tune\n",
"\n",
"<a id=\"try-anyscale-quickstart-tune-vanilla-pytorch-lightning\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=tune-vanilla-pytorch-lightning\">\n",
" <img src=\"../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
"</a>\n",
"<br></br>\n",
"\n",
"(tune-vanilla-pytorch-lightning-ref)=\n",
"\n",
"PyTorch Lightning is a framework which brings structure into training PyTorch models. It\n",
"aims to avoid boilerplate code, so you don't have to write the same training\n",
"loops all over again when building a new model.\n",
"\n",
"```{image} /images/pytorch_lightning_full.png\n",
":align: center\n",
"```\n",
"\n",
"The main abstraction of PyTorch Lightning is the `LightningModule` class, which\n",
"should be extended by your application. There is [a great post on how to transfer your models from vanilla PyTorch to Lightning](https://towardsdatascience.com/from-pytorch-to-pytorch-lightning-a-gentle-introduction-b371b7caaf09).\n",
"\n",
"The class structure of PyTorch Lightning makes it very easy to define and tune model\n",
"parameters. This tutorial will show you how to use Tune to find the best set of\n",
"parameters for your application on the example of training a MNIST classifier. Notably,\n",
"the `LightningModule` does not have to be altered at all for this - so you can\n",
"use it plug and play for your existing models, assuming their parameters are configurable!\n",
"\n",
":::{note}\n",
"To run this example, you will need to install the following:\n",
"\n",
"```bash\n",
"$ pip install \"ray[tune]\" torch torchvision lightning\n",
"```\n",
":::\n",
"\n",
"```{contents}\n",
":backlinks: none\n",
":local: true\n",
"```\n",
"\n",
"## PyTorch Lightning classifier for MNIST\n",
"\n",
"Let's first start with the basic PyTorch Lightning implementation of an MNIST classifier.\n",
"This classifier does not include any tuning code at this point.\n",
"\n",
"Our example builds on the MNIST example from the [blog post we talked about\n",
"earlier](https://towardsdatascience.com/from-pytorch-to-pytorch-lightning-a-gentle-introduction-b371b7caaf09).\n",
"\n",
"First, we run some imports:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "e6e77570",
"metadata": {},
"outputs": [],
"source": [
"import math\n",
"\n",
"import torch\n",
"\n",
"try:\n",
" import lightning.pytorch as pl\n",
"except ModuleNotFoundError:\n",
" import pytorch_lightning as pl\n",
"\n",
"from filelock import FileLock\n",
"from torch.utils.data import DataLoader, random_split\n",
"from torch.nn import functional as F\n",
"from torchvision.datasets import MNIST\n",
"from torchvision import transforms\n",
"import os\n"
]
},
{
"cell_type": "markdown",
"id": "3c442e73",
"metadata": {},
"source": [
"And then there is the Lightning model adapted from the blog post.\n",
"Note that we left out the test set validation and made the model parameters\n",
"configurable through a `config` dict that is passed on initialization.\n",
"Also, we specify a `data_dir` where the MNIST data will be stored. Note that\n",
"we use a `FileLock` for downloading data so that the dataset is only downloaded\n",
"once per node.\n",
"Lastly, we added a new metric, the validation accuracy, to the logs."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "48b20f48",
"metadata": {},
"outputs": [],
"source": [
"class LightningMNISTClassifier(pl.LightningModule):\n",
" \"\"\"\n",
" This has been adapted from\n",
" https://towardsdatascience.com/from-pytorch-to-pytorch-lightning-a-gentle-introduction-b371b7caaf09\n",
" \"\"\"\n",
"\n",
" def __init__(self, config, data_dir=None):\n",
" super(LightningMNISTClassifier, self).__init__()\n",
"\n",
" self.data_dir = data_dir or os.getcwd()\n",
"\n",
" self.layer_1_size = config[\"layer_1_size\"]\n",
" self.layer_2_size = config[\"layer_2_size\"]\n",
" self.lr = config[\"lr\"]\n",
" self.batch_size = config[\"batch_size\"]\n",
"\n",
" # mnist images are (1, 28, 28) (channels, width, height)\n",
" self.layer_1 = torch.nn.Linear(28 * 28, self.layer_1_size)\n",
" self.layer_2 = torch.nn.Linear(self.layer_1_size, self.layer_2_size)\n",
" self.layer_3 = torch.nn.Linear(self.layer_2_size, 10)\n",
"\n",
" def forward(self, x):\n",
" batch_size, channels, width, height = x.size()\n",
" x = x.view(batch_size, -1)\n",
"\n",
" x = self.layer_1(x)\n",
" x = torch.relu(x)\n",
"\n",
" x = self.layer_2(x)\n",
" x = torch.relu(x)\n",
"\n",
" x = self.layer_3(x)\n",
" x = torch.log_softmax(x, dim=1)\n",
"\n",
" return x\n",
"\n",
" def cross_entropy_loss(self, logits, labels):\n",
" return F.nll_loss(logits, labels)\n",
"\n",
" def accuracy(self, logits, labels):\n",
" _, predicted = torch.max(logits.data, 1)\n",
" correct = (predicted == labels).sum().item()\n",
" accuracy = correct / len(labels)\n",
" return torch.tensor(accuracy)\n",
"\n",
" def training_step(self, train_batch, batch_idx):\n",
" x, y = train_batch\n",
" logits = self.forward(x)\n",
" loss = self.cross_entropy_loss(logits, y)\n",
" accuracy = self.accuracy(logits, y)\n",
"\n",
" self.log(\"ptl/train_loss\", loss)\n",
" self.log(\"ptl/train_accuracy\", accuracy)\n",
" return loss\n",
"\n",
" def validation_step(self, val_batch, batch_idx):\n",
" x, y = val_batch\n",
" logits = self.forward(x)\n",
" loss = self.cross_entropy_loss(logits, y)\n",
" accuracy = self.accuracy(logits, y)\n",
" return {\"val_loss\": loss, \"val_accuracy\": accuracy}\n",
"\n",
" def validation_epoch_end(self, outputs):\n",
" avg_loss = torch.stack([x[\"val_loss\"] for x in outputs]).mean()\n",
" avg_acc = torch.stack([x[\"val_accuracy\"] for x in outputs]).mean()\n",
" self.log(\"ptl/val_loss\", avg_loss)\n",
" self.log(\"ptl/val_accuracy\", avg_acc)\n",
"\n",
" @staticmethod\n",
" def download_data(data_dir):\n",
" transform = transforms.Compose(\n",
" [transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]\n",
" )\n",
" with FileLock(os.path.expanduser(\"~/.data.lock\")):\n",
" return MNIST(data_dir, train=True, download=True, transform=transform)\n",
"\n",
" def prepare_data(self):\n",
" mnist_train = self.download_data(self.data_dir)\n",
"\n",
" self.mnist_train, self.mnist_val = random_split(mnist_train, [55000, 5000])\n",
"\n",
" def train_dataloader(self):\n",
" return DataLoader(self.mnist_train, batch_size=int(self.batch_size))\n",
"\n",
" def val_dataloader(self):\n",
" return DataLoader(self.mnist_val, batch_size=int(self.batch_size))\n",
"\n",
" def configure_optimizers(self):\n",
" optimizer = torch.optim.Adam(self.parameters(), lr=self.lr)\n",
" return optimizer\n",
"\n",
"\n",
"def train_mnist(config):\n",
" model = LightningMNISTClassifier(config)\n",
" trainer = pl.Trainer(max_epochs=10, enable_progress_bar=False)\n",
"\n",
" trainer.fit(model)\n"
]
},
{
"cell_type": "markdown",
"id": "da1c3632",
"metadata": {},
"source": [
"And that's it! You can now run `train_mnist(config)` to train the classifier, e.g.\n",
"like so:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "86df3d39",
"metadata": {},
"outputs": [],
"source": [
"def train_mnist_no_tune():\n",
" config = {\"layer_1_size\": 128, \"layer_2_size\": 256, \"lr\": 1e-3, \"batch_size\": 64}\n",
" train_mnist(config)\n"
]
},
{
"cell_type": "markdown",
"id": "edcc0991",
"metadata": {},
"source": [
"## Tuning the model parameters\n",
"\n",
"The parameters above should give you a good accuracy of over 90% already. However,\n",
"we might improve on this simply by changing some of the hyperparameters. For instance,\n",
"maybe we get an even higher accuracy if we used a larger batch size.\n",
"\n",
"Instead of guessing the parameter values, let's use Tune to systematically try out\n",
"parameter combinations and find the best performing set.\n",
"\n",
"First, we need some additional imports:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "34faeb3b",
"metadata": {},
"outputs": [],
"source": [
"try:\n",
" from lightning.pytorch.loggers import TensorBoardLogger\n",
"except ModuleNotFoundError:\n",
" from pytorch_lightning.loggers import TensorBoardLogger\n",
"\n",
"from ray import tune\n",
"from ray.tune import CLIReporter\n",
"from ray.tune.schedulers import ASHAScheduler, PopulationBasedTraining\n",
"from ray.tune.integration.pytorch_lightning import (\n",
" TuneReportCallback,\n",
" TuneReportCheckpointCallback,\n",
")\n"
]
},
{
"cell_type": "markdown",
"id": "f65b9c5f",
"metadata": {},
"source": [
"### Talking to Tune with a PyTorch Lightning callback\n",
"\n",
"PyTorch Lightning introduced [Callbacks](https://pytorch-lightning.readthedocs.io/en/latest/extensions/callbacks.html)\n",
"that can be used to plug custom functions into the training loop. This way the original\n",
"`LightningModule` does not have to be altered at all. Also, we could use the same\n",
"callback for multiple modules.\n",
"\n",
"Ray Tune comes with ready-to-use PyTorch Lightning callbacks. To report metrics\n",
"back to Tune after each validation epoch, we will use the `TuneReportCallback`:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "4bab80bc",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<ray.tune.integration.pytorch_lightning.TuneReportCallback at 0x17b305710>"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"TuneReportCallback(\n",
" {\"loss\": \"ptl/val_loss\", \"mean_accuracy\": \"ptl/val_accuracy\"}, on=\"validation_end\"\n",
")\n"
]
},
{
"cell_type": "markdown",
"id": "286a1070",
"metadata": {},
"source": [
"This callback will take the `val_loss` and `val_accuracy` values\n",
"from the PyTorch Lightning trainer and report them to Tune as the `loss`\n",
"and `mean_accuracy`, respectively.\n",
"\n",
"### Adding the Tune training function\n",
"\n",
"Then we specify our training function. Note that we added the `data_dir` as a\n",
"parameter here to avoid\n",
"that each training run downloads the full MNIST dataset. Instead, we want to access\n",
"a shared data location.\n",
"\n",
"We are also able to specify the number of epochs to train each model, and the number\n",
"of GPUs we want to use for training. We also create a TensorBoard logger that writes\n",
"logfiles directly into Tune's root trial directory - if we didn't do that PyTorch\n",
"Lightning would create subdirectories, and each trial would thus be shown twice in\n",
"TensorBoard, one time for Tune's logs, and another time for PyTorch Lightning's logs."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "74e7d1c2",
"metadata": {},
"outputs": [],
"source": [
"def train_mnist_tune(config, num_epochs=10, num_gpus=0, data_dir=\"~/data\"):\n",
" data_dir = os.path.expanduser(data_dir)\n",
" model = LightningMNISTClassifier(config, data_dir)\n",
" trainer = pl.Trainer(\n",
" max_epochs=num_epochs,\n",
" # If fractional GPUs passed in, convert to int.\n",
" gpus=math.ceil(num_gpus),\n",
" logger=TensorBoardLogger(save_dir=os.getcwd(), name=\"\", version=\".\"),\n",
" enable_progress_bar=False,\n",
" callbacks=[\n",
" TuneReportCallback(\n",
" {\"loss\": \"ptl/val_loss\", \"mean_accuracy\": \"ptl/val_accuracy\"},\n",
" on=\"validation_end\",\n",
" )\n",
" ],\n",
" )\n",
" trainer.fit(model)\n"
]
},
{
"cell_type": "markdown",
"id": "cf0f6d6e",
"metadata": {},
"source": [
"### Configuring the search space\n",
"\n",
"Now we configure the parameter search space. We would like to choose between three\n",
"different layer and batch sizes. The learning rate should be sampled uniformly between\n",
"`0.0001` and `0.1`. The `tune.loguniform()` function is syntactic sugar to make\n",
"sampling between these different orders of magnitude easier, specifically\n",
"we are able to also sample small values."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "a50645e9",
"metadata": {},
"outputs": [],
"source": [
"config = {\n",
" \"layer_1_size\": tune.choice([32, 64, 128]),\n",
" \"layer_2_size\": tune.choice([64, 128, 256]),\n",
" \"lr\": tune.loguniform(1e-4, 1e-1),\n",
" \"batch_size\": tune.choice([32, 64, 128]),\n",
"}\n"
]
},
{
"cell_type": "markdown",
"id": "b1fb9ecd",
"metadata": {},
"source": [
"### Selecting a scheduler\n",
"\n",
"In this example, we use an [Asynchronous Hyperband](https://blog.ml.cmu.edu/2018/12/12/massively-parallel-hyperparameter-optimization/)\n",
"scheduler. This scheduler decides at each iteration which trials are likely to perform\n",
"badly, and stops these trials. This way we don't waste any resources on bad hyperparameter\n",
"configurations."
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "a2596b01",
"metadata": {},
"outputs": [],
"source": [
"num_epochs = 10\n",
"\n",
"scheduler = ASHAScheduler(max_t=num_epochs, grace_period=1, reduction_factor=2)\n"
]
},
{
"cell_type": "markdown",
"id": "9a49ae58",
"metadata": {},
"source": [
"### Changing the CLI output\n",
"\n",
"We instantiate a `CLIReporter` to specify which metrics we would like to see in our\n",
"output tables in the command line. This is optional, but can be used to make sure our\n",
"output tables only include information we would like to see."
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "cd605a16",
"metadata": {},
"outputs": [],
"source": [
"reporter = CLIReporter(\n",
" parameter_columns=[\"layer_1_size\", \"layer_2_size\", \"lr\", \"batch_size\"],\n",
" metric_columns=[\"loss\", \"mean_accuracy\", \"training_iteration\"],\n",
")\n"
]
},
{
"cell_type": "markdown",
"id": "5ec9a305",
"metadata": {},
"source": [
"### Passing constants to the train function\n",
"\n",
"The `data_dir`, `num_epochs` and `num_gpus` we pass to the training function\n",
"are constants. To avoid including them as non-configurable parameters in the `config`\n",
"specification, we can use `tune.with_parameters` to wrap around the training function."
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "332668dc",
"metadata": {},
"outputs": [],
"source": [
"gpus_per_trial = 0\n",
"data_dir = \"~/data\"\n",
"\n",
"train_fn_with_parameters = tune.with_parameters(\n",
" train_mnist_tune, num_epochs=num_epochs, num_gpus=gpus_per_trial, data_dir=data_dir\n",
")\n"
]
},
{
"cell_type": "markdown",
"id": "feef8c39",
"metadata": {},
"source": [
"### Training with GPUs\n",
"\n",
"We can specify how many resources Tune should request for each trial.\n",
"This also includes GPUs.\n",
"\n",
"PyTorch Lightning takes care of moving the training to the GPUs. We\n",
"already made sure that our code is compatible with that, so there's\n",
"nothing more to do here other than to specify the number of GPUs\n",
"we would like to use:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "dc402716",
"metadata": {},
"outputs": [],
"source": [
"resources_per_trial = {\"cpu\": 1, \"gpu\": gpus_per_trial}\n"
]
},
{
"cell_type": "markdown",
"id": "ca050dfa",
"metadata": {},
"source": [
"You can also specify {ref}`fractional GPUs for Tune <tune-parallelism>`,\n",
"allowing multiple trials to share GPUs and thus increase concurrency under resource constraints.\n",
"While the `gpus_per_trial` passed into\n",
"Tune is a decimal value, the `gpus` passed into the `pl.Trainer` should still be an integer.\n",
"Please note that if using fractional GPUs, it is the user's responsibility to\n",
"make sure multiple trials can share GPUs and there is enough memory to do so.\n",
"Ray does not automatically handle this for you.\n",
"\n",
"If you want to use multiple GPUs per trial, you should check out {ref}`Getting Start with Lightning and Ray TorchTrainer <train-pytorch-lightning>`.\n",
"\n",
"### Putting it together\n",
"\n",
"Lastly, we need to create a `Tuner()` object and start Ray Tune with `tuner.fit()`.\n",
"\n",
"The full code looks like this:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "ea182330",
"metadata": {},
"outputs": [],
"source": [
"def tune_mnist_asha(num_samples=10, num_epochs=10, gpus_per_trial=0, data_dir=\"~/data\"):\n",
" config = {\n",
" \"layer_1_size\": tune.choice([32, 64, 128]),\n",
" \"layer_2_size\": tune.choice([64, 128, 256]),\n",
" \"lr\": tune.loguniform(1e-4, 1e-1),\n",
" \"batch_size\": tune.choice([32, 64, 128]),\n",
" }\n",
"\n",
" scheduler = ASHAScheduler(max_t=num_epochs, grace_period=1, reduction_factor=2)\n",
"\n",
" reporter = CLIReporter(\n",
" parameter_columns=[\"layer_1_size\", \"layer_2_size\", \"lr\", \"batch_size\"],\n",
" metric_columns=[\"loss\", \"mean_accuracy\", \"training_iteration\"],\n",
" )\n",
"\n",
" train_fn_with_parameters = tune.with_parameters(\n",
" train_mnist_tune,\n",
" num_epochs=num_epochs,\n",
" num_gpus=gpus_per_trial,\n",
" data_dir=data_dir,\n",
" )\n",
" resources_per_trial = {\"cpu\": 1, \"gpu\": gpus_per_trial}\n",
"\n",
" tuner = tune.Tuner(\n",
" tune.with_resources(train_fn_with_parameters, resources=resources_per_trial),\n",
" tune_config=tune.TuneConfig(\n",
" metric=\"loss\",\n",
" mode=\"min\",\n",
" scheduler=scheduler,\n",
" num_samples=num_samples,\n",
" ),\n",
" run_config=tune.RunConfig(\n",
" name=\"tune_mnist_asha\",\n",
" progress_reporter=reporter,\n",
" ),\n",
" param_space=config,\n",
" )\n",
" results = tuner.fit()\n",
"\n",
" print(\"Best hyperparameters found were: \", results.get_best_result().config)\n"
]
},
{
"cell_type": "markdown",
"id": "1fb96b6c",
"metadata": {},
"source": [
"In the example above, Tune runs 10 trials with different hyperparameter configurations.\n",
"An example output could look like so:\n",
"\n",
"```{code-block} bash\n",
":emphasize-lines: 12\n",
"\n",
" +------------------------------+------------+-------+----------------+----------------+-------------+--------------+----------+-----------------+----------------------+\n",
" | Trial name | status | loc | layer_1_size | layer_2_size | lr | batch_size | loss | mean_accuracy | training_iteration |\n",
" |------------------------------+------------+-------+----------------+----------------+-------------+--------------+----------+-----------------+----------------------|\n",
" | train_mnist_tune_63ecc_00000 | TERMINATED | | 128 | 64 | 0.00121197 | 128 | 0.120173 | 0.972461 | 10 |\n",
" | train_mnist_tune_63ecc_00001 | TERMINATED | | 64 | 128 | 0.0301395 | 128 | 0.454836 | 0.868164 | 4 |\n",
" | train_mnist_tune_63ecc_00002 | TERMINATED | | 64 | 128 | 0.0432097 | 128 | 0.718396 | 0.718359 | 1 |\n",
" | train_mnist_tune_63ecc_00003 | TERMINATED | | 32 | 128 | 0.000294669 | 32 | 0.111475 | 0.965764 | 10 |\n",
" | train_mnist_tune_63ecc_00004 | TERMINATED | | 32 | 256 | 0.000386664 | 64 | 0.133538 | 0.960839 | 8 |\n",
" | train_mnist_tune_63ecc_00005 | TERMINATED | | 128 | 128 | 0.0837395 | 32 | 2.32628 | 0.0991242 | 1 |\n",
" | train_mnist_tune_63ecc_00006 | TERMINATED | | 64 | 128 | 0.000158761 | 128 | 0.134595 | 0.959766 | 10 |\n",
" | train_mnist_tune_63ecc_00007 | TERMINATED | | 64 | 64 | 0.000672126 | 64 | 0.118182 | 0.972903 | 10 |\n",
" | train_mnist_tune_63ecc_00008 | TERMINATED | | 128 | 64 | 0.000502428 | 32 | 0.11082 | 0.975518 | 10 |\n",
" | train_mnist_tune_63ecc_00009 | TERMINATED | | 64 | 256 | 0.00112894 | 32 | 0.13472 | 0.971935 | 8 |\n",
" +------------------------------+------------+-------+----------------+----------------+-------------+--------------+----------+-----------------+----------------------+\n",
"```\n",
"\n",
"As you can see in the `training_iteration` column, trials with a high loss\n",
"(and low accuracy) have been terminated early. The best performing trial used\n",
"`layer_1_size=128`, `layer_2_size=64`, `lr=0.000502428` and\n",
"`batch_size=32`.\n",
"\n",
"## Using Population Based Training to find the best parameters\n",
"\n",
"The `ASHAScheduler` terminates those trials early that show bad performance.\n",
"Sometimes, this stops trials that would get better after more training steps,\n",
"and which might eventually even show better performance than other configurations.\n",
"\n",
"Another popular method for hyperparameter tuning, called\n",
"[Population Based Training](https://deepmind.com/blog/article/population-based-training-neural-networks),\n",
"instead perturbs hyperparameters during the training run. Tune implements PBT, and\n",
"we only need to make some slight adjustments to our code.\n",
"\n",
"### Adding checkpoints to the PyTorch Lightning module\n",
"\n",
"First, we need to introduce\n",
"another callback to save model checkpoints. Since Tune requires a call to\n",
"`tune.report()` after creating a new checkpoint to register it, we will use\n",
"a combined reporting and checkpointing callback:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "7f86e4d8",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<ray.tune.integration.pytorch_lightning.TuneReportCheckpointCallback at 0x17a626090>"
]
},
"execution_count": null,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"TuneReportCheckpointCallback(\n",
" metrics={\"loss\": \"ptl/val_loss\", \"mean_accuracy\": \"ptl/val_accuracy\"},\n",
" filename=\"checkpoint\",\n",
" on=\"validation_end\",\n",
")\n"
]
},
{
"cell_type": "markdown",
"id": "33a76d5b",
"metadata": {},
"source": [
"The `checkpoint` value is the name of the checkpoint file within the\n",
"checkpoint directory.\n",
"\n",
"We also include checkpoint loading in our training function:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "746e962a",
"metadata": {},
"outputs": [],
"source": [
"def train_mnist_tune_checkpoint(config, num_epochs=10, num_gpus=0, data_dir=\"~/data\"):\n",
" data_dir = os.path.expanduser(data_dir)\n",
" kwargs = {\n",
" \"max_epochs\": num_epochs,\n",
" # If fractional GPUs passed in, convert to int.\n",
" \"gpus\": math.ceil(num_gpus),\n",
" \"logger\": TensorBoardLogger(save_dir=os.getcwd(), name=\"\", version=\".\"),\n",
" \"enable_progress_bar\": False,\n",
" \"callbacks\": [\n",
" TuneReportCheckpointCallback(\n",
" metrics={\"loss\": \"ptl/val_loss\", \"mean_accuracy\": \"ptl/val_accuracy\"},\n",
" filename=\"checkpoint\",\n",
" on=\"validation_end\",\n",
" )\n",
" ],\n",
" }\n",
"\n",
" checkpoint = tune.get_checkpoint()\n",
" if checkpoint:\n",
" with checkpoint.as_directory() as checkpoint_dir:\n",
" kwargs[\"resume_from_checkpoint\"] = os.path.join(checkpoint_dir, \"checkpoint\")\n",
"\n",
" model = LightningMNISTClassifier(config=config, data_dir=data_dir)\n",
" trainer = pl.Trainer(**kwargs)\n",
"\n",
" trainer.fit(model)\n"
]
},
{
"cell_type": "markdown",
"id": "39dc7b46",
"metadata": {},
"source": [
"### Configuring and running Population Based Training\n",
"\n",
"We need to call Tune slightly differently:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "e12a1bd5",
"metadata": {},
"outputs": [],
"source": [
"def tune_mnist_pbt(num_samples=10, num_epochs=10, gpus_per_trial=0, data_dir=\"~/data\"):\n",
" config = {\n",
" \"layer_1_size\": tune.choice([32, 64, 128]),\n",
" \"layer_2_size\": tune.choice([64, 128, 256]),\n",
" \"lr\": 1e-3,\n",
" \"batch_size\": 64,\n",
" }\n",
"\n",
" scheduler = PopulationBasedTraining(\n",
" perturbation_interval=4,\n",
" hyperparam_mutations={\n",
" \"lr\": tune.loguniform(1e-4, 1e-1),\n",
" \"batch_size\": [32, 64, 128],\n",
" },\n",
" )\n",
"\n",
" reporter = CLIReporter(\n",
" parameter_columns=[\"layer_1_size\", \"layer_2_size\", \"lr\", \"batch_size\"],\n",
" metric_columns=[\"loss\", \"mean_accuracy\", \"training_iteration\"],\n",
" )\n",
"\n",
" tuner = tune.Tuner(\n",
" tune.with_resources(\n",
" tune.with_parameters(\n",
" train_mnist_tune_checkpoint,\n",
" num_epochs=num_epochs,\n",
" num_gpus=gpus_per_trial,\n",
" data_dir=data_dir,\n",
" ),\n",
" resources={\"cpu\": 1, \"gpu\": gpus_per_trial},\n",
" ),\n",
" tune_config=tune.TuneConfig(\n",
" metric=\"loss\",\n",
" mode=\"min\",\n",
" scheduler=scheduler,\n",
" num_samples=num_samples,\n",
" ),\n",
" run_config=tune.RunConfig(\n",
" name=\"tune_mnist_asha\",\n",
" progress_reporter=reporter,\n",
" ),\n",
" param_space=config,\n",
" )\n",
" results = tuner.fit()\n",
"\n",
" print(\"Best hyperparameters found were: \", results.get_best_result().config)\n"
]
},
{
"cell_type": "markdown",
"id": "6087f807",
"metadata": {},
"source": [
"Instead of passing tune parameters to the `config` dict, we start\n",
"with fixed values, though we are also able to sample some of them, like the\n",
"layer sizes. Additionally, we have to tell PBT how to perturb the hyperparameters.\n",
"Note that the layer sizes are not tuned right here. This is because we cannot simply\n",
"change layer sizes during a training run - which is what would happen in PBT.\n",
"\n",
"To test running both of our main scripts (`tune_mnist_asha` and `tune_mnist_pbt`), all you have to do is specify\n",
"a `data_dir` folder and run the scripts with reasonable parameters:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "eb9faf3e",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"data_dir = \"~/data/\"\n",
"\n",
"tune_mnist_asha(num_samples=1, num_epochs=6, gpus_per_trial=0, data_dir=data_dir)\n",
"tune_mnist_pbt(num_samples=1, num_epochs=6, gpus_per_trial=0, data_dir=data_dir)\n"
]
},
{
"cell_type": "markdown",
"id": "3ae0eea6",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"If you have more resources available (e.g. a GPU), you can modify the above parameters accordingly.\n",
"\n",
"An example output of a run could look like this:\n",
"\n",
"```bash\n",
"+-----------------------------------------+------------+-------+----------------+----------------+-----------+--------------+-----------+-----------------+----------------------+\n",
"| Trial name | status | loc | layer_1_size | layer_2_size | lr | batch_size | loss | mean_accuracy | training_iteration |\n",
"|-----------------------------------------+------------+-------+----------------+----------------+-----------+--------------+-----------+-----------------+----------------------|\n",
"| train_mnist_tune_checkpoint_85489_00000 | TERMINATED | | 128 | 128 | 0.001 | 64 | 0.108734 | 0.973101 | 10 |\n",
"| train_mnist_tune_checkpoint_85489_00001 | TERMINATED | | 128 | 128 | 0.001 | 64 | 0.093577 | 0.978639 | 10 |\n",
"| train_mnist_tune_checkpoint_85489_00002 | TERMINATED | | 128 | 256 | 0.0008 | 32 | 0.0922348 | 0.979299 | 10 |\n",
"| train_mnist_tune_checkpoint_85489_00003 | TERMINATED | | 64 | 256 | 0.001 | 64 | 0.124648 | 0.973892 | 10 |\n",
"| train_mnist_tune_checkpoint_85489_00004 | TERMINATED | | 128 | 64 | 0.001 | 64 | 0.101717 | 0.975079 | 10 |\n",
"| train_mnist_tune_checkpoint_85489_00005 | TERMINATED | | 64 | 64 | 0.001 | 64 | 0.121467 | 0.969146 | 10 |\n",
"| train_mnist_tune_checkpoint_85489_00006 | TERMINATED | | 128 | 256 | 0.00064 | 32 | 0.053446 | 0.987062 | 10 |\n",
"| train_mnist_tune_checkpoint_85489_00007 | TERMINATED | | 128 | 256 | 0.001 | 64 | 0.129804 | 0.973497 | 10 |\n",
"| train_mnist_tune_checkpoint_85489_00008 | TERMINATED | | 64 | 256 | 0.0285125 | 128 | 0.363236 | 0.913867 | 10 |\n",
"| train_mnist_tune_checkpoint_85489_00009 | TERMINATED | | 32 | 256 | 0.001 | 64 | 0.150946 | 0.964201 | 10 |\n",
"+-----------------------------------------+------------+-------+----------------+----------------+-----------+--------------+-----------+-----------------+----------------------+\n",
"```\n",
"\n",
"As you can see, each sample ran the full number of 10 iterations.\n",
"All trials ended with quite good parameter combinations and showed relatively good performances.\n",
"In some runs, the parameters have been perturbed. And the best configuration even reached a\n",
"mean validation accuracy of `0.987062`!\n",
"\n",
"In summary, PyTorch Lightning Modules are easy to extend to use with Tune. It just took\n",
"us importing one or two callbacks and a small wrapper function to get great performing\n",
"parameter configurations.\n",
"\n",
"## More PyTorch Lightning Examples\n",
"\n",
"- {doc}`/tune/examples/includes/mnist_ptl_mini`:\n",
" A minimal example of using [Pytorch Lightning](https://github.com/PyTorchLightning/pytorch-lightning)\n",
" to train a MNIST model. This example utilizes the Ray Tune-provided\n",
" {ref}`PyTorch Lightning callbacks <tune-integration-pytorch-lightning>`.\n",
" See also {ref}`this tutorial for a full walkthrough <tune-pytorch-lightning-ref>`.\n",
"- {ref}`A walkthrough tutorial for using Ray Tune with Pytorch-Lightning <tune-pytorch-lightning-ref>`.\n",
"- {doc}`/tune/examples/includes/mlflow_ptl_example`: Example for using [MLflow](https://github.com/mlflow/mlflow/)\n",
" and [Pytorch Lightning](https://github.com/PyTorchLightning/pytorch-lightning) with Ray Tune."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.16"
},
"orphan": true
},
"nbformat": 4,
"nbformat_minor": 5
}
doc/source/tune/examples/tune-wandb.ipynb
+790
View File
@@ -0,0 +1,790 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "ecad719c",
"metadata": {},
"source": [
"# Using Weights & Biases with Tune\n",
"\n",
"<a id=\"try-anyscale-quickstart-tune-wandb\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=tune-wandb\">\n",
" <img src=\"../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
"</a>\n",
"<br></br>\n",
"\n",
"(tune-wandb-ref)=\n",
"\n",
"[Weights & Biases](https://www.wandb.ai/) (Wandb) is a tool for experiment\n",
"tracking, model optimizaton, and dataset versioning. It is very popular\n",
"in the machine learning and data science community for its superb visualization\n",
"tools.\n",
"\n",
"```{image} /images/wandb_logo_full.png\n",
":align: center\n",
":alt: Weights & Biases\n",
":height: 80px\n",
":target: https://www.wandb.ai/\n",
"```\n",
"\n",
"Ray Tune currently offers two lightweight integrations for Weights & Biases.\n",
"One is the {ref}`WandbLoggerCallback <air-wandb-logger>`, which automatically logs\n",
"metrics reported to Tune to the Wandb API.\n",
"\n",
"The other one is the {ref}`setup_wandb() <air-wandb-setup>` function, which can be\n",
"used with the function API. It automatically\n",
"initializes the Wandb API with Tune's training information. You can just use the\n",
"Wandb API like you would normally do, e.g. using `wandb.log()` to log your training\n",
"process.\n",
"\n",
"```{contents}\n",
":backlinks: none\n",
":local: true\n",
"```\n",
"\n",
"## Running A Weights & Biases Example\n",
"\n",
"In the following example we're going to use both of the above methods, namely the `WandbLoggerCallback` and\n",
"the `setup_wandb` function to log metrics.\n",
"\n",
"As the very first step, make sure you're logged in into wandb on all machines you're running your training on:\n",
"\n",
" wandb login\n",
"\n",
"We can then start with a few crucial imports:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "100bcf8a",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"import ray\n",
"from ray import tune\n",
"from ray.air.integrations.wandb import WandbLoggerCallback, setup_wandb\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "9346c0f6",
"metadata": {},
"source": [
"Next, let's define an easy `train_function` function (a Tune `Trainable`) that reports a random loss to Tune.\n",
"The objective function itself is not important for this example, since we want to focus on the Weights & Biases\n",
"integration primarily."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "e8b4fc4d",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"def train_function(config):\n",
" for i in range(30):\n",
" loss = config[\"mean\"] + config[\"sd\"] * np.random.randn()\n",
" tune.report({\"loss\": loss})\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "831eed42",
"metadata": {},
"source": [
"You can define a\n",
"simple grid-search Tune run using the `WandbLoggerCallback` as follows:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "52988599",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"def tune_with_callback():\n",
" \"\"\"Example for using a WandbLoggerCallback with the function API\"\"\"\n",
" tuner = tune.Tuner(\n",
" train_function,\n",
" tune_config=tune.TuneConfig(\n",
" metric=\"loss\",\n",
" mode=\"min\",\n",
" ),\n",
" run_config=tune.RunConfig(\n",
" callbacks=[WandbLoggerCallback(project=\"Wandb_example\")]\n",
" ),\n",
" param_space={\n",
" \"mean\": tune.grid_search([1, 2, 3, 4, 5]),\n",
" \"sd\": tune.uniform(0.2, 0.8),\n",
" },\n",
" )\n",
" tuner.fit()\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "e24c05fa",
"metadata": {},
"source": [
"To use the `setup_wandb` utility, you simply call this function in your objective.\n",
"Note that we also use `wandb.log(...)` to log the `loss` to Weights & Biases as a dictionary.\n",
"Otherwise, this version of our objective is identical to its original."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "5e30d5e7",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"def train_function_wandb(config):\n",
" wandb = setup_wandb(config, project=\"Wandb_example\")\n",
"\n",
" for i in range(30):\n",
" loss = config[\"mean\"] + config[\"sd\"] * np.random.randn()\n",
" tune.report({\"loss\": loss})\n",
" wandb.log(dict(loss=loss))\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "04040bcb",
"metadata": {},
"source": [
"With the `train_function_wandb` defined, your Tune experiment will set up `wandb` in each trial once it starts!"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "d4fbd368",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"def tune_with_setup():\n",
" \"\"\"Example for using the setup_wandb utility with the function API\"\"\"\n",
" tuner = tune.Tuner(\n",
" train_function_wandb,\n",
" tune_config=tune.TuneConfig(\n",
" metric=\"loss\",\n",
" mode=\"min\",\n",
" ),\n",
" param_space={\n",
" \"mean\": tune.grid_search([1, 2, 3, 4, 5]),\n",
" \"sd\": tune.uniform(0.2, 0.8),\n",
" },\n",
" )\n",
" tuner.fit()\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "f9521481",
"metadata": {},
"source": [
"Finally, you can also define a class-based Tune `Trainable` by using the `setup_wandb` in the `setup()` method and storing the run object as an attribute. Please note that with the class trainable, you have to pass the trial id, name, and group separately:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "d27a7a35",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"class WandbTrainable(tune.Trainable):\n",
" def setup(self, config):\n",
" self.wandb = setup_wandb(\n",
" config,\n",
" trial_id=self.trial_id,\n",
" trial_name=self.trial_name,\n",
" group=\"Example\",\n",
" project=\"Wandb_example\",\n",
" )\n",
"\n",
" def step(self):\n",
" for i in range(30):\n",
" loss = self.config[\"mean\"] + self.config[\"sd\"] * np.random.randn()\n",
" self.wandb.log({\"loss\": loss})\n",
" return {\"loss\": loss, \"done\": True}\n",
"\n",
" def save_checkpoint(self, checkpoint_dir: str):\n",
" pass\n",
"\n",
" def load_checkpoint(self, checkpoint_dir: str):\n",
" pass\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "fa189bb2",
"metadata": {},
"source": [
"Running Tune with this `WandbTrainable` works exactly the same as with the function API.\n",
"The below `tune_trainable` function differs from `tune_decorated` above only in the first argument we pass to\n",
"`Tuner()`:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "6e546cc2",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"def tune_trainable():\n",
" \"\"\"Example for using a WandTrainableMixin with the class API\"\"\"\n",
" tuner = tune.Tuner(\n",
" WandbTrainable,\n",
" tune_config=tune.TuneConfig(\n",
" metric=\"loss\",\n",
" mode=\"min\",\n",
" ),\n",
" param_space={\n",
" \"mean\": tune.grid_search([1, 2, 3, 4, 5]),\n",
" \"sd\": tune.uniform(0.2, 0.8),\n",
" },\n",
" )\n",
" results = tuner.fit()\n",
"\n",
" return results.get_best_result().config\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "0b736172",
"metadata": {},
"source": [
"Since you may not have an API key for Wandb, we can _mock_ the Wandb logger and test all three of our training\n",
"functions as follows.\n",
"If you are logged in into wandb, you can set `mock_api = False` to actually upload your results to Weights & Biases."
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "e0e7f481",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2022-11-02 16:02:45,355\tINFO worker.py:1534 -- Started a local Ray instance. View the dashboard at \u001b[1m\u001b[32mhttp://127.0.0.1:8266 \u001b[39m\u001b[22m\n",
"2022-11-02 16:02:46,513\tINFO wandb.py:282 -- Already logged into W&B.\n"
]
},
{
"data": {
"text/html": [
"<div class=\"tuneStatus\">\n",
" <div style=\"display: flex;flex-direction: row\">\n",
" <div style=\"display: flex;flex-direction: column;\">\n",
" <h3>Tune Status</h3>\n",
" <table>\n",
"<tbody>\n",
"<tr><td>Current time:</td><td>2022-11-02 16:03:13</td></tr>\n",
"<tr><td>Running for: </td><td>00:00:27.28 </td></tr>\n",
"<tr><td>Memory: </td><td>10.8/16.0 GiB </td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
" <div class=\"vDivider\"></div>\n",
" <div class=\"systemInfo\">\n",
" <h3>System Info</h3>\n",
" Using FIFO scheduling algorithm.<br>Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/3.44 GiB heap, 0.0/1.72 GiB objects\n",
" </div>\n",
" \n",
" </div>\n",
" <div class=\"hDivider\"></div>\n",
" <div class=\"trialStatus\">\n",
" <h3>Trial Status</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>status </th><th>loc </th><th style=\"text-align: right;\"> mean</th><th style=\"text-align: right;\"> sd</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> loss</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>train_function_7676d_00000</td><td>TERMINATED</td><td>127.0.0.1:14578</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\">0.411212</td><td style=\"text-align: right;\"> 30</td><td style=\"text-align: right;\"> 0.236137</td><td style=\"text-align: right;\">0.828527</td></tr>\n",
"<tr><td>train_function_7676d_00001</td><td>TERMINATED</td><td>127.0.0.1:14591</td><td style=\"text-align: right;\"> 2</td><td style=\"text-align: right;\">0.756339</td><td style=\"text-align: right;\"> 30</td><td style=\"text-align: right;\"> 5.57185 </td><td style=\"text-align: right;\">3.13156 </td></tr>\n",
"<tr><td>train_function_7676d_00002</td><td>TERMINATED</td><td>127.0.0.1:14593</td><td style=\"text-align: right;\"> 3</td><td style=\"text-align: right;\">0.436643</td><td style=\"text-align: right;\"> 30</td><td style=\"text-align: right;\"> 5.50237 </td><td style=\"text-align: right;\">3.26679 </td></tr>\n",
"<tr><td>train_function_7676d_00003</td><td>TERMINATED</td><td>127.0.0.1:14595</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\">0.295929</td><td style=\"text-align: right;\"> 30</td><td style=\"text-align: right;\"> 5.60986 </td><td style=\"text-align: right;\">3.70388 </td></tr>\n",
"<tr><td>train_function_7676d_00004</td><td>TERMINATED</td><td>127.0.0.1:14596</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\">0.335292</td><td style=\"text-align: right;\"> 30</td><td style=\"text-align: right;\"> 5.61385 </td><td style=\"text-align: right;\">4.74294 </td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
"</div>\n",
"<style>\n",
".tuneStatus {\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".tuneStatus .systemInfo {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus td {\n",
" white-space: nowrap;\n",
"}\n",
".tuneStatus .trialStatus {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus h3 {\n",
" font-weight: bold;\n",
"}\n",
".tuneStatus .hDivider {\n",
" border-bottom-width: var(--jp-border-width);\n",
" border-bottom-color: var(--jp-border-color0);\n",
" border-bottom-style: solid;\n",
"}\n",
".tuneStatus .vDivider {\n",
" border-left-width: var(--jp-border-width);\n",
" border-left-color: var(--jp-border-color0);\n",
" border-left-style: solid;\n",
" margin: 0.5em 1em 0.5em 1em;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<div class=\"trialProgress\">\n",
" <h3>Trial Progress</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>date </th><th>done </th><th>episodes_total </th><th>experiment_id </th><th>experiment_tag </th><th>hostname </th><th style=\"text-align: right;\"> iterations_since_restore</th><th style=\"text-align: right;\"> loss</th><th>node_ip </th><th style=\"text-align: right;\"> pid</th><th style=\"text-align: right;\"> time_since_restore</th><th style=\"text-align: right;\"> time_this_iter_s</th><th style=\"text-align: right;\"> time_total_s</th><th style=\"text-align: right;\"> timestamp</th><th style=\"text-align: right;\"> timesteps_since_restore</th><th>timesteps_total </th><th style=\"text-align: right;\"> training_iteration</th><th>trial_id </th><th style=\"text-align: right;\"> warmup_time</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>train_function_7676d_00000</td><td>2022-11-02_16-02-53</td><td>True </td><td> </td><td>a9f242fa70184d9dadd8952b16fb0ecc</td><td>0_mean=1,sd=0.4112</td><td>Kais-MBP.local.meter</td><td style=\"text-align: right;\"> 30</td><td style=\"text-align: right;\">0.828527</td><td>127.0.0.1</td><td style=\"text-align: right;\">14578</td><td style=\"text-align: right;\"> 0.236137</td><td style=\"text-align: right;\"> 0.00381589</td><td style=\"text-align: right;\"> 0.236137</td><td style=\"text-align: right;\"> 1667430173</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 30</td><td>7676d_00000</td><td style=\"text-align: right;\"> 0.00366998</td></tr>\n",
"<tr><td>train_function_7676d_00001</td><td>2022-11-02_16-03-03</td><td>True </td><td> </td><td>f57118365bcb4c229fe41c5911f05ad6</td><td>1_mean=2,sd=0.7563</td><td>Kais-MBP.local.meter</td><td style=\"text-align: right;\"> 30</td><td style=\"text-align: right;\">3.13156 </td><td>127.0.0.1</td><td style=\"text-align: right;\">14591</td><td style=\"text-align: right;\"> 5.57185 </td><td style=\"text-align: right;\"> 0.00627518</td><td style=\"text-align: right;\"> 5.57185 </td><td style=\"text-align: right;\"> 1667430183</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 30</td><td>7676d_00001</td><td style=\"text-align: right;\"> 0.0027349 </td></tr>\n",
"<tr><td>train_function_7676d_00002</td><td>2022-11-02_16-03-03</td><td>True </td><td> </td><td>394021d4515d4616bae7126668f73b2b</td><td>2_mean=3,sd=0.4366</td><td>Kais-MBP.local.meter</td><td style=\"text-align: right;\"> 30</td><td style=\"text-align: right;\">3.26679 </td><td>127.0.0.1</td><td style=\"text-align: right;\">14593</td><td style=\"text-align: right;\"> 5.50237 </td><td style=\"text-align: right;\"> 0.00494576</td><td style=\"text-align: right;\"> 5.50237 </td><td style=\"text-align: right;\"> 1667430183</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 30</td><td>7676d_00002</td><td style=\"text-align: right;\"> 0.00286222</td></tr>\n",
"<tr><td>train_function_7676d_00003</td><td>2022-11-02_16-03-03</td><td>True </td><td> </td><td>a575e79c9d95485fa37deaa86267aea4</td><td>3_mean=4,sd=0.2959</td><td>Kais-MBP.local.meter</td><td style=\"text-align: right;\"> 30</td><td style=\"text-align: right;\">3.70388 </td><td>127.0.0.1</td><td style=\"text-align: right;\">14595</td><td style=\"text-align: right;\"> 5.60986 </td><td style=\"text-align: right;\"> 0.00689816</td><td style=\"text-align: right;\"> 5.60986 </td><td style=\"text-align: right;\"> 1667430183</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 30</td><td>7676d_00003</td><td style=\"text-align: right;\"> 0.00299597</td></tr>\n",
"<tr><td>train_function_7676d_00004</td><td>2022-11-02_16-03-03</td><td>True </td><td> </td><td>91ce57dcdbb54536b1874666b711350d</td><td>4_mean=5,sd=0.3353</td><td>Kais-MBP.local.meter</td><td style=\"text-align: right;\"> 30</td><td style=\"text-align: right;\">4.74294 </td><td>127.0.0.1</td><td style=\"text-align: right;\">14596</td><td style=\"text-align: right;\"> 5.61385 </td><td style=\"text-align: right;\"> 0.00672579</td><td style=\"text-align: right;\"> 5.61385 </td><td style=\"text-align: right;\"> 1667430183</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 30</td><td>7676d_00004</td><td style=\"text-align: right;\"> 0.00323987</td></tr>\n",
"</tbody>\n",
"</table>\n",
"</div>\n",
"<style>\n",
".trialProgress {\n",
" display: flex;\n",
" flex-direction: column;\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".trialProgress h3 {\n",
" font-weight: bold;\n",
"}\n",
".trialProgress td {\n",
" white-space: nowrap;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2022-11-02 16:03:13,913\tINFO tune.py:788 -- Total run time: 28.53 seconds (27.28 seconds for the tuning loop).\n"
]
},
{
"data": {
"text/html": [
"<div class=\"tuneStatus\">\n",
" <div style=\"display: flex;flex-direction: row\">\n",
" <div style=\"display: flex;flex-direction: column;\">\n",
" <h3>Tune Status</h3>\n",
" <table>\n",
"<tbody>\n",
"<tr><td>Current time:</td><td>2022-11-02 16:03:22</td></tr>\n",
"<tr><td>Running for: </td><td>00:00:08.49 </td></tr>\n",
"<tr><td>Memory: </td><td>9.9/16.0 GiB </td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
" <div class=\"vDivider\"></div>\n",
" <div class=\"systemInfo\">\n",
" <h3>System Info</h3>\n",
" Using FIFO scheduling algorithm.<br>Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/3.44 GiB heap, 0.0/1.72 GiB objects\n",
" </div>\n",
" \n",
" </div>\n",
" <div class=\"hDivider\"></div>\n",
" <div class=\"trialStatus\">\n",
" <h3>Trial Status</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>status </th><th>loc </th><th style=\"text-align: right;\"> mean</th><th style=\"text-align: right;\"> sd</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> loss</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>train_function_wandb_877eb_00000</td><td>TERMINATED</td><td>127.0.0.1:14647</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\">0.738281</td><td style=\"text-align: right;\"> 30</td><td style=\"text-align: right;\"> 1.61319</td><td style=\"text-align: right;\">0.555153</td></tr>\n",
"<tr><td>train_function_wandb_877eb_00001</td><td>TERMINATED</td><td>127.0.0.1:14660</td><td style=\"text-align: right;\"> 2</td><td style=\"text-align: right;\">0.321178</td><td style=\"text-align: right;\"> 30</td><td style=\"text-align: right;\"> 1.72447</td><td style=\"text-align: right;\">2.52109 </td></tr>\n",
"<tr><td>train_function_wandb_877eb_00002</td><td>TERMINATED</td><td>127.0.0.1:14661</td><td style=\"text-align: right;\"> 3</td><td style=\"text-align: right;\">0.202487</td><td style=\"text-align: right;\"> 30</td><td style=\"text-align: right;\"> 1.8159 </td><td style=\"text-align: right;\">2.45412 </td></tr>\n",
"<tr><td>train_function_wandb_877eb_00003</td><td>TERMINATED</td><td>127.0.0.1:14662</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\">0.515434</td><td style=\"text-align: right;\"> 30</td><td style=\"text-align: right;\"> 1.715 </td><td style=\"text-align: right;\">4.51413 </td></tr>\n",
"<tr><td>train_function_wandb_877eb_00004</td><td>TERMINATED</td><td>127.0.0.1:14663</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\">0.216098</td><td style=\"text-align: right;\"> 30</td><td style=\"text-align: right;\"> 1.72827</td><td style=\"text-align: right;\">5.2814 </td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
"</div>\n",
"<style>\n",
".tuneStatus {\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".tuneStatus .systemInfo {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus td {\n",
" white-space: nowrap;\n",
"}\n",
".tuneStatus .trialStatus {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus h3 {\n",
" font-weight: bold;\n",
"}\n",
".tuneStatus .hDivider {\n",
" border-bottom-width: var(--jp-border-width);\n",
" border-bottom-color: var(--jp-border-color0);\n",
" border-bottom-style: solid;\n",
"}\n",
".tuneStatus .vDivider {\n",
" border-left-width: var(--jp-border-width);\n",
" border-left-color: var(--jp-border-color0);\n",
" border-left-style: solid;\n",
" margin: 0.5em 1em 0.5em 1em;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[2m\u001b[36m(train_function_wandb pid=14647)\u001b[0m 2022-11-02 16:03:17,149\tINFO wandb.py:282 -- Already logged into W&B.\n"
]
},
{
"data": {
"text/html": [
"<div class=\"trialProgress\">\n",
" <h3>Trial Progress</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>date </th><th>done </th><th>episodes_total </th><th>experiment_id </th><th>experiment_tag </th><th>hostname </th><th style=\"text-align: right;\"> iterations_since_restore</th><th style=\"text-align: right;\"> loss</th><th>node_ip </th><th style=\"text-align: right;\"> pid</th><th style=\"text-align: right;\"> time_since_restore</th><th style=\"text-align: right;\"> time_this_iter_s</th><th style=\"text-align: right;\"> time_total_s</th><th style=\"text-align: right;\"> timestamp</th><th style=\"text-align: right;\"> timesteps_since_restore</th><th>timesteps_total </th><th style=\"text-align: right;\"> training_iteration</th><th>trial_id </th><th style=\"text-align: right;\"> warmup_time</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>train_function_wandb_877eb_00000</td><td>2022-11-02_16-03-18</td><td>True </td><td> </td><td>7b250c9f31ab484dad1a1fd29823afdf</td><td>0_mean=1,sd=0.7383</td><td>Kais-MBP.local.meter</td><td style=\"text-align: right;\"> 30</td><td style=\"text-align: right;\">0.555153</td><td>127.0.0.1</td><td style=\"text-align: right;\">14647</td><td style=\"text-align: right;\"> 1.61319</td><td style=\"text-align: right;\"> 0.00232315</td><td style=\"text-align: right;\"> 1.61319</td><td style=\"text-align: right;\"> 1667430198</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 30</td><td>877eb_00000</td><td style=\"text-align: right;\"> 0.00391102</td></tr>\n",
"<tr><td>train_function_wandb_877eb_00001</td><td>2022-11-02_16-03-22</td><td>True </td><td> </td><td>5172868368074557a3044ea3a9146673</td><td>1_mean=2,sd=0.3212</td><td>Kais-MBP.local.meter</td><td style=\"text-align: right;\"> 30</td><td style=\"text-align: right;\">2.52109 </td><td>127.0.0.1</td><td style=\"text-align: right;\">14660</td><td style=\"text-align: right;\"> 1.72447</td><td style=\"text-align: right;\"> 0.0152011 </td><td style=\"text-align: right;\"> 1.72447</td><td style=\"text-align: right;\"> 1667430202</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 30</td><td>877eb_00001</td><td style=\"text-align: right;\"> 0.00901699</td></tr>\n",
"<tr><td>train_function_wandb_877eb_00002</td><td>2022-11-02_16-03-22</td><td>True </td><td> </td><td>b13d9bccb1964b4b95e1a858a3ea64c7</td><td>2_mean=3,sd=0.2025</td><td>Kais-MBP.local.meter</td><td style=\"text-align: right;\"> 30</td><td style=\"text-align: right;\">2.45412 </td><td>127.0.0.1</td><td style=\"text-align: right;\">14661</td><td style=\"text-align: right;\"> 1.8159 </td><td style=\"text-align: right;\"> 0.00437403</td><td style=\"text-align: right;\"> 1.8159 </td><td style=\"text-align: right;\"> 1667430202</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 30</td><td>877eb_00002</td><td style=\"text-align: right;\"> 0.00844812</td></tr>\n",
"<tr><td>train_function_wandb_877eb_00003</td><td>2022-11-02_16-03-22</td><td>True </td><td> </td><td>869d7ec7a3544a8387985103e626818f</td><td>3_mean=4,sd=0.5154</td><td>Kais-MBP.local.meter</td><td style=\"text-align: right;\"> 30</td><td style=\"text-align: right;\">4.51413 </td><td>127.0.0.1</td><td style=\"text-align: right;\">14662</td><td style=\"text-align: right;\"> 1.715 </td><td style=\"text-align: right;\"> 0.00247812</td><td style=\"text-align: right;\"> 1.715 </td><td style=\"text-align: right;\"> 1667430202</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 30</td><td>877eb_00003</td><td style=\"text-align: right;\"> 0.00282907</td></tr>\n",
"<tr><td>train_function_wandb_877eb_00004</td><td>2022-11-02_16-03-22</td><td>True </td><td> </td><td>84d3112d66f64325bc469e44b8447ef5</td><td>4_mean=5,sd=0.2161</td><td>Kais-MBP.local.meter</td><td style=\"text-align: right;\"> 30</td><td style=\"text-align: right;\">5.2814 </td><td>127.0.0.1</td><td style=\"text-align: right;\">14663</td><td style=\"text-align: right;\"> 1.72827</td><td style=\"text-align: right;\"> 0.00517201</td><td style=\"text-align: right;\"> 1.72827</td><td style=\"text-align: right;\"> 1667430202</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 30</td><td>877eb_00004</td><td style=\"text-align: right;\"> 0.00272107</td></tr>\n",
"</tbody>\n",
"</table>\n",
"</div>\n",
"<style>\n",
".trialProgress {\n",
" display: flex;\n",
" flex-direction: column;\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".trialProgress h3 {\n",
" font-weight: bold;\n",
"}\n",
".trialProgress td {\n",
" white-space: nowrap;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[2m\u001b[36m(train_function_wandb pid=14660)\u001b[0m 2022-11-02 16:03:20,600\tINFO wandb.py:282 -- Already logged into W&B.\n",
"\u001b[2m\u001b[36m(train_function_wandb pid=14661)\u001b[0m 2022-11-02 16:03:20,600\tINFO wandb.py:282 -- Already logged into W&B.\n",
"\u001b[2m\u001b[36m(train_function_wandb pid=14663)\u001b[0m 2022-11-02 16:03:20,628\tINFO wandb.py:282 -- Already logged into W&B.\n",
"\u001b[2m\u001b[36m(train_function_wandb pid=14662)\u001b[0m 2022-11-02 16:03:20,723\tINFO wandb.py:282 -- Already logged into W&B.\n",
"2022-11-02 16:03:22,565\tINFO tune.py:788 -- Total run time: 8.60 seconds (8.48 seconds for the tuning loop).\n"
]
},
{
"data": {
"text/html": [
"<div class=\"tuneStatus\">\n",
" <div style=\"display: flex;flex-direction: row\">\n",
" <div style=\"display: flex;flex-direction: column;\">\n",
" <h3>Tune Status</h3>\n",
" <table>\n",
"<tbody>\n",
"<tr><td>Current time:</td><td>2022-11-02 16:03:31</td></tr>\n",
"<tr><td>Running for: </td><td>00:00:09.28 </td></tr>\n",
"<tr><td>Memory: </td><td>9.9/16.0 GiB </td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
" <div class=\"vDivider\"></div>\n",
" <div class=\"systemInfo\">\n",
" <h3>System Info</h3>\n",
" Using FIFO scheduling algorithm.<br>Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/3.44 GiB heap, 0.0/1.72 GiB objects\n",
" </div>\n",
" \n",
" </div>\n",
" <div class=\"hDivider\"></div>\n",
" <div class=\"trialStatus\">\n",
" <h3>Trial Status</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>status </th><th>loc </th><th style=\"text-align: right;\"> mean</th><th style=\"text-align: right;\"> sd</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> loss</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>WandbTrainable_8ca33_00000</td><td>TERMINATED</td><td>127.0.0.1:14718</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\">0.397894</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\"> 0.000187159</td><td style=\"text-align: right;\">0.742345</td></tr>\n",
"<tr><td>WandbTrainable_8ca33_00001</td><td>TERMINATED</td><td>127.0.0.1:14737</td><td style=\"text-align: right;\"> 2</td><td style=\"text-align: right;\">0.386883</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\"> 0.000151873</td><td style=\"text-align: right;\">2.5709 </td></tr>\n",
"<tr><td>WandbTrainable_8ca33_00002</td><td>TERMINATED</td><td>127.0.0.1:14738</td><td style=\"text-align: right;\"> 3</td><td style=\"text-align: right;\">0.290693</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\"> 0.00014019 </td><td style=\"text-align: right;\">2.99601 </td></tr>\n",
"<tr><td>WandbTrainable_8ca33_00003</td><td>TERMINATED</td><td>127.0.0.1:14739</td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\">0.33333 </td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\"> 0.00015831 </td><td style=\"text-align: right;\">3.91276 </td></tr>\n",
"<tr><td>WandbTrainable_8ca33_00004</td><td>TERMINATED</td><td>127.0.0.1:14740</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\">0.645479</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\"> 0.000150919</td><td style=\"text-align: right;\">5.47779 </td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
"</div>\n",
"<style>\n",
".tuneStatus {\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".tuneStatus .systemInfo {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus td {\n",
" white-space: nowrap;\n",
"}\n",
".tuneStatus .trialStatus {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus h3 {\n",
" font-weight: bold;\n",
"}\n",
".tuneStatus .hDivider {\n",
" border-bottom-width: var(--jp-border-width);\n",
" border-bottom-color: var(--jp-border-color0);\n",
" border-bottom-style: solid;\n",
"}\n",
".tuneStatus .vDivider {\n",
" border-left-width: var(--jp-border-width);\n",
" border-left-color: var(--jp-border-color0);\n",
" border-left-style: solid;\n",
" margin: 0.5em 1em 0.5em 1em;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[2m\u001b[36m(WandbTrainable pid=14718)\u001b[0m 2022-11-02 16:03:25,742\tINFO wandb.py:282 -- Already logged into W&B.\n"
]
},
{
"data": {
"text/html": [
"<div class=\"trialProgress\">\n",
" <h3>Trial Progress</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>date </th><th>done </th><th>episodes_total </th><th>experiment_id </th><th>hostname </th><th style=\"text-align: right;\"> iterations_since_restore</th><th style=\"text-align: right;\"> loss</th><th>node_ip </th><th style=\"text-align: right;\"> pid</th><th style=\"text-align: right;\"> time_since_restore</th><th style=\"text-align: right;\"> time_this_iter_s</th><th style=\"text-align: right;\"> time_total_s</th><th style=\"text-align: right;\"> timestamp</th><th style=\"text-align: right;\"> timesteps_since_restore</th><th>timesteps_total </th><th style=\"text-align: right;\"> training_iteration</th><th>trial_id </th><th style=\"text-align: right;\"> warmup_time</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>WandbTrainable_8ca33_00000</td><td>2022-11-02_16-03-27</td><td>True </td><td> </td><td>3adb4d0ae0d74d1c9ddd07924b5653b0</td><td>Kais-MBP.local.meter</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\">0.742345</td><td>127.0.0.1</td><td style=\"text-align: right;\">14718</td><td style=\"text-align: right;\"> 0.000187159</td><td style=\"text-align: right;\"> 0.000187159</td><td style=\"text-align: right;\"> 0.000187159</td><td style=\"text-align: right;\"> 1667430207</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 1</td><td>8ca33_00000</td><td style=\"text-align: right;\"> 1.31382</td></tr>\n",
"<tr><td>WandbTrainable_8ca33_00001</td><td>2022-11-02_16-03-31</td><td>True </td><td> </td><td>f1511cfd51f94b3d9cf192181ccc08a9</td><td>Kais-MBP.local.meter</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\">2.5709 </td><td>127.0.0.1</td><td style=\"text-align: right;\">14737</td><td style=\"text-align: right;\"> 0.000151873</td><td style=\"text-align: right;\"> 0.000151873</td><td style=\"text-align: right;\"> 0.000151873</td><td style=\"text-align: right;\"> 1667430211</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 1</td><td>8ca33_00001</td><td style=\"text-align: right;\"> 1.31668</td></tr>\n",
"<tr><td>WandbTrainable_8ca33_00002</td><td>2022-11-02_16-03-31</td><td>True </td><td> </td><td>a7528ec6adf74de0b73aa98ebedab66d</td><td>Kais-MBP.local.meter</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\">2.99601 </td><td>127.0.0.1</td><td style=\"text-align: right;\">14738</td><td style=\"text-align: right;\"> 0.00014019 </td><td style=\"text-align: right;\"> 0.00014019 </td><td style=\"text-align: right;\"> 0.00014019 </td><td style=\"text-align: right;\"> 1667430211</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 1</td><td>8ca33_00002</td><td style=\"text-align: right;\"> 1.32008</td></tr>\n",
"<tr><td>WandbTrainable_8ca33_00003</td><td>2022-11-02_16-03-31</td><td>True </td><td> </td><td>b7af756ca586449ba2d4c44141b53b06</td><td>Kais-MBP.local.meter</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\">3.91276 </td><td>127.0.0.1</td><td style=\"text-align: right;\">14739</td><td style=\"text-align: right;\"> 0.00015831 </td><td style=\"text-align: right;\"> 0.00015831 </td><td style=\"text-align: right;\"> 0.00015831 </td><td style=\"text-align: right;\"> 1667430211</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 1</td><td>8ca33_00003</td><td style=\"text-align: right;\"> 1.31879</td></tr>\n",
"<tr><td>WandbTrainable_8ca33_00004</td><td>2022-11-02_16-03-31</td><td>True </td><td> </td><td>196624f42bcc45c18a26778573a43a2c</td><td>Kais-MBP.local.meter</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\">5.47779 </td><td>127.0.0.1</td><td style=\"text-align: right;\">14740</td><td style=\"text-align: right;\"> 0.000150919</td><td style=\"text-align: right;\"> 0.000150919</td><td style=\"text-align: right;\"> 0.000150919</td><td style=\"text-align: right;\"> 1667430211</td><td style=\"text-align: right;\"> 0</td><td> </td><td style=\"text-align: right;\"> 1</td><td>8ca33_00004</td><td style=\"text-align: right;\"> 1.31945</td></tr>\n",
"</tbody>\n",
"</table>\n",
"</div>\n",
"<style>\n",
".trialProgress {\n",
" display: flex;\n",
" flex-direction: column;\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".trialProgress h3 {\n",
" font-weight: bold;\n",
"}\n",
".trialProgress td {\n",
" white-space: nowrap;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[2m\u001b[36m(WandbTrainable pid=14739)\u001b[0m 2022-11-02 16:03:30,360\tINFO wandb.py:282 -- Already logged into W&B.\n",
"\u001b[2m\u001b[36m(WandbTrainable pid=14740)\u001b[0m 2022-11-02 16:03:30,393\tINFO wandb.py:282 -- Already logged into W&B.\n",
"\u001b[2m\u001b[36m(WandbTrainable pid=14737)\u001b[0m 2022-11-02 16:03:30,454\tINFO wandb.py:282 -- Already logged into W&B.\n",
"\u001b[2m\u001b[36m(WandbTrainable pid=14738)\u001b[0m 2022-11-02 16:03:30,510\tINFO wandb.py:282 -- Already logged into W&B.\n",
"2022-11-02 16:03:31,985\tINFO tune.py:788 -- Total run time: 9.40 seconds (9.27 seconds for the tuning loop).\n"
]
},
{
"data": {
"text/plain": [
"{'mean': 1, 'sd': 0.3978937765393781, 'wandb': {'project': 'Wandb_example'}}"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import os\n",
"\n",
"mock_api = True\n",
"\n",
"if mock_api:\n",
" os.environ.setdefault(\"WANDB_MODE\", \"disabled\")\n",
" os.environ.setdefault(\"WANDB_API_KEY\", \"abcd\")\n",
" ray.init(\n",
" runtime_env={\"env_vars\": {\"WANDB_MODE\": \"disabled\", \"WANDB_API_KEY\": \"abcd\"}}\n",
" )\n",
"\n",
"tune_with_callback()\n",
"tune_with_setup()\n",
"tune_trainable()\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "2f6e9138",
"metadata": {},
"source": [
"This completes our Tune and Wandb walk-through.\n",
"In the following sections you can find more details on the API of the Tune-Wandb integration.\n",
"\n",
"## Tune Wandb API Reference\n",
"\n",
"### WandbLoggerCallback\n",
"\n",
"(air-wandb-logger)=\n",
"\n",
"```{eval-rst}\n",
".. autoclass:: ray.air.integrations.wandb.WandbLoggerCallback\n",
" :noindex:\n",
"```\n",
"\n",
"### setup_wandb\n",
"\n",
"(air-wandb-setup)=\n",
"\n",
"```{eval-rst}\n",
".. autofunction:: ray.air.integrations.wandb.setup_wandb\n",
" :noindex:\n",
"```"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
},
"orphan": true
},
"nbformat": 4,
"nbformat_minor": 5
}
doc/source/tune/examples/tune-xgboost.ipynb
+940
View File
@@ -0,0 +1,940 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "edce67b9",
"metadata": {},
"source": [
"# Tuning XGBoost hyperparameters with Ray Tune\n",
"\n",
"<a id=\"try-anyscale-quickstart-tune-xgboost\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=tune-xgboost\">\n",
" <img src=\"../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
"</a>\n",
"<br></br>\n",
"\n",
"(tune-xgboost-ref)=\n",
"\n",
"This tutorial demonstrates how to optimize XGBoost models using Ray Tune. You'll learn:\n",
"- The basics of XGBoost and its key hyperparameters\n",
"- How to train a simple XGBoost classifier (without hyperparameter tuning)\n",
"- How to use Ray Tune to find optimal hyperparameters\n",
"- Advanced techniques like early stopping and GPU acceleration\n",
"\n",
"XGBoost is currently one of the most popular machine learning algorithms. It performs\n",
"very well on a large selection of tasks, and was the key to success in many Kaggle\n",
"competitions.\n",
"\n",
"```{image} /images/xgboost_logo.png\n",
":align: center\n",
":alt: XGBoost\n",
":target: https://xgboost.readthedocs.io/en/latest/\n",
":width: 200px\n",
"```\n",
"\n",
"```{contents}\n",
":depth: 2\n",
"```\n",
"\n",
":::{note}\n",
"To run this tutorial, you will need to install the following:\n",
"\n",
"```bash\n",
"$ pip install -q \"ray[tune]\" scikit-learn xgboost\n",
"```\n",
":::\n",
"\n",
"## What is XGBoost\n",
"\n",
"\n",
"XGBoost (e**X**treme **G**radient **Boost**ing) is a powerful and efficient implementation of gradient boosted [decision trees](https://en.wikipedia.org/wiki/Decision_tree). It has become one of the most popular machine learning algorithms due to its:\n",
"\n",
"1. Performance: Consistently strong results across many types of problems\n",
"2. Speed: Highly optimized implementation that can leverage GPU acceleration \n",
"3. Flexibility: Works with many types of prediction problems (classification, regression, ranking)\n",
"\n",
"Key Concepts:\n",
"- Uses an ensemble of simple decision trees\n",
"- Trees are built sequentially, with each tree correcting errors from previous trees\n",
"- Employs gradient descent to minimize a loss function\n",
"- Even though single trees can have high bias, using a boosted ensemble can result in better predictions and reduced bias\n",
"\n",
"\n",
":::{figure} /images/tune-xgboost-ensemble.svg\n",
":alt: Single vs. ensemble learning\n",
"\n",
"A single decision tree (left) might be able to get to an accuracy of 70%\n",
"for a binary classification task. By combining the output of several small\n",
"decision trees, an ensemble learner (right) might end up with a higher accuracy\n",
"of 90%.\n",
":::\n",
"\n",
"Boosting algorithms start with a single small decision tree and evaluate how well\n",
"it predicts the given examples. When building the next tree, those samples that have\n",
"been misclassified before have a higher chance of being used to generate the tree.\n",
"This is useful because it avoids overfitting to samples that can be easily classified\n",
"and instead tries to come up with models that are able to classify hard examples, too.\n",
"Please see [here for a more thorough introduction to bagging and boosting algorithms](https://towardsdatascience.com/ensemble-methods-bagging-boosting-and-stacking-c9214a10a205).\n",
"\n",
"There are many boosting algorithms. In their core, they are all very similar. XGBoost\n",
"uses second-level derivatives to find splits that maximize the *gain* (the inverse of\n",
"the *loss*) - hence the name. In practice, XGBoost usually shows the best performance\n",
"against other boosting algorithms, although LightGBM tends to be [faster and more\n",
"memory efficient](https://xgboosting.com/xgboost-vs-lightgbm/), especially for large datasets.\n",
"\n",
"## Training a simple XGBoost classifier\n",
"\n",
"Let's first see how a simple XGBoost classifier can be trained. We'll use the\n",
"`breast_cancer`-Dataset included in the `sklearn` dataset collection. This is\n",
"a binary classification dataset. Given 30 different input features, our task is to\n",
"learn to identify subjects with breast cancer and those without.\n",
"\n",
"Here is the full code to train a simple XGBoost model:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "63611b7f",
"metadata": {},
"outputs": [],
"source": [
"SMOKE_TEST = False"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "be0b8321",
"metadata": {
"tags": [
"hide-cell"
]
},
"outputs": [],
"source": [
"SMOKE_TEST = True"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "77b3c71c",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Accuracy: 0.9650\n"
]
}
],
"source": [
"import sklearn.datasets\n",
"import sklearn.metrics\n",
"from sklearn.model_selection import train_test_split\n",
"import xgboost as xgb\n",
"\n",
"\n",
"def train_breast_cancer(config):\n",
" # Load dataset\n",
" data, labels = sklearn.datasets.load_breast_cancer(return_X_y=True)\n",
" # Split into train and test set\n",
" train_x, test_x, train_y, test_y = train_test_split(data, labels, test_size=0.25)\n",
" # Build input matrices for XGBoost\n",
" train_set = xgb.DMatrix(train_x, label=train_y)\n",
" test_set = xgb.DMatrix(test_x, label=test_y)\n",
" # Train the classifier\n",
" results = {}\n",
" bst = xgb.train(\n",
" config,\n",
" train_set,\n",
" evals=[(test_set, \"eval\")],\n",
" evals_result=results,\n",
" verbose_eval=False,\n",
" )\n",
" return results\n",
"\n",
"\n",
"results = train_breast_cancer(\n",
" {\"objective\": \"binary:logistic\", \"eval_metric\": [\"logloss\", \"error\"]}\n",
")\n",
"accuracy = 1.0 - results[\"eval\"][\"error\"][-1]\n",
"print(f\"Accuracy: {accuracy:.4f}\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "ec2a13f8",
"metadata": {},
"source": [
"As you can see, the code is quite simple. First, the dataset is loaded and split\n",
"into a `test` and `train` set. The XGBoost model is trained with `xgb.train()`.\n",
"XGBoost automatically evaluates metrics we specified on the test set. In our case\n",
"it calculates the *logloss* and the prediction *error*, which is the percentage of\n",
"misclassified examples. To calculate the accuracy, we just have to subtract the error\n",
"from `1.0`. Even in this simple example, most runs result\n",
"in a good accuracy of over `0.90`.\n",
"\n",
"Maybe you have noticed the `config` parameter we pass to the XGBoost algorithm. This\n",
"is a {class}`dict` in which you can specify parameters for the XGBoost algorithm. In this\n",
"simple example, the only parameters we passed are the `objective` and `eval_metric` parameters.\n",
"The value `binary:logistic` tells XGBoost that we aim to train a logistic regression model for\n",
"a binary classification task. You can find an overview over all valid objectives\n",
"[here in the XGBoost documentation](https://xgboost.readthedocs.io/en/latest/parameter.html#learning-task-parameters).\n",
"\n",
"## Scaling XGBoost Training with Ray Train\n",
"\n",
"In {doc}`/train/getting-started-xgboost`, we covered how to scale XGBoost single-model training with *Ray Train*.\n",
"For the rest of this tutorial, we will focus on how to optimize the hyperparameters of the XGBoost model using *Ray Tune*.\n",
"\n",
"## XGBoost Hyperparameters\n",
"\n",
"Even with the default settings, XGBoost was able to get to a good accuracy on the\n",
"breast cancer dataset. However, as in many machine learning algorithms, there are\n",
"many knobs to tune which might lead to even better performance. Let's explore some of\n",
"them below.\n",
"\n",
"### Maximum tree depth\n",
"\n",
"Remember that XGBoost internally uses many decision tree models to come up with\n",
"predictions. When training a decision tree, we need to tell the algorithm how\n",
"large the tree may get. The parameter for this is called the tree *depth*.\n",
"\n",
":::{figure} /images/tune-xgboost-depth.svg\n",
":align: center\n",
":alt: Decision tree depth\n",
"\n",
"In this image, the left tree has a depth of 2, and the right tree a depth of 3.\n",
"Note that with each level, $2^{(d-1)}$ splits are added, where *d* is the depth\n",
"of the tree.\n",
":::\n",
"\n",
"Tree depth is a property that concerns the model complexity. If you only allow short\n",
"trees, the models are likely not very precise - they underfit the data. If you allow\n",
"very large trees, the single models are likely to overfit to the data. In practice,\n",
"a number between `2` and `6` is often a good starting point for this parameter.\n",
"\n",
"XGBoost's default value is `3`.\n",
"\n",
"### Minimum child weight\n",
"\n",
"When a decision tree creates new leaves, it splits up the remaining data at one node\n",
"into two groups. If there are only few samples in one of these groups, it often\n",
"doesn't make sense to split it further. One of the reasons for this is that the\n",
"model is harder to train when we have fewer samples.\n",
"\n",
":::{figure} /images/tune-xgboost-weight.svg\n",
":align: center\n",
":alt: Minimum child weight\n",
"\n",
"In this example, we start with 100 examples. At the first node, they are split\n",
"into 4 and 96 samples, respectively. In the next step, our model might find\n",
"that it doesn't make sense to split the 4 examples more. It thus only continues\n",
"to add leaves on the right side.\n",
":::\n",
"\n",
"The parameter used by the model to decide if it makes sense to split a node is called\n",
"the *minimum child weight*. In the case of linear regression, this is just the absolute\n",
"number of nodes requried in each child. In other objectives, this value is determined\n",
"using the weights of the examples, hence the name.\n",
"\n",
"The larger the value, the more constrained the trees are and the less deep they will be.\n",
"This parameter thus also affects the model complexity. Thus, for noisy or small datasets, \n",
"smaller values are preferred. Values can range between 0 and infinity and are dependent on\n",
"the sample size. For our case with only 500 examples in the breast cancer dataset, values \n",
"between `0` and `10` should be sensible.\n",
"\n",
"XGBoost's default value is `1`.\n",
"\n",
"### Subsample size\n",
"\n",
"Each decision tree we add is trained on a subsample of the total training dataset.\n",
"The probabilities for the samples are weighted according to the XGBoost algorithm,\n",
"but we can decide on which fraction of the samples we want to train each decision\n",
"tree on.\n",
"\n",
"Setting this value to `0.7` would mean that we randomly sample `70%` of the\n",
"training dataset before each training iteration. Lower values lead to more\n",
"diverse trees and higher values to more similar trees. Lower values help\n",
"prevent overfitting.\n",
"\n",
"XGBoost's default value is `1`.\n",
"\n",
"### Learning rate / Eta\n",
"\n",
"Remember that XGBoost sequentially trains many decision trees, and that later trees\n",
"are more likely trained on data that has been misclassified by prior trees. In effect\n",
"this means that earlier trees make decisions for easy samples (i.e. those samples that\n",
"can easily be classified) and later trees make decisions for harder samples. It is then\n",
"sensible to assume that the later trees are less accurate than earlier trees.\n",
"\n",
"To address this fact, XGBoost uses a parameter called *Eta*, which is sometimes called\n",
"the *learning rate*. Don't confuse this with learning rates from gradient descent!\n",
"The original [paper on stochastic gradient boosting](https://www.researchgate.net/publication/222573328_Stochastic_Gradient_Boosting)\n",
"introduces this parameter like so:\n",
"\n",
"$$\n",
"F_m(x) = F_{m-1}(x) + \\eta \\cdot \\gamma_{lm} \\textbf{1}(x \\in R_{lm})\n",
"$$\n",
"\n",
"This is just a complicated way to say that when we train we new decision tree,\n",
"represented by $\\gamma_{lm} \\textbf{1}(x \\in R_{lm})$, we want to dampen\n",
"its effect on the previous prediction $F_{m-1}(x)$ with a factor\n",
"$\\eta$.\n",
"\n",
"Typical values for this parameter are between `0.01` and `` 0.3` ``.\n",
"\n",
"XGBoost's default value is `0.3`.\n",
"\n",
"### Number of boost rounds\n",
"\n",
"Lastly, we can decide on how many boosting rounds we perform, which means how\n",
"many decision trees we ultimately train. When we do heavy subsampling or use small\n",
"learning rate, it might make sense to increase the number of boosting rounds.\n",
"\n",
"XGBoost's default value is `10`.\n",
"\n",
"### Putting it together\n",
"\n",
"Let's see how this looks like in code! We just need to adjust our `config` dict:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "35073e88",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Accuracy: 0.9231\n"
]
}
],
"source": [
"config = {\n",
" \"objective\": \"binary:logistic\",\n",
" \"eval_metric\": [\"logloss\", \"error\"],\n",
" \"max_depth\": 2,\n",
" \"min_child_weight\": 0,\n",
" \"subsample\": 0.8,\n",
" \"eta\": 0.2,\n",
"}\n",
"results = train_breast_cancer(config)\n",
"accuracy = 1.0 - results[\"eval\"][\"error\"][-1]\n",
"print(f\"Accuracy: {accuracy:.4f}\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "69cf0c13",
"metadata": {},
"source": [
"The rest stays the same. Please note that we do not adjust the `num_boost_rounds` here.\n",
"The result should also show a high accuracy of over 90%.\n",
"\n",
"## Tuning the configuration parameters\n",
"\n",
"XGBoosts default parameters already lead to a good accuracy, and even our guesses in the\n",
"last section should result in accuracies well above 90%. However, our guesses were\n",
"just that: guesses. Often we do not know what combination of parameters would actually\n",
"lead to the best results on a machine learning task.\n",
"\n",
"Unfortunately, there are infinitely many combinations of hyperparameters we could try\n",
"out. Should we combine `max_depth=3` with `subsample=0.8` or with `subsample=0.9`?\n",
"What about the other parameters?\n",
"\n",
"This is where hyperparameter tuning comes into play. By using tuning libraries such as\n",
"Ray Tune we can try out combinations of hyperparameters. Using sophisticated search\n",
"strategies, these parameters can be selected so that they are likely to lead to good\n",
"results (avoiding an expensive *exhaustive search*). Also, trials that do not perform\n",
"well can be preemptively stopped to reduce waste of computing resources. Lastly, Ray Tune\n",
"also takes care of training these runs in parallel, greatly increasing search speed.\n",
"\n",
"Let's start with a basic example on how to use Tune for this. We just need to make\n",
"a few changes to our code-block:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "ff856a82",
"metadata": {
"tags": [
"hide-output"
]
},
"outputs": [
{
"data": {
"text/html": [
"<div class=\"tuneStatus\">\n",
" <div style=\"display: flex;flex-direction: row\">\n",
" <div style=\"display: flex;flex-direction: column;\">\n",
" <h3>Tune Status</h3>\n",
" <table>\n",
"<tbody>\n",
"<tr><td>Current time:</td><td>2025-02-11 16:13:34</td></tr>\n",
"<tr><td>Running for: </td><td>00:00:01.87 </td></tr>\n",
"<tr><td>Memory: </td><td>22.5/36.0 GiB </td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
" <div class=\"vDivider\"></div>\n",
" <div class=\"systemInfo\">\n",
" <h3>System Info</h3>\n",
" Using FIFO scheduling algorithm.<br>Logical resource usage: 1.0/12 CPUs, 0/0 GPUs\n",
" </div>\n",
" \n",
" </div>\n",
" <div class=\"hDivider\"></div>\n",
" <div class=\"trialStatus\">\n",
" <h3>Trial Status</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>status </th><th>loc </th><th style=\"text-align: right;\"> eta</th><th style=\"text-align: right;\"> max_depth</th><th style=\"text-align: right;\"> min_child_weight</th><th style=\"text-align: right;\"> subsample</th><th style=\"text-align: right;\"> acc</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>train_breast_cancer_31c9f_00000</td><td>TERMINATED</td><td>127.0.0.1:89735</td><td style=\"text-align: right;\">0.0434196 </td><td style=\"text-align: right;\"> 8</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\"> 0.530351</td><td style=\"text-align: right;\">0.909091</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\"> 0.0114911 </td></tr>\n",
"<tr><td>train_breast_cancer_31c9f_00001</td><td>TERMINATED</td><td>127.0.0.1:89734</td><td style=\"text-align: right;\">0.0115669 </td><td style=\"text-align: right;\"> 6</td><td style=\"text-align: right;\"> 2</td><td style=\"text-align: right;\"> 0.996519</td><td style=\"text-align: right;\">0.615385</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\"> 0.01138 </td></tr>\n",
"<tr><td>train_breast_cancer_31c9f_00002</td><td>TERMINATED</td><td>127.0.0.1:89740</td><td style=\"text-align: right;\">0.00124339 </td><td style=\"text-align: right;\"> 7</td><td style=\"text-align: right;\"> 3</td><td style=\"text-align: right;\"> 0.536078</td><td style=\"text-align: right;\">0.629371</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\"> 0.0096581 </td></tr>\n",
"<tr><td>train_breast_cancer_31c9f_00003</td><td>TERMINATED</td><td>127.0.0.1:89742</td><td style=\"text-align: right;\">0.000400434</td><td style=\"text-align: right;\"> 6</td><td style=\"text-align: right;\"> 3</td><td style=\"text-align: right;\"> 0.90014 </td><td style=\"text-align: right;\">0.601399</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\"> 0.0103199 </td></tr>\n",
"<tr><td>train_breast_cancer_31c9f_00004</td><td>TERMINATED</td><td>127.0.0.1:89738</td><td style=\"text-align: right;\">0.0121308 </td><td style=\"text-align: right;\"> 6</td><td style=\"text-align: right;\"> 3</td><td style=\"text-align: right;\"> 0.843156</td><td style=\"text-align: right;\">0.629371</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\"> 0.00843 </td></tr>\n",
"<tr><td>train_breast_cancer_31c9f_00005</td><td>TERMINATED</td><td>127.0.0.1:89733</td><td style=\"text-align: right;\">0.0344144 </td><td style=\"text-align: right;\"> 2</td><td style=\"text-align: right;\"> 3</td><td style=\"text-align: right;\"> 0.513071</td><td style=\"text-align: right;\">0.895105</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\"> 0.00800109</td></tr>\n",
"<tr><td>train_breast_cancer_31c9f_00006</td><td>TERMINATED</td><td>127.0.0.1:89737</td><td style=\"text-align: right;\">0.0530037 </td><td style=\"text-align: right;\"> 7</td><td style=\"text-align: right;\"> 2</td><td style=\"text-align: right;\"> 0.920801</td><td style=\"text-align: right;\">0.965035</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\"> 0.0117419 </td></tr>\n",
"<tr><td>train_breast_cancer_31c9f_00007</td><td>TERMINATED</td><td>127.0.0.1:89741</td><td style=\"text-align: right;\">0.000230442</td><td style=\"text-align: right;\"> 3</td><td style=\"text-align: right;\"> 3</td><td style=\"text-align: right;\"> 0.946852</td><td style=\"text-align: right;\">0.608392</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\"> 0.00917387</td></tr>\n",
"<tr><td>train_breast_cancer_31c9f_00008</td><td>TERMINATED</td><td>127.0.0.1:89739</td><td style=\"text-align: right;\">0.00166323 </td><td style=\"text-align: right;\"> 4</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\"> 0.588879</td><td style=\"text-align: right;\">0.636364</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\"> 0.011095 </td></tr>\n",
"<tr><td>train_breast_cancer_31c9f_00009</td><td>TERMINATED</td><td>127.0.0.1:89736</td><td style=\"text-align: right;\">0.0753618 </td><td style=\"text-align: right;\"> 3</td><td style=\"text-align: right;\"> 3</td><td style=\"text-align: right;\"> 0.55103 </td><td style=\"text-align: right;\">0.909091</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\"> 0.00776482</td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
"</div>\n",
"<style>\n",
".tuneStatus {\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".tuneStatus .systemInfo {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus td {\n",
" white-space: nowrap;\n",
"}\n",
".tuneStatus .trialStatus {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus h3 {\n",
" font-weight: bold;\n",
"}\n",
".tuneStatus .hDivider {\n",
" border-bottom-width: var(--jp-border-width);\n",
" border-bottom-color: var(--jp-border-color0);\n",
" border-bottom-style: solid;\n",
"}\n",
".tuneStatus .vDivider {\n",
" border-left-width: var(--jp-border-width);\n",
" border-left-color: var(--jp-border-color0);\n",
" border-left-style: solid;\n",
" margin: 0.5em 1em 0.5em 1em;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2025-02-11 16:13:34,649\tINFO tune.py:1009 -- Wrote the latest version of all result files and experiment state to '~/ray_results/train_breast_cancer_2025-02-11_16-13-31' in 0.0057s.\n",
"2025-02-11 16:13:34,652\tINFO tune.py:1041 -- Total run time: 1.88 seconds (1.86 seconds for the tuning loop).\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[36m(train_breast_cancer pid=90413)\u001b[0m Checkpoint successfully created at: Checkpoint(filesystem=local, path=~/ray_results/train_breast_cancer_2025-02-11_16-17-11/train_breast_cancer_b412c_00000_0_eta=0.0200,max_depth=4,min_child_weight=2,subsample=0.7395_2025-02-11_16-17-11/checkpoint_000000)\n",
"\u001b[36m(train_breast_cancer pid=90413)\u001b[0m Checkpoint successfully created at: Checkpoint(filesystem=local, path=~/ray_results/train_breast_cancer_2025-02-11_16-17-11/train_breast_cancer_b412c_00000_0_eta=0.0200,max_depth=4,min_child_weight=2,subsample=0.7395_2025-02-11_16-17-11/checkpoint_000001)\n",
"\u001b[36m(train_breast_cancer pid=90413)\u001b[0m Checkpoint successfully created at: Checkpoint(filesystem=local, path=~/ray_results/train_breast_cancer_2025-02-11_16-17-11/train_breast_cancer_b412c_00000_0_eta=0.0200,max_depth=4,min_child_weight=2,subsample=0.7395_2025-02-11_16-17-11/checkpoint_000002)\n",
"\u001b[36m(train_breast_cancer pid=90413)\u001b[0m Checkpoint successfully created at: Checkpoint(filesystem=local, path=~/ray_results/train_breast_cancer_2025-02-11_16-17-11/train_breast_cancer_b412c_00000_0_eta=0.0200,max_depth=4,min_child_weight=2,subsample=0.7395_2025-02-11_16-17-11/checkpoint_000003)\n",
"\u001b[36m(train_breast_cancer pid=90413)\u001b[0m Checkpoint successfully created at: Checkpoint(filesystem=local, path=~/ray_results/train_breast_cancer_2025-02-11_16-17-11/train_breast_cancer_b412c_00000_0_eta=0.0200,max_depth=4,min_child_weight=2,subsample=0.7395_2025-02-11_16-17-11/checkpoint_000004)\n",
"\u001b[36m(train_breast_cancer pid=90413)\u001b[0m Checkpoint successfully created at: Checkpoint(filesystem=local, path=~/ray_results/train_breast_cancer_2025-02-11_16-17-11/train_breast_cancer_b412c_00000_0_eta=0.0200,max_depth=4,min_child_weight=2,subsample=0.7395_2025-02-11_16-17-11/checkpoint_000005)\n",
"\u001b[36m(train_breast_cancer pid=90413)\u001b[0m Checkpoint successfully created at: Checkpoint(filesystem=local, path=~/ray_results/train_breast_cancer_2025-02-11_16-17-11/train_breast_cancer_b412c_00000_0_eta=0.0200,max_depth=4,min_child_weight=2,subsample=0.7395_2025-02-11_16-17-11/checkpoint_000006)\n",
"\u001b[36m(train_breast_cancer pid=90413)\u001b[0m Checkpoint successfully created at: Checkpoint(filesystem=local, path=~/ray_results/train_breast_cancer_2025-02-11_16-17-11/train_breast_cancer_b412c_00000_0_eta=0.0200,max_depth=4,min_child_weight=2,subsample=0.7395_2025-02-11_16-17-11/checkpoint_000007)\n",
"\u001b[36m(train_breast_cancer pid=90413)\u001b[0m Checkpoint successfully created at: Checkpoint(filesystem=local, path=~/ray_results/train_breast_cancer_2025-02-11_16-17-11/train_breast_cancer_b412c_00000_0_eta=0.0200,max_depth=4,min_child_weight=2,subsample=0.7395_2025-02-11_16-17-11/checkpoint_000008)\n",
"\u001b[36m(train_breast_cancer pid=90413)\u001b[0m Checkpoint successfully created at: Checkpoint(filesystem=local, path=~/ray_results/train_breast_cancer_2025-02-11_16-17-11/train_breast_cancer_b412c_00000_0_eta=0.0200,max_depth=4,min_child_weight=2,subsample=0.7395_2025-02-11_16-17-11/checkpoint_000009)\n"
]
}
],
"source": [
"import sklearn.datasets\n",
"import sklearn.metrics\n",
"\n",
"from ray import tune\n",
"\n",
"\n",
"def train_breast_cancer(config):\n",
" # Load dataset\n",
" data, labels = sklearn.datasets.load_breast_cancer(return_X_y=True)\n",
" # Split into train and test set\n",
" train_x, test_x, train_y, test_y = train_test_split(data, labels, test_size=0.25)\n",
" # Build input matrices for XGBoost\n",
" train_set = xgb.DMatrix(train_x, label=train_y)\n",
" test_set = xgb.DMatrix(test_x, label=test_y)\n",
" # Train the classifier\n",
" results = {}\n",
" xgb.train(\n",
" config,\n",
" train_set,\n",
" evals=[(test_set, \"eval\")],\n",
" evals_result=results,\n",
" verbose_eval=False,\n",
" )\n",
" # Return prediction accuracy\n",
" accuracy = 1.0 - results[\"eval\"][\"error\"][-1]\n",
" tune.report({\"mean_accuracy\": accuracy, \"done\": True})\n",
"\n",
"\n",
"config = {\n",
" \"objective\": \"binary:logistic\",\n",
" \"eval_metric\": [\"logloss\", \"error\"],\n",
" \"max_depth\": tune.randint(1, 9),\n",
" \"min_child_weight\": tune.choice([1, 2, 3]),\n",
" \"subsample\": tune.uniform(0.5, 1.0),\n",
" \"eta\": tune.loguniform(1e-4, 1e-1),\n",
"}\n",
"tuner = tune.Tuner(\n",
" train_breast_cancer,\n",
" tune_config=tune.TuneConfig(num_samples=10),\n",
" param_space=config,\n",
")\n",
"results = tuner.fit()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "4999e858",
"metadata": {},
"source": [
"As you can see, the changes in the actual training function are minimal. Instead of\n",
"returning the accuracy value, we report it back to Tune using `tune.report()`.\n",
"Our `config` dictionary only changed slightly. Instead of passing hard-coded\n",
"parameters, we tell Tune to choose values from a range of valid options. There are\n",
"a number of options we have here, all of which are explained in\n",
"{ref}`the Tune docs <tune-search-space>`.\n",
"\n",
"For a brief explanation, this is what they do:\n",
"\n",
"- `tune.randint(min, max)` chooses a random integer value between *min* and *max*.\n",
" Note that *max* is exclusive, so it will not be sampled.\n",
"- `tune.choice([a, b, c])` chooses one of the items of the list at random. Each item\n",
" has the same chance to be sampled.\n",
"- `tune.uniform(min, max)` samples a floating point number between *min* and *max*.\n",
" Note that *max* is exclusive here, too.\n",
"- `tune.loguniform(min, max)` samples a floating point number between *min* and *max*,\n",
" but applies a logarithmic transformation to these boundaries first. Thus, this makes\n",
" it easy to sample values from different orders of magnitude.\n",
"\n",
"The `num_samples=10` option we pass to the `TuneConfig()` means that we sample 10 different\n",
"hyperparameter configurations from this search space.\n",
"\n",
"The output of our training run coud look like this:\n",
"\n",
"```{code-block} bash\n",
":emphasize-lines: 14\n",
"\n",
" Number of trials: 10/10 (10 TERMINATED)\n",
" +---------------------------------+------------+-------+-------------+-------------+--------------------+-------------+----------+--------+------------------+\n",
" | Trial name | status | loc | eta | max_depth | min_child_weight | subsample | acc | iter | total time (s) |\n",
" |---------------------------------+------------+-------+-------------+-------------+--------------------+-------------+----------+--------+------------------|\n",
" | train_breast_cancer_b63aa_00000 | TERMINATED | | 0.000117625 | 2 | 2 | 0.616347 | 0.916084 | 1 | 0.0306492 |\n",
" | train_breast_cancer_b63aa_00001 | TERMINATED | | 0.0382954 | 8 | 2 | 0.581549 | 0.937063 | 1 | 0.0357082 |\n",
" | train_breast_cancer_b63aa_00002 | TERMINATED | | 0.000217926 | 1 | 3 | 0.528428 | 0.874126 | 1 | 0.0264609 |\n",
" | train_breast_cancer_b63aa_00003 | TERMINATED | | 0.000120929 | 8 | 1 | 0.634508 | 0.958042 | 1 | 0.036406 |\n",
" | train_breast_cancer_b63aa_00004 | TERMINATED | | 0.00839715 | 5 | 1 | 0.730624 | 0.958042 | 1 | 0.0389378 |\n",
" | train_breast_cancer_b63aa_00005 | TERMINATED | | 0.000732948 | 8 | 2 | 0.915863 | 0.958042 | 1 | 0.0382841 |\n",
" | train_breast_cancer_b63aa_00006 | TERMINATED | | 0.000856226 | 4 | 1 | 0.645209 | 0.916084 | 1 | 0.0357089 |\n",
" | train_breast_cancer_b63aa_00007 | TERMINATED | | 0.00769908 | 7 | 1 | 0.729443 | 0.909091 | 1 | 0.0390737 |\n",
" | train_breast_cancer_b63aa_00008 | TERMINATED | | 0.00186339 | 5 | 3 | 0.595744 | 0.944056 | 1 | 0.0343912 |\n",
" | train_breast_cancer_b63aa_00009 | TERMINATED | | 0.000950272 | 3 | 2 | 0.835504 | 0.965035 | 1 | 0.0348201 |\n",
" +---------------------------------+------------+-------+-------------+-------------+--------------------+-------------+----------+--------+------------------+\n",
"```\n",
"\n",
"The best configuration we found used `eta=0.000950272`, `max_depth=3`,\n",
"`min_child_weight=2`, `subsample=0.835504` and reached an accuracy of\n",
"`0.965035`.\n",
"\n",
"## Early stopping\n",
"\n",
"Currently, Tune samples 10 different hyperparameter configurations and trains a full\n",
"XGBoost on all of them. In our small example, training is very fast. However,\n",
"if training takes longer, a significant amount of computer resources is spent on trials\n",
"that will eventually show a bad performance, e.g. a low accuracy. It would be good\n",
"if we could identify these trials early and stop them, so we don't waste any resources.\n",
"\n",
"This is where Tune's *Schedulers* shine. A Tune `TrialScheduler` is responsible\n",
"for starting and stopping trials. Tune implements a number of different schedulers, each\n",
"described {ref}`in the Tune documentation <tune-schedulers>`.\n",
"For our example, we will use the `AsyncHyperBandScheduler` or `ASHAScheduler`.\n",
"\n",
"The basic idea of this scheduler: We sample a number of hyperparameter configurations.\n",
"Each of these configurations is trained for a specific number of iterations.\n",
"After these iterations, only the best performing hyperparameters are retained. These\n",
"are selected according to some loss metric, usually an evaluation loss. This cycle is\n",
"repeated until we end up with the best configuration.\n",
"\n",
"The `ASHAScheduler` needs to know three things:\n",
"\n",
"1. Which metric should be used to identify badly performing trials?\n",
"2. Should this metric be maximized or minimized?\n",
"3. How many iterations does each trial train for?\n",
"\n",
"There are more parameters, which are explained in the\n",
"{ref}`documentation <tune-scheduler-hyperband>`.\n",
"\n",
"Lastly, we have to report the loss metric to Tune. We do this with a `Callback` that\n",
"XGBoost accepts and calls after each evaluation round. Ray Tune comes\n",
"with {ref}`two XGBoost callbacks <tune-integration-xgboost>`\n",
"we can use for this. The `TuneReportCallback` just reports the evaluation\n",
"metrics back to Tune. The `TuneReportCheckpointCallback` also saves\n",
"checkpoints after each evaluation round. We will just use the latter in this\n",
"example so that we can retrieve the saved model later.\n",
"\n",
"These parameters from the `eval_metrics` configuration setting are then automatically\n",
"reported to Tune via the callback. Here, the raw error will be reported, not the accuracy.\n",
"To display the best reached accuracy, we will inverse it later.\n",
"\n",
"We will also load the best checkpointed model so that we can use it for predictions.\n",
"The best model is selected with respect to the `metric` and `mode` parameters we\n",
"pass to the `TunerConfig()`."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "d08b5b0a",
"metadata": {
"tags": [
"hide-output"
]
},
"outputs": [
{
"data": {
"text/html": [
"<div class=\"tuneStatus\">\n",
" <div style=\"display: flex;flex-direction: row\">\n",
" <div style=\"display: flex;flex-direction: column;\">\n",
" <h3>Tune Status</h3>\n",
" <table>\n",
"<tbody>\n",
"<tr><td>Current time:</td><td>2025-02-11 16:13:35</td></tr>\n",
"<tr><td>Running for: </td><td>00:00:01.05 </td></tr>\n",
"<tr><td>Memory: </td><td>22.5/36.0 GiB </td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
" <div class=\"vDivider\"></div>\n",
" <div class=\"systemInfo\">\n",
" <h3>System Info</h3>\n",
" Using AsyncHyperBand: num_stopped=1<br>Bracket: Iter 8.000: -0.6414526407118444 | Iter 4.000: -0.6439705872452343 | Iter 2.000: -0.6452721030145259 | Iter 1.000: -0.6459394399519567<br>Logical resource usage: 1.0/12 CPUs, 0/0 GPUs\n",
" </div>\n",
" \n",
" </div>\n",
" <div class=\"hDivider\"></div>\n",
" <div class=\"trialStatus\">\n",
" <h3>Trial Status</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>status </th><th>loc </th><th style=\"text-align: right;\"> eta</th><th style=\"text-align: right;\"> max_depth</th><th style=\"text-align: right;\"> min_child_weight</th><th style=\"text-align: right;\"> subsample</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> eval-logloss</th><th style=\"text-align: right;\"> eval-error</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>train_breast_cancer_32eb5_00000</td><td>TERMINATED</td><td>127.0.0.1:89763</td><td style=\"text-align: right;\">0.000830475</td><td style=\"text-align: right;\"> 5</td><td style=\"text-align: right;\"> 1</td><td style=\"text-align: right;\"> 0.675899</td><td style=\"text-align: right;\"> 10</td><td style=\"text-align: right;\"> 0.0169384</td><td style=\"text-align: right;\"> 0.640195</td><td style=\"text-align: right;\"> 0.342657</td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
"</div>\n",
"<style>\n",
".tuneStatus {\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".tuneStatus .systemInfo {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus td {\n",
" white-space: nowrap;\n",
"}\n",
".tuneStatus .trialStatus {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus h3 {\n",
" font-weight: bold;\n",
"}\n",
".tuneStatus .hDivider {\n",
" border-bottom-width: var(--jp-border-width);\n",
" border-bottom-color: var(--jp-border-color0);\n",
" border-bottom-style: solid;\n",
"}\n",
".tuneStatus .vDivider {\n",
" border-left-width: var(--jp-border-width);\n",
" border-left-color: var(--jp-border-color0);\n",
" border-left-style: solid;\n",
" margin: 0.5em 1em 0.5em 1em;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2025-02-11 16:13:35,717\tINFO tune.py:1009 -- Wrote the latest version of all result files and experiment state to '~/ray_results/train_breast_cancer_2025-02-11_16-13-34' in 0.0018s.\n",
"2025-02-11 16:13:35,719\tINFO tune.py:1041 -- Total run time: 1.05 seconds (1.04 seconds for the tuning loop).\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best model parameters: {'objective': 'binary:logistic', 'eval_metric': ['logloss', 'error'], 'max_depth': 5, 'min_child_weight': 1, 'subsample': 0.675899175238225, 'eta': 0.0008304750981897656}\n",
"Best model total accuracy: 0.6573\n"
]
}
],
"source": [
"import sklearn.datasets\n",
"import sklearn.metrics\n",
"from ray.tune.schedulers import ASHAScheduler\n",
"from sklearn.model_selection import train_test_split\n",
"import xgboost as xgb\n",
"\n",
"from ray import tune\n",
"from ray.tune.integration.xgboost import TuneReportCheckpointCallback\n",
"\n",
"\n",
"def train_breast_cancer(config: dict):\n",
" # This is a simple training function to be passed into Tune\n",
" # Load dataset\n",
" data, labels = sklearn.datasets.load_breast_cancer(return_X_y=True)\n",
" # Split into train and test set\n",
" train_x, test_x, train_y, test_y = train_test_split(data, labels, test_size=0.25)\n",
" # Build input matrices for XGBoost\n",
" train_set = xgb.DMatrix(train_x, label=train_y)\n",
" test_set = xgb.DMatrix(test_x, label=test_y)\n",
" # Train the classifier, using the Tune callback\n",
" xgb.train(\n",
" config,\n",
" train_set,\n",
" evals=[(test_set, \"eval\")],\n",
" verbose_eval=False,\n",
" # `TuneReportCheckpointCallback` defines the checkpointing frequency and format.\n",
" callbacks=[TuneReportCheckpointCallback(frequency=1)],\n",
" )\n",
"\n",
"\n",
"def get_best_model_checkpoint(results):\n",
" best_result = results.get_best_result()\n",
"\n",
" # `TuneReportCheckpointCallback` provides a helper method to retrieve the\n",
" # model from a checkpoint.\n",
" best_bst = TuneReportCheckpointCallback.get_model(best_result.checkpoint)\n",
"\n",
" accuracy = 1.0 - best_result.metrics[\"eval-error\"]\n",
" print(f\"Best model parameters: {best_result.config}\")\n",
" print(f\"Best model total accuracy: {accuracy:.4f}\")\n",
" return best_bst\n",
"\n",
"\n",
"def tune_xgboost(smoke_test=False):\n",
" search_space = {\n",
" # You can mix constants with search space objects.\n",
" \"objective\": \"binary:logistic\",\n",
" \"eval_metric\": [\"logloss\", \"error\"],\n",
" \"max_depth\": tune.randint(1, 9),\n",
" \"min_child_weight\": tune.choice([1, 2, 3]),\n",
" \"subsample\": tune.uniform(0.5, 1.0),\n",
" \"eta\": tune.loguniform(1e-4, 1e-1),\n",
" }\n",
" # This will enable aggressive early stopping of bad trials.\n",
" scheduler = ASHAScheduler(\n",
" max_t=10, grace_period=1, reduction_factor=2 # 10 training iterations\n",
" )\n",
"\n",
" tuner = tune.Tuner(\n",
" train_breast_cancer,\n",
" tune_config=tune.TuneConfig(\n",
" metric=\"eval-logloss\",\n",
" mode=\"min\",\n",
" scheduler=scheduler,\n",
" num_samples=1 if smoke_test else 10,\n",
" ),\n",
" param_space=search_space,\n",
" )\n",
" results = tuner.fit()\n",
" return results\n",
"\n",
"\n",
"results = tune_xgboost(smoke_test=SMOKE_TEST)\n",
"\n",
"# Load the best model checkpoint.\n",
"best_bst = get_best_model_checkpoint(results)\n",
"\n",
"# You could now do further predictions with\n",
"# best_bst.predict(...)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "20732fe4",
"metadata": {},
"source": [
"The output of our run could look like this:\n",
"\n",
"```{code-block} bash\n",
":emphasize-lines: 7\n",
"\n",
" Number of trials: 10/10 (10 TERMINATED)\n",
" +---------------------------------+------------+-------+-------------+-------------+--------------------+-------------+--------+------------------+----------------+--------------+\n",
" | Trial name | status | loc | eta | max_depth | min_child_weight | subsample | iter | total time (s) | eval-logloss | eval-error |\n",
" |---------------------------------+------------+-------+-------------+-------------+--------------------+-------------+--------+------------------+----------------+--------------|\n",
" | train_breast_cancer_ba275_00000 | TERMINATED | | 0.00205087 | 2 | 1 | 0.898391 | 10 | 0.380619 | 0.678039 | 0.090909 |\n",
" | train_breast_cancer_ba275_00001 | TERMINATED | | 0.000183834 | 4 | 3 | 0.924939 | 1 | 0.0228798 | 0.693009 | 0.111888 |\n",
" | train_breast_cancer_ba275_00002 | TERMINATED | | 0.0242721 | 7 | 2 | 0.501551 | 10 | 0.376154 | 0.54472 | 0.06993 |\n",
" | train_breast_cancer_ba275_00003 | TERMINATED | | 0.000449692 | 5 | 3 | 0.890212 | 1 | 0.0234981 | 0.692811 | 0.090909 |\n",
" | train_breast_cancer_ba275_00004 | TERMINATED | | 0.000376393 | 7 | 2 | 0.883609 | 1 | 0.0231569 | 0.692847 | 0.062937 |\n",
" | train_breast_cancer_ba275_00005 | TERMINATED | | 0.00231942 | 3 | 3 | 0.877464 | 2 | 0.104867 | 0.689541 | 0.083916 |\n",
" | train_breast_cancer_ba275_00006 | TERMINATED | | 0.000542326 | 1 | 2 | 0.578584 | 1 | 0.0213971 | 0.692765 | 0.083916 |\n",
" | train_breast_cancer_ba275_00007 | TERMINATED | | 0.0016801 | 1 | 2 | 0.975302 | 1 | 0.02226 | 0.691999 | 0.083916 |\n",
" | train_breast_cancer_ba275_00008 | TERMINATED | | 0.000595756 | 8 | 3 | 0.58429 | 1 | 0.0221152 | 0.692657 | 0.06993 |\n",
" | train_breast_cancer_ba275_00009 | TERMINATED | | 0.000357845 | 8 | 1 | 0.637776 | 1 | 0.022635 | 0.692859 | 0.090909 |\n",
" +---------------------------------+------------+-------+-------------+-------------+--------------------+-------------+--------+------------------+----------------+--------------+\n",
"\n",
"\n",
" Best model parameters: {'objective': 'binary:logistic', 'eval_metric': ['logloss', 'error'], 'max_depth': 7, 'min_child_weight': 2, 'subsample': 0.5015513240240503, 'eta': 0.024272050872920895}\n",
" Best model total accuracy: 0.9301\n",
"```\n",
"\n",
"As you can see, most trials have been stopped only after a few iterations. Only the\n",
"two most promising trials were run for the full 10 iterations.\n",
"\n",
"You can also ensure that all available resources are being used as the scheduler\n",
"terminates trials, freeing them up. This can be done through the\n",
"`ResourceChangingScheduler`. An example of this can be found here:\n",
"{doc}`/tune/examples/includes/xgboost_dynamic_resources_example`.\n",
"\n",
"## Using fractional GPUs\n",
"\n",
"You can often accelerate your training by using GPUs in addition to CPUs. However,\n",
"you usually don't have as many GPUs as you have trials to run. For instance, if you\n",
"run 10 Tune trials in parallel, you usually don't have access to 10 separate GPUs.\n",
"\n",
"Tune supports *fractional GPUs*. This means that each task is assigned a fraction\n",
"of the GPU memory for training. For 10 tasks, this could look like this:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7d1b20a3",
"metadata": {
"tags": [
"hide-output"
]
},
"outputs": [],
"source": [
"config = {\n",
" \"objective\": \"binary:logistic\",\n",
" \"eval_metric\": [\"logloss\", \"error\"],\n",
" \"tree_method\": \"gpu_hist\",\n",
" \"max_depth\": tune.randint(1, 9),\n",
" \"min_child_weight\": tune.choice([1, 2, 3]),\n",
" \"subsample\": tune.uniform(0.5, 1.0),\n",
" \"eta\": tune.loguniform(1e-4, 1e-1),\n",
"}\n",
"\n",
"tuner = tune.Tuner(\n",
" tune.with_resources(train_breast_cancer, resources={\"cpu\": 1, \"gpu\": 0.1}),\n",
" tune_config=tune.TuneConfig(num_samples=1 if SMOKE_TEST else 10),\n",
" param_space=config,\n",
")\n",
"results = tuner.fit()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "ee131861",
"metadata": {},
"source": [
"Each task thus works with 10% of the available GPU memory. You also have to tell\n",
"XGBoost to use the `gpu_hist` tree method, so it knows it should use the GPU.\n",
"\n",
"## Conclusion\n",
"\n",
"You should now have a basic understanding on how to train XGBoost models and on how\n",
"to tune the hyperparameters to yield the best results. In our simple example,\n",
"Tuning the parameters didn't make a huge difference for the accuracy.\n",
"But in larger applications, intelligent hyperparameter tuning can make the\n",
"difference between a model that doesn't seem to learn at all, and a model\n",
"that outperforms all the other ones.\n",
"\n",
"## More XGBoost Examples\n",
"\n",
"- {doc}`/tune/examples/includes/xgboost_dynamic_resources_example`:\n",
" Trains a basic XGBoost model with Tune with the class-based API and a ResourceChangingScheduler, ensuring all resources are being used at all time.\n",
"- {doc}`/train/getting-started-xgboost`: Shows how to scale XGBoost single-model training with *Ray Train* (as opposed to hyperparameter tuning with Ray Tune).\n",
"\n",
"## Learn More\n",
"\n",
"- [XGBoost Hyperparameter Tuning - A Visual Guide](https://kevinvecmanis.io/machine%20learning/hyperparameter%20tuning/dataviz/python/2019/05/11/XGBoost-Tuning-Visual-Guide.html)\n",
"- [Notes on XGBoost Parameter Tuning](https://xgboost.readthedocs.io/en/latest/tutorials/param_tuning.html)\n",
"- [Doing XGBoost Hyperparameter Tuning the smart way](https://towardsdatascience.com/doing-xgboost-hyper-parameter-tuning-the-smart-way-part-1-of-2-f6d255a45dde)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "xgboost-tune",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.11"
},
"orphan": true
},
"nbformat": 4,
"nbformat_minor": 5
}
doc/source/tune/examples/tune_analyze_results.ipynb
+1013
View File
File diff suppressed because one or more lines are too long
doc/source/tune/examples/tune_mnist_keras.ipynb
+271
View File
@@ -0,0 +1,271 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "3b05af3b",
"metadata": {},
"source": [
"(tune-mnist-keras)=\n",
"\n",
"# Using Keras & TensorFlow with Tune\n",
"\n",
"<a id=\"try-anyscale-quickstart-tune_mnist_keras\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=tune_mnist_keras\">\n",
" <img src=\"../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
"</a>\n",
"<br></br>\n",
"\n",
"```{image} /images/tf_keras_logo.jpeg\n",
":align: center\n",
":alt: Keras & TensorFlow Logo\n",
":height: 120px\n",
":target: https://keras.io\n",
"```\n",
"\n",
"```{contents}\n",
":backlinks: none\n",
":local: true\n",
"```\n",
"\n",
"## Prerequisites\n",
"\n",
"- `pip install \"ray[tune]\" tensorflow==2.18.0 filelock`\n",
"\n",
"## Example"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "19e3c389",
"metadata": {
"tags": [
"hide-output"
]
},
"outputs": [
{
"data": {
"text/html": [
"<div class=\"tuneStatus\">\n",
" <div style=\"display: flex;flex-direction: row\">\n",
" <div style=\"display: flex;flex-direction: column;\">\n",
" <h3>Tune Status</h3>\n",
" <table>\n",
"<tbody>\n",
"<tr><td>Current time:</td><td>2025-02-13 15:22:41</td></tr>\n",
"<tr><td>Running for: </td><td>00:00:41.76 </td></tr>\n",
"<tr><td>Memory: </td><td>21.4/36.0 GiB </td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
" <div class=\"vDivider\"></div>\n",
" <div class=\"systemInfo\">\n",
" <h3>System Info</h3>\n",
" Using AsyncHyperBand: num_stopped=0<br>Bracket: Iter 320.000: None | Iter 80.000: None | Iter 20.000: None<br>Logical resource usage: 2.0/12 CPUs, 0/0 GPUs\n",
" </div>\n",
" \n",
" </div>\n",
" <div class=\"hDivider\"></div>\n",
" <div class=\"trialStatus\">\n",
" <h3>Trial Status</h3>\n",
" <table>\n",
"<thead>\n",
"<tr><th>Trial name </th><th>status </th><th>loc </th><th style=\"text-align: right;\"> hidden</th><th style=\"text-align: right;\"> learning_rate</th><th style=\"text-align: right;\"> momentum</th><th style=\"text-align: right;\"> iter</th><th style=\"text-align: right;\"> total time (s)</th><th style=\"text-align: right;\"> accuracy</th></tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td>train_mnist_533a2_00000</td><td>TERMINATED</td><td>127.0.0.1:36365</td><td style=\"text-align: right;\"> 371</td><td style=\"text-align: right;\"> 0.0799367 </td><td style=\"text-align: right;\"> 0.588387</td><td style=\"text-align: right;\"> 12</td><td style=\"text-align: right;\"> 20.8515</td><td style=\"text-align: right;\"> 0.984583</td></tr>\n",
"<tr><td>train_mnist_533a2_00001</td><td>TERMINATED</td><td>127.0.0.1:36364</td><td style=\"text-align: right;\"> 266</td><td style=\"text-align: right;\"> 0.0457424 </td><td style=\"text-align: right;\"> 0.22303 </td><td style=\"text-align: right;\"> 12</td><td style=\"text-align: right;\"> 19.5277</td><td style=\"text-align: right;\"> 0.96495 </td></tr>\n",
"<tr><td>train_mnist_533a2_00002</td><td>TERMINATED</td><td>127.0.0.1:36368</td><td style=\"text-align: right;\"> 157</td><td style=\"text-align: right;\"> 0.0190286 </td><td style=\"text-align: right;\"> 0.537132</td><td style=\"text-align: right;\"> 12</td><td style=\"text-align: right;\"> 16.6606</td><td style=\"text-align: right;\"> 0.95385 </td></tr>\n",
"<tr><td>train_mnist_533a2_00003</td><td>TERMINATED</td><td>127.0.0.1:36363</td><td style=\"text-align: right;\"> 451</td><td style=\"text-align: right;\"> 0.0433488 </td><td style=\"text-align: right;\"> 0.18925 </td><td style=\"text-align: right;\"> 12</td><td style=\"text-align: right;\"> 22.0514</td><td style=\"text-align: right;\"> 0.966283</td></tr>\n",
"<tr><td>train_mnist_533a2_00004</td><td>TERMINATED</td><td>127.0.0.1:36367</td><td style=\"text-align: right;\"> 276</td><td style=\"text-align: right;\"> 0.0336728 </td><td style=\"text-align: right;\"> 0.430171</td><td style=\"text-align: right;\"> 12</td><td style=\"text-align: right;\"> 20.0884</td><td style=\"text-align: right;\"> 0.964767</td></tr>\n",
"<tr><td>train_mnist_533a2_00005</td><td>TERMINATED</td><td>127.0.0.1:36366</td><td style=\"text-align: right;\"> 208</td><td style=\"text-align: right;\"> 0.071015 </td><td style=\"text-align: right;\"> 0.419166</td><td style=\"text-align: right;\"> 12</td><td style=\"text-align: right;\"> 17.933 </td><td style=\"text-align: right;\"> 0.976083</td></tr>\n",
"<tr><td>train_mnist_533a2_00006</td><td>TERMINATED</td><td>127.0.0.1:36475</td><td style=\"text-align: right;\"> 312</td><td style=\"text-align: right;\"> 0.00692959</td><td style=\"text-align: right;\"> 0.714595</td><td style=\"text-align: right;\"> 12</td><td style=\"text-align: right;\"> 13.058 </td><td style=\"text-align: right;\"> 0.944017</td></tr>\n",
"<tr><td>train_mnist_533a2_00007</td><td>TERMINATED</td><td>127.0.0.1:36479</td><td style=\"text-align: right;\"> 169</td><td style=\"text-align: right;\"> 0.0694114 </td><td style=\"text-align: right;\"> 0.664904</td><td style=\"text-align: right;\"> 12</td><td style=\"text-align: right;\"> 10.7991</td><td style=\"text-align: right;\"> 0.9803 </td></tr>\n",
"<tr><td>train_mnist_533a2_00008</td><td>TERMINATED</td><td>127.0.0.1:36486</td><td style=\"text-align: right;\"> 389</td><td style=\"text-align: right;\"> 0.0370836 </td><td style=\"text-align: right;\"> 0.665592</td><td style=\"text-align: right;\"> 12</td><td style=\"text-align: right;\"> 14.018 </td><td style=\"text-align: right;\"> 0.977833</td></tr>\n",
"<tr><td>train_mnist_533a2_00009</td><td>TERMINATED</td><td>127.0.0.1:36487</td><td style=\"text-align: right;\"> 389</td><td style=\"text-align: right;\"> 0.0676138 </td><td style=\"text-align: right;\"> 0.52372 </td><td style=\"text-align: right;\"> 12</td><td style=\"text-align: right;\"> 14.0043</td><td style=\"text-align: right;\"> 0.981833</td></tr>\n",
"</tbody>\n",
"</table>\n",
" </div>\n",
"</div>\n",
"<style>\n",
".tuneStatus {\n",
" color: var(--jp-ui-font-color1);\n",
"}\n",
".tuneStatus .systemInfo {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus td {\n",
" white-space: nowrap;\n",
"}\n",
".tuneStatus .trialStatus {\n",
" display: flex;\n",
" flex-direction: column;\n",
"}\n",
".tuneStatus h3 {\n",
" font-weight: bold;\n",
"}\n",
".tuneStatus .hDivider {\n",
" border-bottom-width: var(--jp-border-width);\n",
" border-bottom-color: var(--jp-border-color0);\n",
" border-bottom-style: solid;\n",
"}\n",
".tuneStatus .vDivider {\n",
" border-left-width: var(--jp-border-width);\n",
" border-left-color: var(--jp-border-color0);\n",
" border-left-style: solid;\n",
" margin: 0.5em 1em 0.5em 1em;\n",
"}\n",
"</style>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2025-02-13 15:22:41,843\tINFO tune.py:1009 -- Wrote the latest version of all result files and experiment state to '~/ray_results/exp' in 0.0048s.\n",
"2025-02-13 15:22:41,846\tINFO tune.py:1041 -- Total run time: 41.77 seconds (41.75 seconds for the tuning loop).\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best hyperparameters found were: {'threads': 2, 'learning_rate': 0.07993666231835218, 'momentum': 0.5883866709655042, 'hidden': 371} | Accuracy: 0.98458331823349\n"
]
}
],
"source": [
"import os\n",
"\n",
"from filelock import FileLock\n",
"from tensorflow.keras.datasets import mnist\n",
"\n",
"from ray import tune\n",
"from ray.tune.schedulers import AsyncHyperBandScheduler\n",
"from ray.tune.integration.keras import TuneReportCheckpointCallback\n",
"\n",
"\n",
"def train_mnist(config):\n",
" # https://github.com/tensorflow/tensorflow/issues/32159\n",
" import tensorflow as tf\n",
"\n",
" batch_size = 128\n",
" num_classes = 10\n",
" epochs = 12\n",
"\n",
" with FileLock(os.path.expanduser(\"~/.data.lock\")):\n",
" (x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
" x_train, x_test = x_train / 255.0, x_test / 255.0\n",
" model = tf.keras.models.Sequential(\n",
" [\n",
" tf.keras.layers.Flatten(input_shape=(28, 28)),\n",
" tf.keras.layers.Dense(config[\"hidden\"], activation=\"relu\"),\n",
" tf.keras.layers.Dropout(0.2),\n",
" tf.keras.layers.Dense(num_classes, activation=\"softmax\"),\n",
" ]\n",
" )\n",
"\n",
" model.compile(\n",
" loss=\"sparse_categorical_crossentropy\",\n",
" optimizer=tf.keras.optimizers.SGD(learning_rate=config[\"learning_rate\"], momentum=config[\"momentum\"]),\n",
" metrics=[\"accuracy\"],\n",
" )\n",
"\n",
" model.fit(\n",
" x_train,\n",
" y_train,\n",
" batch_size=batch_size,\n",
" epochs=epochs,\n",
" verbose=0,\n",
" validation_data=(x_test, y_test),\n",
" callbacks=[TuneReportCheckpointCallback(metrics={\"accuracy\": \"accuracy\"})],\n",
" )\n",
"\n",
"\n",
"def tune_mnist():\n",
" sched = AsyncHyperBandScheduler(\n",
" time_attr=\"training_iteration\", max_t=400, grace_period=20\n",
" )\n",
"\n",
" tuner = tune.Tuner(\n",
" tune.with_resources(train_mnist, resources={\"cpu\": 2, \"gpu\": 0}),\n",
" tune_config=tune.TuneConfig(\n",
" metric=\"accuracy\",\n",
" mode=\"max\",\n",
" scheduler=sched,\n",
" num_samples=10,\n",
" ),\n",
" run_config=tune.RunConfig(\n",
" name=\"exp\",\n",
" stop={\"accuracy\": 0.99},\n",
" ),\n",
" param_space={\n",
" \"threads\": 2,\n",
" \"learning_rate\": tune.uniform(0.001, 0.1),\n",
" \"momentum\": tune.uniform(0.1, 0.9),\n",
" \"hidden\": tune.randint(32, 512),\n",
" },\n",
" )\n",
" results = tuner.fit()\n",
" return results\n",
"\n",
" \n",
"\n",
"results = tune_mnist()\n",
"print(f\"Best hyperparameters found were: {results.get_best_result().config} | Accuracy: {results.get_best_result().metrics['accuracy']}\")\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "d7e46189",
"metadata": {},
"source": [
"This should output something like:\n",
"\n",
"```\n",
"Best hyperparameters found were: {'threads': 2, 'learning_rate': 0.07607440973606909, 'momentum': 0.7715363277240616, 'hidden': 452} | Accuracy: 0.98458331823349\n",
"```\n",
"\n",
"## More Keras and TensorFlow Examples\n",
"\n",
"- {doc}`/tune/examples/includes/pbt_memnn_example`: Example of training a Memory NN on bAbI with Keras using PBT.\n",
"- {doc}`/tune/examples/includes/tf_mnist_example`: Converts the Advanced TF2.0 MNIST example to use Tune\n",
" with the Trainable. This uses `tf.function`.\n",
" Original code from tensorflow: https://www.tensorflow.org/tutorials/quickstart/advanced\n",
"- {doc}`/tune/examples/includes/pbt_tune_cifar10_with_keras`:\n",
" A contributed example of tuning a Keras model on CIFAR10 with the PopulationBasedTraining scheduler.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "tune-keras",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.11"
},
"orphan": true
},
"nbformat": 4,
"nbformat_minor": 5
}
doc/source/tune/faq.rst
+708
View File
@@ -0,0 +1,708 @@
.. _tune-faq:
Ray Tune FAQ
------------
Here we try to answer questions that come up often.
If you still have questions after reading this FAQ, let us know!
.. contents::
:local:
:depth: 1
What are Hyperparameters?
~~~~~~~~~~~~~~~~~~~~~~~~~
What are *hyperparameters?* And how are they different from *model parameters*?
In supervised learning, we train a model with labeled data so the model can properly identify new data values.
Everything about the model is defined by a set of parameters, such as the weights in a linear regression. These
are *model parameters*; they are learned during training.
.. image:: /images/hyper-model-parameters.png
In contrast, the *hyperparameters* define structural details about the kind of model itself, like whether or not
we are using a linear regression or classification, what architecture is best for a neural network,
how many layers, what kind of filters, etc. They are defined before training, not learned.
.. image:: /images/hyper-network-params.png
Other quantities considered *hyperparameters* include learning rates, discount rates, etc. If we want our training
process and resulting model to work well, we first need to determine the optimal or near-optimal set of *hyperparameters*.
How do we determine the optimal *hyperparameters*? The most direct approach is to perform a loop where we pick
a candidate set of values from some reasonably inclusive list of possible values, train a model, compare the results
achieved with previous loop iterations, and pick the set that performed best. This process is called
*Hyperparameter Tuning* or *Optimization* (HPO). And *hyperparameters* are specified over a configured and confined
search space, collectively defined for each *hyperparameter* in a ``config`` dictionary.
.. TODO: We *really* need to improve this section.
Which search algorithm/scheduler should I choose?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Ray Tune offers :ref:`many different search algorithms <tune-search-alg>`
and :ref:`schedulers <tune-schedulers>`.
Deciding on which to use mostly depends on your problem:
* Is it a small or large problem (how long does it take to train? How costly
are the resources, like GPUs)? Can you run many trials in parallel?
* How many hyperparameters would you like to tune?
* What values are valid for hyperparameters?
**If your model returns incremental results** (eg. results per epoch in deep learning,
results per each added tree in GBDTs, etc.) using early stopping usually allows for sampling
more configurations, as unpromising trials are pruned before they run their full course.
Please note that not all search algorithms can use information from pruned trials.
Early stopping cannot be used without incremental results - in case of the functional API,
that means that ``tune.report()`` has to be called more than once - usually in a loop.
**If your model is small**, you can usually try to run many different configurations.
A **random search** can be used to generate configurations. You can also grid search
over some values. You should probably still use
:ref:`ASHA for early termination of bad trials <tune-scheduler-hyperband>` (if your problem
supports early stopping).
**If your model is large**, you can try to either use
**Bayesian Optimization-based search algorithms** like :ref:`BayesOpt <bayesopt>`
to get good parameter configurations after few
trials. :ref:`Ax <tune-ax>` is similar but more robust to noisy data.
Please note that these algorithms only work well with **a small number of hyperparameters**.
Alternatively, you can use :ref:`Population Based Training <tune-scheduler-pbt>` which
works well with few trials, e.g. 8 or even 4. However, this will output a hyperparameter *schedule* rather
than one fixed set of hyperparameters.
**If you have a small number of hyperparameters**, Bayesian Optimization methods
work well. Take a look at :ref:`BOHB <tune-scheduler-bohb>` or :ref:`Optuna <tune-optuna>`
with the :ref:`ASHA <tune-scheduler-hyperband>` scheduler to combine the
benefits of Bayesian Optimization with early stopping.
**If you only have continuous values for hyperparameters** this will work well
with most Bayesian Optimization methods. Discrete or categorical variables still
work, but less good with an increasing number of categories.
**If you have many categorical values for hyperparameters**, consider using random search,
or a TPE-based Bayesian Optimization algorithm such as :ref:`Optuna <tune-optuna>` or
:ref:`HyperOpt <tune-hyperopt>`.
**Our go-to solution** is usually to use **random search** with
:ref:`ASHA for early stopping <tune-scheduler-hyperband>` for smaller problems.
Use :ref:`BOHB <tune-scheduler-bohb>` for **larger problems** with a **small number of hyperparameters**
and :ref:`Population Based Training <tune-scheduler-pbt>` for **larger problems** with a
**large number of hyperparameters** if a learning schedule is acceptable.
How do I choose hyperparameter ranges?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A good start is to look at the papers that introduced the algorithms, and also
to see what other people are using.
Most algorithms also have sensible defaults for some of their parameters.
For instance, `XGBoost's parameter overview <https://xgboost.readthedocs.io/en/latest/parameter.html>`_
reports to use ``max_depth=6`` for the maximum decision tree depth. Here, anything
between 2 and 10 might make sense (though that naturally depends on your problem).
For **learning rates**, we suggest using a **loguniform distribution** between
**1e-5** and **1e-1**: ``tune.loguniform(1e-5, 1e-1)``.
For **batch sizes**, we suggest trying **powers of 2**, for instance, 2, 4, 8,
16, 32, 64, 128, 256, etc. The magnitude depends on your problem. For easy
problems with lots of data, use higher batch sizes, for harder problems with
not so much data, use lower batch sizes.
For **layer sizes** we also suggest trying **powers of 2**. For small problems
(e.g. Cartpole), use smaller layer sizes. For larger problems, try larger ones.
For **discount factors** in reinforcement learning we suggest sampling uniformly
between 0.9 and 1.0. Depending on the problem, a much stricter range above 0.97
or even above 0.99 can make sense (e.g. for Atari).
How can I use nested/conditional search spaces?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sometimes you might need to define parameters whose value depend on the value
of other parameters. Ray Tune offers some methods to define these.
Nested spaces
'''''''''''''
You can nest hyperparameter definition in sub dictionaries:
.. literalinclude:: doc_code/faq.py
:language: python
:start-after: __basic_config_start__
:end-before: __basic_config_end__
The trial config will be nested exactly like the input config.
Conditional spaces
''''''''''''''''''
:ref:`Custom and conditional search spaces are explained in detail here <tune_custom-search>`.
In short, you can pass custom functions to ``tune.sample_from()`` that can
return values that depend on other values:
.. literalinclude:: doc_code/faq.py
:language: python
:start-after: __conditional_spaces_start__
:end-before: __conditional_spaces_end__
Conditional grid search
'''''''''''''''''''''''
If you would like to grid search over two parameters that depend on each other,
this might not work out of the box. For instance say that *a* should be a value
between 5 and 10 and *b* should be a value between 0 and a. In this case, we
cannot use ``tune.sample_from`` because it doesn't support grid searching.
The solution here is to create a list of valid *tuples* with the help of a
helper function, like this:
.. literalinclude:: doc_code/faq.py
:language: python
:start-after: __iter_start__
:end-before: __iter_end__
Your trainable then can do something like ``a, b = config["ab"]`` to split
the a and b variables and use them afterwards.
How does early termination (e.g. Hyperband/ASHA) work?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Early termination algorithms look at the intermediately reported values,
e.g. what is reported to them via ``tune.report()`` after each training
epoch. After a certain number of steps, they then remove the worst
performing trials and keep only the best performing trials. Goodness of a trial
is determined by ordering them by the objective metric, for instance accuracy
or loss.
In ASHA, you can decide how many trials are early terminated.
``reduction_factor=4`` means that only 25% of all trials are kept each
time they are reduced. With ``grace_period=n`` you can force ASHA to
train each trial at least for ``n`` epochs.
Why are all my trials returning "1" iteration?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**This is most likely applicable for the Tune function API.**
Ray Tune counts iterations internally every time ``tune.report()`` is
called. If you only call ``tune.report()`` once at the end of the training,
the counter has only been incremented once. If you're using the class API,
the counter is increased after calling ``step()``.
Note that it might make sense to report metrics more often than once. For
instance, if you train your algorithm for 1000 timesteps, consider reporting
intermediate performance values every 100 steps. That way, schedulers
like Hyperband/ASHA can terminate bad performing trials early.
What are all these extra outputs?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You'll notice that Ray Tune not only reports hyperparameters (from the
``config``) or metrics (passed to ``tune.report()``), but also some other
outputs.
.. code-block:: bash
Result for easy_objective_c64c9112:
date: 2020-10-07_13-29-18
done: false
experiment_id: 6edc31257b564bf8985afeec1df618ee
experiment_tag: 7_activation=tanh,height=-53.116,steps=100,width=13.885
hostname: ubuntu
iterations: 0
iterations_since_restore: 1
mean_loss: 4.688385317424468
neg_mean_loss: -4.688385317424468
node_ip: 192.168.1.115
pid: 5973
time_since_restore: 7.605552673339844e-05
time_this_iter_s: 7.605552673339844e-05
time_total_s: 7.605552673339844e-05
timestamp: 1602102558
timesteps_since_restore: 0
training_iteration: 1
trial_id: c64c9112
See the :ref:`tune-autofilled-metrics` section for a glossary.
How do I set resources?
~~~~~~~~~~~~~~~~~~~~~~~
If you want to allocate specific resources to a trial, you can use the
``tune.with_resources`` and wrap it around you trainable together with
a dict or a :class:`PlacementGroupFactory <ray.tune.execution.placement_groups.PlacementGroupFactory>` object:
.. literalinclude:: doc_code/faq.py
:dedent:
:language: python
:start-after: __resources_start__
:end-before: __resources_end__
The example above showcases three things:
1. The `cpu` and `gpu` options set how many CPUs and GPUs are available for
each trial, respectively. **Trials cannot request more resources** than these
(exception: see 3).
2. It is possible to request **fractional GPUs**. A value of 0.5 means that
half of the memory of the GPU is made available to the trial. You will have
to make sure yourself that your model still fits on the fractional memory.
3. You can request custom resources you supplied to Ray when starting the cluster.
Trials will only be scheduled on single nodes that can provide all resources you
requested.
One important thing to keep in mind is that each Ray worker (and thus each
Ray Tune Trial) will only be scheduled on **one machine**. That means if
you for instance request 2 GPUs for your trial, but your cluster consists
of 4 machines with 1 GPU each, the trial will never be scheduled.
In other words, you will have to make sure that your Ray cluster
has machines that can actually fulfill your resource requests.
In some cases your trainable might want to start other remote actors, for instance if you're
leveraging distributed training via Ray Train. In these cases, you can use
:ref:`placement groups <ray-placement-group-doc-ref>` to request additional resources:
.. literalinclude:: doc_code/faq.py
:dedent:
:language: python
:start-after: __resources_pgf_start__
:end-before: __resources_pgf_end__
Here, you're requesting 2 additional CPUs for remote tasks. These two additional
actors do not necessarily have to live on the same node as your main trainable.
In fact, you can control this via the ``strategy`` parameter. In this example, ``PACK``
will try to schedule the actors on the same node, but allows them to be scheduled
on other nodes as well. Please refer to the
:ref:`placement groups documentation <ray-placement-group-doc-ref>` to learn more
about these placement strategies.
You can also allocate specific resources to a trial based on a custom rule via lambda functions.
For instance, if you want to allocate GPU resources to trials based on a setting in your param space:
.. literalinclude:: doc_code/faq.py
:dedent:
:language: python
:start-after: __resources_lambda_start__
:end-before: __resources_lambda_end__
Why is my training stuck and Ray reporting that pending actor or tasks cannot be scheduled?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is usually caused by Ray actors or tasks being started by the
trainable without the trainable resources accounting for them, leading to a deadlock.
This can also be "stealthily" caused by using other libraries in the trainable that are
based on Ray, such as Modin. In order to fix the issue, request additional resources for
the trial using :ref:`placement groups <ray-placement-group-doc-ref>`, as outlined in
the section above.
For example, if your trainable is using Modin dataframes, operations on those will spawn
Ray tasks. By allocating an additional CPU bundle to the trial, those tasks will be able
to run without being starved of resources.
.. literalinclude:: doc_code/faq.py
:dedent:
:language: python
:start-after: __modin_start__
:end-before: __modin_end__
How can I pass further parameter values to my trainable?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Ray Tune expects your trainable functions to accept only up to two parameters,
``config`` and ``checkpoint_dir``. But sometimes there are cases where
you want to pass constant arguments, like the number of epochs to run,
or a dataset to train on. Ray Tune offers a wrapper function to achieve
just that, called :func:`tune.with_parameters() <ray.tune.with_parameters>`:
.. literalinclude:: doc_code/faq.py
:language: python
:start-after: __huge_data_start__
:end-before: __huge_data_end__
This function works similarly to ``functools.partial``, but it stores
the parameters directly in the Ray object store. This means that you
can pass even huge objects like datasets, and Ray makes sure that these
are efficiently stored and retrieved on your cluster machines.
:func:`tune.with_parameters() <ray.tune.with_parameters>`
also works with class trainables. Please see
:func:`tune.with_parameters() <ray.tune.with_parameters>` for more details and examples.
How can I reproduce experiments?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Reproducing experiments and experiment results means that you get the exact same
results when running an experiment again and again. To achieve this, the
conditions have to be exactly the same each time you run the experiment.
In terms of ML training and tuning, this mostly concerns
the random number generators that are used for sampling in various places of the
training and tuning lifecycle.
Random number generators are used to create randomness, for instance to sample a hyperparameter
value for a parameter you defined. There is no true randomness in computing, rather
there are sophisticated algorithms that generate numbers that *seem* to be random and
fulfill all properties of a random distribution. These algorithms can be *seeded* with
an initial state, after which the generated random numbers are always the same.
.. literalinclude:: doc_code/faq.py
:language: python
:start-after: __seeded_1_start__
:end-before: __seeded_1_end__
The most commonly used random number generators from Python libraries are those in the
native ``random`` submodule and the ``numpy.random`` module.
.. literalinclude:: doc_code/faq.py
:language: python
:start-after: __seeded_2_start__
:end-before: __seeded_2_end__
In your tuning and training run, there are several places where randomness occurs, and
at all these places we will have to introduce seeds to make sure we get the same behavior.
* **Search algorithm**: Search algorithms have to be seeded to generate the same
hyperparameter configurations in each run. Some search algorithms can be explicitly instantiated with a
random seed (look for a ``seed`` parameter in the constructor). For others, try to use
the above code block.
* **Schedulers**: Schedulers like Population Based Training rely on resampling some
of the parameters, requiring randomness. Use the code block above to set the initial
seeds.
* **Training function**: In addition to initializing the configurations, the training
functions themselves have to use seeds. This could concern e.g. the data splitting.
You should make sure to set the seed at the start of your training function.
PyTorch and TensorFlow use their own RNGs, which have to be initialized, too:
.. literalinclude:: doc_code/faq.py
:language: python
:start-after: __torch_tf_seeds_start__
:end-before: __torch_tf_seeds_end__
You should thus seed both Ray Tune's schedulers and search algorithms, and the
training code. The schedulers and search algorithms should always be seeded with the
same seed. This is also true for the training code, but often it is beneficial that
the seeds differ *between different training runs*.
Here's a blueprint on how to do all this in your training code:
.. literalinclude:: doc_code/faq.py
:language: python
:start-after: __torch_seed_example_start__
:end-before: __torch_seed_example_end__
**Please note** that it is not always possible to control all sources of non-determinism.
For instance, if you use schedulers like ASHA or PBT, some trials might finish earlier
than other trials, affecting the behavior of the schedulers. Which trials finish first
can however depend on the current system load, network communication, or other factors
in the environment that we cannot control with random seeds. This is also true for search
algorithms such as Bayesian Optimization, which take previous results into account when
sampling new configurations. This can be tackled by
using the **synchronous modes** of PBT and Hyperband, where the schedulers wait for all trials to
finish an epoch before deciding which trials to promote.
We strongly advise to try reproduction on smaller toy problems first before relying
on it for larger experiments.
.. _tune-bottlenecks:
How can I avoid bottlenecks?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sometimes you might run into a message like this:
.. code-block::
The `experiment_checkpoint` operation took 2.43 seconds to complete, which may be a performance bottleneck
Most commonly, the ``experiment_checkpoint`` operation is throwing this warning, but it might be something else,
like ``process_trial_result``.
These operations should usually take less than 500ms to complete. When it consistently takes longer, this might
indicate a problem or inefficiencies. To get rid of this message, it is important to understand where it comes
from.
These are the main reasons this problem comes up:
**The Trial config is very large**
This is the case if you e.g. try to pass a dataset or other large object via the ``config`` parameter.
If this is the case, the dataset is serialized and written to disk repeatedly during experiment
checkpointing, which takes a long time.
**Solution**: Use :func:`tune.with_parameters <ray.tune.with_parameters>` to pass large objects to
function trainables via the objects store. For class trainables you can do this manually via ``ray.put()``
and ``ray.get()``. If you need to pass a class definition, consider passing an
indicator (e.g. a string) instead and let the trainable select the class instead. Generally, your config
dictionary should only contain primitive types, like numbers or strings.
**The Trial result is very large**
This is the case if you return objects, data, or other large objects via the return value of ``step()`` in
your class trainable or to ``tune.report()`` in your function trainable. The effect is the same as above:
The results are repeatedly serialized and written to disk, and this can take a long time.
**Solution**: Use checkpoint by writing data to the trainable's current working directory instead. There are various ways
to do that depending on whether you are using class or functional Trainable API.
**You are training a large number of trials on a cluster, or you are saving huge checkpoints**
**Solution**: You can use :ref:`cloud checkpointing <tune-cloud-checkpointing>` to save logs and checkpoints to a specified `storage_path`.
This is the preferred way to deal with this. All syncing will be taken care of automatically, as all nodes
are able to access the cloud storage. Additionally, your results will be safe, so even when you're working on
pre-emptible instances, you won't lose any of your data.
**You are reporting results too often**
Each result is processed by the search algorithm, trial scheduler, and callbacks (including loggers and the
trial syncer). If you're reporting a large number of results per trial (e.g. multiple results per second),
this can take a long time.
**Solution**: The solution here is obvious: Just don't report results that often. In class trainables, ``step()``
should maybe process a larger chunk of data. In function trainables, you can report only every n-th iteration
of the training loop. Try to balance the number of results you really need to make scheduling or searching
decisions. If you need more fine grained metrics for logging or tracking, consider using a separate logging
mechanism for this instead of the Ray Tune-provided progress logging of results.
How can I develop and test Tune locally?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
First, follow the instructions in :ref:`python-develop` to develop Tune without compiling Ray.
After Ray is set up, run ``pip install -r ray/python/ray/tune/requirements-dev.txt`` to install all packages
required for Tune development. Now, to run all Tune tests simply run:
.. code-block:: shell
pytest ray/python/ray/tune/tests/
If you plan to submit a pull request, we recommend you to run unit tests locally beforehand to speed up the review process.
Even though we have hooks to run unit tests automatically for each pull request, it's usually quicker to run them
on your machine first to avoid any obvious mistakes.
How can I get started contributing to Tune?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
We use GitHub to track issues, feature requests, and bugs. Take a look at the
ones labeled `"good first issue" <https://github.com/ray-project/ray/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+label%3A%22good-first-issue%22>`__ and `"help wanted" <https://github.com/ray-project/ray/issues?q=is%3Aopen+is%3Aissue+label%3A%22help-wanted%22>`__ for a place to start.
Look for issues with "[tune]" in the title.
.. note::
If raising a new issue or PR related to Tune, be sure to include "[tune]" in the title and add a ``tune`` label.
.. _tune-reproducible:
How can I make my Tune experiments reproducible?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Exact reproducibility of machine learning runs is hard to achieve. This
is even more true in a distributed setting, as more non-determinism is
introduced. For instance, if two trials finish at the same time, the
convergence of the search algorithm might be influenced by which trial
result is processed first. This depends on the searcher - for random search,
this shouldn't make a difference, but for most other searchers it will.
If you try to achieve some amount of reproducibility, there are two
places where you'll have to set random seeds:
1. On the driver program, e.g. for the search algorithm. This will ensure
that at least the initial configurations suggested by the search
algorithms are the same.
2. In the trainable (if required). Neural networks are usually initialized
with random numbers, and many classical ML algorithms, like GBDTs, make use of
randomness. Thus you'll want to make sure to set a seed here
so that the initialization is always the same.
Here is an example that will always produce the same result (except for trial
runtimes).
.. literalinclude:: doc_code/faq.py
:language: python
:start-after: __reproducible_start__
:end-before: __reproducible_end__
Some searchers use their own random states to sample new configurations.
These searchers usually accept a ``seed`` parameter that can be passed on
initialization. Other searchers use Numpy's ``np.random`` interface -
these seeds can be then set with ``np.random.seed()``. We don't offer an
interface to do this in the searcher classes as setting a random seed
globally could have side effects. For instance, it could influence the
way your dataset is split. Thus, we leave it up to the user to make
these global configuration changes.
How can I use large datasets in Tune?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You often will want to compute a large object (e.g., training data, model weights) on the driver and use that
object within each trial.
Tune provides a wrapper function ``tune.with_parameters()`` that allows you to broadcast large objects to your trainable.
Objects passed with this wrapper will be stored on the :ref:`Ray object store <objects-in-ray>` and will
be automatically fetched and passed to your trainable as a parameter.
.. tip:: If the objects are small in size or already exist in the :ref:`Ray Object Store <objects-in-ray>`, there's no need to use ``tune.with_parameters()``. You can use `partials <https://docs.python.org/3/library/functools.html#functools.partial>`__ or pass in directly to ``config`` instead.
.. literalinclude:: doc_code/faq.py
:language: python
:start-after: __large_data_start__
:end-before: __large_data_end__
.. _tune-cloud-syncing:
How can I upload my Tune results to cloud storage?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
See :ref:`tune-cloud-checkpointing`.
Make sure that worker nodes have the write access to the cloud storage.
Failing to do so would cause error messages like ``Error message (1): fatal error: Unable to locate credentials``.
For AWS set up, this involves adding an IamInstanceProfile configuration for worker nodes.
Please :ref:`see here for more tips <aws-cluster-s3>`.
.. _tune-kubernetes:
How can I use Tune with Kubernetes?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You should configure shared storage. See this user guide: :ref:`tune-storage-options`.
.. _tune-docker:
How can I use Tune with Docker?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You should configure shared storage. See this user guide: :ref:`tune-storage-options`.
.. _tune-default-search-space:
How do I configure search spaces?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can specify a grid search or sampling distribution via the dict passed into ``Tuner(param_space=...)``.
.. literalinclude:: doc_code/faq.py
:dedent:
:language: python
:start-after: __grid_search_start__
:end-before: __grid_search_end__
By default, each random variable and grid search point is sampled once.
To take multiple random samples, add ``num_samples: N`` to the experiment config.
If `grid_search` is provided as an argument, the grid will be repeated ``num_samples`` of times.
.. literalinclude:: doc_code/faq.py
:language: python
:start-after: __grid_search_2_start__
:end-before: __grid_search_2_end__
Note that search spaces may not be interoperable across different search algorithms.
For example, for many search algorithms, you will not be able to use a ``grid_search`` or ``sample_from`` parameters.
Read about this in the :ref:`Search Space API <tune-search-space>` page.
.. _tune-working-dir:
How do I access relative filepaths in my Tune training function?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Let's say you launch a Tune experiment with ``my_script.py`` from inside ``~/code``.
By default, Tune changes the working directory of each worker to its corresponding trial
directory (e.g. ``~/ray_results/exp_name/trial_0000x``). This guarantees separate working
directories for each worker process, avoiding conflicts when saving trial-specific outputs.
You can configure this by setting the `RAY_CHDIR_TO_TRIAL_DIR=0` environment variable.
This explicitly tells Tune to not change the working directory
to the trial directory, giving access to paths relative to the original working directory.
One caveat is that the working directory is now shared between workers, so the
:meth:`tune.get_context().get_trial_dir() <ray.tune.TuneContext.get_trial_dir>`
API should be used to get the path for saving trial-specific outputs.
.. literalinclude:: doc_code/faq.py
:dedent:
:emphasize-lines: 3, 10, 11, 12, 16
:language: python
:start-after: __no_chdir_start__
:end-before: __no_chdir_end__
.. warning::
The `TUNE_ORIG_WORKING_DIR` environment variable was the original workaround for
accessing paths relative to the original working directory. This environment
variable is deprecated, and the `RAY_CHDIR_TO_TRIAL_DIR` environment variable above
should be used instead.
.. _tune-multi-tenancy:
How can I run multiple Ray Tune jobs on the same cluster at the same time (multi-tenancy)?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Running multiple Ray Tune runs on the same cluster at the same
time is not officially supported. We do not test this workflow and we recommend
using a separate cluster for each tuning job.
The reasons for this are:
1. When multiple Ray Tune jobs run at the same time, they compete for resources.
One job could run all its trials at the same time, while the other job waits
for a long time until it gets resources to run the first trial.
2. If it is easy to start a new Ray cluster on your infrastructure, there is often
no cost benefit to running one large cluster instead of multiple smaller
clusters. For instance, running one cluster of 32 instances incurs almost the same
cost as running 4 clusters with 8 instances each.
3. Concurrent jobs are harder to debug. If a trial of job A fills the disk,
trials from job B on the same node are impacted. In practice, it's hard
to reason about these conditions from the logs if something goes wrong.
Previously, some internal implementations in Ray Tune assumed that you only have one job
running at a time. A symptom was when trials from job A used parameters specified in job B,
leading to unexpected results.
Please refer to
`this GitHub issue <https://github.com/ray-project/ray/issues/30091#issuecomment-1431676976>`__
for more context and a workaround if you run into this issue.
.. _tune-iterative-experimentation:
How can I continue training a completed Tune experiment for longer and with new configurations (iterative experimentation)?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Let's say that I have a Tune experiment that has completed with the following configurations:
.. literalinclude:: /tune/doc_code/faq.py
:language: python
:start-after: __iter_experimentation_initial_start__
:end-before: __iter_experimentation_initial_end__
Now, I want to continue training from a checkpoint (e.g., the best one) generated by the previous experiment,
and search over a new hyperparameter search space, for another ``10`` epochs.
:ref:`tune-fault-tolerance-ref` explains that the usage of :meth:`Tuner.restore <ray.tune.Tuner.restore>`
is meant for resuming an *unfinished* experiment that was interrupted in the middle,
according to the *exact configuration* that was supplied in the initial training run.
Therefore, ``Tuner.restore`` is not suitable for our desired behavior.
This style of "iterative experimentation" should be done with *new* Tune experiments
rather than restoring a single experiment over and over and modifying the experiment spec.
See the following for an example of how to create a new experiment that builds off of the old one:
.. literalinclude:: /tune/doc_code/faq.py
:language: python
:start-after: __iter_experimentation_resume_start__
:end-before: __iter_experimentation_resume_end__
doc/source/tune/getting-started.rst
+167
View File
@@ -0,0 +1,167 @@
.. _tune-tutorial:
.. TODO: make this an executable notebook later on.
Getting Started with Ray Tune
=============================
This tutorial will walk you through the process of setting up a Tune experiment.
To get started, we take a PyTorch model and show you how to leverage Ray Tune to
optimize the hyperparameters of this model.
Specifically, we'll leverage early stopping and Bayesian Optimization via HyperOpt to do so.
.. tip:: If you have suggestions on how to improve this tutorial,
please `let us know <https://github.com/ray-project/ray/issues/new/choose>`_!
To run this example, you will need to install the following:
.. code-block:: bash
$ pip install "ray[tune]" torch torchvision
Setting Up a PyTorch Model to Tune
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To start off, let's first import some dependencies.
We import some PyTorch and TorchVision modules to help us create a model and train it.
Also, we'll import Ray Tune to help us optimize the model.
As you can see we use a so-called scheduler, in this case the ``ASHAScheduler``
that we will use for tuning the model later in this tutorial.
.. literalinclude:: /../../python/ray/tune/tests/tutorial.py
:language: python
:start-after: __tutorial_imports_begin__
:end-before: __tutorial_imports_end__
Then, let's define a simple PyTorch model that we'll be training.
If you're not familiar with PyTorch, the simplest way to define a model is to implement a ``nn.Module``.
This requires you to set up your model with ``__init__`` and then implement a ``forward`` pass.
In this example we're using a small convolutional neural network consisting of one 2D convolutional layer, a fully
connected layer, and a softmax function.
.. literalinclude:: /../../python/ray/tune/tests/tutorial.py
:language: python
:start-after: __model_def_begin__
:end-before: __model_def_end__
Below, we have implemented functions for training and evaluating your PyTorch model.
We define a ``train`` and a ``test`` function for that purpose.
If you know how to do this, skip ahead to the next section.
.. dropdown:: Training and evaluating the model
.. literalinclude:: /../../python/ray/tune/tests/tutorial.py
:language: python
:start-after: __train_def_begin__
:end-before: __train_def_end__
.. _tutorial-tune-setup:
Setting up a ``Tuner`` for a Training Run with Tune
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Below, we define a function that trains the PyTorch model for multiple epochs.
This function will be executed on a separate :ref:`Ray Actor (process) <actor-guide>` underneath the hood,
so we need to communicate the performance of the model back to Tune (which is on the main Python process).
To do this, we call :func:`tune.report() <ray.tune.report>` in our training function,
which sends the performance value back to Tune. Since the function is executed on the separate process,
make sure that the function is :ref:`serializable by Ray <serialization-guide>`.
.. literalinclude:: /../../python/ray/tune/tests/tutorial.py
:language: python
:start-after: __train_func_begin__
:end-before: __train_func_end__
Let's run one trial by calling :ref:`Tuner.fit <tune-run-ref>` and :ref:`randomly sample <tune-search-space>`
from a uniform distribution for learning rate and momentum.
.. literalinclude:: /../../python/ray/tune/tests/tutorial.py
:language: python
:start-after: __eval_func_begin__
:end-before: __eval_func_end__
``Tuner.fit`` returns an :ref:`ResultGrid object <tune-analysis-docs>`.
You can use this to plot the performance of this trial.
.. literalinclude:: /../../python/ray/tune/tests/tutorial.py
:language: python
:start-after: __plot_begin__
:end-before: __plot_end__
.. note:: Tune will automatically run parallel trials across all available cores/GPUs on your machine or cluster.
To limit the number of concurrent trials, use the :ref:`ConcurrencyLimiter <limiter>`.
Early Stopping with Adaptive Successive Halving (ASHAScheduler)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Let's integrate early stopping into our optimization process. Let's use :ref:`ASHA <tune-scheduler-hyperband>`, a scalable algorithm for `principled early stopping`_.
.. _`principled early stopping`: https://blog.ml.cmu.edu/2018/12/12/massively-parallel-hyperparameter-optimization/
On a high level, ASHA terminates trials that are less promising and allocates more time and resources to more promising trials.
As our optimization process becomes more efficient, we can afford to **increase the search space by 5x**, by adjusting the parameter ``num_samples``.
ASHA is implemented in Tune as a "Trial Scheduler".
These Trial Schedulers can early terminate bad trials, pause trials, clone trials, and alter hyperparameters of a running trial.
See :ref:`the TrialScheduler documentation <tune-schedulers>` for more details of available schedulers and library integrations.
.. literalinclude:: /../../python/ray/tune/tests/tutorial.py
:language: python
:start-after: __run_scheduler_begin__
:end-before: __run_scheduler_end__
You can run the below in a Jupyter notebook to visualize trial progress.
.. literalinclude:: /../../python/ray/tune/tests/tutorial.py
:language: python
:start-after: __plot_scheduler_begin__
:end-before: __plot_scheduler_end__
.. image:: /images/tune-df-plot.png
:scale: 50%
:align: center
You can also use :ref:`TensorBoard <tensorboard>` for visualizing results.
.. code:: bash
$ tensorboard --logdir {logdir}
Using Search Algorithms in Tune
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In addition to :ref:`TrialSchedulers <tune-schedulers>`, you can further optimize your hyperparameters
by using an intelligent search technique like Bayesian Optimization.
To do this, you can use a Tune :ref:`Search Algorithm <tune-search-alg>`.
Search Algorithms leverage optimization algorithms to intelligently navigate the given hyperparameter space.
Note that each library has a specific way of defining the search space.
.. literalinclude:: /../../python/ray/tune/tests/tutorial.py
:language: python
:start-after: __run_searchalg_begin__
:end-before: __run_searchalg_end__
.. note:: Tune allows you to use some search algorithms in combination with different trial schedulers. See :ref:`this page for more details <tune-schedulers>`.
Evaluating Your Model after Tuning
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can evaluate the best trained model using the :ref:`ExperimentAnalysis object <tune-analysis-docs>` to retrieve the best model:
.. literalinclude:: /../../python/ray/tune/tests/tutorial.py
:language: python
:start-after: __run_analysis_begin__
:end-before: __run_analysis_end__
Next Steps
----------
* Check out the :ref:`Tune tutorials <tune-guides>` for guides on using Tune with your preferred machine learning library.
* Browse our :ref:`gallery of examples <tune-examples-others>` to see how to use Tune with PyTorch, XGBoost, Tensorflow, etc.
* `Let us know <https://github.com/ray-project/ray/issues>`__ if you ran into issues or have any questions by opening an issue on our GitHub.
* To check how your application is doing, you can use the :ref:`Ray dashboard <observability-getting-started>`.
doc/source/tune/images/ax.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 7.1 KiB

doc/source/tune/images/bayesopt.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.8 KiB

doc/source/tune/images/bohb.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.7 KiB

doc/source/tune/images/dragonfly.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 9.3 KiB

doc/source/tune/images/flaml.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.9 KiB

doc/source/tune/images/hebo.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

doc/source/tune/images/hyperopt.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 6.0 KiB

doc/source/tune/images/nevergrad.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 7.0 KiB

doc/source/tune/images/optuna.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 8.3 KiB

doc/source/tune/images/pbt.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 50 KiB

doc/source/tune/images/ray-tune-parcoords.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 546 KiB

doc/source/tune/images/ray-tune-tensorboard.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 84 KiB

doc/source/tune/images/ray-tune-viskit.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 58 KiB

doc/source/tune/images/serve_thumb.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 26 KiB

doc/source/tune/images/sigopt.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 6.8 KiB

doc/source/tune/images/sklearn_thumb.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 26 KiB

doc/source/tune/images/skopt.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.2 KiB

doc/source/tune/images/tune_flow.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 40 KiB

Some files were not shown because too many files have changed in this diff Show More