wehub/ray-project--ray
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:17:40 +08:00
commit f1825c8ceb
10096 changed files with 2364182 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/rllib/algorithm-config.rst
+289
View File
@@ -0,0 +1,289 @@
.. _rllib-algo-configuration-docs:
AlgorithmConfig API
===================
.. include:: /_includes/rllib/new_api_stack.rst
RLlib's :py:class:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig` API is
the auto-validated and type-safe gateway into configuring and building an RLlib
:py:class:`~ray.rllib.algorithms.algorithm.Algorithm`.
In essence, you first create an instance of :py:class:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig`
and then call some of its methods to set various configuration options. RLlib uses the following `black <https://github.com/psf/black>`__-compliant format
in all parts of its code.
Note that you can chain together more than one method call, including the constructor:
.. testcode::
from ray.rllib.algorithms.algorithm_config import AlgorithmConfig
config = (
# Create an `AlgorithmConfig` instance.
AlgorithmConfig()
# Change the learning rate.
.training(lr=0.0005)
# Change the number of Learner actors.
.learners(num_learners=2)
)
.. hint::
For value checking and type-safety reasons, you should never set attributes in your
:py:class:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig`
directly, but always go through the proper methods:
.. testcode::
# WRONG!
config.env = "CartPole-v1" # <- don't set attributes directly
# CORRECT!
config.environment(env="CartPole-v1") # call the proper method
Algorithm specific config classes
---------------------------------
You don't use the base ``AlgorithmConfig`` class directly in practice, but always its algorithm-specific
subclasses, such as :py:class:`~ray.rllib.algorithms.ppo.ppo.PPOConfig`. Each subclass comes
with its own set of additional arguments to the :py:meth:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig.training`
method.
Normally, you should pick the specific :py:class:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig`
subclass that matches the :py:class:`~ray.rllib.algorithms.algorithm.Algorithm`
you would like to run your learning experiments with. For example, if you would like to
use :ref:`IMPALA <impala>` as your algorithm, you should import its specific config class:
.. testcode::
from ray.rllib.algorithms.impala import IMPALAConfig
config = (
# Create an `IMPALAConfig` instance.
IMPALAConfig()
# Specify the RL environment.
.environment("CartPole-v1")
# Change the learning rate.
.training(lr=0.0004)
)
To change algorithm-specific settings, here for ``IMPALA``, also use the
:py:meth:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig.training` method:
.. testcode::
# Change an IMPALA-specific setting (the entropy coefficient).
config.training(entropy_coeff=0.01)
You can build the :py:class:`~ray.rllib.algorithms.impala.IMPALA` instance directly from the
config object through calling the
:py:meth:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig.build_algo` method:
.. testcode::
# Build the algorithm instance.
impala = config.build_algo()
.. testcode::
:hide:
impala.stop()
The config object stored inside any built :py:class:`~ray.rllib.algorithms.algorithm.Algorithm` instance
is a copy of your original config. This allows you to further alter your original config object and
build another algorithm instance without affecting the previously built one:
.. testcode::
# Further alter the config without affecting the previously built IMPALA object ...
config.training(lr=0.00123)
# ... and build a new IMPALA from it.
another_impala = config.build_algo()
.. testcode::
:hide:
another_impala.stop()
If you are working with `Ray Tune <https://docs.ray.io/en/latest/tune/index.html>`__,
pass your :py:class:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig`
instance into the constructor of the :py:class:`~ray.tune.tuner.Tuner`:
.. code-block:: python
from ray import tune
tuner = tune.Tuner(
"IMPALA",
param_space=config, # <- your RLlib AlgorithmConfig object
..
)
# Run the experiment with Ray Tune.
results = tuner.fit()
.. _rllib-algo-configuration-generic-settings:
Generic config settings
-----------------------
Most config settings are generic and apply to all of RLlib's :py:class:`~ray.rllib.algorithms.algorithm.Algorithm` classes.
The following sections walk you through the most important config settings users should pay close attention to before
diving further into other config settings and before starting with hyperparameter fine tuning.
RL Environment
~~~~~~~~~~~~~~
To configure, which :ref:`RL environment <rllib-environments-doc>` your algorithm trains against, use the ``env`` argument to the
:py:meth:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig.environment` method:
.. testcode::
config.environment("Humanoid-v5")
See this :ref:`RL environment guide <rllib-environments-doc>` for more details.
.. tip::
Install both `Atari <https://ale.farama.org/environments/>`__ and
`MuJoCo <https://gymnasium.farama.org/environments/mujoco>`__ to be able to run
all of RLlib's :ref:`tuned examples <rllib-tuned-examples-docs>`:
.. code-block:: bash
pip install "gymnasium[atari,accept-rom-license,mujoco]"
Learning rate `lr`
~~~~~~~~~~~~~~~~~~
Set the learning rate for updating your models through the ``lr`` argument to the
:py:meth:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig.training` method:
.. testcode::
config.training(lr=0.0001)
.. _rllib-algo-configuration-train-batch-size:
Train batch size
~~~~~~~~~~~~~~~~
Set the train batch size, per Learner actor,
through the ``train_batch_size_per_learner`` argument to the :py:meth:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig.training`
method:
.. testcode::
config.training(train_batch_size_per_learner=256)
.. note::
You can compute the total, effective train batch size through multiplying
``train_batch_size_per_learner`` with ``(num_learners or 1)``.
Or you can also just check the value of your config's
:py:attr:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig.total_train_batch_size` property:
.. testcode::
config.training(train_batch_size_per_learner=256)
config.learners(num_learners=2)
print(config.total_train_batch_size) # expect: 512 = 256 * 2
Discount factor `gamma`
~~~~~~~~~~~~~~~~~~~~~~~
Set the `RL discount factor <https://www.envisioning.io/vocab/discount-factor?utm_source=chatgpt.com>`__
through the ``gamma`` argument to the :py:meth:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig.training`
method:
.. testcode::
config.training(gamma=0.995)
Scaling with `num_env_runners` and `num_learners`
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. todo (sven): link to scaling guide, once separated out in its own rst.
Set the number of :py:class:`~ray.rllib.env.env_runner.EnvRunner` actors used to collect training samples
through the ``num_env_runners`` argument to the :py:meth:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig.env_runners`
method:
.. testcode::
config.env_runners(num_env_runners=4)
# Also use `num_envs_per_env_runner` to vectorize your environment on each EnvRunner actor.
# Note that this option is only available in single-agent setups.
# The Ray Team is working on a solution for this restriction.
config.env_runners(num_envs_per_env_runner=10)
Set the number of :py:class:`~ray.rllib.core.learner.learner.Learner` actors used to update your models
through the ``num_learners`` argument to the :py:meth:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig.learners`
method. This should correspond to the number of GPUs you have available for training.
.. testcode::
config.learners(num_learners=2)
Disable `explore` behavior
~~~~~~~~~~~~~~~~~~~~~~~~~~
Switch off/on exploratory behavior
through the ``explore`` argument to the :py:meth:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig.env_runners`
method. To compute actions, the :py:class:`~ray.rllib.env.env_runner.EnvRunner` calls `forward_exploration()` on the RLModule when ``explore=True``
and `forward_inference()` when ``explore=False``. The default value is ``explore=True``.
.. testcode::
# Disable exploration behavior.
# When False, the EnvRunner calls `forward_inference()` on the RLModule to compute
# actions instead of `forward_exploration()`.
config.env_runners(explore=False)
Rollout length
~~~~~~~~~~~~~~
Set the number of timesteps that each :py:class:`~ray.rllib.env.env_runner.EnvRunner` steps
through with each of its RL environment copies through the ``rollout_fragment_length`` argument.
Pass this argument to the :py:meth:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig.env_runners`
method. Note that some algorithms, like :py:class:`~ray.rllib.algorithms.ppo.PPO`,
set this value automatically, based on the :ref:`train batch size <rllib-algo-configuration-train-batch-size>`,
number of :py:class:`~ray.rllib.env.env_runner.EnvRunner` actors and number of envs per
:py:class:`~ray.rllib.env.env_runner.EnvRunner`.
.. testcode::
config.env_runners(rollout_fragment_length=50)
All available methods and their settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Besides the previously described most common settings, the :py:class:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig`
class and its algo-specific subclasses come with many more configuration options.
To structure things more semantically, :py:class:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig` groups
its various config settings into the following categories, each represented by its own method:
- :ref:`Config settings for the RL environment <rllib-config-env>`
- :ref:`Config settings for training behavior (including algo-specific settings) <rllib-config-training>`
- :ref:`Config settings for EnvRunners <rllib-config-env-runners>`
- :ref:`Config settings for Learners <rllib-config-learners>`
- :ref:`Config settings for adding callbacks <rllib-config-callbacks>`
- :ref:`Config settings for multi-agent setups <rllib-config-multi_agent>`
- :ref:`Config settings for offline RL <rllib-config-offline_data>`
- :ref:`Config settings for evaluating policies <rllib-config-evaluation>`
- :ref:`Config settings for the DL framework <rllib-config-framework>`
- :ref:`Config settings for reporting and logging behavior <rllib-config-reporting>`
- :ref:`Config settings for checkpointing <rllib-config-checkpointing>`
- :ref:`Config settings for debugging <rllib-config-debugging>`
- :ref:`Experimental config settings <rllib-config-experimental>`
To familiarize yourself with the vast number of RLlib's different config options, you can browse through
`RLlib's examples folder <https://github.com/ray-project/ray/tree/master/rllib/examples>`__ or take a look at this
:ref:`examples folder overview page <rllib-examples-overview-docs>`.
Each example script usually introduces a new config setting or shows you how to implement specific customizations through
a combination of setting certain config options and adding custom code to your experiment.
doc/source/rllib/checkpoints.rst
+465
View File
@@ -0,0 +1,465 @@
.. _rllib-checkpoints-docs:
Checkpointing
=============
.. include:: /_includes/rllib/new_api_stack.rst
RLlib offers a powerful checkpointing system for all its major classes, allowing you to save the
states of :py:class:`~ray.rllib.algorithms.algorithm.Algorithm` instances and their subcomponents
to local disk or cloud storage, and restore previously run experiment states and individual subcomponents.
This system allows you to continue training models from a previous state or deploy bare-bones PyTorch
models into production.
.. figure:: images/checkpointing/save_and_restore.svg
:width: 500
:align: left
**Saving to and restoring from disk or cloud storage**: Use the :py:meth:`~ray.rllib.utils.checkpoints.Checkpointable.save_to_path` method
to write the current state of any :py:meth:`~ray.rllib.utils.checkpoints.Checkpointable` component or your entire Algorithm to
disk or cloud storage. To load a saved state back into a running component or into your Algorithm, use
the :py:meth:`~ray.rllib.utils.checkpoints.Checkpointable.restore_from_path` method.
A checkpoint is a directory on disk or some `PyArrow <https://arrow.apache.org/>`__-supported cloud location, like
`gcs <https://cloud.google.com/storage>`__ or `S3 <https://aws.amazon.com/de/s3/>`__.
It contains architecture information, such as the class and the constructor arguments for creating a new instance,
a ``pickle`` or ``msgpack`` file with state information, and a human readable ``metadata.json`` file with information about the Ray version,
git commit, and checkpoint version.
You can generate a new :py:class:`~ray.rllib.algorithms.algorithm.Algorithm` instance or other subcomponent,
like an :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule`, from an existing checkpoint using
the :py:meth:`~ray.rllib.utils.checkpoints.Checkpointable.from_checkpoint` method.
For example, you can deploy a previously trained :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule`, without
any of the other RLlib components, into production.
.. figure:: images/checkpointing/from_checkpoint.svg
:width: 750
:align: left
**Creating a new instance directly from a checkpoint**: Use the ``classmethod``
:py:meth:`~ray.rllib.utils.checkpoints.Checkpointable.from_checkpoint` to instantiate objects directly
from a checkpoint. RLlib first uses the saved meta data to create a bare-bones instance of the originally
checkpointed object, and then restores its state from the state information in the checkpoint dir.
Another possibility is to load only a certain subcomponent's state into the containing
higher-level object. For example, you may want to load only the state of your :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule`,
located inside your :py:class:`~ray.rllib.algorithms.algorithm.Algorithm`, but leave all the other components
as-is.
Checkpointable API
------------------
RLlib manages checkpointing through the :py:class:`~ray.rllib.utils.checkpoints.Checkpointable` API,
which exposes the following three main methods:
- :py:meth:`~ray.rllib.utils.checkpoints.Checkpointable.save_to_path` for creating a new checkpoint
- :py:meth:`~ray.rllib.utils.checkpoints.Checkpointable.restore_from_path` for loading a state from a checkpoint into a running object
- :py:meth:`~ray.rllib.utils.checkpoints.Checkpointable.from_checkpoint` for creating a new object from a checkpoint
RLlib classes, which thus far support the :py:class:`~ray.rllib.utils.checkpoints.Checkpointable` API are:
- :py:class:`~ray.rllib.algorithms.algorithm.Algorithm`
- :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` (and :py:class:`~ray.rllib.core.rl_module.multi_rl_module.MultiRLModule`)
- :py:class:`~ray.rllib.env.env_runner.EnvRunner` (thus, also :py:class:`~ray.rllib.env.single_agent_env_runner.SingleAgentEnvRunner` and :py:class:`~ray.rllib.env.multi_agent_env_runner.MultiAgentEnvRunner`)
- :py:class:`~ray.rllib.connectors.connector_v2.ConnectorV2` (thus, also :py:class:`~ray.rllib.connectors.connector_pipeline_v2.ConnectorPipelineV2`)
- :py:class:`~ray.rllib.core.learner.learner_group.LearnerGroup`
- :py:class:`~ray.rllib.core.learner.learner.Learner`
.. _rllib-checkpoints-save-to-path:
Creating a new checkpoint with `save_to_path()`
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You create a new checkpoint from an instantiated RLlib object through the
:py:meth:`~ray.rllib.utils.checkpoints.Checkpointable.save_to_path` method.
The following are two examples, single- and multi-agent, using the :py:class:`~ray.rllib.algorithms.algorithm.Algorithm` class, showing
how to create checkpoints:
.. tab-set::
.. tab-item:: Single-agent setup
.. testcode::
from ray.rllib.algorithms.ppo import PPOConfig
# Configure and build an initial algorithm.
config = (
PPOConfig()
.environment("Pendulum-v1")
)
ppo = config.build()
# Train for one iteration, then save to a checkpoint.
print(ppo.train())
checkpoint_dir = ppo.save_to_path()
print(f"saved algo to {checkpoint_dir}")
.. testcode::
:hide:
_weights_check = ppo.get_module("default_policy").get_state()
ppo.stop()
.. tab-item:: Multi-agent setup
.. testcode::
from ray.rllib.algorithms.ppo import PPOConfig
from ray.rllib.examples.envs.classes.multi_agent import MultiAgentPendulum
from ray.tune import register_env
register_env("multi-pendulum", lambda cfg: MultiAgentPendulum({"num_agents": 2}))
# Configure and build an initial algorithm.
multi_agent_config = (
PPOConfig()
.environment("multi-pendulum")
.multi_agent(
policies={"p0", "p1"},
# Agent IDs are 0 and 1 -> map to p0 and p1, respectively.
policy_mapping_fn=lambda aid, eps, **kw: f"p{aid}"
)
)
ppo = multi_agent_config.build()
# Train for one iteration, then save to a checkpoint.
print(ppo.train())
multi_agent_checkpoint_dir = ppo.save_to_path()
print(f"saved multi-agent algo to {multi_agent_checkpoint_dir}")
.. testcode::
:hide:
ppo.stop()
.. note::
When running your experiments with `Ray Tune <https://docs.ray.io/en/latest/tune/index.html>`__,
Tune calls the :py:meth:`~ray.rllib.utils.checkpoints.Checkpointable.save_to_path`
method automatically on the :py:class:`~ray.rllib.algorithms.algorithm.Algorithm` instance, whenever the training
iteration matches the checkpoint frequency configured through Tune. The default location where Tune creates these checkpoints
is ``~/ray_results/[your experiment name]/[Tune trial name]/checkpoint_[sequence number]``.
Checkpoint versions
+++++++++++++++++++
RLlib uses a checkpoint versioning system to figure out how to restore an Algorithm or any
subcomponent from a given directory.
From Ray 2.40 on, you can find the checkpoint version in the human readable ``metadata.json``
file inside all checkpoint directories.
Also starting from `Ray 2.40`, RLlib checkpoints are backward compatible. This means that
a checkpoint created with Ray `2.x` can be read and handled by `Ray 2.x+n`, as long as `x >= 40`.
The Ray team ensures backward compatibility with
`comprehensive CI tests on checkpoints taken with previous Ray versions <https://github.com/ray-project/ray/blob/master/rllib/utils/tests/test_checkpointable.py>`__.
.. _rllib-checkpoints-structure-of-checkpoint-dir:
Structure of a checkpoint directory
+++++++++++++++++++++++++++++++++++
After saving your PPO's state in the ``checkpoint_dir`` directory, or somewhere in ``~/ray_results/`` if you use Ray Tune,
the directory looks like the following:
.. code-block:: shell
$ cd [your algo checkpoint dir]
$ ls -la
.
..
env_runner/
learner_group/
algorithm_state.pkl
class_and_ctor_args.pkl
metadata.json
Subdirectories inside a checkpoint dir, like ``env_runner/``, hint at a subcomponent's own checkpoint data.
For example, an :py:class:`~ray.rllib.algorithms.algorithm.Algorithm` always also saves its
:py:class:`~ray.rllib.env.env_runner.EnvRunner` state and :py:class:`~ray.rllib.core.learner.learner_group.LearnerGroup` state.
.. note::
Each of the subcomponent's directories themselves contain a ``metadata.json`` file, a ``class_and_ctor_args.pkl`` file,
and a ``pickle`` or ``msgpack`` state file, all serving the same purpose as their counterparts in the main algorithm checkpoint directory.
For example, inside the ``learner_group/`` subdirectory, you would find the :py:class:`~ray.rllib.core.learner.learner_group.LearnerGroup`'s own
architecture, state, and meta information:
.. code-block:: shell
$ cd env_runner/
$ ls -la
.
..
state.pkl
class_and_ctor_args.pkl
metadata.json
See :ref:`RLlib component tree <rllib-checkpoints-component-tree>` for details.
The ``metadata.json`` file exists for your convenience only and RLlib doesn't need it.
.. note::
The ``metadata.json`` file contains information about the Ray version used to create the checkpoint,
the Ray commit, the RLlib checkpoint version, and the names of the state- and constructor-information
files in the same directory.
.. code-block:: shell
$ more metadata.json
{
"class_and_ctor_args_file": "class_and_ctor_args.pkl",
"state_file": "state",
"ray_version": ..,
"ray_commit": ..,
"checkpoint_version": "2.1"
}
The ``class_and_ctor_args.pkl`` file stores meta information needed to construct a "fresh" object, without any particular state.
This information, as the filename suggests, contains the class of the saved object and its constructor arguments and keyword arguments.
RLlib uses this file to create the initial new object when calling :py:meth:`~ray.rllib.utils.checkpoints.Checkpointable.from_checkpoint`.
Finally, the ``.._state.[pkl|msgpack]`` file contains the pickled or msgpacked state dict of the saved object.
RLlib obtains this state dict, when saving a checkpoint, through calling the object's
:py:meth:`~ray.rllib.utils.checkpoints.Checkpointable.get_state` method.
.. note::
Support for ``msgpack`` based checkpoints is experimental, but might become the default in the future.
Unlike ``pickle``, ``msgpack`` has the advantage of being independent of the python-version, thus allowing
users to recover experiment and model states from old checkpoints they have generated with older python
versions.
The Ray team is working on completely separating state from architecture within checkpoints, meaning all state
information should go into the ``state.msgpack`` file, which is python-version independent,
whereas all architecture information should go into the ``class_and_ctor_args.pkl`` file, which still depends on
the python version. At the time of loading from checkpoint, the user would have to provide the latter/architecture part
of the checkpoint.
`See here for an example that illustrates this in more detail <https://github.com/ray-project/ray/blob/master/rllib/examples/checkpoints/change_config_during_training.py>`__.
.. _rllib-checkpoints-component-tree:
RLlib component tree
+++++++++++++++++++++++
The following is the structure of the RLlib component tree, showing under which name you can
access a subcomponent's own checkpoint within the higher-level checkpoint. At the highest level
is the :py:class:`~ray.rllib.algorithms.algorithm.Algorithm` class:
.. code-block:: shell
algorithm/
learner_group/
learner/
rl_module/
default_policy/ # <- single-agent case
[module ID 1]/ # <- multi-agent case
[module ID 2]/ # ...
env_runner/
env_to_module_connector/
module_to_env_connector/
.. note::
The ``env_runner/`` subcomponent currently doesn't hold a copy of the :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule`
checkpoint because it's already saved under ``learner/``. The Ray team is working on resolving
this issue, probably through soft-linking to avoid duplicate files and unnecessary disk usage.
.. _rllib-checkpoints-from-checkpoint:
Creating instances from a checkpoint with `from_checkpoint`
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Once you have a checkpoint of either a trained :py:class:`~ray.rllib.algorithms.algorithm.Algorithm` or
any of its :ref:`subcomponents <rllib-checkpoints-component-tree>`, you can recreate new objects directly
from this checkpoint.
The following are two examples:
.. tab-set::
.. tab-item:: Create a new Algorithm from a checkpoint
To recreate an entire :py:class:`~ray.rllib.algorithms.algorithm.Algorithm`
instance from a checkpoint, you can do the following:
.. testcode::
# Import the correct class to create from scratch using the checkpoint.
from ray.rllib.algorithms.algorithm import Algorithm
# Use the already existing checkpoint in `checkpoint_dir`.
new_ppo = Algorithm.from_checkpoint(checkpoint_dir)
# Confirm the `new_ppo` matches the originally checkpointed one.
assert new_ppo.config.env == "Pendulum-v1"
# Continue training.
new_ppo.train()
.. testcode::
:hide:
new_ppo.stop()
.. tab-item:: Create a new RLModule from an Algorithm checkpoint
Creating a new RLModule from an Algorithm checkpoint is useful when deploying trained models
into production or evaluating them in a separate process while training is ongoing.
To recreate only the :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` from
the algorithm's checkpoint, you can do the following.
.. testcode::
from pathlib import Path
import torch
# Import the correct class to create from scratch using the checkpoint.
from ray.rllib.core.rl_module.rl_module import RLModule
# Use the already existing checkpoint in `checkpoint_dir`, but go further down
# into its subdirectory for the single RLModule.
# See the preceding section on "RLlib component tree" for the various elements in the RLlib
# component tree.
rl_module_checkpoint_dir = Path(checkpoint_dir) / "learner_group" / "learner" / "rl_module" / "default_policy"
# Now that you have the correct subdirectory, create the actual RLModule.
rl_module = RLModule.from_checkpoint(rl_module_checkpoint_dir)
# Run a forward pass to compute action logits.
# Use a dummy Pendulum observation tensor (3d) and add a batch dim (B=1).
results = rl_module.forward_inference(
{"obs": torch.tensor([0.5, 0.25, -0.3]).unsqueeze(0).float()}
)
print(results)
See this `example of how to run policy inference after training <https://github.com/ray-project/ray/blob/master/rllib/examples/inference/policy_inference_after_training.py>`__
and this `example of how to run policy inference with an LSTM <https://github.com/ray-project/ray/blob/master/rllib/examples/inference/policy_inference_after_training_w_connector.py>`__.
.. hint::
Because your :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` is also a
`PyTorch Module <https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module>`__,
you can easily export your model to `ONNX <https://onnx.ai/>`__, `IREE <https://iree.dev/>`__,
or other deployment-friendly formats.
See this `example script supporting ONNX <https://github.com/ray-project/ray/blob/master/rllib/examples/inference/policy_inference_after_training.py>`__ for more details.
Restoring state from a checkpoint with `restore_from_path`
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Normally, the :py:meth:`~ray.rllib.utils.checkpoints.Checkpointable.save_to_path` and
:py:meth:`~ray.rllib.utils.checkpoints.Checkpointable.from_checkpoint` methods are all you need to create
checkpoints and re-create instances from them.
However, sometimes, you already have an instantiated object up and running and would like to "load" another
state into it. For example, consider training two :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` networks
through multi-agent training, playing against each other in a self-play fashion. After a while, you would like to swap out,
without interrupting your experiment, one of the ``RLModules`` with a third one that you have saved to disk or cloud storage a while back.
This is where the :py:meth:`~ray.rllib.utils.checkpoints.Checkpointable.restore_from_path` method comes in handy.
It loads a state into an already running object, for example your Algorithm, or into a subcomponent of that object,
for example a particular :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` within your :py:class:`~ray.rllib.algorithms.algorithm.Algorithm`.
.. tab-set::
.. tab-item:: Continue training
When using RLlib directly, meaning without Ray Tune, the problem of loading a state
into a running instance is straightforward:
.. testcode::
# Recreate the preceding PPO from the config.
new_ppo = config.build()
# Load the state stored previously in `checkpoint_dir` into the
# running algorithm instance.
new_ppo.restore_from_path(checkpoint_dir)
# Run another training iteration.
new_ppo.train()
.. testcode::
:hide:
new_ppo.stop()
.. tab-item:: Continue training with Ray Tune
However, when running through Ray Tune, you don't have direct access to the
Algorithm object or any of its subcomponents.
You can use :ref:`RLlib's callbacks APIs <rllib-callback-docs>` to inject custom code and solve for this.
Also, see here for an
`example on how to continue training with a different config <https://github.com/ray-project/ray/blob/master/rllib/examples/checkpoints/change_config_during_training.py>`__.
.. testcode::
from ray import tune
# Reuse the preceding PPOConfig (`config`).
# Inject custom callback code that runs right after algorithm's initialization.
config.callbacks(
on_algorithm_init=(
lambda algorithm, _dir=checkpoint_dir, **kw: algorithm.restore_from_path(_dir)
),
)
# Run the experiment, continuing from the checkpoint, through Ray Tune.
results = tune.Tuner(
config.algo_class,
param_space=config,
run_config=tune.RunConfig(stop={"num_env_steps_sampled_lifetime": 4000})
).fit()
.. tab-item:: Swap out one RLModule and continue multi-agent training
In the :ref:`preceding section on save_to_path <rllib-checkpoints-save-to-path>`, you created
a single-agent checkpoint with the ``default_policy`` ModuleID, and a multi-agent checkpoint with two ModuleIDs,
``p0`` and ``p1``.
Here is how you can continue training the multi-agent experiment, but swap out ``p1`` with
the state of the ``default_policy`` from the single-agent experiment.
You can use :ref:`RLlib's callbacks APIs <rllib-callback-docs>` to inject custom
code into a Ray Tune experiment:
.. testcode::
# Reuse the preceding multi-agent PPOConfig (`multi_agent_config`).
# But swap out ``p1`` with the state of the ``default_policy`` from the
# single-agent run, using a callback and the correct path through the
# RLlib component tree:
multi_rl_module_component_tree = "learner_group/learner/rl_module"
# Inject custom callback code that runs right after algorithm's initialization.
def _on_algo_init(algorithm, **kwargs):
algorithm.restore_from_path(
# Checkpoint was single-agent (has "default_policy" subdir).
path=Path(checkpoint_dir) / multi_rl_module_component_tree / "default_policy",
# Algo is multi-agent (has "p0" and "p1" subdirs).
component=multi_rl_module_component_tree + "/p1",
)
# Inject callback.
multi_agent_config.callbacks(on_algorithm_init=_on_algo_init)
# Run the experiment through Ray Tune.
results = tune.Tuner(
multi_agent_config.algo_class,
param_space=multi_agent_config,
run_config=tune.RunConfig(stop={"num_env_steps_sampled_lifetime": 4000})
).fit()
.. testcode::
:hide:
from ray.rllib.utils.test_utils import check
_weights_check_2 = multi_agent_config.build().get_module("p1").get_state()
check(_weights_check, _weights_check_2)
doc/source/rllib/connector-v2.rst
+197
View File
@@ -0,0 +1,197 @@
.. _connector-v2-docs:
.. grid:: 1 2 3 4
:gutter: 1
:class-container: container pb-3
.. grid-item-card::
:img-top: /rllib/images/connector_v2/connector_generic.svg
:class-img-top: pt-2 w-75 d-block mx-auto fixed-height-img
.. button-ref:: connector-v2-docs
ConnectorV2 overview (this page)
.. grid-item-card::
:img-top: /rllib/images/connector_v2/env_to_module_connector.svg
:class-img-top: pt-2 w-75 d-block mx-auto fixed-height-img
.. button-ref:: env-to-module-pipeline-docs
Env-to-module pipelines
.. grid-item-card::
:img-top: /rllib/images/connector_v2/learner_connector.svg
:class-img-top: pt-2 w-75 d-block mx-auto fixed-height-img
.. button-ref:: learner-pipeline-docs
Learner connector pipelines
ConnectorV2 and ConnectorV2 pipelines
=====================================
.. toctree::
:hidden:
env-to-module-connector
learner-connector
.. include:: /_includes/rllib/new_api_stack.rst
RLlib stores and transports all trajectory data in the form of :py:class:`~ray.rllib.env.single_agent_episode.SingleAgentEpisode`
or :py:class:`~ray.rllib.env.multi_agent_episode.MultiAgentEpisode` objects.
**Connector pipelines** are the components that translate this episode data into tensor batches
readable by neural network models right before the model forward pass.
.. figure:: images/connector_v2/generic_connector_pipeline.svg
:width: 1000
:align: left
**Generic ConnectorV2 Pipeline**: All pipelines consist of one or more :py:class:`~ray.rllib.connectors.connector_v2.ConnectorV2` pieces.
When calling the pipeline, you pass in a list of Episodes, the :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` instance,
and a batch, which initially might be an empty dict.
Each :py:class:`~ray.rllib.connectors.connector_v2.ConnectorV2` piece in the pipeline takes its predecessor's output,
starting on the left side with the batch, performs some transformations on the episodes, the batch, or both, and passes everything
on to the next piece. Thereby, all :py:class:`~ray.rllib.connectors.connector_v2.ConnectorV2` pieces can read from and write to the
provided episodes, add any data from these episodes to the batch, or change the data that's already in the batch.
The pipeline then returns the output batch of the last piece.
.. note::
Note that the batch output of the pipeline lives only as long as the succeeding
:py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` forward pass or `Env.step()` call. RLlib discards the data afterwards.
The list of episodes, however, may persist longer. For example, if a env-to-module pipeline reads an observation from an episode,
mutates that observation, and then writes it back into the episode, the subsequent module-to-env pipeline is able to see the changed observation.
Also, the Learner pipeline operates on the same episodes that have already passed through both env-to-module and module-to-env pipelines
and thus might have undergone changes.
Three ConnectorV2 pipeline types
--------------------------------
There are three different types of connector pipelines in RLlib:
1) :ref:`Env-to-module pipeline <env-to-module-pipeline-docs>`, which creates tensor batches for action computing forward passes.
2) Module-to-env pipeline (documentation pending), which translates a model's output into RL environment actions.
3) :ref:`Learner connector pipeline <learner-pipeline-docs>`, which creates the train batch for a model update.
The :py:class:`~ray.rllib.connectors.connector_v2.ConnectorV2` API is an extremely powerful tool for
customizing your RLlib experiments and algorithms. It allows you to take full control over accessing, changing, and re-assembling
the episode data collected from your RL environments or your offline RL input files as well as controlling the exact
nature and shape of the tensor batches that RLlib feeds into your models for computing actions or losses.
.. figure:: images/connector_v2/location_of_connector_pipelines_in_rllib.svg
:width: 900
:align: left
**ConnectorV2 Pipelines**: Connector pipelines convert episodes into batched data, which your model can process
(env-to-module and Learner) or convert your model's output into action batches, which your possibly vectorized RL environment needs for
stepping (module-to-env).
The env-to-module pipeline, located on an :py:class:`~ray.rllib.env.env_runner.EnvRunner`, takes a list of
episodes as input and outputs a batch for an :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` forward pass
that computes the next action. The module-to-env pipeline on the same :py:class:`~ray.rllib.env.env_runner.EnvRunner`
takes the output of that :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` and converts it into actions
for the next call to your RL environment's `step()` method.
Lastly, a Learner connector pipeline, located on a :py:class:`~ray.rllib.core.learner.learner.Learner`
worker, converts a list of episodes into a train batch for the next :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` update.
The succeeding pages discuss the three pipeline types in more detail, however, all three have in common:
* All connector pipelines are sequences of one or more :py:class:`~ray.rllib.connectors.connector_v2.ConnectorV2` pieces. You can nest these as well, meaning some of the pieces may be connector pipelines themselves.
* All connector pieces and -pipelines are Python callables, overriding the :py:meth:`~ray.rllib.connectors.connector_v2.ConnectorV2.__call__` method.
* The call signatures are uniform across the different pipeline types. The main, mandatory arguments are the list of episodes, the batch to-be-built, and the :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` instance. See the :py:meth:`~ray.rllib.connectors.connector_v2.ConnectorV2.__call__` method for more details.
* All connector pipelines can read from and write to the provided list of episodes as well as the batch and thereby perform data transforms as required.
Batch construction phases and formats
-------------------------------------
When you push a list of input episodes through a connector pipeline, the pipeline constructs a batch from the given data.
This batch always starts as an empty python dictionary and undergoes different formats and phases while passing through the different
pieces of the pipeline.
The following applies to all :ref:`env-to-module <env-to-module-pipeline-docs>` and learner connector pipelines (documentation in progress).
.. figure:: images/connector_v2/pipeline_batch_phases_single_agent.svg
:width: 1000
:align: left
**Batch construction phases and formats**: In the standard single-agent case, where only one ModuleID (``DEFAULT_MODULE_ID``) exists,
the batch starts as an empty dictionary (left) and then undergoes a "collect data" phase, in which connector pieces add individual items
to the batch by storing them under a) the column name, for example ``obs`` or ``rewards``, and b) under the episode ID, from which they extracted
the item.
In most cases, your custom connector pieces operate during this phase. Once all custom pieces have performed their data insertions and transforms,
the :py:class:`~ray.rllib.connectors.common.agent_to_module_mapping.AgentToModuleMapping` default piece performs a
"reorganize by ModuleID" operation (center), during which the batch's dictionary hierarchy changes to having the ModuleID (``DEFAULT_MODULE_ID``) at
the top level and the column names thereunder. On the lowest level in the batch, data items still reside in python lists.
Finally, the :py:class:`~ray.rllib.connectors.common.batch_individual_items.BatchIndividualItems` default piece creates NumPy arrays
out of the python lists, thereby batching all data (right).
For multi-agent setups, where there are more than one ModuleIDs the
:py:class:`~ray.rllib.connectors.common.agent_to_module_mapping.AgentToModuleMapping` default connector piece makes sure that
the constructed output batch maps module IDs to the respective module's forward batch:
.. figure:: images/connector_v2/pipeline_batch_phases_multi_agent.svg
:width: 1100
:align: left
**Batch construction for multi-agent**: In a multi-agent setup, the default :py:class:`~ray.rllib.connectors.common.agent_to_module_mapping.AgentToModuleMapping`
connector piece reorganizes the batch by ``ModuleID``, then column names, such that a
:py:class:`~ray.rllib.core.rl_module.multi_rl_module.MultiRLModule` can loop through its sub-modules and provide each with a batch
for the forward pass.
RLlib's :py:class:`~ray.rllib.core.rl_module.multi_rl_module.MultiRLModule` can split up the forward passes into
individual submodules' forward passes using the individual batches under the respective ``ModuleIDs``.
See :ref:`here for how to write your own multi-module or multi-agent forward logic <implementing-custom-multi-rl-modules>`
and override this default behavior of :py:class:`~ray.rllib.core.rl_module.multi_rl_module.MultiRLModule`.
Finally, if you have a stateful :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule`, for example an LSTM, RLlib adds two additional
default connector pieces to the pipeline, :py:class:`~ray.rllib.connectors.common.add_time_dim_to_batch_and_zero_pad.AddTimeDimToBatchAndZeroPad`
and :py:class:`~ray.rllib.connectors.common.add_states_from_episodes_to_batch.AddStatesFromEpisodesToBatch`:
.. figure:: images/connector_v2/pipeline_batch_phases_single_agent_w_states.svg
:width: 900
:align: left
**Batch construction for stateful models**: For stateful :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` instances,
RLlib automatically adds additional two default connector pieces to the pipeline. The
:py:class:`~ray.rllib.connectors.common.add_time_dim_to_batch_and_zero_pad.AddTimeDimToBatchAndZeroPad` piece converts all lists of individual data
items on the lowest batch level into sequences of a fixed length (``max_seq_len``, see note below for how to set this) and automatically zero-pads
these if it encounters an episode end.
The :py:class:`~ray.rllib.connectors.common.add_states_from_episodes_to_batch.AddStatesFromEpisodesToBatch` piece adds the previously generated
``state_out`` values of your :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` under the ``state_in`` column name to the batch. Note that
RLlib only adds the ``state_in`` values for the first timestep in each sequence and therefore also doesn't add a time dimension to the data in the
``state_in`` column.
.. note::
To change the zero-padded sequence length for the :py:class:`~ray.rllib.connectors.common.add_time_dim_to_batch_and_zero_pad.AddTimeDimToBatchAndZeroPad`
connector, set in your config for custom models:
.. code-block:: python
config.rl_module(model_config={"max_seq_len": ...})
And for RLlib's default models:
.. code-block:: python
from ray.rllib.core.rl_module.default_model_config import DefaultModelConfig
config.rl_module(model_config=DefaultModelConfig(max_seq_len=...))
.. Debugging ConnectorV2 Pipelines
.. ===============================
.. TODO (sven): Move the following to the "how to contribute to RLlib" page and rename that page "how to develop, debug and contribute to RLlib?"
.. You can debug your custom ConnectorV2 pipelines (and any RLlib component in general) through the following simple steps:
.. Run without any remote :py:class:`~ray.rllib.env.env_runner.EnvRunner` workers. After defining your :py:class:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig` object, do: `config.env_runners(num_env_runners=0)`.
.. Run without any remote :py:class:`~ray.rllib.core.learner.learner.Learner` workers. After defining your :py:class:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig` object, do: `config.learners(num_learners=0)`.
.. Switch off Ray Tune, if applicable. After defining your :py:class:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig` object, do: `algo = config.build()`, then `while True: algo.train()`.
.. Set a breakpoint in the ConnectorV2 piece (or any other RLlib component) you would like to debug and start the experiment script in your favorite IDE in debugging mode.
.. .. figure:: images/debugging_rllib_in_ide.png
doc/source/rllib/doc_code/advanced_api.py
+125
View File
@@ -0,0 +1,125 @@
# flake8: noqa
# __rllib-adv_api_counter_begin__
import ray
@ray.remote
class Counter:
def __init__(self):
self.count = 0
def inc(self, n):
self.count += n
def get(self):
return self.count
# on the driver
counter = Counter.options(name="global_counter").remote()
print(ray.get(counter.get.remote())) # get the latest count
# in your envs
counter = ray.get_actor("global_counter")
counter.inc.remote(1) # async call to increment the global count
# __rllib-adv_api_counter_end__
# __rllib-adv_api_explore_begin__
from ray.rllib.algorithms.algorithm_config import AlgorithmConfig
config = AlgorithmConfig().env_runners(
exploration_config={
# Special `type` key provides class information
"type": "StochasticSampling",
# Add any needed constructor args here.
"constructor_arg": "value",
}
)
# __rllib-adv_api_explore_end__
# __rllib-adv_api_evaluation_1_begin__
from ray.rllib.algorithms.algorithm_config import AlgorithmConfig
# Run one evaluation step on every 3rd `Algorithm.train()` call.
config = AlgorithmConfig().evaluation(
evaluation_interval=3,
)
# __rllib-adv_api_evaluation_1_end__
# __rllib-adv_api_evaluation_2_begin__
# Every time we run an evaluation step, run it for exactly 10 episodes.
config = AlgorithmConfig().evaluation(
evaluation_duration=10,
evaluation_duration_unit="episodes",
)
# Every time we run an evaluation step, run it for (close to) 200 timesteps.
config = AlgorithmConfig().evaluation(
evaluation_duration=200,
evaluation_duration_unit="timesteps",
)
# __rllib-adv_api_evaluation_2_end__
# __rllib-adv_api_evaluation_3_begin__
# Every time we run an evaluation step, run it for exactly 10 episodes, no matter,
# how many eval workers we have.
config = AlgorithmConfig().evaluation(
evaluation_duration=10,
evaluation_duration_unit="episodes",
# What if number of eval workers is non-dividable by 10?
# -> Run 7 episodes (1 per eval worker), then run 3 more episodes only using
# evaluation workers 1-3 (evaluation workers 4-7 remain idle during that time).
evaluation_num_env_runners=7,
)
# __rllib-adv_api_evaluation_3_end__
# __rllib-adv_api_evaluation_4_begin__
# Run evaluation and training at the same time via threading and make sure they roughly
# take the same time, such that the next `Algorithm.train()` call can execute
# immediately and not have to wait for a still ongoing (e.g. b/c of very long episodes)
# evaluation step:
config = AlgorithmConfig().evaluation(
evaluation_interval=2,
# run evaluation and training in parallel
evaluation_parallel_to_training=True,
# automatically end evaluation when train step has finished
evaluation_duration="auto",
evaluation_duration_unit="timesteps", # <- this setting is ignored; RLlib
# will always run by timesteps (not by complete
# episodes) in this duration=auto mode
)
# __rllib-adv_api_evaluation_4_end__
# __rllib-adv_api_evaluation_5_begin__
# Switching off exploration behavior for evaluation workers
# (see rllib/algorithms/algorithm.py). Use any keys in this sub-dict that are
# also supported in the main Algorithm config.
config = AlgorithmConfig().evaluation(
evaluation_config=AlgorithmConfig.overrides(explore=False),
)
# ... which is a more type-checked version of the old-style:
# config = AlgorithmConfig().evaluation(
# evaluation_config={"explore": False},
# )
# __rllib-adv_api_evaluation_5_end__
# __rllib-adv_api_evaluation_6_begin__
# Having an environment that occasionally blocks completely for e.g. 10min would
# also affect (and block) training. Here is how you can defend your evaluation setup
# against oft-crashing or -stalling envs (or other unstable components on your evaluation
# workers).
config = AlgorithmConfig().evaluation(
evaluation_interval=1,
evaluation_parallel_to_training=True,
evaluation_duration="auto",
evaluation_duration_unit="timesteps", # <- default anyway
evaluation_force_reset_envs_before_iteration=True, # <- default anyway
)
# __rllib-adv_api_evaluation_6_end__
doc/source/rllib/doc_code/custom_gym_env.py
+40
View File
@@ -0,0 +1,40 @@
# __rllib-custom-gym-env-begin__
import gymnasium as gym
import numpy as np
import ray
from ray.rllib.algorithms.ppo import PPOConfig
class SimpleCorridor(gym.Env):
def __init__(self, config):
self.end_pos = config["corridor_length"]
self.cur_pos = 0.0
self.action_space = gym.spaces.Discrete(2) # right/left
self.observation_space = gym.spaces.Box(0.0, self.end_pos, shape=(1,))
def reset(self, *, seed=None, options=None):
self.cur_pos = 0.0
return np.array([self.cur_pos]), {}
def step(self, action):
if action == 0 and self.cur_pos > 0.0: # move right (towards goal)
self.cur_pos -= 1.0
elif action == 1: # move left (towards start)
self.cur_pos += 1.0
if self.cur_pos >= self.end_pos:
return np.array([0.0]), 1.0, True, True, {}
else:
return np.array([self.cur_pos]), -0.1, False, False, {}
ray.init()
config = PPOConfig().environment(SimpleCorridor, env_config={"corridor_length": 5})
algo = config.build()
for _ in range(3):
print(algo.train())
algo.stop()
# __rllib-custom-gym-env-end__
doc/source/rllib/doc_code/dreamerv3_inference.py
+67
View File
@@ -0,0 +1,67 @@
import gymnasium as gym
import numpy as np
import tree # pip install dm_tree
from ray.rllib.algorithms.dreamerv3.dreamerv3 import DreamerV3Config
from ray.rllib.core.columns import Columns
from ray.rllib.utils.framework import convert_to_tensor
env_name = "CartPole-v1"
# Use the vector env API.
env = gym.make_vec(env_name, num_envs=1, vectorization_mode="sync")
terminated = truncated = False
# Reset the env.
obs, _ = env.reset()
# Every time, we start a new episode, we should set is_first to True for the upcoming
# action inference.
is_first = 1.0
# Create the algorithm from a simple config.
config = (
DreamerV3Config()
.environment("CartPole-v1")
.training(model_size="XS", training_ratio=1024)
)
algo = config.build()
# Extract the actual RLModule from the local (Dreamer) EnvRunner.
rl_module = algo.env_runner.module
# Get initial states from RLModule (note that these are always B=1, so this matches
# our num_envs=1; if you are using a vector env >1, you would have to repeat the
# returned states `num_env` times to get the correct batch size):
states = rl_module.get_initial_state()
# Batch the states to B=1.
states = tree.map_structure(lambda s: s.unsqueeze(0), states)
while not terminated and not truncated:
# Use the RLModule for action computations directly.
# DreamerV3 expects this particular batch format:
# obs=[B, T, ...]
# prev. states=[B, ...]
# `is_first`=[B]
batch = {
# States is already batched (see above).
Columns.STATE_IN: states,
# `obs` is already batched (due to vector env), but needs time-rank.
Columns.OBS: convert_to_tensor(obs, framework="torch")[None],
# Set to True at beginning of episode.
"is_first": convert_to_tensor(is_first, "torch")[None],
}
outs = rl_module.forward_inference(batch)
# Alternatively, call `forward_exploration` in case you want stochastic, non-greedy
# actions.
# outs = rl_module.forward_exploration(batch)
# Extract actions (remove time-rank) from outs.
actions = outs[Columns.ACTIONS].numpy()[0]
# Extract states from out. States are returned as batched.
states = outs[Columns.STATE_OUT]
# Perform a step in the env. Note that actions are still batched, which
# is ok, because we have a vector env.
obs, reward, terminated, truncated, info = env.step(actions)
# Not at the beginning of the episode anymore.
is_first = 0.0
doc/source/rllib/doc_code/replay_buffer_demo.py
+175
View File
@@ -0,0 +1,175 @@
# Demonstration of RLlib's ReplayBuffer workflow
from typing import Optional
import random
import numpy as np
from ray import tune
from ray.rllib.utils.replay_buffers import ReplayBuffer, StorageUnit
from ray.rllib.utils.annotations import override
from ray.rllib.utils.typing import SampleBatchType
from ray.rllib.utils.replay_buffers.utils import validate_buffer_config
from ray.rllib.examples.envs.classes.random_env import RandomEnv
from ray.rllib.policy.sample_batch import SampleBatch, concat_samples
from ray.rllib.algorithms.dqn.dqn import DQNConfig
# __sphinx_doc_replay_buffer_type_specification__begin__
config = (
DQNConfig()
.api_stack(
enable_env_runner_and_connector_v2=False, enable_rl_module_and_learner=False
)
.training(replay_buffer_config={"type": ReplayBuffer})
)
another_config = (
DQNConfig()
.api_stack(
enable_env_runner_and_connector_v2=False, enable_rl_module_and_learner=False
)
.training(replay_buffer_config={"type": "ReplayBuffer"})
)
yet_another_config = (
DQNConfig()
.api_stack(
enable_env_runner_and_connector_v2=False, enable_rl_module_and_learner=False
)
.training(
replay_buffer_config={"type": "ray.rllib.utils.replay_buffers.ReplayBuffer"}
)
)
validate_buffer_config(config)
validate_buffer_config(another_config)
validate_buffer_config(yet_another_config)
# After validation, all three configs yield the same effective config
assert (
config.replay_buffer_config
== another_config.replay_buffer_config
== yet_another_config.replay_buffer_config
)
# __sphinx_doc_replay_buffer_type_specification__end__
# __sphinx_doc_replay_buffer_basic_interaction__begin__
# We choose fragments because it does not impose restrictions on our batch to be added
buffer = ReplayBuffer(capacity=2, storage_unit=StorageUnit.FRAGMENTS)
dummy_batch = SampleBatch({"a": [1], "b": [2]})
buffer.add(dummy_batch)
buffer.sample(2)
# Because elements can be sampled multiple times, we receive a concatenated version
# of dummy_batch `{a: [1, 1], b: [2, 2,]}`.
# __sphinx_doc_replay_buffer_basic_interaction__end__
# __sphinx_doc_replay_buffer_own_buffer__begin__
class LessSampledReplayBuffer(ReplayBuffer):
@override(ReplayBuffer)
def sample(
self, num_items: int, evict_sampled_more_then: int = 30, **kwargs
) -> Optional[SampleBatchType]:
"""Evicts experiences that have been sampled > evict_sampled_more_then times."""
idxes = [random.randint(0, len(self) - 1) for _ in range(num_items)]
often_sampled_idxes = list(
filter(lambda x: self._hit_count[x] >= evict_sampled_more_then, set(idxes))
)
sample = self._encode_sample(idxes)
self._num_timesteps_sampled += sample.count
for idx in often_sampled_idxes:
del self._storage[idx]
self._hit_count = np.append(
self._hit_count[:idx], self._hit_count[idx + 1 :]
)
return sample
config = (
DQNConfig()
.api_stack(
enable_env_runner_and_connector_v2=False, enable_rl_module_and_learner=False
)
.environment(env="CartPole-v1")
.training(replay_buffer_config={"type": LessSampledReplayBuffer})
)
tune.Tuner(
"DQN",
param_space=config,
run_config=tune.RunConfig(
stop={"training_iteration": 1},
),
).fit()
# __sphinx_doc_replay_buffer_own_buffer__end__
# __sphinx_doc_replay_buffer_advanced_usage_storage_unit__begin__
# This line will make our buffer store only complete episodes found in a batch
config.training(replay_buffer_config={"storage_unit": StorageUnit.EPISODES})
less_sampled_buffer = LessSampledReplayBuffer(**config.replay_buffer_config)
# Gather some random experiences
env = RandomEnv()
terminated = truncated = False
batch = SampleBatch({})
t = 0
while not terminated and not truncated:
obs, reward, terminated, truncated, info = env.step([0, 0])
# Note that in order for RLlib to find out about start and end of an episode,
# "t" and "terminateds" have to properly mark an episode's trajectory
one_step_batch = SampleBatch(
{
"obs": [obs],
"t": [t],
"reward": [reward],
"terminateds": [terminated],
"truncateds": [truncated],
}
)
batch = concat_samples([batch, one_step_batch])
t += 1
less_sampled_buffer.add(batch)
for i in range(10):
assert len(less_sampled_buffer._storage) == 1
less_sampled_buffer.sample(num_items=1, evict_sampled_more_then=9)
assert len(less_sampled_buffer._storage) == 0
# __sphinx_doc_replay_buffer_advanced_usage_storage_unit__end__
# __sphinx_doc_replay_buffer_advanced_usage_underlying_buffers__begin__
config = (
DQNConfig()
.api_stack(
enable_env_runner_and_connector_v2=False, enable_rl_module_and_learner=False
)
.training(
replay_buffer_config={
"type": "MultiAgentReplayBuffer",
"underlying_replay_buffer_config": {
"type": LessSampledReplayBuffer,
# We can specify the default call argument
# for the sample method of the underlying buffer method here.
"evict_sampled_more_then": 20,
},
}
)
.environment(env="CartPole-v1")
)
tune.Tuner(
"DQN",
param_space=config.to_dict(),
run_config=tune.RunConfig(
stop={"env_runners/episode_return_mean": 40, "training_iteration": 7},
),
).fit()
# __sphinx_doc_replay_buffer_advanced_usage_underlying_buffers__end__
doc/source/rllib/doc_code/rllib_on_ray_readme.py
+170
View File
@@ -0,0 +1,170 @@
# __quick_start_begin__
import gymnasium as gym
import numpy as np
import torch
from typing import Dict, Tuple, Any, Optional
from ray.rllib.algorithms.ppo import PPOConfig
# Define your problem using python and Farama-Foundation's gymnasium API:
class SimpleCorridor(gym.Env):
"""Corridor environment where an agent must learn to move right to reach the exit.
---------------------
| S | 1 | 2 | 3 | G | S=start; G=goal; corridor_length=5
---------------------
Actions:
0: Move left
1: Move right
Observations:
A single float representing the agent's current position (index)
starting at 0.0 and ending at corridor_length
Rewards:
-0.1 for each step
+1.0 when reaching the goal
Episode termination:
When the agent reaches the goal (position >= corridor_length)
"""
def __init__(self, config):
self.end_pos = config["corridor_length"]
self.cur_pos = 0.0
self.action_space = gym.spaces.Discrete(2) # 0=left, 1=right
self.observation_space = gym.spaces.Box(0.0, self.end_pos, (1,), np.float32)
def reset(
self, *, seed: Optional[int] = None, options: Optional[Dict] = None
) -> Tuple[np.ndarray, Dict]:
"""Reset the environment for a new episode.
Args:
seed: Random seed for reproducibility
options: Additional options (not used in this environment)
Returns:
Initial observation of the new episode and an info dict.
"""
super().reset(seed=seed) # Initialize RNG if seed is provided
self.cur_pos = 0.0
# Return initial observation.
return np.array([self.cur_pos], np.float32), {}
def step(self, action: int) -> Tuple[np.ndarray, float, bool, bool, Dict]:
"""Take a single step in the environment based on the provided action.
Args:
action: 0 for left, 1 for right
Returns:
A tuple of (observation, reward, terminated, truncated, info):
observation: Agent's new position
reward: Reward from taking the action (-0.1 or +1.0)
terminated: Whether episode is done (reached goal)
truncated: Whether episode was truncated (always False here)
info: Additional information (empty dict)
"""
# Walk left if action is 0 and we're not at the leftmost position
if action == 0 and self.cur_pos > 0:
self.cur_pos -= 1
# Walk right if action is 1
elif action == 1:
self.cur_pos += 1
# Set `terminated` flag when end of corridor (goal) reached.
terminated = self.cur_pos >= self.end_pos
truncated = False
# +1 when goal reached, otherwise -0.1.
reward = 1.0 if terminated else -0.1
return np.array([self.cur_pos], np.float32), reward, terminated, truncated, {}
# Create an RLlib Algorithm instance from a PPOConfig object.
print("Setting up the PPO configuration...")
config = (
PPOConfig().environment(
# Env class to use (our custom gymnasium environment).
SimpleCorridor,
# Config dict passed to our custom env's constructor.
# Use corridor with 20 fields (including start and goal).
env_config={"corridor_length": 20},
)
# Parallelize environment rollouts for faster training.
.env_runners(num_env_runners=3)
# Use a smaller network for this simple task
.training(model={"fcnet_hiddens": [64, 64]})
)
# Construct the actual PPO algorithm object from the config.
algo = config.build_algo()
rl_module = algo.get_module()
# Train for n iterations and report results (mean episode rewards).
# Optimal reward calculation:
# - Need at least 19 steps to reach the goal (from position 0 to 19)
# - Each step (except last) gets -0.1 reward: 18 * (-0.1) = -1.8
# - Final step gets +1.0 reward
# - Total optimal reward: -1.8 + 1.0 = -0.8
print("\nStarting training loop...")
for i in range(5):
results = algo.train()
# Log the metrics from training results
print(f"Iteration {i+1}")
print(f" Training metrics: {results['env_runners']}")
# Save the trained algorithm (optional)
checkpoint_dir = algo.save()
print(f"\nSaved model checkpoint to: {checkpoint_dir}")
print("\nRunning inference with the trained policy...")
# Create a test environment with a shorter corridor to verify the agent's behavior
env = SimpleCorridor({"corridor_length": 10})
# Get the initial observation (should be: [0.0] for the starting position).
obs, info = env.reset()
terminated = truncated = False
total_reward = 0.0
step_count = 0
# Play one episode and track the agent's trajectory
print("\nAgent trajectory:")
positions = [float(obs[0])] # Track positions for visualization
while not terminated and not truncated and step_count < 1000:
# Compute an action given the current observation
action_logits = rl_module.forward_inference(
{"obs": torch.from_numpy(obs).unsqueeze(0)}
)["action_dist_inputs"].numpy()[
0
] # [0]: Batch dimension=1
# Get the action with highest probability
action = np.argmax(action_logits)
# Log the agent's decision
action_name = "LEFT" if action == 0 else "RIGHT"
print(f" Step {step_count}: Position {obs[0]:.1f}, Action: {action_name}")
# Apply the computed action in the environment
obs, reward, terminated, truncated, info = env.step(action)
positions.append(float(obs[0]))
# Sum up rewards
total_reward += reward
step_count += 1
# Report final results
print(f"\nEpisode complete:")
print(f" Steps taken: {step_count}")
print(f" Total reward: {total_reward:.2f}")
print(f" Final position: {obs[0]:.1f}")
# Verify the agent has learned the optimal policy
if total_reward > -0.5 and obs[0] >= 9.0:
print(" Success! The agent has learned the optimal policy (always move right).")
else:
print(" Failure! The agent didn't reach the goal within 1000 timesteps.")
# __quick_start_end__
doc/source/rllib/doc_code/sa_episode.py
+270
View File
@@ -0,0 +1,270 @@
# flake8: noqa
import copy
# __rllib-sa-episode-01-begin__
from ray.rllib.env.single_agent_episode import SingleAgentEpisode
# Construct a new episode (without any data in it yet).
episode = SingleAgentEpisode()
assert len(episode) == 0
episode.add_env_reset(observation="obs_0", infos="info_0")
# Even with the initial obs/infos, the episode is still considered len=0.
assert len(episode) == 0
# Fill the episode with some fake data (5 timesteps).
for i in range(5):
episode.add_env_step(
observation=f"obs_{i+1}",
action=f"act_{i}",
reward=f"rew_{i}",
terminated=False,
truncated=False,
infos=f"info_{i+1}",
)
assert len(episode) == 5
# __rllib-sa-episode-01-end__
# __rllib-sa-episode-02-begin__
# We can now access information from the episode via its getter APIs.
from ray.rllib.utils.test_utils import check
# Get the very first observation ("reset observation"). Note that a single observation
# is returned here (not a list of size 1 or a batch of size 1).
check(episode.get_observations(0), "obs_0")
# ... which is the same as using the indexing operator on the Episode's
# `observations` property:
check(episode.observations[0], "obs_0")
# You can also get several observations at once by providing a list of indices:
check(episode.get_observations([1, 2]), ["obs_1", "obs_2"])
# .. or a slice of observations by providing a python slice object:
check(episode.get_observations(slice(1, 3)), ["obs_1", "obs_2"])
# Note that when passing only a single index, a single item is returned.
# Whereas when passing a list of indices or a slice, a list of items is returned.
# Similarly for getting rewards:
# Get the last reward.
check(episode.get_rewards(-1), "rew_4")
# ... which is the same as using the slice operator on the `rewards` property:
check(episode.rewards[-1], "rew_4")
# Similarly for getting actions:
# Get the first action in the episode (single item, not batched).
# This works regardless of the action space.
check(episode.get_actions(0), "act_0")
# ... which is the same as using the indexing operator on the `actions` property:
check(episode.actions[0], "act_0")
# Finally, you can slice the entire episode using the []-operator with a slice notation:
sliced_episode = episode[3:4]
check(list(sliced_episode.observations), ["obs_3", "obs_4"])
check(list(sliced_episode.actions), ["act_3"])
check(list(sliced_episode.rewards), ["rew_3"])
# __rllib-sa-episode-02-end__
import copy # noqa
episode_2 = copy.deepcopy(episode)
# __rllib-sa-episode-03-begin__
# Episodes start in the non-numpy'ized state (in which data is stored
# under the hood in lists).
assert episode.is_numpy is False
# Call `to_numpy()` to convert all stored data from lists of individual (possibly
# complex) items to numpy arrays. Note that RLlib normally performs this method call,
# so users don't need to call `to_numpy()` themselves.
episode.to_numpy()
assert episode.is_numpy is True
# __rllib-sa-episode-03-end__
episode = episode_2
# __rllib-sa-episode-04-begin__
# An ongoing episode (of length 5):
assert len(episode) == 5
assert episode.is_done is False
# During an `EnvRunner.sample()` rollout, when enough data has been collected into
# one or more Episodes, the `EnvRunner` calls the `cut()` method, interrupting
# the ongoing Episode and returning a new continuation chunk (with which the
# `EnvRunner` can continue collecting data during the next call to `sample()`):
continuation_episode = episode.cut()
# The length is still 5, but the length of the continuation chunk is 0.
assert len(episode) == 5
assert len(continuation_episode) == 0
# Thanks to the lookback buffer, we can still access the most recent observation
# in the continuation chunk:
check(continuation_episode.get_observations(-1), "obs_5")
# __rllib-sa-episode-04-end__
# __rllib-sa-episode-05-begin__
# Construct a new episode (with some data in its lookback buffer).
episode = SingleAgentEpisode(
observations=["o0", "o1", "o2", "o3"],
actions=["a0", "a1", "a2"],
rewards=[0.0, 1.0, 2.0],
len_lookback_buffer=3,
)
# Since our lookback buffer is 3, all data already specified in the constructor should
# now be in the lookback buffer (and not be part of the `episode` chunk), meaning
# the length of `episode` should still be 0.
assert len(episode) == 0
# .. and trying to get the first reward will hence lead to an IndexError.
try:
episode.get_rewards(0)
except IndexError:
pass
# Get the last 3 rewards (using the lookback buffer).
check(episode.get_rewards(slice(-3, None)), [0.0, 1.0, 2.0])
# Assuming the episode actually started with `obs_0` (reset obs),
# then `obs_1` + `act_0` + reward=0.0, but your model always requires a 1D reward tensor
# of shape (5,) with the 5 most recent rewards in it.
# You could try to code for this by manually filling the missing 2 timesteps with zeros:
last_5_rewards = [0.0, 0.0] + episode.get_rewards(slice(-3, None))
# However, this will become extremely tedious, especially when moving to (possibly more
# complex) observations and actions.
# Instead, `SingleAgentEpisode` getters offer some useful options to solve this problem:
last_5_rewards = episode.get_rewards(slice(-5, None), fill=0.0)
# Note that the `fill` argument allows you to even go further back into the past, provided
# you are ok with filling timesteps that are not covered by the lookback buffer with
# a fixed value.
# __rllib-sa-episode-05-end__
# __rllib-sa-episode-06-begin__
# Construct a new episode (len=3 and lookback buffer=3).
episode = SingleAgentEpisode(
observations=[
"o-3",
"o-2",
"o-1", # <- lookback # noqa
"o0",
"o1",
"o2",
"o3", # <- actual episode data # noqa
],
actions=[
"a-3",
"a-2",
"a-1", # <- lookback # noqa
"a0",
"a1",
"a2", # <- actual episode data # noqa
],
rewards=[
-3.0,
-2.0,
-1.0, # <- lookback # noqa
0.0,
1.0,
2.0, # <- actual episode data # noqa
],
len_lookback_buffer=3,
)
assert len(episode) == 3
# In case you want to loop through global timesteps 0 to 2 (timesteps -3, -2, and -1
# being the lookback buffer) and at each such global timestep look 2 timesteps back,
# you can do so easily using the `neg_index_as_lookback` arg like so:
for global_ts in [0, 1, 2]:
rewards = episode.get_rewards(
slice(global_ts - 2, global_ts + 1),
# Switch behavior of negative indices from "from-the-end" to
# "into the lookback buffer":
neg_index_as_lookback=True,
)
print(rewards)
# The expected output should be:
# [-2.0, -1.0, 0.0] # global ts=0 (plus looking back 2 ts)
# [-1.0, 0.0, 1.0] # global ts=1 (plus looking back 2 ts)
# [0.0, 1.0, 2.0] # global ts=2 (plus looking back 2 ts)
# __rllib-sa-episode-06-end__
# Looking back from ts=1, get the previous 4 rewards AND fill with 0.0
# in case we go over the beginning (ts=0). So we would expect
# [0.0, 0.0, 0.0, r0] to be returned here, where r0 is the very first received
# reward in the episode:
episode.get_rewards(slice(-4, 0), neg_index_as_lookback=True, fill=0.0)
# Note the use of fill=0.0 here (fill everything that's out of range with this
# value) AND the argument `neg_index_as_lookback=True`, which interprets
# negative indices as being left of ts=0 (e.g. -1 being the timestep before
# ts=0).
import gymnasium as gym
import numpy as np
# Assuming we had a complex action space (nested gym.spaces.Dict) with one or
# more elements being Discrete or MultiDiscrete spaces:
# 1) The `fill=...` argument would still work, filling all spaces (Boxes,
# Discrete) with that provided value.
# 2) Setting the flag `one_hot_discrete=True` would convert those discrete
# sub-components automatically into one-hot (or multi-one-hot) tensors.
# This simplifies the task of having to provide the previous 4 (nested and
# partially discrete/multi-discrete) actions for each timestep within a training
# batch, thereby filling timesteps before the episode started with 0.0s and
# one-hot'ing the discrete/multi-discrete components in these actions:
episode = SingleAgentEpisode(
action_space=gym.spaces.Dict(
{
"a": gym.spaces.Discrete(3),
"b": gym.spaces.MultiDiscrete([2, 3]),
"c": gym.spaces.Box(-1.0, 1.0, (2,)),
}
)
)
# ... fill episode with data ...
episode.add_env_reset(observation=0)
# ... from a few steps.
episode.add_env_step(
observation=1,
action={"a": 0, "b": np.array([1, 2]), "c": np.array([0.5, -0.5], np.float32)},
reward=1.0,
)
# In your connector
prev_4_a = []
# Note here that len(episode) does NOT include the lookback buffer.
for ts in range(len(episode)):
prev_4_a.append(
episode.get_actions(
indices=slice(ts - 4, ts),
# Make sure negative indices are interpreted as
# "into lookback buffer"
neg_index_as_lookback=True,
# Zero-out everything even further before the lookback buffer.
fill=0.0,
# Take care of discrete components (get ready as NN input).
one_hot_discrete=True,
)
)
# Finally, convert from list of batch items to a struct (same as action space)
# of batched (numpy) arrays, in which all leafs have B==len(prev_4_a).
from ray.rllib.utils.spaces.space_utils import batch
prev_4_actions_col = batch(prev_4_a)
doc/source/rllib/env-to-module-connector.rst
+582
View File
@@ -0,0 +1,582 @@
.. _env-to-module-pipeline-docs:
.. grid:: 1 2 3 4
:gutter: 1
:class-container: container pb-3
.. grid-item-card::
:img-top: /rllib/images/connector_v2/connector_generic.svg
:class-img-top: pt-2 w-75 d-block mx-auto fixed-height-img
.. button-ref:: connector-v2-docs
ConnectorV2 overview
.. grid-item-card::
:img-top: /rllib/images/connector_v2/env_to_module_connector.svg
:class-img-top: pt-2 w-75 d-block mx-auto fixed-height-img
.. button-ref:: env-to-module-pipeline-docs
Env-to-module pipelines (this page)
.. grid-item-card::
:img-top: /rllib/images/connector_v2/learner_connector.svg
:class-img-top: pt-2 w-75 d-block mx-auto fixed-height-img
.. button-ref:: learner-pipeline-docs
Learner pipelines
Env-to-module pipelines
=======================
.. include:: /_includes/rllib/new_api_stack.rst
On each :py:class:`~ray.rllib.env.env_runner.EnvRunner` resides one env-to-module pipeline
responsible for handling the data flow from the `gymnasium.Env <https://gymnasium.farama.org/api/env/>`__ to the :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule`.
.. figure:: images/connector_v2/env_runner_connector_pipelines.svg
:width: 1000
:align: left
**EnvRunner ConnectorV2 Pipelines**: Both env-to-module and module-to-env pipelines are located on the :py:class:`~ray.rllib.env.env_runner.EnvRunner`
workers. The env-to-module pipeline sits between the RL environment, a `gymnasium.Env <https://gymnasium.farama.org/api/env/>`__, and the
:py:class:`~ray.rllib.core.rl_module.rl_module.RLModule`, and translates ongoing episodes into batches for the model's `forward_...()` methods.
.. The module-to-env pipeline serves the other direction, converting the output of the :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule`, such as action logits and action distribution parameters, to actual actions understandable by the `gymnasium.Env <https://gymnasium.farama.org/api/env/>`__ and used in the env's next `step()` call.
The env-to-module pipeline, when called, performs transformations from a list of ongoing :ref:`Episode objects <single-agent-episode-docs>` to an
``RLModule``-readable tensor batch and RLlib passes this generated batch as the first argument into the
:py:meth:`~ray.rllib.core.rl_module.rl_module.RLModule.forward_inference` or :py:meth:`~ray.rllib.core.rl_module.rl_module.RLModule.forward_exploration`
methods of the :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule`, depending on your exploration settings.
.. hint::
Set `config.exploration(explore=True)` in your :py:class:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig` to have RLlib call the
:py:meth:`~ray.rllib.core.rl_module.rl_module.RLModule.forward_exploration` method with the connector's output.
Otherwise, RLlib calls :py:meth:`~ray.rllib.core.rl_module.rl_module.RLModule.forward_inference`.
Note also that normally these two methods only differ in that actions are sampled when ``explore=True`` and
greedily picked when ``explore=False``. However, the exact behavior in each case depends on your :ref:`RLModule's implementation <rlmodule-guide>`.
.. _default-env-to-module-pipeline:
Default env-to-module behavior
------------------------------
By default RLlib populates every env-to-module pipeline with the following built-in connector pieces.
* :py:class:`~ray.rllib.connectors.common.add_observations_from_episodes_to_batch.AddObservationsFromEpisodesToBatch`: Places the most recent observation from each ongoing episode into the batch. The column name is ``obs``. Note that if you have a vector of ``N`` environments per :py:class:`~ray.rllib.env.env_runner.EnvRunner`, your batch size is also ``N``.
* *Relevant for stateful models only:* :py:class:`~ray.rllib.connectors.common.add_time_dim_to_batch_and_zero_pad.AddTimeDimToBatchAndZeroPad`: If the :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` is stateful, adds a single timestep, second axis to all data to make it sequential.
* *Relevant for stateful models only:* :py:class:`~ray.rllib.connectors.common.add_states_from_episodes_to_batch.AddStatesFromEpisodesToBatch`: If the :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` is stateful, places the most recent state outputs of the module as new state inputs into the batch. The column name is ``state_in`` and the values don't have a time-dimension.
* *For multi-agent only:* :py:class:`~ray.rllib.connectors.common.agent_to_module_mapping.AgentToModuleMapping`: Maps per-agent data to the respective per-module data depending on your defined agent-to-module mapping function.
* :py:class:`~ray.rllib.connectors.common.batch_individual_items.BatchIndividualItems`: Converts all data in the batch, which thus far are lists of individual items, into batched structures meaning NumPy arrays, whose 0th axis is the batch axis.
* :py:class:`~ray.rllib.connectors.common.numpy_to_tensor.NumpyToTensor`: Converts all NumPy arrays in the batch into framework specific tensors and moves these to the GPU, if required.
You can disable all the preceding default connector pieces by setting `config.env_runners(add_default_connectors_to_env_to_module_pipeline=False)`
in your :ref:`algorithm config <rllib-algo-configuration-docs>`.
Note that the order of these transforms is very relevant for the functionality of the pipeline.
See :ref:`here on how to write and add your own connector pieces <writing_custom_env_to_module_connectors>` to the pipeline.
Constructing an env-to-module connector
---------------------------------------
Normally, you wouldn't have to construct the env-to-module connector pipeline yourself. RLlib's :py:class:`~ray.rllib.env.env_runner.EnvRunner`
actors initially perform this operation. However, if you would like to test or debug either the default pipeline or a custom one,
use the following code snippet as a starting point:
.. testcode::
import gymnasium as gym
from ray.rllib.algorithms.ppo import PPOConfig
from ray.rllib.env.single_agent_episode import SingleAgentEpisode
# Start with an algorithm config.
config = (
PPOConfig()
.environment("CartPole-v1")
)
# Create an env to generate some episode data.
env = gym.make("CartPole-v1")
# Build the env-to-module connector through the config object.
env_to_module = config.build_env_to_module_connector(env=env, spaces=None)
Alternatively, in case there is no ``env`` object available, you should pass in the ``spaces`` argument instead.
RLlib requires either of these pieces of information to compute the correct output observation space of the pipeline, so that the
:py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` can receive the correct input space for its own setup procedure.
The structure of the `spaces` argument should ideally be:
.. code-block:: python
spaces = {
"__env__": ([env observation space], [env action space]), # <- may be vectorized
"__env_single__": ([env observation space], [env action space]), # <- never vectorized!
"[module ID, e.g. 'default_policy']": ([module observation space], [module action space]),
... # <- more modules in multi-agent case
}
However, for single-agent cases, it may be enough to provide the non-vectorized, single observation-
and action spaces only:
.. testcode::
# No `env` available? Use `spaces` instead:
env_to_module = config.build_env_to_module_connector(
env=None,
spaces={
# At minimum, pass in a 2-tuple of the single, non-vectorized
# observation- and action spaces:
"__env_single__": (env.observation_space, env.action_space),
},
)
To test the actual behavior or the created pipeline, look at these code snippets
for stateless- and stateful :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` cases:
.. tab-set::
.. tab-item:: Stateless RLModule
.. testcode::
from ray.rllib.env.single_agent_episode import SingleAgentEpisode
# Create two SingleAgentEpisode instances. You pass these to the connector pipeline
# as input.
episode1 = SingleAgentEpisode()
episode2 = SingleAgentEpisode()
# Fill episodes with some data, as if we were currently stepping through them
# to collect samples.
# - episode 1 (two timesteps)
obs, _ = env.reset()
episode1.add_env_reset(observation=obs)
action = 0
obs, _, _, _, _ = env.step(action)
episode1.add_env_step(observation=obs, action=action, reward=1.0)
# - episode 2 (just one timestep)
obs, _ = env.reset()
episode2.add_env_reset(observation=obs)
# Call the connector on the two running episodes.
batch = {}
batch = env_to_module(
episodes=[episode1, episode2],
batch=batch,
rl_module=None, # in stateless case, RLModule is not strictly required
explore=True,
)
# Print out the resulting batch.
print(batch)
.. tab-item:: Stateful RLModule (RNN)
.. testcode::
from ray.rllib.core.rl_module.default_model_config import DefaultModelConfig
from ray.rllib.env.single_agent_episode import SingleAgentEpisode
# Alter the config to use the default LSTM model of RLlib.
config.rl_module(model_config=DefaultModelConfig(use_lstm=True))
# For stateful RLModules, we do need to pass in the RLModule to every call to the
# connector. so construct an instance here.
rl_module_spec = config.get_rl_module_spec(env=env)
rl_module = rl_module_spec.build()
# Create a SingleAgentEpisode instance. You pass this to the connector pipeline
# as input.
episode = SingleAgentEpisode()
# Initialize episode with first (reset) observation.
obs, _ = env.reset()
episode.add_env_reset(observation=obs)
# Call the connector on the running episode.
batch = {}
batch = env_to_module(
episodes=[episode],
batch=batch,
rl_module=rl_module, # in stateful case, RLModule is required
explore=True,
)
# Print out the resulting batch.
print(batch)
You can see that the pipeline extracted the current observations from the two
running episodes and placed them under the ``obs`` column into the forward batch.
The batch has a size of two, because we had two episodes, and should look similar to this:
.. code-block:: text
{'obs': tensor([[ 0.0212, -0.1996, -0.0414, 0.2848],
[ 0.0292, 0.0259, -0.0322, -0.0004]])}
In the stateful case, you can also expect the ``STATE_IN`` columns to be present.
Note that because of the LSTM layer, the internal state of the module consists of two components, ``c`` and ``h``:
.. code-block:: text
{
'obs': tensor(
[[ 0.0212, -0.1996, -0.0414, 0.2848],
[ 0.0292, 0.0259, -0.0322, -0.0004]]
),
'state_in': {
# Note: The shape of each state tensor here is
# (B=2, [num LSTM-layers=1], [LSTM cell size]).
'h': tensor([[[0., 0., .., 0.]]]),
'c': tensor([[[0., 0., ... 0.]]]),
},
}
.. hint::
You are free to design the internal states of your custom :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` classes
however you like. You only need to override the :py:meth:`~ray.rllib.core.rl_module.rl_module.RLModule.get_initial_state` method and make sure
you return a new state of any nested structure and shape from your `forward_..()` methods under the fixed ``state_out`` key.
See `here for an example <https://github.com/ray-project/ray/blob/master/rllib/examples/rl_modules/classes/lstm_containing_rlm.py>`__
of an RLModule class with a custom LSTM layer in it.
.. _writing_custom_env_to_module_connectors:
Writing custom env-to-module connectors
---------------------------------------
You can customize the default env-to-module pipeline that RLlib creates through specifying a function in your
:py:class:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig`, which takes an optional RL environment object (`env`) and an optional `spaces`
dictionary as input arguments and returns a single :py:class:`~ray.rllib.connectors.connector_v2.ConnectorV2` piece or a list thereof.
RLlib prepends these :py:class:`~ray.rllib.connectors.connector_v2.ConnectorV2` instances to the
:ref:`default env-to-module pipeline <default-env-to-module-pipeline>` in the order returned,
unless you set `add_default_connectors_to_env_to_module_pipeline=False` in your config, in which case RLlib exclusively uses the provided
:py:class:`~ray.rllib.connectors.connector_v2.ConnectorV2` pieces without any automatically added default behavior.
For example, to prepend a custom ConnectorV2 piece to the env-to-module pipeline, you can do this in your config:
.. testcode::
:skipif: True
# Your builder function must accept an optional `gymnasium.Env` and an optional `spaces` dict
# as arguments.
config.env_runners(
env_to_module_connector=lambda env, spaces, device: MyEnvToModuleConnector(..),
)
If you want to add multiple custom pieces to the pipeline, return them as a list:
.. testcode::
:skipif: True
# Return a list of connector pieces to make RLlib add all of them to your
# env-to-module pipeline.
config.env_runners(
env_to_module_connector=lambda env, spaces, device: [
MyEnvToModuleConnector(..),
MyOtherEnvToModuleConnector(..),
AndOneMoreConnector(..),
],
)
RLlib adds the connector pieces returned by your function to the beginning of the env-to-module pipeline,
before the previously described default connector pieces that RLlib provides automatically:
.. figure:: images/connector_v2/custom_pieces_in_env_to_module_pipeline.svg
:width: 1000
:align: left
**Inserting custom ConnectorV2 pieces into the env-to-module pipeline**: RLlib inserts custom connector pieces, such
as observation preprocessors, before the default pieces. This way, if your custom connectors alter the input episodes
in any way, for example by changing the observations as in an :ref:`ObservationPreprocessor <observation-preprocessors>`,
the tailing default pieces automatically add these changed observations to the batch.
.. _observation-preprocessors:
Observation preprocessors
~~~~~~~~~~~~~~~~~~~~~~~~~
The simplest way of customizing an env-to-module pipeline is to write your own
:py:class:`~ray.rllib.connectors.env_to_module.observation_preprocessor.SingleAgentObservationPreprocessor` subclass, implement two methods,
and point your config to the new class:
.. testcode::
import gymnasium as gym
import numpy as np
from ray.rllib.connectors.env_to_module.observation_preprocessor import SingleAgentObservationPreprocessor
class IntObservationToOneHotTensor(SingleAgentObservationPreprocessor):
"""Converts int observations (Discrete) into one-hot tensors (Box)."""
def recompute_output_observation_space(self, in_obs_space, in_act_space):
# Based on the input observation space, either from the preceding connector piece or
# directly from the environment, return the output observation space of this connector
# piece.
# Implementing this method is crucial for the pipeline to know its output
# spaces, which are an important piece of information to construct the succeeding
# RLModule.
return gym.spaces.Box(0.0, 1.0, (in_obs_space.n,), np.float32)
def preprocess(self, observation, episode):
# Convert an input observation (int) into a one-hot (float) tensor.
# Note that 99% of all connectors in RLlib operate on NumPy arrays.
new_obs = np.zeros(shape=self.observation_space.shape, dtype=np.float32)
new_obs[observation] = 1.0
return new_obs
Note that any observation preprocessor actually changes the underlying episodes object in place, but and doesn't contribute anything to
the batch under construction. Because RLlib always inserts any user defined preprocessor (and other custom
:py:class:`~ray.rllib.connectors.connector_v2.ConnectorV2`
pieces) before the default pieces, the :py:class:`~ray.rllib.connectors.common.add_observations_from_episodes_to_batch.AddObservationsFromEpisodesToBatch`
default piece then automatically takes care of adding the preprocessed and updated observation from the episode to the batch:
Now you can use the custom preprocessor in environments with integer observations, for example the
`FrozenLake <https://gymnasium.farama.org/environments/toy_text/frozen_lake/>`__ RL environment:
.. testcode::
from ray.rllib.algorithms.ppo import PPOConfig
config = (
PPOConfig()
# Configure a simple 2x2 grid-world.
# ____
# |S | <- S=start position
# | G| <- G=goal position
# ----
.environment("FrozenLake-v1", env_config={"desc": ["SF", "FG"]})
# Plug your custom connector piece into the env-to-module pipeline.
.env_runners(
env_to_module_connector=(
lambda env, spaces, device: IntObservationToOneHotTensor()
),
)
)
algo = config.build()
# Train one iteration.
print(algo.train())
.. _observation-preprocessors-adding-rewards-to-obs:
Example: Adding recent rewards to the batch
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Assume you wrote a custom :ref:`RLModule <rlmodule-guide>` that requires the last three received
rewards as input in the calls to any of its `forward_..()` methods.
You can use the same :py:class:`~ray.rllib.connectors.env_to_module.observation_preprocessor.SingleAgentObservationPreprocessor`
API to achieve this.
In the following example, you extract the last three rewards from the ongoing episode and concatenate
them with the observation to form a new observation tensor.
Note that you also have to change the observation space returned by the connector, since
there are now three more values in each observation:
.. testcode::
import gymnasium as gym
import numpy as np
from ray.rllib.connectors.env_to_module.observation_preprocessor import SingleAgentObservationPreprocessor
class AddPastThreeRewards(SingleAgentObservationPreprocessor):
"""Extracts last three rewards from episode and concatenates them to the observation tensor."""
def recompute_output_observation_space(self, in_obs_space, in_act_space):
# Based on the input observation space (), return the output observation
# space. Implementing this method is crucial for the pipeline to know its output
# spaces, which are an important piece of information to construct the succeeding
# RLModule.
assert isinstance(in_obs_space, gym.spaces.Box) and len(in_obs_space.shape) == 1
return gym.spaces.Box(-100.0, 100.0, (in_obs_space.shape[0] + 3,), np.float32)
def preprocess(self, observation, episode):
# Extract the last 3 rewards from the ongoing episode using a python `slice` object.
# Alternatively, you can also pass in a list of indices, [-3, -2, -1].
past_3_rewards = episode.get_rewards(indices=slice(-3, None))
# Concatenate the rewards to the actual observation.
new_observation = np.concatenate([
observation, np.array(past_3_rewards, np.float32)
])
# Return the new observation.
return new_observation
.. note::
Note that the preceding example should work without any further action required on your model,
whether it's a custom one or a default one provided by RLlib, as long as the model determines its input layer's
size through its own ``self.observation_space`` attribute. The connector pipeline correctly captures the observation
space changes, from the environment's 1D-Box to the reward-enhanced, larger 1D-Box and
passes this new observation space to your RLModule's :py:meth:`~ray.rllib.core.rl_module.rl_module.RLModule.setup`
method.
Example: Preprocessing observations in multi-agent setups
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In multi-agent setups, you have two options for preprocessing your agents' individual observations
through customizing your env-to-module pipeline:
1) Agent-by-agent: Using the same API as in the previous examples,
:py:class:`~ray.rllib.connectors.env_to_module.observation_preprocessor.SingleAgentObservationPreprocessor`,
you can apply a single preprocessing logic across all agents. However, in case you need one distinct preprocessing
logic per ``AgentID``, lookup the agent information from the provided ``episode`` argument in the
:py:meth:`~ray.rllib.connectors.env_to_module.observation_preprocessor.SingleAgentObservationPreprocessor.preprocess` method:
.. testcode::
:skipif: True
def recompute_output_observation_space(self, in_obs_space, in_act_space):
# `in_obs_space` is a `Dict` space, mapping agent IDs to individual agents' spaces.
# Alter this dict according to which agents you want to preprocess observations for
# and return the new `Dict` space.
# For example:
return gym.spaces.Dict({
"some_agent_id": [obs space],
"other_agent_id": [another obs space],
...
})
def preprocess(self, observation, episode):
# Skip preprocessing for certain agent ID(s).
if episode.agent_id != "some_agent_id":
return observation
# Preprocess other agents' observations.
...
1) Multi-agent preprocessor with access to the entire multi-agent observation dict: Alternatively, you can subclass the
:py:class:`~ray.rllib.connectors.env_to_module.observation_preprocessor.MultiAgentObservationPreprocessor` API and
override the same two methods, ``recompute_output_observation_space`` and ``preprocess``.
See here for a `2-agent observation preprocessor example <https://github.com/ray-project/ray/blob/master/rllib/examples/connectors/multi_agent_observation_preprocessor.py>`__
showing how to enhance each agents' observations through adding information from the respective other agent to the observations.
Use :py:class:`~ray.rllib.connectors.env_to_module.observation_preprocessor.MultiAgentObservationPreprocessor` whenever you need to
preprocess observations of an agent by lookup information from other agents, for example their own observations, but also rewards and
previous actions.
Example: Adding new columns to the batch
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
So far, you have altered the observations in the input episodes, either by
:ref:`manipulating them directly <observation-preprocessors>` or
:ref:`adding additional information like rewards to them <observation-preprocessors-adding-rewards-to-obs>`.
RLlib's default env-to-module connectors add the observations found in the episodes to the batch under the ``obs`` column.
If you would like to create a new column in the batch, you can subclass :py:class:`~ray.rllib.connectors.connector_v2.ConnectorV2` directly
and implement its :py:meth:`~ray.rllib.connectors.connector_v2.ConnectorV2.__call__` method. This way, if you have an
:py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` that requires certain custom columns to be present in the input batch,
write a custom connector piece following this example here:
.. testcode::
import numpy as np
from ray.rllib.connectors.connector_v2 import ConnectorV2
class AddNewColumnToBatch(ConnectorV2):
def __init__(
self,
input_observation_space=None,
input_action_space=None,
*,
col_name: str = "last_3_rewards_mean",
):
super().__init__(input_observation_space, input_action_space)
self.col_name = col_name
def __call__(self, *, episodes, batch, rl_module, explore, shared_data, **kwargs):
# Use the convenience `single_agent_episode_iterator` to loop through given episodes.
# Even if `episodes` are a list of MultiAgentEpisodes, RLlib splits them up into
# their single-agent subcomponents.
for sa_episode in self.single_agent_episode_iterator(episodes):
# Compute some example new-data item for your `batch` (to be added
# under a new column).
# Here, we compile the average over the last 3 rewards.
last_3_rewards = sa_episode.get_rewards(
indices=[-3, -2, -1],
fill=0.0, # at beginning of episode, fill with 0s
)
new_data_item = np.mean(last_3_rewards)
# Use the convenience utility: `add_item_to_batch` to add a new value to
# a new or existing column.
self.add_batch_item(
batch=batch,
column=self.col_name,
item_to_add=new_data_item,
single_agent_episode=sa_episode,
)
# Return the altered batch (with the new column in it).
return batch
.. testcode::
:hide:
config = (
PPOConfig()
.environment("CartPole-v1")
.env_runners(
env_to_module_connector=lambda env, spaces, device: AddNewColumnToBatch()
)
)
env = gym.make("CartPole-v1")
env_to_module = config.build_env_to_module_connector(env=env, spaces=None)
episode = SingleAgentEpisode()
obs, _ = env.reset()
episode.add_env_reset(observation=obs)
action = 0
obs, _, _, _, _ = env.step(action)
episode.add_env_step(observation=obs, action=action, reward=1.0)
batch = {}
batch = env_to_module(
episodes=[episode],
batch=batch,
rl_module=None, # in stateless case, RLModule is not strictly required
explore=True,
)
# Print out the resulting batch.
print(batch)
You should see the new column in the batch, after running through this connector piece.
Note, though, that if your :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` also requires the new information
in the train batch, you would also need to add the same custom connector piece to your Algorithm's
:py:class:`~ray.rllib.connectors.learner.learner_connector_pipeline.LearnerConnectorPipeline`.
See :ref:`the Learner connector pipeline documentation <learner-pipeline-docs>` for more details on how to customize it.
doc/source/rllib/external-envs.rst
+200
View File
@@ -0,0 +1,200 @@
.. _rllib-external-env-setups-doc:
External Environments and Applications
======================================
.. include:: /_includes/rllib/new_api_stack.rst
In many situations, it doesn't make sense for an RL environment to be "stepped" by RLlib.
For example, if you train a policy inside a complex simulator that operates its own execution loop,
like a game engine or a robotics simulation. A natural and user friendly approach is to flip this setup around
and - instead of RLlib "stepping" the env - allow the agents in the simulation to fully control
their own stepping. An external RLlib-powered service would be available for either querying
individual actions or for accepting batched sample data. The service would cover the task
of training the policies, but wouldn't pose any restrictions on when and how often per second the simulation
should step.
.. figure:: images/envs/external_env_setup_client_inference.svg
:align: left
:width: 600
**External application with client-side inference**: An external simulator (for example a game engine)
connects to RLlib, which runs as a server through a tcp-cabable, custom EnvRunner.
The simulator sends batches of data from time to time to the server and in turn receives weights updates.
For better performance, actions are computed locally on the client side.
RLlib provides an `external messaging protocol <https://github.com/ray-project/ray/blob/master/rllib/env/external/rllink.py>`__
called :ref:`RLlink <rllink-protocol-docs>` for this purpose as well as the option to customize your :py:class:`~ray.rllib.env.env_runner.EnvRunner` class
toward communicating through :ref:`RLlink <rllink-protocol-docs>` with one or more clients.
An example, `tcp-based EnvRunner implementation with RLlink is available here <https://github.com/ray-project/ray/blob/master/rllib/examples/envs/env_connecting_to_rllib_w_tcp_client.py>`__.
It also contains a dummy (CartPole) client that can be used for testing and as a template for how your external application or simulator should
utilize the :ref:`RLlink <rllink-protocol-docs>` protocol.
.. note::
External application support is still work-in-progress on RLlib's new API stack. The Ray team
is working on more examples for custom EnvRunner implementations (besides
`the already available tcp-based one <https://github.com/ray-project/ray/blob/master/rllib/env/tcp_client_inference_env_runner.py>`__)
as well as various client-side, non-python RLlib-adapters, for example for popular game engines and other
simulation software.
.. _rllink-protocol-docs:
The RLlink Protocol
-------------------
RLlink is a simple, stateful protocol designed for communication between a reinforcement learning (RL) server (ex., RLlib) and an
external client acting as an environment simulator. The protocol enables seamless exchange of RL-specific data such as episodes,
configuration, and model weights, while also facilitating on-policy training workflows.
Key Features
~~~~~~~~~~~~
- **Stateful Design**: The protocol maintains some state through sequences of message exchanges (ex., request-response pairs like `GET_CONFIG` -> `SET_CONFIG`).
- **Strict Request-Response Design**: The protocol is strictly (client) request -> (server) response based. Due to the necessity to let the client simulation run in its own execution loop, the server side refrains from sending any unsolicited messages to the clients.
- **RL-Specific Capabilities**: Tailored for RL workflows, including episode handling, model weight updates, and configuration management.
- **Flexible Sampling**: Supports both on-policy and off-policy data collection modes.
- **JSON**: For reasons of better debugging and faster iterations, the first versions of RLlink are entirely JSON-based, non-encrypted, and non-secure.
Message Structure
~~~~~~~~~~~~~~~~~
RLlink messages consist of a header and a body:
- **Header**: 8-byte length field indicating the size of the body, for example `00000016` for a body of length 16 (thus, in total, the message size).
- **Body**: JSON-encoded content with a `type` field indicating the message type.
Example Messages: PING and EPISODES_AND_GET_STATE
+++++++++++++++++++++++++++++++++++++++++++++++++
Here is a complete simple example message for the `PING` message. Note the 8-byte header
encoding the size of the following body to be of length `16`, followed by the message body with the mandatory "type" field.
.. code-block::
00000016{"type": "PING"}
The `PING` message should be sent by the client after initiation of a new connection. The server
then responds with:
.. code-block::
00000016{"type": "PONG"}
Here is an example of an `EPISODES_AND_GET_STATE` message sent by the client to the server and carrying
a batch of sampling data. With the same message, the client asks the server to send back the updated model weights.
.. _example-rllink-episode-and-get-state-msg:
.. code-block:: javascript
{
"type": "EPISODES_AND_GET_STATE",
"episodes": [
{
"obs": [[...]], // List of observations
"actions": [...], // List of actions
"rewards": [...], // List of rewards
"is_terminated": false,
"is_truncated": false
}
],
"env_steps": 128
}
Overview of all Message Types
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Requests: Client → Server
+++++++++++++++++++++++++
- **``PING``**
- Example: ``{"type": "PING"}``
- Purpose: Initial handshake to establish communication.
- Expected Response: ``{"type": "PONG"}``.
- **``GET_CONFIG``**
- Example: ``{"type": "GET_CONFIG"}``
- Purpose: Request the relevant configuration (for example, how many timesteps to collect for a single `EPISODES_AND_GET_STATE` message; see below).
- Expected Response: ``{"type": "SET_CONFIG", "env_steps_per_sample": 500, "force_on_policy": true}``.
- **``EPISODES_AND_GET_STATE``**
- Example: :ref:`See here for an example message <example-rllink-episode-and-get-state-msg>`
- Purpose: Combine ``EPISODES`` and ``GET_STATE`` into a single request. This is useful for workflows requiring on-policy (synchronous) updates to model weights after data collection.
- Body:
- ``episodes``: A list of JSON objects (dicts), each with mandatory keys "obs" (list of observations in the episode), "actions" (list of actions in the episode), "rewards" (list of rewards in the episode), "is_terminated" (bool), and "is_truncated" (bool). Note that the "obs" list has one item more than the lists for "actions" and "rewards" due to the initial "reset" observation.
- ``weights_seq_no``: Sequence number for the model weights version, ensuring synchronization.
- Expected Response: ``{"type": "SET_STATE", "weights_seq_no": 123, "mlir_file": ".. [b64 encoded string of the binary .mlir file with the model in it] .."}``.
Responses: Server → Client
++++++++++++++++++++++++++
- **``PONG``**
- Example: ``{"type": "PONG"}``
- Purpose: Acknowledgment of the ``PING`` request to confirm connectivity.
- **``SET_STATE``**
- Example: ``{"type": "SET_STATE", "weights_seq_no": 123, "onnx_file": "... [base64 encoded ONNX file] ..."}``
- Purpose: Provide the client with the current state (for example, model weights).
- Body:
- ``onnx_file``: Base64-encoded, compressed ONNX model file.
- ``weights_seq_no``: Sequence number for the model weights, ensuring synchronization.
- **``SET_CONFIG``**
- Purpose: Send relevant configuration details to the client.
- Body:
- ``env_steps_per_sample``: Number of total env steps collected for one ``EPISODES_AND_GET_STATE`` message.
- ``force_on_policy``: Whether on-policy sampling is enforced. If true, the client should wait after sending the ``EPISODES_AND_GET_STATE`` message for the ``SET_STATE`` response before continuing to collect the next round of samples.
Workflow Examples
+++++++++++++++++
**Initial Handshake**
1. Client sends ``PING``.
2. Server responds with ``PONG``.
**Configuration Request**
1. Client sends ``GET_CONFIG``.
2. Server responds with ``SET_CONFIG``.
**Training (on-policy)**
1. Client collects on-policy data and sends ``EPISODES_AND_GET_STATE``.
2. Server processes the episodes and responds with ``SET_STATE``.
.. note::
This protocol is an initial draft of the attempt to develop a widely adapted protocol for communication between an external
client and a remote RL-service. Expect many changes, enhancements, and upgrades as it moves toward maturity, including
adding a safety layer and compression.
For now, however, it offers a lightweight, simple, yet powerful interface for integrating external environments with RL
frameworks.
Example: External client connecting to tcp-based EnvRunner
----------------------------------------------------------
An example `tcp-based EnvRunner implementation with RLlink is available here <https://github.com/ray-project/ray/blob/master/rllib/env/tcp_client_inference_env_runner.py>`__.
See `here for the full end-to-end example <https://github.com/ray-project/ray/blob/master/rllib/examples/envs/env_connecting_to_rllib_w_tcp_client.py>`__.
Feel free to alter the underlying logic of your custom EnvRunner, for example, you could implement a shared memory based
communication layer (instead of the tcp-based one).
doc/source/rllib/getting-started.rst
+487
View File
@@ -0,0 +1,487 @@
.. _rllib-getting-started:
Getting Started
===============
.. include:: /_includes/rllib/new_api_stack.rst
.. _rllib-in-60min:
RLlib in 60 minutes
-------------------
.. figure:: images/rllib-index-header.svg
In this tutorial, you learn how to design, customize, and run an end-to-end RLlib learning experiment
from scratch. This includes picking and configuring an :py:class:`~ray.rllib.algorithms.algorithm.Algorithm`,
running a couple of training iterations, saving the state of your
:py:class:`~ray.rllib.algorithms.algorithm.Algorithm` from time to time, running a separate
evaluation loop, and finally utilizing one of the checkpoints to deploy your trained model
to an environment outside of RLlib and compute actions.
You also learn how to customize your :ref:`RL environment <rllib-key-concepts-environments>`
and your :ref:`neural network model <rllib-key-concepts-rl-modules>`.
Installation
~~~~~~~~~~~~
First, install RLlib, `PyTorch <https://pytorch.org>`__, and `Farama Gymnasium <https://gymnasium.farama.org>`__ as shown below:
.. code-block:: bash
pip install "ray[rllib]" torch "gymnasium[atari,accept-rom-license,mujoco]"
.. _rllib-python-api:
Python API
~~~~~~~~~~
RLlib's Python API provides all the flexibility required for applying the library to any
type of RL problem.
You manage RLlib experiments through an instance of the :py:class:`~ray.rllib.algorithms.algorithm.Algorithm`
class. An :py:class:`~ray.rllib.algorithms.algorithm.Algorithm` typically holds a neural
network for computing actions, called ``policy``, the :ref:`RL environment <rllib-key-concepts-environments>`
that you want to optimize against, a loss function, an optimizer, and some code describing the
algorithm's execution logic, like determining when to collect samples, when to update your model, etc..
In :ref:`multi-agent training <rllib-multi-agent-environments-doc>`,
:py:class:`~ray.rllib.algorithms.algorithm.Algorithm` manages the querying and optimization of multiple policies at once.
Through the algorithm's interface, you can train the policy, compute actions, or store your
algorithm's state through checkpointing.
Configure and build the algorithm
+++++++++++++++++++++++++++++++++
You first create an :py:class:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig` instance
and change some default settings through the config object's various methods.
For example, you can set the :ref:`RL environment <rllib-key-concepts-environments>`
you want to use by calling the config's :py:meth:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig.environment`
method:
.. testcode::
from ray.rllib.algorithms.ppo import PPOConfig
# Create a config instance for the PPO algorithm.
config = (
PPOConfig()
.environment("Pendulum-v1")
)
To scale your setup and define how many :py:class:`~ray.rllib.env.env_runner.EnvRunner` actors you want to leverage,
you can call the :py:meth:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig.env_runners` method.
``EnvRunners`` are used to collect samples for training updates from your :ref:`environment <rllib-key-concepts-environments>`.
.. testcode::
config.env_runners(num_env_runners=2)
For training-related settings or any algorithm-specific settings, use the
:py:meth:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig.training` method:
.. testcode::
config.training(
lr=0.0002,
train_batch_size_per_learner=2000,
num_epochs=10,
)
Finally, you build the actual :py:class:`~ray.rllib.algorithms.algorithm.Algorithm` instance
through calling your config's :py:meth:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig.build_algo`
method.
.. testcode::
# Build the Algorithm (PPO).
ppo = config.build_algo()
.. note::
See here to learn about all the :ref:`methods you can use to configure your Algorithm <rllib-algo-configuration-docs>`.
Run the algorithm
+++++++++++++++++
After you built your :ref:`PPO <ppo>` from its configuration, you can ``train`` it for a number of
iterations through calling the :py:meth:`~ray.rllib.algorithms.algorithm.Algorithm.train` method,
which returns a result dictionary that you can pretty-print for debugging purposes:
.. testcode::
from pprint import pprint
for _ in range(4):
pprint(ppo.train())
Checkpoint the algorithm
++++++++++++++++++++++++
To save the current state of your :py:class:`~ray.rllib.algorithms.algorithm.Algorithm`,
create a checkpoint through calling its :py:meth:`~ray.rllib.algorithms.algorithm.Algorithm.save_to_path` method,
which returns the directory of the saved checkpoint.
Instead of not passing any arguments to this call and letting the :py:class:`~ray.rllib.algorithms.algorithm.Algorithm` decide where to save
the checkpoint, you can also provide a checkpoint directory yourself:
.. testcode::
checkpoint_path = ppo.save_to_path()
# OR:
# ppo.save_to_path([a checkpoint location of your choice])
Evaluate the algorithm
++++++++++++++++++++++
RLlib supports setting up a separate :py:class:`~ray.rllib.env.env_runner_group.EnvRunnerGroup`
for the sole purpose of evaluating your model from time to time on the :ref:`RL environment <rllib-key-concepts-environments>`.
Use your config's :py:meth:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig.evaluation` method
to set up the details. By default, RLlib doesn't perform evaluation during training and only reports the
results of collecting training samples with its "regular" :py:class:`~ray.rllib.env.env_runner_group.EnvRunnerGroup`.
.. testcode::
:hide:
ppo.stop()
.. testcode::
config.evaluation(
# Run one evaluation round every iteration.
evaluation_interval=1,
# Create 2 eval EnvRunners in the extra EnvRunnerGroup.
evaluation_num_env_runners=2,
# Run evaluation for exactly 10 episodes. Note that because you have
# 2 EnvRunners, each one runs through 5 episodes.
evaluation_duration_unit="episodes",
evaluation_duration=10,
)
# Rebuild the PPO, but with the extra evaluation EnvRunnerGroup
ppo_with_evaluation = config.build_algo()
for _ in range(3):
pprint(ppo_with_evaluation.train())
.. testcode::
:hide:
ppo_with_evaluation.stop()
.. _rllib-with-ray-tune:
RLlib with Ray Tune
+++++++++++++++++++
All online RLlib :py:class:`~ray.rllib.algorithms.algorithm.Algorithm` classes are compatible with
the :ref:`Ray Tune API <tune-api-ref>`.
.. note::
The offline RL algorithms, like :ref:`BC <bc>`, :ref:`CQL <cql>`, and :ref:`MARWIL <marwil>`
require more work on :ref:`Tune <tune-main>` and :ref:`Ray Data <data>`
to add Ray Tune support.
This integration allows for utilizing your configured :py:class:`~ray.rllib.algorithms.algorithm.Algorithm` in
:ref:`Ray Tune <tune-main>` experiments.
For example, the following code performs a hyper-parameter sweep of your :ref:`PPO <ppo>`, creating three ``Trials``,
one for each of the configured learning rates:
.. testcode::
from ray import tune
from ray.rllib.algorithms.ppo import PPOConfig
config = (
PPOConfig()
.environment("Pendulum-v1")
# Specify a simple tune hyperparameter sweep.
.training(
lr=tune.grid_search([0.001, 0.0005, 0.0001]),
)
)
# Create a Tuner instance to manage the trials.
tuner = tune.Tuner(
config.algo_class,
param_space=config,
# Specify a stopping criterion. Note that the criterion has to match one of the
# pretty printed result metrics from the results returned previously by
# ``.train()``. Also note that -1100 is not a good episode return for
# Pendulum-v1, we are using it here to shorten the experiment time.
run_config=tune.RunConfig(
stop={"env_runners/episode_return_mean": -1100.0},
),
)
# Run the Tuner and capture the results.
results = tuner.fit()
Note that each :py:class:`~ray.tune.trial.Trial` creates a separate
:py:class:`~ray.rllib.algorithms.algorithm.Algorithm` instance as a :ref:`Ray actor <actor-guide>`,
assigns compute resources to each ``Trial``, and runs them in parallel, if possible,
on your Ray cluster:
.. code-block:: text
Trial status: 3 RUNNING
Current time: 2025-01-17 18:47:33. Total running time: 3min 0s
Logical resource usage: 9.0/12 CPUs, 0/0 GPUs
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Trial name status lr iter total time (s) episode_return_mean .._sampled_lifetime │
├───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ PPO_Pendulum-v1_b5c41_00000 RUNNING 0.001 29 86.2426 -998.449 108000 │
│ PPO_Pendulum-v1_b5c41_00001 RUNNING 0.0005 25 74.4335 -997.079 100000 │
│ PPO_Pendulum-v1_b5c41_00002 RUNNING 0.0001 20 60.0421 -960.293 80000 │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
``Tuner.fit()`` returns a ``ResultGrid`` object that allows for a detailed analysis of the
training process and for retrieving the :ref:`checkpoints <rllib-checkpoints-docs>` of the trained
algorithms and their models:
.. testcode::
# Get the best result of the final iteration, based on a particular metric.
best_result = results.get_best_result(
metric="env_runners/episode_return_mean",
mode="max",
scope="last",
)
# Get the best checkpoint corresponding to the best result
# from the preceding experiment.
best_checkpoint = best_result.checkpoint
Deploy a trained model for inference
++++++++++++++++++++++++++++++++++++
After training, you might want to deploy your models into a new environment, for example
to run inference in production. For this purpose, you can use the checkpoint directory created
in the preceding example. To read more about checkpoints, model deployments, and restoring algorithm state,
see this :ref:`page on checkpointing <rllib-checkpoints-docs>` here.
Here is how you would create a new model instance from the checkpoint and run inference through
a single episode of your RL environment. Note in particular the use of the
:py:meth:`~ray.rllib.utils.checkpoints.Checkpointable.from_checkpoint` method to create
the model and the
:py:meth:`~ray.rllib.core.rl_module.rl_module.RLModule.forward_inference`
method to compute actions:
.. testcode::
from pathlib import Path
import gymnasium as gym
import numpy as np
import torch
from ray.rllib.core.rl_module import RLModule
# Create only the neural network (RLModule) from our algorithm checkpoint.
# See here (https://docs.ray.io/en/master/rllib/checkpoints.html)
# to learn more about checkpointing and the specific "path" used.
rl_module = RLModule.from_checkpoint(
Path(best_checkpoint.path)
/ "learner_group"
/ "learner"
/ "rl_module"
/ "default_policy"
)
# Create the RL environment to test against (same as was used for training earlier).
env = gym.make("Pendulum-v1")
episode_return = 0.0
done = False
# Reset the env to get the initial observation.
obs, info = env.reset()
while not done:
# Uncomment this line to render the env.
# env.render()
# Compute the next action from a batch (B=1) of observations.
obs_batch = torch.from_numpy(obs).unsqueeze(0) # add batch B=1 dimension
model_outputs = rl_module.forward_inference({"obs": obs_batch})
# Extract the action distribution parameters from the output and dissolve batch dim.
action_dist_params = model_outputs["action_dist_inputs"][0].numpy()
# We have continuous actions -> take the mean (max likelihood).
greedy_action = np.clip(
action_dist_params[0:1], # 0=mean, 1=log(stddev), [0:1]=use mean, but keep shape=(1,)
a_min=env.action_space.low[0],
a_max=env.action_space.high[0],
)
# For discrete actions, you should take the argmax over the logits:
# greedy_action = np.argmax(action_dist_params)
# Send the action to the environment for the next step.
obs, reward, terminated, truncated, info = env.step(greedy_action)
# Perform env-loop bookkeeping.
episode_return += reward
done = terminated or truncated
print(f"Reached episode return of {episode_return}.")
.. note::
To watch the agent play the environment in a local pygame window, pass
``render_mode="human"`` to ``gym.make(...)`` and uncomment the ``env.render()``
call in the loop above. This requires a local display, so it's omitted here
to keep the example runnable in headless environments.
Alternatively, if you still have an :py:class:`~ray.rllib.algorithms.algorithm.Algorithm` instance up and running
in your script, you can get the :py:class:`~ray.rllib.core.rl_module.rl_module.RLModule` through the
:py:meth:`~ray.rllib.algorithms.algorithm.Algorithm.get_module` method:
.. code-block:: python
rl_module = ppo.get_module("default_policy") # Equivalent to `rl_module = ppo.get_module()`
Customizing your RL environment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In the preceding examples, your :ref:`RL environment <rllib-key-concepts-environments>` was
a `Farama gymnasium <gymnasium.farama.org>`__ pre-registered one,
like ``Pendulum-v1`` or ``CartPole-v1``. However, if you would like to run your
experiments against a custom one, see this tab below for a less-than-50-lines example.
See here for an :ref:`in-depth guide on how to setup RL environments in RLlib <rllib-environments-doc>`
and how to customize them.
.. dropdown:: Quickstart: Custom RL environment
:animate: fade-in-slide-down
.. testcode::
import gymnasium as gym
from ray.rllib.algorithms.ppo import PPOConfig
# Define your custom env class by subclassing gymnasium.Env:
class ParrotEnv(gym.Env):
"""Environment in which the agent learns to repeat the seen observations.
Observations are float numbers indicating the to-be-repeated values,
e.g. -1.0, 5.1, or 3.2.
The action space is the same as the observation space.
Rewards are `r=-abs([observation] - [action])`, for all steps.
"""
def __init__(self, config=None):
# Since actions should repeat observations, their spaces must be the same.
self.observation_space = config.get(
"obs_act_space",
gym.spaces.Box(-1.0, 1.0, (1,), np.float32),
)
self.action_space = self.observation_space
self._cur_obs = None
self._episode_len = 0
def reset(self, *, seed=None, options=None):
"""Resets the environment, starting a new episode."""
# Reset the episode len.
self._episode_len = 0
# Sample a random number from our observation space.
self._cur_obs = self.observation_space.sample()
# Return initial observation.
return self._cur_obs, {}
def step(self, action):
"""Takes a single step in the episode given `action`."""
# Set `terminated` and `truncated` flags to True after 10 steps.
self._episode_len += 1
terminated = truncated = self._episode_len >= 10
# Compute the reward: `r = -abs([obs] - [action])`
reward = -sum(abs(self._cur_obs - action))
# Set a new observation (random sample).
self._cur_obs = self.observation_space.sample()
return self._cur_obs, reward, terminated, truncated, {}
# Point your config to your custom env class:
config = (
PPOConfig()
.environment(
ParrotEnv,
# Add `env_config={"obs_act_space": [some Box space]}` to customize.
)
)
# Build a PPO algorithm and train it.
ppo_w_custom_env = config.build_algo()
ppo_w_custom_env.train()
.. testcode::
:hide:
ppo_w_custom_env.stop()
Customizing your models
~~~~~~~~~~~~~~~~~~~~~~~
In the preceding examples, because you didn't specify anything in your
:py:class:`~ray.rllib.algorithms.algorithm_config.AlgorithmConfig`, RLlib provided a default
neural network model. If you would like to either reconfigure the type and size of RLlib's default models,
for example define the number of hidden layers and their activation functions,
or even write your own custom models from scratch using PyTorch, see here
for a :ref:`detailed guide on the RLModule class <rlmodule-guide>`.
See this tab below for a 30-lines example.
.. dropdown:: Quickstart: Custom RLModule
:animate: fade-in-slide-down
.. testcode::
import torch
from ray.rllib.core.columns import Columns
from ray.rllib.core.rl_module.torch import TorchRLModule
# Define your custom env class by subclassing `TorchRLModule`:
class CustomTorchRLModule(TorchRLModule):
def setup(self):
# You have access here to the following already set attributes:
# self.observation_space
# self.action_space
# self.inference_only
# self.model_config # <- a dict with custom settings
input_dim = self.observation_space.shape[0]
hidden_dim = self.model_config["hidden_dim"]
output_dim = self.action_space.n
# Define and assign your torch subcomponents.
self._policy_net = torch.nn.Sequential(
torch.nn.Linear(input_dim, hidden_dim),
torch.nn.ReLU(),
torch.nn.Linear(hidden_dim, output_dim),
)
def _forward(self, batch, **kwargs):
# Push the observations from the batch through our `self._policy_net`.
action_logits = self._policy_net(batch[Columns.OBS])
# Return parameters for the default action distribution, which is
# `TorchCategorical` (due to our action space being `gym.spaces.Discrete`).
return {Columns.ACTION_DIST_INPUTS: action_logits}
doc/source/rllib/hierarchical-envs.rst
+54
View File
@@ -0,0 +1,54 @@
.. _rllib-hierarchical-environments-doc:
Hierarchical Environments
=========================
.. include:: /_includes/rllib/new_api_stack.rst
You can implement hierarchical training as a special case of multi-agent RL. For example, consider a two-level hierarchy of policies,
where a top-level policy issues high level tasks that are executed at a finer timescale by one or more low-level policies.
The following timeline shows one step of the top-level policy, which corresponds to four low-level actions:
.. code-block:: text
top-level: action_0 -------------------------------------> action_1 ->
low-level: action_0 -> action_1 -> action_2 -> action_3 -> action_4 ->
Alternatively, you could implement an environment, in which the two agent types don't act at the same time (overlappingly),
but the low-level agents wait for the high-level agent to issue an action, then act n times before handing
back control to the high-level agent:
.. code-block:: text
top-level: action_0 -----------------------------------> action_1 ->
low-level: ---------> action_0 -> action_1 -> action_2 ------------>
You can implement any of these hierarchical action patterns as a multi-agent environment with various
types of agents, for example a high-level agent and a low-level agent. When set up using the correct
agent to module mapping functions, from RLlib's perspective, the problem becomes a simple independent
multi-agent problem with different types of policies.
Your configuration might look something like the following:
.. testcode::
from ray.rllib.algorithms.ppo import PPOConfig
config = (
PPOConfig()
.multi_agent(
policies={"top_level", "low_level"},
policy_mapping_fn=(
lambda aid, eps, **kw: "low_level" if aid.startswith("low_level") else "top_level"
),
policies_to_train=["top_level"],
)
)
In this setup, the appropriate rewards at any hierarchy level should be provided by the multi-agent env implementation.
The environment class is also responsible for routing between agents, for example conveying `goals <https://arxiv.org/pdf/1703.01161.pdf>`__ from higher-level
agents to lower-level agents as part of the lower-level agent observation.
See `this runnable example of a hierarchical env <https://github.com/ray-project/ray/blob/master/rllib/examples/hierarchical/hierarchical_training.py>`__.
doc/source/rllib/images/MemoryTrackingCallbacks.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 74 KiB

doc/source/rllib/images/acrobot-v1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.6 KiB

doc/source/rllib/images/algos/appo-architecture.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 727 KiB

doc/source/rllib/images/algos/bc-architecture.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 819 KiB

doc/source/rllib/images/algos/cql-architecture.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 1.2 MiB

doc/source/rllib/images/algos/curiosity-architecture.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 560 KiB

doc/source/rllib/images/algos/dqn-architecture.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 786 KiB

doc/source/rllib/images/algos/dreamerv3-architecture.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 663 KiB

doc/source/rllib/images/algos/icm-architecture.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 560 KiB

doc/source/rllib/images/algos/impala-architecture.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 726 KiB

doc/source/rllib/images/algos/marwil-architecture.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 1.1 MiB

doc/source/rllib/images/algos/ppo-architecture.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 502 KiB

doc/source/rllib/images/algos/sac-architecture.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 658 KiB

doc/source/rllib/images/apex-arch.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 112 KiB

doc/source/rllib/images/apex.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 62 KiB

doc/source/rllib/images/catalog/catalog_and_rlm_diagram.svg
+16
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 13 KiB

doc/source/rllib/images/catalog/catalog_rlmspecs_diagram.svg
+16
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 7.1 KiB

doc/source/rllib/images/catalog/ppo_catalog_and_rlm_diagram.svg
+16
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 18 KiB

doc/source/rllib/images/checkpointing/from_checkpoint.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 65 KiB

doc/source/rllib/images/checkpointing/save_and_restore.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 116 KiB

doc/source/rllib/images/connector-diagram.svg
+16
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

doc/source/rllib/images/connector_v2/connector_generic.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 272 KiB

doc/source/rllib/images/connector_v2/custom_pieces_in_env_to_module_pipeline.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 204 KiB

doc/source/rllib/images/connector_v2/custom_pieces_in_learner_pipeline.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 182 KiB

doc/source/rllib/images/connector_v2/env_runner_connector_pipelines.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 392 KiB

doc/source/rllib/images/connector_v2/env_to_module_connector.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 266 KiB

doc/source/rllib/images/connector_v2/frame_stacking_connector_setup.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 176 KiB

doc/source/rllib/images/connector_v2/generic_connector_pipeline.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 117 KiB

doc/source/rllib/images/connector_v2/learner_connector.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 40 KiB

doc/source/rllib/images/connector_v2/learner_connector_pipeline.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 87 KiB

doc/source/rllib/images/connector_v2/location_of_connector_pipelines_in_rllib.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 398 KiB

doc/source/rllib/images/connector_v2/module_to_env_connector.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 266 KiB

doc/source/rllib/images/connector_v2/pipeline_batch_phases_multi_agent.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 398 KiB

doc/source/rllib/images/connector_v2/pipeline_batch_phases_single_agent.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 344 KiB

doc/source/rllib/images/connector_v2/pipeline_batch_phases_single_agent_w_states.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 620 KiB

doc/source/rllib/images/custom_metric.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 26 KiB

doc/source/rllib/images/debugging_rllib_in_ide.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 246 KiB

doc/source/rllib/images/dqn-arch.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 83 KiB

doc/source/rllib/images/dreamerv3/atari100k_1_vs_4gpus.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 1.6 MiB

doc/source/rllib/images/dreamerv3/dmc_1_vs_4gpus.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 254 KiB

doc/source/rllib/images/dreamerv3/dreamerv3.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 653 KiB

doc/source/rllib/images/dreamerv3/flappy_bird_env.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 96 KiB

doc/source/rllib/images/dreamerv3/pong_1_2_and_4gpus.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 136 KiB

doc/source/rllib/images/envs/env_loop_concept.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 17 MiB

doc/source/rllib/images/envs/env_runners.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 6.5 MiB

doc/source/rllib/images/envs/external_env_logo.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 581 KiB

doc/source/rllib/images/envs/external_env_setup_client_inference.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 612 KiB

doc/source/rllib/images/envs/external_env_setup_server_inference.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 617 KiB

doc/source/rllib/images/envs/hierarchical_env_logo.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 15 KiB

doc/source/rllib/images/envs/multi_agent_env_logo.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 554 KiB

doc/source/rllib/images/envs/multi_agent_episode_complex_order.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 85 KiB

doc/source/rllib/images/envs/multi_agent_episode_simultaneous.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 81 KiB

doc/source/rllib/images/envs/multi_agent_episode_turn_based.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 59 KiB

doc/source/rllib/images/envs/multi_agent_setup.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 626 KiB

doc/source/rllib/images/envs/single_agent_env_logo.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 546 KiB

doc/source/rllib/images/envs/single_agent_setup.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 602 KiB

doc/source/rllib/images/episodes/sa_episode.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 183 KiB

doc/source/rllib/images/episodes/sa_episode_cut_and_lookback.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 179 KiB

doc/source/rllib/images/episodes/sa_episode_finalized.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 116 KiB

doc/source/rllib/images/episodes/sa_episode_getters.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 224 KiB

doc/source/rllib/images/episodes/sa_episode_non_finalized.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 108 KiB

doc/source/rllib/images/episodes/sa_episode_non_finalized_vs_finalized.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 103 KiB

doc/source/rllib/images/episodes/usage_of_episodes.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 434 KiB

doc/source/rllib/images/impala.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 37 KiB

doc/source/rllib/images/metrics_logger_api.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 280 KiB

doc/source/rllib/images/metrics_logger_hierarchy.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 107 KiB

doc/source/rllib/images/offline/docs_rllib_offline_bc_episode_return_mean.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 47 KiB

doc/source/rllib/images/offline/docs_rllib_offline_prelearner.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 204 KiB

doc/source/rllib/images/offline/key_layers.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 186 KiB

doc/source/rllib/images/policy_classes_overview.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 70 KiB

doc/source/rllib/images/ppo.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 30 KiB

doc/source/rllib/images/pytorch.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

doc/source/rllib/images/rl_modules/rl_module_in_env_runner.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 3.5 MiB

doc/source/rllib/images/rl_modules/rl_module_in_learner.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 106 KiB

doc/source/rllib/images/rl_modules/rl_module_overview.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 178 KiB

doc/source/rllib/images/rllib-batch-modes.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 131 KiB

doc/source/rllib/images/rllib-example-run-help.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 28 KiB

doc/source/rllib/images/rllib-exploration-api-table.svg
+5372
View File
File diff suppressed because it is too large Load Diff
Side by Side

After

Width:  |  Height:  |  Size: 326 KiB

doc/source/rllib/images/rllib-index-header.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 1.1 MiB

doc/source/rllib/images/rllib-logo.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 8.8 KiB

doc/source/rllib/images/rllib-logo.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 38 KiB

doc/source/rllib/images/rllib-new-api-stack-simple.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 263 KiB

doc/source/rllib/images/rllib-new-api-stack.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 20 MiB

doc/source/rllib/images/rllib-offline.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.3 MiB

doc/source/rllib/images/rllib-sample-collection.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 111 KiB

doc/source/rllib/images/rllib-training-inside-a-unity3d-env.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 264 KiB

doc/source/rllib/images/rllib-trajectory-view-example.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 207 KiB

Some files were not shown because too many files have changed in this diff Show More