wehub/ray-project--ray
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:17:40 +08:00
commit f1825c8ceb
10096 changed files with 2364182 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/ray-core/scheduling/accelerators.rst
+839
View File
@@ -0,0 +1,839 @@
.. _gpu-support:
.. _accelerator-support:
Accelerator Support
===================
Accelerators like GPUs are critical for many machine learning apps.
Ray Core natively supports many accelerators as pre-defined :ref:`resource <core-resources>` types and allows tasks and actors to specify their accelerator :ref:`resource requirements <resource-requirements>`.
The accelerators natively supported by Ray Core are:
.. list-table::
:header-rows: 1
* - Accelerator
- Ray Resource Name
- Support Level
* - NVIDIA GPU
- GPU
- Fully tested, supported by the Ray team
* - AMD GPU
- GPU
- Experimental, supported by the community
* - Intel GPU
- GPU
- Experimental, supported by the community
* - `AWS Neuron Core <https://awsdocs-neuron.readthedocs-hosted.com/en/latest/general/arch/model-architecture-fit.html>`_
- neuron_cores
- Experimental, supported by the community
* - Google TPU
- TPU
- Fully tested, supported by the Ray team
* - Intel Gaudi
- HPU
- Experimental, supported by the community
* - Huawei Ascend
- NPU
- Experimental, supported by the community
* - Rebellions RBLN
- RBLN
- Experimental, supported by the community
* - METAX GPU
- GPU
- Experimental, supported by the community
* - FuriosaAI
- FURIOSA
- Experimental, supported by the community
Starting Ray nodes with accelerators
------------------------------------
By default, Ray sets the quantity of accelerator resources of a node to the physical quantities of accelerators auto detected by Ray.
If you need to, you can :ref:`override <specify-node-resources>` this.
.. tab-set::
.. tab-item:: NVIDIA GPU
:sync: NVIDIA GPU
.. tip::
You can set the ``CUDA_VISIBLE_DEVICES`` environment variable before starting a Ray node
to limit the NVIDIA GPUs that are visible to Ray.
For example, ``CUDA_VISIBLE_DEVICES=1,3 ray start --head --num-gpus=2``
lets Ray only see devices 1 and 3.
.. tab-item:: AMD GPU
:sync: AMD GPU
.. tip::
You can set the ``ROCR_VISIBLE_DEVICES`` environment variable before starting a Ray node
to limit the AMD GPUs that are visible to Ray.
For example, ``ROCR_VISIBLE_DEVICES=1,3 ray start --head --num-gpus=2``
lets Ray only see devices 1 and 3.
.. tab-item:: Intel GPU
:sync: Intel GPU
.. tip::
You can set the ``ONEAPI_DEVICE_SELECTOR`` environment variable before starting a Ray node
to limit the Intel GPUs that are visible to Ray.
For example, ``ONEAPI_DEVICE_SELECTOR=1,3 ray start --head --num-gpus=2``
lets Ray only see devices 1 and 3.
.. tab-item:: AWS Neuron Core
:sync: AWS Neuron Core
.. tip::
You can set the ``NEURON_RT_VISIBLE_CORES`` environment variable before starting a Ray node
to limit the AWS Neuron Cores that are visible to Ray.
For example, ``NEURON_RT_VISIBLE_CORES=1,3 ray start --head --resources='{"neuron_cores": 2}'``
lets Ray only see devices 1 and 3.
See the `Amazon documentation <https://awslabs.github.io/data-on-eks/docs/category/inference-on-eks>`_ for more examples of Ray on Neuron with EKS as an orchestration substrate.
.. tab-item:: Google TPU
:sync: Google TPU
.. tip::
You can set the ``TPU_VISIBLE_CHIPS`` environment variable before starting a Ray node
to limit the Google TPUs that are visible to Ray.
For example, ``TPU_VISIBLE_CHIPS=1,3 ray start --head --resources='{"TPU": 2}'``
lets Ray only see devices 1 and 3.
.. tab-item:: Intel Gaudi
:sync: Intel Gaudi
.. tip::
You can set the ``HABANA_VISIBLE_MODULES`` environment variable before starting a Ray node
to limit the Intel Gaudi HPUs that are visible to Ray.
For example, ``HABANA_VISIBLE_MODULES=1,3 ray start --head --resources='{"HPU": 2}'``
lets Ray only see devices 1 and 3.
.. tab-item:: Huawei Ascend
:sync: Huawei Ascend
.. tip::
You can set the ``ASCEND_RT_VISIBLE_DEVICES`` environment variable before starting a Ray node
to limit the Huawei Ascend NPUs that are visible to Ray.
For example, ``ASCEND_RT_VISIBLE_DEVICES=1,3 ray start --head --resources='{"NPU": 2}'``
lets Ray only see devices 1 and 3.
.. tab-item:: Rebellions RBLN
:sync: Rebellions RBLN
.. tip::
You can set the ``RBLN_DEVICES`` environment variable before starting a Ray node
to limit the Rebellions RBLNs that are visible to Ray.
For example, ``RBLN_DEVICES=1,3 ray start --head --resources='{"RBLN": 2}'``
lets Ray only see devices 1 and 3.
.. tab-item:: METAX GPU
:sync: METAX GPU
.. tip::
You can set the ``CUDA_VISIBLE_DEVICES`` environment variable before starting a Ray node
to limit the METAX GPUs that are visible to Ray.
For example, ``CUDA_VISIBLE_DEVICES=1,3 ray start --head --num-gpus=2``
lets Ray only see devices 1 and 3.
.. tab-item:: FuriosaAI
:sync: FuriosaAI
.. tip::
You can set the ``FURIOSA_DEVICES`` environment variable before starting a Ray node
to limit the FuriosaAI NPUs that are visible to Ray, using ``npu:<id>`` tokens.
For example, ``FURIOSA_DEVICES=npu:1,npu:3 ray start --head``
lets Ray only see devices 1 and 3 (Ray auto-detects the count).
Bare integer IDs (e.g., ``FURIOSA_DEVICES=1,3``) are also accepted on read.
.. note::
When using the ``furiosa_llm.LLM`` Python API inside a Ray task or actor,
pass the assigned devices explicitly; ``LLM(devices=None)`` would
allocate all visible NPUs and bypass Ray's per-worker isolation::
from furiosa_llm import LLM
llm = LLM(model_path, devices=os.environ["FURIOSA_DEVICES"])
``furiosa-llm`` also accepts the PE-level form ``npu:X:Y``
(e.g., ``npu:0:0-3`` for fused PE 0-3 of NPU 0), but Ray currently
treats each NPU as a single resource and does not preserve PE
ranges through worker scheduling.
.. note::
There's nothing preventing you from specifying a larger number of
accelerator resources (e.g., ``num_gpus``) than the true number of accelerators on the machine given Ray resources are :ref:`logical <logical-resources>`.
In this case, Ray acts as if the machine has the number of accelerators you specified
for the purposes of scheduling tasks and actors that require accelerators.
Trouble only occurs if those tasks and actors
attempt to actually use accelerators that don't exist.
Using accelerators in Tasks and Actors
--------------------------------------
If a task or actor requires accelerators, you can specify the corresponding :ref:`resource requirements <resource-requirements>` (e.g. ``@ray.remote(num_gpus=1)``).
Ray then schedules the task or actor to a node that has enough free accelerator resources
and assign accelerators to the task or actor by setting the corresponding environment variable (e.g. ``CUDA_VISIBLE_DEVICES``) before running the task or actor code.
.. tab-set::
.. tab-item:: NVIDIA GPU
:sync: NVIDIA GPU
.. testcode::
import os
import ray
ray.init(num_gpus=2)
@ray.remote(num_gpus=1)
class GPUActor:
def ping(self):
print("GPU IDs: {}".format(ray.get_runtime_context().get_accelerator_ids()["GPU"]))
print("CUDA_VISIBLE_DEVICES: {}".format(os.environ["CUDA_VISIBLE_DEVICES"]))
@ray.remote(num_gpus=1)
def gpu_task():
print("GPU IDs: {}".format(ray.get_runtime_context().get_accelerator_ids()["GPU"]))
print("CUDA_VISIBLE_DEVICES: {}".format(os.environ["CUDA_VISIBLE_DEVICES"]))
gpu_actor = GPUActor.remote()
ray.get(gpu_actor.ping.remote())
# The actor uses the first GPU so the task uses the second one.
ray.get(gpu_task.remote())
.. testoutput::
:options: +MOCK
(GPUActor pid=52420) GPU IDs: [0]
(GPUActor pid=52420) CUDA_VISIBLE_DEVICES: 0
(gpu_task pid=51830) GPU IDs: [1]
(gpu_task pid=51830) CUDA_VISIBLE_DEVICES: 1
.. tab-item:: AMD GPU
:sync: AMD GPU
.. testcode::
:hide:
ray.shutdown()
.. testcode::
:skipif: True
import os
import ray
ray.init(num_gpus=2)
@ray.remote(num_gpus=1)
class GPUActor:
def ping(self):
print("GPU IDs: {}".format(ray.get_runtime_context().get_accelerator_ids()["GPU"]))
print("ROCR_VISIBLE_DEVICES: {}".format(os.environ["ROCR_VISIBLE_DEVICES"]))
@ray.remote(num_gpus=1)
def gpu_task():
print("GPU IDs: {}".format(ray.get_runtime_context().get_accelerator_ids()["GPU"]))
print("ROCR_VISIBLE_DEVICES: {}".format(os.environ["ROCR_VISIBLE_DEVICES"]))
gpu_actor = GPUActor.remote()
ray.get(gpu_actor.ping.remote())
# The actor uses the first GPU so the task uses the second one.
ray.get(gpu_task.remote())
.. testoutput::
:options: +MOCK
(GPUActor pid=52420) GPU IDs: [0]
(GPUActor pid=52420) ROCR_VISIBLE_DEVICES: 0
(gpu_task pid=51830) GPU IDs: [1]
(gpu_task pid=51830) ROCR_VISIBLE_DEVICES: 1
.. tab-item:: Intel GPU
:sync: Intel GPU
.. testcode::
:hide:
ray.shutdown()
.. testcode::
:skipif: True
import os
import ray
ray.init(num_gpus=2)
@ray.remote(num_gpus=1)
class GPUActor:
def ping(self):
print("GPU IDs: {}".format(ray.get_runtime_context().get_accelerator_ids()["GPU"]))
print("ONEAPI_DEVICE_SELECTOR: {}".format(os.environ["ONEAPI_DEVICE_SELECTOR"]))
@ray.remote(num_gpus=1)
def gpu_task():
print("GPU IDs: {}".format(ray.get_runtime_context().get_accelerator_ids()["GPU"]))
print("ONEAPI_DEVICE_SELECTOR: {}".format(os.environ["ONEAPI_DEVICE_SELECTOR"]))
gpu_actor = GPUActor.remote()
ray.get(gpu_actor.ping.remote())
# The actor uses the first GPU so the task uses the second one.
ray.get(gpu_task.remote())
.. testoutput::
:options: +MOCK
(GPUActor pid=52420) GPU IDs: [0]
(GPUActor pid=52420) ONEAPI_DEVICE_SELECTOR: 0
(gpu_task pid=51830) GPU IDs: [1]
(gpu_task pid=51830) ONEAPI_DEVICE_SELECTOR: 1
.. tab-item:: AWS Neuron Core
:sync: AWS Neuron Core
.. testcode::
:hide:
ray.shutdown()
.. testcode::
import os
import ray
ray.init(resources={"neuron_cores": 2})
@ray.remote(resources={"neuron_cores": 1})
class NeuronCoreActor:
def ping(self):
print("Neuron Core IDs: {}".format(ray.get_runtime_context().get_accelerator_ids()["neuron_cores"]))
print("NEURON_RT_VISIBLE_CORES: {}".format(os.environ["NEURON_RT_VISIBLE_CORES"]))
@ray.remote(resources={"neuron_cores": 1})
def neuron_core_task():
print("Neuron Core IDs: {}".format(ray.get_runtime_context().get_accelerator_ids()["neuron_cores"]))
print("NEURON_RT_VISIBLE_CORES: {}".format(os.environ["NEURON_RT_VISIBLE_CORES"]))
neuron_core_actor = NeuronCoreActor.remote()
ray.get(neuron_core_actor.ping.remote())
# The actor uses the first Neuron Core so the task uses the second one.
ray.get(neuron_core_task.remote())
.. testoutput::
:options: +MOCK
(NeuronCoreActor pid=52420) Neuron Core IDs: [0]
(NeuronCoreActor pid=52420) NEURON_RT_VISIBLE_CORES: 0
(neuron_core_task pid=51830) Neuron Core IDs: [1]
(neuron_core_task pid=51830) NEURON_RT_VISIBLE_CORES: 1
.. tab-item:: Google TPU
:sync: Google TPU
.. testcode::
:hide:
ray.shutdown()
.. testcode::
import os
import ray
ray.init(resources={"TPU": 2})
@ray.remote(resources={"TPU": 1})
class TPUActor:
def ping(self):
print("TPU IDs: {}".format(ray.get_runtime_context().get_accelerator_ids()["TPU"]))
print("TPU_VISIBLE_CHIPS: {}".format(os.environ["TPU_VISIBLE_CHIPS"]))
@ray.remote(resources={"TPU": 1})
def tpu_task():
print("TPU IDs: {}".format(ray.get_runtime_context().get_accelerator_ids()["TPU"]))
print("TPU_VISIBLE_CHIPS: {}".format(os.environ["TPU_VISIBLE_CHIPS"]))
tpu_actor = TPUActor.remote()
ray.get(tpu_actor.ping.remote())
# The actor uses the first TPU so the task uses the second one.
ray.get(tpu_task.remote())
.. testoutput::
:options: +MOCK
(TPUActor pid=52420) TPU IDs: [0]
(TPUActor pid=52420) TPU_VISIBLE_CHIPS: 0
(tpu_task pid=51830) TPU IDs: [1]
(tpu_task pid=51830) TPU_VISIBLE_CHIPS: 1
.. tab-item:: Intel Gaudi
:sync: Intel Gaudi
.. testcode::
:hide:
ray.shutdown()
.. testcode::
import os
import ray
ray.init(resources={"HPU": 2})
@ray.remote(resources={"HPU": 1})
class HPUActor:
def ping(self):
print("HPU IDs: {}".format(ray.get_runtime_context().get_accelerator_ids()["HPU"]))
print("HABANA_VISIBLE_MODULES: {}".format(os.environ["HABANA_VISIBLE_MODULES"]))
@ray.remote(resources={"HPU": 1})
def hpu_task():
print("HPU IDs: {}".format(ray.get_runtime_context().get_accelerator_ids()["HPU"]))
print("HABANA_VISIBLE_MODULES: {}".format(os.environ["HABANA_VISIBLE_MODULES"]))
hpu_actor = HPUActor.remote()
ray.get(hpu_actor.ping.remote())
# The actor uses the first HPU so the task uses the second one.
ray.get(hpu_task.remote())
.. testoutput::
:options: +MOCK
(HPUActor pid=52420) HPU IDs: [0]
(HPUActor pid=52420) HABANA_VISIBLE_MODULES: 0
(hpu_task pid=51830) HPU IDs: [1]
(hpu_task pid=51830) HABANA_VISIBLE_MODULES: 1
.. tab-item:: Huawei Ascend
:sync: Huawei Ascend
.. testcode::
:hide:
ray.shutdown()
.. testcode::
import os
import ray
ray.init(resources={"NPU": 2})
@ray.remote(resources={"NPU": 1})
class NPUActor:
def ping(self):
print("NPU IDs: {}".format(ray.get_runtime_context().get_accelerator_ids()["NPU"]))
print("ASCEND_RT_VISIBLE_DEVICES: {}".format(os.environ["ASCEND_RT_VISIBLE_DEVICES"]))
@ray.remote(resources={"NPU": 1})
def npu_task():
print("NPU IDs: {}".format(ray.get_runtime_context().get_accelerator_ids()["NPU"]))
print("ASCEND_RT_VISIBLE_DEVICES: {}".format(os.environ["ASCEND_RT_VISIBLE_DEVICES"]))
npu_actor = NPUActor.remote()
ray.get(npu_actor.ping.remote())
# The actor uses the first NPU so the task uses the second one.
ray.get(npu_task.remote())
.. testoutput::
:options: +MOCK
(NPUActor pid=52420) NPU IDs: [0]
(NPUActor pid=52420) ASCEND_RT_VISIBLE_DEVICES: 0
(npu_task pid=51830) NPU IDs: [1]
(npu_task pid=51830) ASCEND_RT_VISIBLE_DEVICES: 1
.. tab-item:: Rebellions RBLN
:sync: Rebellions RBLN
.. testcode::
:hide:
ray.shutdown()
.. testcode::
import os
import ray
ray.init(resources={"RBLN": 2})
@ray.remote(resources={"RBLN": 1})
class RBLNActor:
def ping(self):
print("RBLN IDs: {}".format(ray.get_runtime_context().get_accelerator_ids()["RBLN"]))
print("RBLN_DEVICES: {}".format(os.environ["RBLN_DEVICES"]))
@ray.remote(resources={"RBLN": 1})
def rbln_task():
print("RBLN IDs: {}".format(ray.get_runtime_context().get_accelerator_ids()["RBLN"]))
print("RBLN_DEVICES: {}".format(os.environ["RBLN_DEVICES"]))
rbln_actor = RBLNActor.remote()
ray.get(rbln_actor.ping.remote())
# The actor uses the first RBLN so the task uses the second one.
ray.get(rbln_task.remote())
.. testoutput::
:options: +MOCK
(RBLNActor pid=52420) RBLN IDs: [0]
(RBLNActor pid=52420) RBLN_DEVICES: 0
(rbln_task pid=51830) RBLN IDs: [1]
(rbln_task pid=51830) RBLN_DEVICES: 1
.. tab-item:: METAX GPU
:sync: METAX GPU
.. testcode::
:hide:
ray.shutdown()
.. testcode::
import os
import ray
ray.init(num_gpus=2)
@ray.remote(num_gpus=1)
class GPUActor:
def ping(self):
print("GPU IDs: {}".format(ray.get_runtime_context().get_accelerator_ids()["GPU"]))
print("CUDA_VISIBLE_DEVICES: {}".format(os.environ["CUDA_VISIBLE_DEVICES"]))
@ray.remote(num_gpus=1)
def gpu_task():
print("GPU IDs: {}".format(ray.get_runtime_context().get_accelerator_ids()["GPU"]))
print("CUDA_VISIBLE_DEVICES: {}".format(os.environ["CUDA_VISIBLE_DEVICES"]))
gpu_actor = GPUActor.remote()
ray.get(gpu_actor.ping.remote())
# The actor uses the first GPU so the task uses the second one.
ray.get(gpu_task.remote())
.. testoutput::
:options: +MOCK
(GPUActor pid=52420) GPU IDs: [0]
(GPUActor pid=52420) CUDA_VISIBLE_DEVICES: 0
(gpu_task pid=51830) GPU IDs: [1]
(gpu_task pid=51830) CUDA_VISIBLE_DEVICES: 1
.. tab-item:: FuriosaAI
:sync: FuriosaAI
.. testcode::
:hide:
ray.shutdown()
.. testcode::
import os
import ray
ray.init(resources={"FURIOSA": 2})
@ray.remote(resources={"FURIOSA": 1})
class RNGDActor:
def ping(self):
print("RNGD IDs: {}".format(ray.get_runtime_context().get_accelerator_ids()["FURIOSA"]))
print("FURIOSA_DEVICES: {}".format(os.environ["FURIOSA_DEVICES"]))
@ray.remote(resources={"FURIOSA": 1})
def rngd_task():
print("RNGD IDs: {}".format(ray.get_runtime_context().get_accelerator_ids()["FURIOSA"]))
print("FURIOSA_DEVICES: {}".format(os.environ["FURIOSA_DEVICES"]))
rngd_actor = RNGDActor.remote()
ray.get(rngd_actor.ping.remote())
# The actor uses the first RNGD so the task uses the second one.
ray.get(rngd_task.remote())
.. testoutput::
:options: +MOCK
(RNGDActor pid=52420) RNGD IDs: ['0']
(RNGDActor pid=52420) FURIOSA_DEVICES: npu:0
(rngd_task pid=51830) RNGD IDs: ['1']
(rngd_task pid=51830) FURIOSA_DEVICES: npu:1
Inside a task or actor, :func:`ray.get_runtime_context().get_accelerator_ids() <ray.runtime_context.RuntimeContext.get_accelerator_ids>` returns a
list of accelerator IDs that are available to the task or actor.
Typically, it is not necessary to call ``get_accelerator_ids()`` because Ray
automatically sets the corresponding environment variable (e.g. ``CUDA_VISIBLE_DEVICES``),
which most ML frameworks respect for purposes of accelerator assignment.
**Note:** The remote function or actor defined above doesn't actually use any
accelerators. Ray schedules it on a node which has at least one accelerator, and
reserves one accelerator for it while it is being executed, however it is up to the
function to actually make use of the accelerator. This is typically done through an
external library like TensorFlow. Here is an example that actually uses accelerators.
In order for this example to work, you need to install the GPU version of
TensorFlow.
.. testcode::
@ray.remote(num_gpus=1)
def gpu_task():
import tensorflow as tf
# Create a TensorFlow session. TensorFlow restricts itself to use the
# GPUs specified by the CUDA_VISIBLE_DEVICES environment variable.
tf.Session()
**Note:** It is certainly possible for the person to
ignore assigned accelerators and to use all of the accelerators on the machine. Ray does
not prevent this from happening, and this can lead to too many tasks or actors using the
same accelerator at the same time. However, Ray does automatically set the
environment variable (e.g. ``CUDA_VISIBLE_DEVICES``), which restricts the accelerators used
by most deep learning frameworks assuming it's not overridden by the user.
Fractional Accelerators
-----------------------
Ray supports :ref:`fractional resource requirements <fractional-resource-requirements>`
so multiple tasks and actors can share the same accelerator.
.. tab-set::
.. tab-item:: NVIDIA GPU
:sync: NVIDIA GPU
.. testcode::
:hide:
ray.shutdown()
.. testcode::
ray.init(num_cpus=4, num_gpus=1)
@ray.remote(num_gpus=0.25)
def f():
import time
time.sleep(1)
# The four tasks created here can execute concurrently
# and share the same GPU.
ray.get([f.remote() for _ in range(4)])
.. tab-item:: AMD GPU
:sync: AMD GPU
.. testcode::
:hide:
ray.shutdown()
.. testcode::
ray.init(num_cpus=4, num_gpus=1)
@ray.remote(num_gpus=0.25)
def f():
import time
time.sleep(1)
# The four tasks created here can execute concurrently
# and share the same GPU.
ray.get([f.remote() for _ in range(4)])
.. tab-item:: Intel GPU
:sync: Intel GPU
.. testcode::
:hide:
ray.shutdown()
.. testcode::
ray.init(num_cpus=4, num_gpus=1)
@ray.remote(num_gpus=0.25)
def f():
import time
time.sleep(1)
# The four tasks created here can execute concurrently
# and share the same GPU.
ray.get([f.remote() for _ in range(4)])
.. tab-item:: AWS Neuron Core
:sync: AWS Neuron Core
AWS Neuron Core doesn't support fractional resource.
.. tab-item:: Google TPU
:sync: Google TPU
Google TPU doesn't support fractional resource.
.. tab-item:: Intel Gaudi
:sync: Intel Gaudi
Intel Gaudi doesn't support fractional resource.
.. tab-item:: Huawei Ascend
:sync: Huawei Ascend
.. testcode::
:hide:
ray.shutdown()
.. testcode::
ray.init(num_cpus=4, resources={"NPU": 1})
@ray.remote(resources={"NPU": 0.25})
def f():
import time
time.sleep(1)
# The four tasks created here can execute concurrently
# and share the same NPU.
ray.get([f.remote() for _ in range(4)])
.. tab-item:: Rebellions RBLN
:sync: Rebellions RBLN
Rebellions RBLN doesn't support fractional resources.
.. tab-item:: METAX GPU
:sync: METAX GPU
.. testcode::
:hide:
ray.shutdown()
.. testcode::
ray.init(num_cpus=4, num_gpus=1)
@ray.remote(num_gpus=0.25)
def f():
import time
time.sleep(1)
# The four tasks created here can execute concurrently
# and share the same GPU.
ray.get([f.remote() for _ in range(4)])
.. tab-item:: FuriosaAI
:sync: FuriosaAI
FuriosaAI doesn't support fractional resources.
**Note:** It is the user's responsibility to make sure that the individual tasks
don't use more than their share of the accelerator memory.
Pytorch and TensorFlow can be configured to limit its memory usage.
When Ray assigns accelerators of a node to tasks or actors with fractional resource requirements,
it packs one accelerator before moving on to the next one to avoid fragmentation.
.. testcode::
:hide:
ray.shutdown()
.. testcode::
ray.init(num_gpus=3)
@ray.remote(num_gpus=0.5)
class FractionalGPUActor:
def ping(self):
print("GPU id: {}".format(ray.get_runtime_context().get_accelerator_ids()["GPU"]))
fractional_gpu_actors = [FractionalGPUActor.remote() for _ in range(3)]
# Ray tries to pack GPUs if possible.
[ray.get(fractional_gpu_actors[i].ping.remote()) for i in range(3)]
.. testoutput::
:options: +MOCK
(FractionalGPUActor pid=57417) GPU id: [0]
(FractionalGPUActor pid=57416) GPU id: [0]
(FractionalGPUActor pid=57418) GPU id: [1]
.. _gpu-leak:
Workers not Releasing GPU Resources
-----------------------------------
Currently, when a worker executes a task that uses a GPU (e.g.,
through TensorFlow), the task may allocate memory on the GPU and may not release
it when the task finishes executing. This can lead to problems the next time a
task tries to use the same GPU. To address the problem, Ray disables the worker
process reuse between GPU tasks by default, where the GPU resources is released after
the task process exits. Since this adds overhead to GPU task scheduling,
you can re-enable worker reuse by setting ``max_calls=0``
in the :func:`ray.remote <ray.remote>` decorator.
.. testcode::
# By default, ray does not reuse workers for GPU tasks to prevent
# GPU resource leakage.
@ray.remote(num_gpus=1, max_calls=0)
def leak_gpus():
import tensorflow as tf
# This task allocates memory on the GPU and then never release it.
tf.Session()
.. _accelerator-types:
Accelerator Types
-----------------
Ray supports resource specific accelerator types. The `accelerator_type` option can be used to force to a task or actor to run on a node with a specific type of accelerator.
Under the hood, the accelerator type option is implemented as a :ref:`custom resource requirement <custom-resources>` of ``"accelerator_type:<type>": 0.001``.
This forces the task or actor to be placed on a node with that particular accelerator type available.
This also lets the multi-node-type autoscaler know that there is demand for that type of resource, potentially triggering the launch of new nodes providing that accelerator.
.. testcode::
:hide:
ray.shutdown()
import ray.util.accelerators
v100_resource_name = f"accelerator_type:{ray.util.accelerators.NVIDIA_TESLA_V100}"
ray.init(num_gpus=4, resources={v100_resource_name: 1})
.. testcode::
from ray.util.accelerators import NVIDIA_TESLA_V100
@ray.remote(num_gpus=1, accelerator_type=NVIDIA_TESLA_V100)
def train(data):
return "This function was run on a node with a Tesla V100 GPU"
ray.get(train.remote(1))
See :ref:`ray.util.accelerators <accelerator_types>` for available accelerator types.
doc/source/ray-core/scheduling/index.rst
+152
View File
@@ -0,0 +1,152 @@
.. _ray-scheduling:
Scheduling
==========
This page provides an overview of how Ray decides to schedule tasks and actors to nodes.
.. DJS 19 Sept 2025: There should be an overview of all features and configs that impact scheduling here.
This should include descriptions for default values and behaviors, and links to things like default labels or resource definitions that can be used for scheduling without customization.
Labels
------
Labels provide a simplified solution for controlling scheduling for tasks, actors, and placement group bundles using default and custom labels. See :doc:`./labels`.
Labels are a beta feature. As this feature becomes stable, the Ray team recommends using labels to replace the following patterns:
- NodeAffinitySchedulingStrategy when `soft=false`. Use the default `ray.io/node-id` label instead.
- The `accelerator_type` option for tasks and actors. Use the default `ray.io/accelerator-type` label instead.
.. note::
A legacy pattern recommended using custom resources for label-based scheduling. We now recommend only using custom resources when you need to manage scheduling using numeric values.
.. _ray-scheduling-resources:
Resources
---------
Each task or actor has the :ref:`specified resource requirements <resource-requirements>`.
Given that, a node can be in one of the following states:
- Feasible: the node has the required resources to run the task or actor.
Depending on the current availability of these resources, there are two sub-states:
- Available: the node has the required resources and they are free now.
- Unavailable: the node has the required resources but they are currently being used by other tasks or actors.
- Infeasible: the node doesn't have the required resources. For example a CPU-only node is infeasible for a GPU task.
Resource requirements are **hard** requirements meaning that only feasible nodes are eligible to run the task or actor.
If there are feasible nodes, Ray will either choose an available node or wait until an unavailable node to become available
depending on other factors discussed below.
If all nodes are infeasible, the task or actor cannot be scheduled until feasible nodes are added to the cluster.
.. _ray-scheduling-strategies:
Scheduling Strategies
---------------------
Tasks or actors support a :func:`scheduling_strategy <ray.remote>` option to specify the strategy used to decide the best node among feasible nodes.
Currently the supported strategies are the followings.
"DEFAULT"
~~~~~~~~~
``"DEFAULT"`` is the default strategy used by Ray.
Ray schedules tasks or actors onto a group of the top k nodes.
Specifically, the nodes are sorted to first favor those that already have tasks or actors scheduled (for locality),
then to favor those that have low resource utilization (for load balancing).
Within the top k group, nodes are chosen randomly to further improve load-balancing and mitigate delays from cold-start in large clusters.
Implementation-wise, Ray calculates a score for each node in a cluster based on the utilization of its logical resources.
If the utilization is below a threshold (controlled by the OS environment variable ``RAY_scheduler_spread_threshold``, default is 0.5), the score is 0,
otherwise it is the resource utilization itself (score 1 means the node is fully utilized).
Ray selects the best node for scheduling by randomly picking from the top k nodes with the lowest scores.
The value of ``k`` is the max of (number of nodes in the cluster * ``RAY_scheduler_top_k_fraction`` environment variable) and ``RAY_scheduler_top_k_absolute`` environment variable.
By default, it's 20% of the total number of nodes.
Currently Ray handles actors that don't require any resources (i.e., ``num_cpus=0`` with no other resources) specially by randomly choosing a node in the cluster without considering resource utilization.
Since nodes are randomly chosen, actors that don't require any resources are effectively SPREAD across the cluster.
.. literalinclude:: ../doc_code/scheduling.py
:language: python
:start-after: __default_scheduling_strategy_start__
:end-before: __default_scheduling_strategy_end__
"SPREAD"
~~~~~~~~
``"SPREAD"`` strategy will try to spread the tasks or actors among available nodes.
.. literalinclude:: ../doc_code/scheduling.py
:language: python
:start-after: __spread_scheduling_strategy_start__
:end-before: __spread_scheduling_strategy_end__
PlacementGroupSchedulingStrategy
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
:py:class:`~ray.util.scheduling_strategies.PlacementGroupSchedulingStrategy` will schedule the task or actor to where the placement group is located.
This is useful for actor gang scheduling. See :ref:`Placement Group <ray-placement-group-doc-ref>` for more details.
NodeAffinitySchedulingStrategy
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
:py:class:`~ray.util.scheduling_strategies.NodeAffinitySchedulingStrategy` is a low-level strategy that allows a task or actor to be scheduled onto a particular node specified by its node id.
The ``soft`` flag specifies whether the task or actor is allowed to run somewhere else if the specified node doesn't exist (e.g. if the node dies)
or is infeasible because it does not have the resources required to run the task or actor.
In these cases, if ``soft`` is True, the task or actor will be scheduled onto a different feasible node.
Otherwise, the task or actor will fail with :py:class:`~ray.exceptions.TaskUnschedulableError` or :py:class:`~ray.exceptions.ActorUnschedulableError`.
As long as the specified node is alive and feasible, the task or actor will only run there
regardless of the ``soft`` flag. This means if the node currently has no available resources, the task or actor will wait until resources
become available.
This strategy should *only* be used if other high level scheduling strategies (e.g. :ref:`placement group <ray-placement-group-doc-ref>`) cannot give the
desired task or actor placements. It has the following known limitations:
- It's a low-level strategy which prevents optimizations by a smart scheduler.
- It cannot fully utilize an autoscaling cluster since node ids must be known when the tasks or actors are created.
- It can be difficult to make the best static placement decision
especially in a multi-tenant cluster: for example, an application won't know what else is being scheduled onto the same nodes.
.. literalinclude:: ../doc_code/scheduling.py
:language: python
:start-after: __node_affinity_scheduling_strategy_start__
:end-before: __node_affinity_scheduling_strategy_end__
.. _ray-scheduling-locality:
Locality-Aware Scheduling
-------------------------
By default, Ray prefers available nodes that have large task arguments local
to avoid transferring data over the network. If there are multiple large task arguments,
the node with most object bytes local is preferred.
This takes precedence over the ``"DEFAULT"`` scheduling strategy,
which means Ray will try to run the task on the locality preferred node regardless of the node resource utilization.
However, if the locality preferred node is not available, Ray may run the task somewhere else.
When other scheduling strategies are specified,
they have higher precedence and data locality is no longer considered.
.. note::
Locality-aware scheduling is only for tasks not actors.
.. literalinclude:: ../doc_code/scheduling.py
:language: python
:start-after: __locality_aware_scheduling_start__
:end-before: __locality_aware_scheduling_end__
More about Ray Scheduling
-------------------------
.. toctree::
:maxdepth: 1
labels
resources
accelerators
placement-group
memory-management
ray-oom-prevention
doc/source/ray-core/scheduling/labels.md
+223
View File
@@ -0,0 +1,223 @@
---
description: "Learn about using labels to control how Ray schedules tasks, actors, and placement groups to nodes in your Kubernetes cluster."
---
(labels)=
# Use labels to control scheduling
In Ray version 2.49.0 and above, you can use labels to control scheduling for KubeRay. Labels are a beta feature.
This page provides a conceptual overview and usage instructions for labels. Labels are key-value pairs that provide a human-readable configuration for users to control how Ray schedules tasks, actors, and placement group bundles to specific nodes.
```{note}
Ray labels share the same syntax and formatting restrictions as Kubernetes labels, but are conceptually distinct. See the [Kubernetes docs on labels and selectors](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set).
```
## How do labels work?
The following is a high-level overview of how you use labels to control scheduling:
- Ray sets default labels that describe the underlying compute. See [](defaults).
- You define custom labels as key-value pairs. See [](custom).
- You specify *label selectors* in your Ray code to define label requirements. You can specify these requirements at the task, actor, or placement group bundle level. See [](label-selectors).
- Ray schedules tasks, actors, or placement group bundles based on the specified label selectors.
- In Ray 2.50.0 and above, if you're using a dynamic cluster with autoscaler V2 enabled, the cluster scales up to add new nodes from a designated worker group to fulfill label requirements.
(defaults)=
## Default node labels
```{note}
Ray reserves all labels under ray.io namespace.
```
During cluster initialization or as autoscaling events add nodes to your cluster, Ray assigns the following default labels to each node:
| Label | Description |
| --- | --- |
| `ray.io/node-id` | A unique ID generated for the node. |
| `ray.io/accelerator-type` | The accelerator type of the node, for example `L4`. CPU-only machines don't have the label. See {ref}`accelerator types <accelerator-types>` for a mapping of values. |
```{note}
You can override default values using `ray start` parameters.
```
The following are examples of default labels:
```python
"ray.io/accelerator-type": "L4" # Default label indicating the machine has Nvidia L4 GPU
```
(custom)=
## Define custom labels
You can add custom labels to your nodes using the `--labels` or `--labels-file` parameter when running `ray start`.
```bash
# Examples 1: Start a head node with cpu-family and test-label labels
ray start --head --labels="cpu-family=amd,test-label=test-value"
# Example 2: Start a head node with labels from a label file
ray start --head --labels-files='./test-labels-file'
# The file content can be the following (should be a valid YAML file):
# "test-label": "test-value"
# "test-label-2": "test-value-2"
```
```{note}
You can use `ray.init(labels={"key": "value"})` to set labels for the node in a local Ray cluster to test labels locally.
```
(label-selectors)=
## Specify label selectors
You add label selector logic to your Ray code when defining Ray tasks, actors, or placement group bundles. Label selectors define the label requirements for matching your Ray code to a node in your Ray cluster.
Label selectors specify the following:
- The key of the label.
- Operator logic for matching.
- The value or values to match on.
The following table shows the basic syntax for label selector operator logic:
| Operator | Description | Example syntax |
| --- | --- | --- |
| Equals | Label matches exactly one value. | `{“key”: “value”}`
| Not equal | Label matches anything by one value. | `{“key”: “!value”}`
| In | Label matches one of the provided values. | `{“key”: “in(val1,val2)”}`
| Not in | Label matches none of the provided values. | `{“key”: “!in(val1,val2)”}`
You can specify one or more label selectors as a dict. When specifying multiple label selectors, the candidate node must meet all requirements. The following example configuration uses a custom label to require an `m5.16xlarge` EC2 instance and a default label to require node ID to be 123:
```python
label_selector={"instance_type": "m5.16xlarge", "ray.io/node-id": "123"}
```
## Specify label requirements for tasks and actors
Use the following syntax to add label selectors to tasks and actors:
```python
# An example for specifing label_selector in task's @ray.remote annotation
@ray.remote(label_selector={"label_name":"label_value"})
def f():
pass
# An example of specifying label_selector in actor's @ray.remote annotation
@ray.remote(label_selector={"ray.io/accelerator-type": "H100"})
class Actor:
pass
# An example of specifying label_selector in task's options
@ray.remote
def test_task_label_in_options():
pass
test_task_label_in_options.options(label_selector={"test-lable-key": "test-label-value"}).remote()
# An example of specifying label_selector in actor's options
@ray.remote
class Actor:
pass
actor_1 = Actor.options(
label_selector={"ray.io/accelerator-type": "H100"},
).remote()
```
## Specify label requirements for placement group bundles
Use the `bundle_label_selector` option to add label selector to placement group bundles. See the following examples:
```python
# All bundles require the same labels:
ray.util.placement_group(
bundles=[{"GPU": 1}, {"GPU": 1}],
bundle_label_selector=[{"ray.io/accelerator-type": "H100"}] * 2,
)
# Bundles require different labels:
ray.util.placement_group(
bundles=[{"CPU": 1}] + [{"GPU": 1}] * 2,
bundle_label_selector=[{"ray.io/market-type": "spot"}] + [{"ray.io/accelerator-type": "H100"}] * 2
)
```
## Specify label requirements for a local RayCluster
The following test script showcases how users can use `ray.init(labels={"key": "value"})` to test labels locally.
```python
import ray
# 1. Initialize Ray with a specific label for the local node
print("Initializing Ray with labels={'env': 'dev', 'hardware': 'mock_gpu'}...")
ray.init(labels={"env": "dev", "hardware": "mock_gpu"})
@ray.remote
def get_labels():
# Verify the labels are visible in the runtime context
return ray.get_runtime_context().get_node_labels()
@ray.remote(label_selector={"hardware": "mock_gpu"})
def matching_task():
return "Successfully scheduled on mock_gpu!"
@ray.remote(label_selector={"hardware": "real_h100"})
def non_matching_task():
return "This should not run!"
try:
# Test 1: Verify node labels are set correctly
current_labels = ray.get(get_labels.remote())
print(f"Node labels detected: {current_labels}")
assert current_labels.get("hardware") == "mock_gpu"
# Test 2: Run a task with a matching label selector
print("Running task with matching label selector...")
result = ray.get(matching_task.remote())
print(f"Result: {result}")
# Test 3: Attempt to run a task with a non-matching label selector
print("Running task with non-matching label selector (expecting it to stay pending)...")
ref = non_matching_task.remote()
ready, not_ready = ray.wait([ref], timeout=5)
if not ready and not_ready:
print("Success: Task with non-matching label stayed pending as expected.")
finally:
ray.shutdown()
```
## Using labels with autoscaler
Autoscaler V2 supports label-based scheduling. To enable autoscaler to scale up nodes to fulfill label requirements, you need to create multiple worker groups for different label requirement combinations and specify all the corresponding labels in the `rayStartParams` field in the Ray cluster configuration. For example:
```python
rayStartParams: {
labels: "region=me-central1,ray.io/accelerator-type=H100"
}
```
## Monitor nodes using labels
The Ray dashboard automatically shows the following information:
- Labels for each node. See {py:attr}`ray.util.state.common.NodeState.labels`.
- Label selectors set for each task, actor, or placement group bundle. See {py:attr}`ray.util.state.common.TaskState.label_selector` and {py:attr}`ray.util.state.common.ActorState.label_selector`.
Within a task, you can programmatically obtain the node label from the RuntimeContextAPI using `ray.get_runtime_context().get_node_labels()`. This returns a Python dict. See the following example:
```python
@ray.remote
def test_task_label():
node_labels = ray.get_runtime_context().get_node_labels()
print(f"[test_task_label] node labels: {node_labels}")
"""
Example output:
(test_task_label pid=68487) [test_task_label] node labels: {'test-label-1': 'test-value-1', 'test-label-key': 'test-label-value', 'test-label-2': 'test-value-2'}
"""
```
You can also access information about node label and label selector information using the state API and state CLI.
doc/source/ray-core/scheduling/memory-management.rst
+280
View File
@@ -0,0 +1,280 @@
.. _memory:
Memory Management
=================
This page describes how memory management works in Ray.
Also view :ref:`Debugging Out of Memory <troubleshooting-out-of-memory>` to learn how to troubleshoot out-of-memory issues.
Concepts
~~~~~~~~
There are several ways that Ray applications use memory:
..
https://docs.google.com/drawings/d/1wHHnAJZ-NsyIv3TUXQJTYpPz6pjB6PUm2M40Zbfb1Ak/edit
.. image:: ../images/memory.svg
Ray system memory: this is memory used internally by Ray
- **GCS**: memory used for storing the list of nodes and actors present in the cluster. The amount of memory used for these purposes is typically quite small.
- **Raylet**: memory used by the C++ raylet process running on each node. This cannot be controlled, but is typically quite small.
Application memory: this is memory used by your application
- **Worker heap**: memory used by your application (e.g., in Python code or TensorFlow), best measured as the *resident set size (RSS)* of your application minus its *shared memory usage (SHR)* in commands such as ``top``. The reason you need to subtract *SHR* is that object store shared memory is reported by the OS as shared with each worker. Not subtracting *SHR* will result in double counting memory usage.
- **Object store memory**: memory used when your application creates objects in the object store via ``ray.put`` and when it returns values from remote functions. Objects are reference counted and evicted when they fall out of scope. An object store server runs on each node. By default, when starting an instance, Ray reserves 30% of available memory. The size of the object store can be controlled by `--object-store-memory <https://docs.ray.io/en/master/cluster/cli.html#cmdoption-ray-start-object-store-memory>`_. The memory is by default allocated to ``/dev/shm`` (shared memory) for Linux. For MacOS, Ray uses ``/tmp`` (disk), which can impact the performance compared to Linux. In Ray 1.3+, objects are :ref:`spilled to disk <object-spilling>` if the object store fills up.
- **Object store shared memory**: memory used when your application reads objects via ``ray.get``. Note that if an object is already present on the node, this does not cause additional allocations. This allows large objects to be efficiently shared among many actors and tasks.
ObjectRef Reference Counting
----------------------------
Ray implements distributed reference counting so that any ``ObjectRef`` in scope in the cluster is pinned in the object store. This includes local python references, arguments to pending tasks, and IDs serialized inside of other objects.
.. _debug-with-ray-memory:
Debugging using 'ray memory'
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``ray memory`` command can be used to help track down what ``ObjectRef`` references are in scope and may be causing an ``ObjectStoreFullError``.
Running ``ray memory`` from the command line while a Ray application is running will give you a dump of all of the ``ObjectRef`` references that are currently held by the driver, actors, and tasks in the cluster.
.. code-block:: text
======== Object references status: 2021-02-23 22:02:22.072221 ========
Grouping by node address... Sorting by object size...
--- Summary for node address: 192.168.0.15 ---
Mem Used by Objects Local References Pinned Count Pending Tasks Captured in Objects Actor Handles
287 MiB 4 0 0 1 0
--- Object references for node address: 192.168.0.15 ---
IP Address PID Type Object Ref Size Reference Type Call Site
192.168.0.15 6465 Driver ffffffffffffffffffffffffffffffffffffffff0100000001000000 15 MiB LOCAL_REFERENCE (put object)
| test.py:
<module>:17
192.168.0.15 6465 Driver a67dc375e60ddd1affffffffffffffffffffffff0100000001000000 15 MiB LOCAL_REFERENCE (task call)
| test.py:
:<module>:18
192.168.0.15 6465 Driver ffffffffffffffffffffffffffffffffffffffff0100000002000000 18 MiB CAPTURED_IN_OBJECT (put object) |
test.py:
<module>:19
192.168.0.15 6465 Driver ffffffffffffffffffffffffffffffffffffffff0100000004000000 21 MiB LOCAL_REFERENCE (put object) |
test.py:
<module>:20
192.168.0.15 6465 Driver ffffffffffffffffffffffffffffffffffffffff0100000003000000 218 MiB LOCAL_REFERENCE (put object) |
test.py:
<module>:20
--- Aggregate object store stats across all nodes ---
Plasma memory usage 0 MiB, 4 objects, 0.0% full
Each entry in this output corresponds to an ``ObjectRef`` that's currently pinning an object in the object store along with where the reference is (in the driver, in a worker, etc.), what type of reference it is (see below for details on the types of references), the size of the object in bytes, the process ID and IP address where the object was instantiated, and where in the application the reference was created.
``ray memory`` comes with features to make the memory debugging experience more effective. For example, you can add arguments ``sort-by=OBJECT_SIZE`` and ``group-by=STACK_TRACE``, which may be particularly helpful for tracking down the line of code where a memory leak occurs. You can see the full suite of options by running ``ray memory --help``.
There are five types of references that can keep an object pinned:
**1. Local ObjectRef references**
.. testcode::
import ray
@ray.remote
def f(arg):
return arg
a = ray.put(None)
b = f.remote(None)
In this example, we create references to two objects: one that is ``ray.put()`` in the object store and another that's the return value from ``f.remote()``.
.. code-block:: text
--- Summary for node address: 192.168.0.15 ---
Mem Used by Objects Local References Pinned Count Pending Tasks Captured in Objects Actor Handles
30 MiB 2 0 0 0 0
--- Object references for node address: 192.168.0.15 ---
IP Address PID Type Object Ref Size Reference Type Call Site
192.168.0.15 6867 Driver ffffffffffffffffffffffffffffffffffffffff0100000001000000 15 MiB LOCAL_REFERENCE (put object) |
test.py:
<module>:12
192.168.0.15 6867 Driver a67dc375e60ddd1affffffffffffffffffffffff0100000001000000 15 MiB LOCAL_REFERENCE (task call)
| test.py:
:<module>:13
In the output from ``ray memory``, we can see that each of these is marked as a ``LOCAL_REFERENCE`` in the driver process, but the annotation in the "Reference Creation Site" indicates that the first was created as a "put object" and the second from a "task call."
**2. Objects pinned in memory**
.. testcode::
import numpy as np
a = ray.put(np.zeros(1))
b = ray.get(a)
del a
In this example, we create a ``numpy`` array and then store it in the object store. Then, we fetch the same numpy array from the object store and delete its ``ObjectRef``. In this case, the object is still pinned in the object store because the deserialized copy (stored in ``b``) points directly to the memory in the object store.
.. code-block:: text
--- Summary for node address: 192.168.0.15 ---
Mem Used by Objects Local References Pinned Count Pending Tasks Captured in Objects Actor Handles
243 MiB 0 1 0 0 0
--- Object references for node address: 192.168.0.15 ---
IP Address PID Type Object Ref Size Reference Type Call Site
192.168.0.15 7066 Driver ffffffffffffffffffffffffffffffffffffffff0100000001000000 243 MiB PINNED_IN_MEMORY test.
py:<module>:19
The output from ``ray memory`` displays this as the object being ``PINNED_IN_MEMORY``. If we ``del b``, the reference can be freed.
**3. Pending task references**
.. testcode::
@ray.remote
def f(arg):
while True:
pass
a = ray.put(None)
b = f.remote(a)
In this example, we first create an object via ``ray.put()`` and then submit a task that depends on the object.
.. code-block:: text
--- Summary for node address: 192.168.0.15 ---
Mem Used by Objects Local References Pinned Count Pending Tasks Captured in Objects Actor Handles
25 MiB 1 1 1 0 0
--- Object references for node address: 192.168.0.15 ---
IP Address PID Type Object Ref Size Reference Type Call Site
192.168.0.15 7207 Driver a67dc375e60ddd1affffffffffffffffffffffff0100000001000000 ? LOCAL_REFERENCE (task call)
| test.py:
:<module>:29
192.168.0.15 7241 Worker ffffffffffffffffffffffffffffffffffffffff0100000001000000 10 MiB PINNED_IN_MEMORY (deserialize task arg)
__main__.f
192.168.0.15 7207 Driver ffffffffffffffffffffffffffffffffffffffff0100000001000000 15 MiB USED_BY_PENDING_TASK (put object) |
test.py:
<module>:28
While the task is running, we see that ``ray memory`` shows both a ``LOCAL_REFERENCE`` and a ``USED_BY_PENDING_TASK`` reference for the object in the driver process. The worker process also holds a reference to the object because the Python ``arg`` is directly referencing the memory in the plasma, so it can't be evicted; therefore it is ``PINNED_IN_MEMORY``.
**4. Serialized ObjectRef references**
.. testcode::
@ray.remote
def f(arg):
while True:
pass
a = ray.put(None)
b = f.remote([a])
In this example, we again create an object via ``ray.put()``, but then pass it to a task wrapped in another object (in this case, a list).
.. code-block:: text
--- Summary for node address: 192.168.0.15 ---
Mem Used by Objects Local References Pinned Count Pending Tasks Captured in Objects Actor Handles
15 MiB 2 0 1 0 0
--- Object references for node address: 192.168.0.15 ---
IP Address PID Type Object Ref Size Reference Type Call Site
192.168.0.15 7411 Worker ffffffffffffffffffffffffffffffffffffffff0100000001000000 ? LOCAL_REFERENCE (deserialize task arg)
__main__.f
192.168.0.15 7373 Driver a67dc375e60ddd1affffffffffffffffffffffff0100000001000000 ? LOCAL_REFERENCE (task call)
| test.py:
:<module>:38
192.168.0.15 7373 Driver ffffffffffffffffffffffffffffffffffffffff0100000001000000 15 MiB USED_BY_PENDING_TASK (put object)
| test.py:
<module>:37
Now, both the driver and the worker process running the task hold a ``LOCAL_REFERENCE`` to the object in addition to it being ``USED_BY_PENDING_TASK`` on the driver. If this was an actor task, the actor could even hold a ``LOCAL_REFERENCE`` after the task completes by storing the ``ObjectRef`` in a member variable.
**5. Captured ObjectRef references**
.. testcode::
a = ray.put(None)
b = ray.put([a])
del a
In this example, we first create an object via ``ray.put()``, then capture its ``ObjectRef`` inside of another ``ray.put()`` object, and delete the first ``ObjectRef``. In this case, both objects are still pinned.
.. code-block:: text
--- Summary for node address: 192.168.0.15 ---
Mem Used by Objects Local References Pinned Count Pending Tasks Captured in Objects Actor Handles
233 MiB 1 0 0 1 0
--- Object references for node address: 192.168.0.15 ---
IP Address PID Type Object Ref Size Reference Type Call Site
192.168.0.15 7473 Driver ffffffffffffffffffffffffffffffffffffffff0100000001000000 15 MiB CAPTURED_IN_OBJECT (put object) |
test.py:
<module>:41
192.168.0.15 7473 Driver ffffffffffffffffffffffffffffffffffffffff0100000002000000 218 MiB LOCAL_REFERENCE (put object) |
test.py:
<module>:42
In the output of ``ray memory``, we see that the second object displays as a normal ``LOCAL_REFERENCE``, but the first object is listed as ``CAPTURED_IN_OBJECT``.
.. _memory-aware-scheduling:
Memory Aware Scheduling
~~~~~~~~~~~~~~~~~~~~~~~
By default, Ray does not take into account the potential memory usage of a task or actor when scheduling. This is simply because it cannot estimate ahead of time how much memory is required. However, if you know how much memory a task or actor requires, you can specify it in the resource requirements of its ``ray.remote`` decorator to enable memory-aware scheduling:
.. important::
Specifying a memory requirement does NOT impose any limits on memory usage. The requirements are used for admission control during scheduling only (similar to how CPU scheduling works in Ray). It is up to the task itself to not use more memory than it requested.
To tell the Ray scheduler a task or actor requires a certain amount of available memory to run, set the ``memory`` argument. The Ray scheduler will then reserve the specified amount of available memory during scheduling, similar to how it handles CPU and GPU resources:
.. testcode::
# reserve 500MiB of available memory to place this task
@ray.remote(memory=500 * 1024 * 1024)
def some_function(x):
pass
# reserve 2.5GiB of available memory to place this actor
@ray.remote(memory=2500 * 1024 * 1024)
class SomeActor:
def __init__(self, a, b):
pass
In the above example, the memory quota is specified statically by the decorator, but you can also set them dynamically at runtime using ``.options()`` as follows:
.. testcode::
# override the memory quota to 100MiB when submitting the task
some_function.options(memory=100 * 1024 * 1024).remote(x=1)
# override the memory quota to 1GiB when creating the actor
SomeActor.options(memory=1000 * 1024 * 1024).remote(a=1, b=2)
Questions or Issues?
--------------------
.. include:: /_includes/_help.rst
doc/source/ray-core/scheduling/placement-group.rst
+899
View File
@@ -0,0 +1,899 @@
Placement Groups
================
.. _ray-placement-group-doc-ref:
Placement groups allow users to atomically reserve groups of resources across multiple nodes, a concept commonly known as gang scheduling.
After atomically reserving resources, you can use placement groups to schedule Ray tasks and actors packed together for locality (PACK), or spread apart (SPREAD).
Placement groups are generally used for gang-scheduling actors, but also support tasks.
Here are some real-world use cases:
- **Distributed Machine Learning Training**: Distributed Training (e.g., :ref:`Ray Train <train-docs>` and :ref:`Ray Tune <tune-main>`) uses the placement group APIs to enable gang scheduling. In these settings, all resources for a trial must be available at the same time. Gang scheduling is a critical technique to enable all-or-nothing scheduling for deep learning training.
- **Fault tolerance in distributed training**: Placement groups can be used to configure fault tolerance. In Ray Tune, it can be beneficial to pack related resources from a single trial together, so that a node failure impacts a low number of trials. In libraries that support elastic training (e.g., XGBoost-Ray), spreading the resources across multiple nodes can help to ensure that training continues even when a node dies.
Key Concepts
------------
Bundles
~~~~~~~
A **bundle** is a collection of "resources." It could be a single resource, ``{"CPU": 1}``, or a group of resources, ``{"CPU": 1, "GPU": 4}``.
A bundle is a unit of reservation for placement groups. "Scheduling a bundle" means we find a node that fits the bundle and reserve the resources specified by the bundle.
A bundle must be able to fit on a single node on the Ray cluster. For example, if you have an 8 CPU node and a 1 CPU node and want to schedule a bundle that requires ``{"CPU": 9}``,
Ray can't schedule the ``{"CPU": 9}``, because there's no single node with 9 CPU's.
Placement Group
~~~~~~~~~~~~~~~
A **placement group** reserves the resources from the cluster. Tasks or actors must use the :ref:`PlacementGroupSchedulingStrategy <ray-placement-group-schedule-tasks-actors-ref>` to use the reserved resources.
- Ray represents placement groups with a list of bundles. For example, ``{"CPU": 1} * 4`` means you'd like to reserve 4 bundles, each with 1 CPU.
- Bundles are then placed according to the :ref:`placement strategies <pgroup-strategy>` across nodes on the cluster.
- After Ray creates the placement group, tasks or actors can be then scheduled according to the placement group and even on individual bundles.
Create a Placement Group (Reserve Resources)
--------------------------------------------
You can create a placement group using :func:`ray.util.placement_group`.
Placement groups take in a list of bundles and a :ref:`placement strategy <pgroup-strategy>`.
Bundles are specified by a list of dictionaries, e.g., ``[{"CPU": 1}, {"CPU": 1, "GPU": 1}]``).
- ``CPU`` corresponds to ``num_cpus`` as used in :func:`ray.remote <ray.remote>`.
- ``GPU`` corresponds to ``num_gpus`` as used in :func:`ray.remote <ray.remote>`.
- ``memory`` corresponds to ``memory`` as used in :func:`ray.remote <ray.remote>`
- Other resources corresponds to ``resources`` as used in :func:`ray.remote <ray.remote>` (E.g., ``ray.init(resources={"disk": 1})`` can have a bundle of ``{"disk": 1}``).
Placement group scheduling is asynchronous. The `ray.util.placement_group` returns immediately.
.. tab-set::
.. tab-item:: Python
.. literalinclude:: ../doc_code/placement_group_example.py
:language: python
:start-after: __create_pg_start__
:end-before: __create_pg_end__
.. tab-item:: Java
.. code-block:: java
// Initialize Ray.
Ray.init();
// Construct a list of bundles.
Map<String, Double> bundle = ImmutableMap.of("CPU", 1.0);
List<Map<String, Double>> bundles = ImmutableList.of(bundle);
// Make a creation option with bundles and strategy.
PlacementGroupCreationOptions options =
new PlacementGroupCreationOptions.Builder()
.setBundles(bundles)
.setStrategy(PlacementStrategy.STRICT_SPREAD)
.build();
PlacementGroup pg = PlacementGroups.createPlacementGroup(options);
.. tab-item:: C++
.. code-block:: c++
// Initialize Ray.
ray::Init();
// Construct a list of bundles.
std::vector<std::unordered_map<std::string, double>> bundles{{{"CPU", 1.0}}};
// Make a creation option with bundles and strategy.
ray::internal::PlacementGroupCreationOptions options{
false, "my_pg", bundles, ray::internal::PlacementStrategy::PACK};
ray::PlacementGroup pg = ray::CreatePlacementGroup(options);
You can block your program until the placement group is ready using one of two APIs:
* :func:`ready <ray.util.placement_group.PlacementGroup.ready>`, which is compatible with ``ray.get``
* :func:`wait <ray.util.placement_group.PlacementGroup.wait>`, which blocks the program until the placement group is ready)
.. tab-set::
.. tab-item:: Python
.. literalinclude:: ../doc_code/placement_group_example.py
:language: python
:start-after: __ready_pg_start__
:end-before: __ready_pg_end__
.. tab-item:: Java
.. code-block:: java
// Wait for the placement group to be ready within the specified time(unit is seconds).
boolean ready = pg.wait(60);
Assert.assertTrue(ready);
// You can look at placement group states using this API.
List<PlacementGroup> allPlacementGroup = PlacementGroups.getAllPlacementGroups();
for (PlacementGroup group: allPlacementGroup) {
System.out.println(group);
}
.. tab-item:: C++
.. code-block:: c++
// Wait for the placement group to be ready within the specified time(unit is seconds).
bool ready = pg.Wait(60);
assert(ready);
// You can look at placement group states using this API.
std::vector<ray::PlacementGroup> all_placement_group = ray::GetAllPlacementGroups();
for (const ray::PlacementGroup &group : all_placement_group) {
std::cout << group.GetName() << std::endl;
}
Let's verify the placement group is successfully created.
.. code-block:: bash
# This API is only available when you download Ray via `pip install "ray[default]"`
ray list placement-groups
.. code-block:: bash
======== List: 2023-04-07 01:15:05.682519 ========
Stats:
------------------------------
Total: 1
Table:
------------------------------
PLACEMENT_GROUP_ID NAME CREATOR_JOB_ID STATE
0 3cd6174711f47c14132155039c0501000000 01000000 CREATED
The placement group is successfully created. Out of the ``{"CPU": 2, "GPU": 2}`` resources, the placement group reserves ``{"CPU": 1, "GPU": 1}``.
The reserved resources can only be used when you schedule tasks or actors with a placement group.
The diagram below demonstrates the "1 CPU and 1 GPU" bundle that the placement group reserved.
.. image:: ../images/pg_image_1.png
:align: center
Ray creates placement groups atomically. If a bundle can't fit in any of the current nodes, Ray reserves no resources for the placement group.
To illustrate this, you can create another placement group with these two bundles ``{"CPU":1}, {"GPU": 2}``.
.. tab-set::
.. tab-item:: Python
.. literalinclude:: ../doc_code/placement_group_example.py
:language: python
:start-after: __create_pg_failed_start__
:end-before: __create_pg_failed_end__
You can verify the new placement group is pending creation.
.. code-block:: bash
# This API is only available when you download Ray via `pip install "ray[default]"`
ray list placement-groups
.. code-block:: bash
======== List: 2023-04-07 01:16:23.733410 ========
Stats:
------------------------------
Total: 2
Table:
------------------------------
PLACEMENT_GROUP_ID NAME CREATOR_JOB_ID STATE
0 3cd6174711f47c14132155039c0501000000 01000000 CREATED
1 e1b043bebc751c3081bddc24834d01000000 01000000 PENDING <---- the new placement group.
You can also verify that the ``{"CPU": 1, "GPU": 2}`` bundles can't be allocated, using the ``ray status`` CLI command.
.. code-block:: bash
ray status
.. code-block:: bash
Resources
---------------------------------------------------------------
Usage:
0.0/2.0 CPU (0.0 used of 1.0 reserved in placement groups)
0.0/2.0 GPU (0.0 used of 1.0 reserved in placement groups)
0B/3.46GiB memory
0B/1.73GiB object_store_memory
Demands:
{'CPU': 1.0} * 1, {'GPU': 2.0} * 1 (PACK): 1+ pending placement groups <--- 1 placement group is pending creation.
The current cluster has ``{"CPU": 2, "GPU": 2}``. We already created a ``{"CPU": 1, "GPU": 1}`` bundle, so the cluster only has 1 CPU and 1 GPU left.
If you try to schedule a placement group with these 2 bundles ``{"CPU": 1}, {"GPU": 2}``, Ray doesn't create the placement group and doesn't reserve any resources,
including the ``{"CPU": 1}`` bundle.
.. image:: ../images/pg_image_2.png
:align: center
When the placement group cannot be scheduled in any way, it is called "infeasible".
Imagine you schedule ``{"CPU": 4}`` bundle, but you only have a single node with 2 CPUs. There's no way to create this bundle in your cluster.
The Ray Autoscaler is aware of placement groups, and auto-scales the cluster to ensure pending groups can be placed as needed.
If the Autoscaler can't provide resources to schedule a placement group, Ray does *not* print a warning about infeasible groups and tasks and actors that use the groups.
You can observe the scheduling state of the placement group from the :ref:`dashboard or state APIs <ray-placement-group-observability-ref>`.
.. note::
When a placement group with GPUs is reserved successfully, the bundles are not necessarily ordered by GPU physical rank.
That is, adjacent bundles don't necessarily map to adjacent physical GPUs.
.. _ray-placement-group-schedule-tasks-actors-ref:
Schedule Tasks and Actors to Placement Groups (Use Reserved Resources)
----------------------------------------------------------------------
In the previous section, we created a placement group that reserved ``{"CPU": 1, "GPU: 1"}`` from a 2 CPU and 2 GPU node.
Now let's schedule an actor to the placement group.
You can schedule actors or tasks to a placement group using
:class:`options(scheduling_strategy=PlacementGroupSchedulingStrategy(...)) <ray.util.scheduling_strategies.PlacementGroupSchedulingStrategy>`.
.. tab-set::
.. tab-item:: Python
.. literalinclude:: ../doc_code/placement_group_example.py
:language: python
:start-after: __schedule_pg_start__
:end-before: __schedule_pg_end__
.. tab-item:: Java
.. code-block:: java
public static class Counter {
private int value;
public Counter(int initValue) {
this.value = initValue;
}
public int getValue() {
return value;
}
public static String ping() {
return "pong";
}
}
// Create GPU actors on a gpu bundle.
for (int index = 0; index < 1; index++) {
Ray.actor(Counter::new, 1)
.setPlacementGroup(pg, 0)
.remote();
}
.. tab-item:: C++
.. code-block:: c++
class Counter {
public:
Counter(int init_value) : value(init_value){}
int GetValue() {return value;}
std::string Ping() {
return "pong";
}
private:
int value;
};
// Factory function of Counter class.
static Counter *CreateCounter() {
return new Counter();
};
RAY_REMOTE(&Counter::Ping, &Counter::GetValue, CreateCounter);
// Create GPU actors on a gpu bundle.
for (int index = 0; index < 1; index++) {
ray::Actor(CreateCounter)
.SetPlacementGroup(pg, 0)
.Remote(1);
}
.. note::
By default, Ray actors require 1 logical CPU at schedule time, but after being scheduled, they do not acquire any CPU resources.
In other words, by default, actors cannot get scheduled on a zero-cpu node, but an infinite number of them can run on any non-zero cpu node.
Thus, when scheduling an actor with the default resource requirements and a placement group, the placement group has to be created with a bundle containing at least 1 CPU
(since the actor requires 1 CPU for scheduling). However, after the actor is created, it doesn't consume any placement group resources.
To avoid any surprises, always specify resource requirements explicitly for actors. If resources are specified explicitly, they are required both at schedule time and at execution time.
The actor is scheduled now! One bundle can be used by multiple tasks and actors (i.e., the bundle to task (or actor) is a one-to-many relationship).
In this case, since the actor uses 1 CPU, 1 GPU remains from the bundle.
You can verify this from the CLI command ``ray status``. You can see the 1 CPU is reserved by the placement group, and 1.0 is used (by the actor we created).
.. code-block:: bash
ray status
.. code-block:: bash
Resources
---------------------------------------------------------------
Usage:
1.0/2.0 CPU (1.0 used of 1.0 reserved in placement groups) <---
0.0/2.0 GPU (0.0 used of 1.0 reserved in placement groups)
0B/4.29GiB memory
0B/2.00GiB object_store_memory
Demands:
(no resource demands)
You can also verify the actor is created using ``ray list actors``.
.. code-block:: bash
# This API is only available when you download Ray via `pip install "ray[default]"`
ray list actors --detail
.. code-block:: bash
- actor_id: b5c990f135a7b32bfbb05e1701000000
class_name: Actor
death_cause: null
is_detached: false
job_id: '01000000'
name: ''
node_id: b552ca3009081c9de857a31e529d248ba051a4d3aeece7135dde8427
pid: 8795
placement_group_id: d2e660ac256db230dbe516127c4a01000000 <------
ray_namespace: e5b19111-306c-4cd8-9e4f-4b13d42dff86
repr_name: ''
required_resources:
CPU_group_d2e660ac256db230dbe516127c4a01000000: 1.0
serialized_runtime_env: '{}'
state: ALIVE
Since 1 GPU remains, let's create a new actor that requires 1 GPU.
This time, we also specify the ``placement_group_bundle_index``. Each bundle is given an "index" within the placement group.
For example, a placement group of 2 bundles ``[{"CPU": 1}, {"GPU": 1}]`` has index 0 bundle ``{"CPU": 1}``
and index 1 bundle ``{"GPU": 1}``. Since we only have 1 bundle, we only have index 0. If you don't specify a bundle, the actor (or task)
is scheduled on a random bundle that has unallocated reserved resources.
.. tab-set::
.. tab-item:: Python
.. literalinclude:: ../doc_code/placement_group_example.py
:language: python
:start-after: __schedule_pg_3_start__
:end-before: __schedule_pg_3_end__
We succeed to schedule the GPU actor! The below image describes 2 actors scheduled into the placement group.
.. image:: ../images/pg_image_3.png
:align: center
You can also verify that the reserved resources are all used, with the ``ray status`` command.
.. code-block:: bash
ray status
.. code-block:: bash
Resources
---------------------------------------------------------------
Usage:
1.0/2.0 CPU (1.0 used of 1.0 reserved in placement groups)
1.0/2.0 GPU (1.0 used of 1.0 reserved in placement groups) <----
0B/4.29GiB memory
0B/2.00GiB object_store_memory
.. _pgroup-strategy:
Placement Strategy
------------------
One of the features the placement group provides is to add placement constraints among bundles.
For example, you'd like to pack your bundles to the same
node or spread out to multiple nodes as much as possible. You can specify the strategy via ``strategy`` argument.
This way, you can make sure your actors and tasks can be scheduled with certain placement constraints.
The example below creates a placement group with 2 bundles with a PACK strategy;
both bundles have to be created in the same node. Note that it is a soft policy. If the bundles cannot be packed
into a single node, they are spread to other nodes. If you'd like to avoid the problem, you can instead use `STRICT_PACK`
policies, which fail to create placement groups if placement requirements cannot be satisfied.
.. literalinclude:: ../doc_code/placement_group_example.py
:language: python
:start-after: __strategy_pg_start__
:end-before: __strategy_pg_end__
The image below demonstrates the PACK policy. Three of the ``{"CPU": 2}`` bundles are located in the same node.
.. image:: ../images/pg_image_4.png
:align: center
The image below demonstrates the SPREAD policy. Each of three of the ``{"CPU": 2}`` bundles are located in three different nodes.
.. image:: ../images/pg_image_5.png
:align: center
Ray supports four placement group strategies. The default scheduling policy is ``PACK``.
**STRICT_PACK**
All bundles must be placed into a single node on the cluster. Use this strategy when you want to maximize the locality.
**PACK**
All provided bundles are packed onto a single node on a best-effort basis.
If strict packing is not feasible (i.e., some bundles do not fit on the node), bundles can be placed onto other nodes.
**STRICT_SPREAD**
Each bundle must be scheduled in a separate node.
**SPREAD**
Each bundle is spread onto separate nodes on a best-effort basis.
If strict spreading is not feasible, bundles can be placed on overlapping nodes.
Remove Placement Groups (Free Reserved Resources)
-------------------------------------------------
By default, a placement group's lifetime is scoped to the driver that creates placement groups
(unless you make it a :ref:`detached placement group <placement-group-detached>`). When the placement group is created from
a :ref:`detached actor <actor-lifetimes>`, the lifetime is scoped to the detached actor.
In Ray, the driver is the Python script that calls ``ray.init``.
Reserved resources (bundles) from the placement group are automatically freed when the driver or detached actor
that creates placement group exits. To free the reserved resources manually, remove the placement
group using the :func:`remove_placement_group <ray.util.remove_placement_group>` API (which is also an asynchronous API).
.. note::
When you remove the placement group, actors or tasks that still use the reserved resources are
forcefully killed.
.. tab-set::
.. tab-item:: Python
.. literalinclude:: ../doc_code/placement_group_example.py
:language: python
:start-after: __remove_pg_start__
:end-before: __remove_pg_end__
.. tab-item:: Java
.. code-block:: java
PlacementGroups.removePlacementGroup(placementGroup.getId());
PlacementGroup removedPlacementGroup = PlacementGroups.getPlacementGroup(placementGroup.getId());
Assert.assertEquals(removedPlacementGroup.getState(), PlacementGroupState.REMOVED);
.. tab-item:: C++
.. code-block:: c++
ray::RemovePlacementGroup(placement_group.GetID());
ray::PlacementGroup removed_placement_group = ray::GetPlacementGroup(placement_group.GetID());
assert(removed_placement_group.GetState(), ray::PlacementGroupState::REMOVED);
.. _ray-placement-group-observability-ref:
Observe and Debug Placement Groups
----------------------------------
Ray provides several useful tools to inspect the placement group states and resource usage.
- **Ray Status** is a CLI tool for viewing the resource usage and scheduling resource requirements of placement groups.
- **Ray Dashboard** is a UI tool for inspecting placement group states.
- **Ray State API** is a CLI for inspecting placement group states.
.. tab-set::
.. tab-item:: ray status (CLI)
The CLI command ``ray status`` provides the autoscaling status of the cluster.
It provides the "resource demands" from unscheduled placement groups as well as the resource reservation status.
.. code-block:: bash
Resources
---------------------------------------------------------------
Usage:
1.0/2.0 CPU (1.0 used of 1.0 reserved in placement groups)
0.0/2.0 GPU (0.0 used of 1.0 reserved in placement groups)
0B/4.29GiB memory
0B/2.00GiB object_store_memory
.. tab-item:: Dashboard
The :ref:`dashboard job view <dash-jobs-view>` provides the placement group table that displays the scheduling state and metadata of the placement group.
.. note::
Ray dashboard is only available when you install Ray is with ``pip install "ray[default]"``.
.. tab-item:: Ray State API
:ref:`Ray state API <state-api-overview-ref>` is a CLI tool for inspecting the state of Ray resources (tasks, actors, placement groups, etc.).
``ray list placement-groups`` provides the metadata and the scheduling state of the placement group.
``ray list placement-groups --detail`` provides statistics and scheduling state in a greater detail.
.. note::
State API is only available when you install Ray is with ``pip install "ray[default]"``
Inspect Placement Group Scheduling State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
With the above tools, you can see the state of the placement group. The definition of states are specified in the following files:
- `High level state <https://github.com/ray-project/ray/blob/03a9d2166988b16b7cbf51dac0e6e586455b28d8/src/ray/protobuf/gcs.proto#L579>`_
- `Details <https://github.com/ray-project/ray/blob/03a9d2166988b16b7cbf51dac0e6e586455b28d8/src/ray/protobuf/gcs.proto#L524>`_
.. image:: ../images/pg_image_6.png
:align: center
[Advanced] Child Tasks and Actors
---------------------------------
By default, child actors and tasks don't share the same placement group that the parent uses.
To automatically schedule child actors or tasks to the same placement group,
set ``placement_group_capture_child_tasks`` to True.
.. tab-set::
.. tab-item:: Python
.. literalinclude:: ../doc_code/placement_group_capture_child_tasks_example.py
:language: python
:start-after: __child_capture_pg_start__
:end-before: __child_capture_pg_end__
.. tab-item:: Java
It's not implemented for Java APIs yet.
When ``placement_group_capture_child_tasks`` is True, but you don't want to schedule
child tasks and actors to the same placement group, specify ``PlacementGroupSchedulingStrategy(placement_group=None)``.
.. literalinclude:: ../doc_code/placement_group_capture_child_tasks_example.py
:language: python
:start-after: __child_capture_disable_pg_start__
:end-before: __child_capture_disable_pg_end__
[Advanced] Named Placement Group
--------------------------------
Within a :ref:`namespace <namespaces-guide>`, you can *name* a placement group.
You can use the name of a placement group to retrieve the placement group from any job
in the Ray cluster, as long as the job is within the same namespace.
This is useful if you can't directly pass the placement group handle to
the actor or task that needs it, or if you are trying to
access a placement group launched by another driver.
The placement group is destroyed when the original creation job completes if its
lifetime isn't `detached`. You can avoid this by using a :ref:`detached placement group <placement-group-detached>`
Note that this feature requires that you specify a
:ref:`namespace <namespaces-guide>` associated with it, or else you can't retrieve the
placement group across jobs.
.. tab-set::
.. tab-item:: Python
.. literalinclude:: ../doc_code/placement_group_example.py
:language: python
:start-after: __get_pg_start__
:end-before: __get_pg_end__
.. tab-item:: Java
.. code-block:: java
// Create a placement group with a unique name.
Map<String, Double> bundle = ImmutableMap.of("CPU", 1.0);
List<Map<String, Double>> bundles = ImmutableList.of(bundle);
PlacementGroupCreationOptions options =
new PlacementGroupCreationOptions.Builder()
.setBundles(bundles)
.setStrategy(PlacementStrategy.STRICT_SPREAD)
.setName("global_name")
.build();
PlacementGroup pg = PlacementGroups.createPlacementGroup(options);
pg.wait(60);
...
// Retrieve the placement group later somewhere.
PlacementGroup group = PlacementGroups.getPlacementGroup("global_name");
Assert.assertNotNull(group);
.. tab-item:: C++
.. code-block:: c++
// Create a placement group with a globally unique name.
std::vector<std::unordered_map<std::string, double>> bundles{{{"CPU", 1.0}}};
ray::PlacementGroupCreationOptions options{
true/*global*/, "global_name", bundles, ray::PlacementStrategy::STRICT_SPREAD};
ray::PlacementGroup pg = ray::CreatePlacementGroup(options);
pg.Wait(60);
...
// Retrieve the placement group later somewhere.
ray::PlacementGroup group = ray::GetGlobalPlacementGroup("global_name");
assert(!group.Empty());
We also support non-global named placement group in C++, which means that the placement group name is only valid within the job and cannot be accessed from another job.
.. code-block:: c++
// Create a placement group with a job-scope-unique name.
std::vector<std::unordered_map<std::string, double>> bundles{{{"CPU", 1.0}}};
ray::PlacementGroupCreationOptions options{
false/*non-global*/, "non_global_name", bundles, ray::PlacementStrategy::STRICT_SPREAD};
ray::PlacementGroup pg = ray::CreatePlacementGroup(options);
pg.Wait(60);
...
// Retrieve the placement group later somewhere in the same job.
ray::PlacementGroup group = ray::GetPlacementGroup("non_global_name");
assert(!group.Empty());
.. _placement-group-detached:
[Advanced] Detached Placement Group
-----------------------------------
By default, the lifetimes of placement groups belong to the driver and actor.
- If the placement group is created from a driver, it is destroyed when the driver is terminated.
- If it is created from a detached actor, it is killed when the detached actor is killed.
To keep the placement group alive regardless of its job or detached actor, specify
`lifetime="detached"`. For example:
.. tab-set::
.. tab-item:: Python
.. literalinclude:: ../doc_code/placement_group_example.py
:language: python
:start-after: __detached_pg_start__
:end-before: __detached_pg_end__
.. tab-item:: Java
The lifetime argument isn't implemented for Java APIs yet.
Let's terminate the current script and start a new Python script. Call ``ray list placement-groups``, and you can see the placement group is not removed.
Note that Ray decouples the lifetime option and the name option. If you only specify
the name without specifying ``lifetime="detached"``, then you can only retrieve the placement group
while the driver where you created the placement group is still running.
It's recommended to always specify the name when creating the detached placement group. If you don't,
there is no way to retrieve the placement group from another process, and there is no way
to kill it once you exit the driver script that created the placement group.
.. _ray-placement-group-ft-ref:
[Advanced] Fault Tolerance
--------------------------
Rescheduling Bundles on a Dead Node
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If nodes that contain some bundles of a placement group die, Ray tries to reschedule the lost bundles on different nodes.
This means that the initial creation of placement group is "atomic," but after the initial creation,
there could be partial placement groups. Actors or tasks running on bundles on the remaining live nodes continue to run.
Note that rescheduling bundles have higher scheduling priority than other placement group scheduling.
Provide Resources for Partially Lost Bundles
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If there aren't enough resources to schedule the partially lost bundles,
the placement group waits, assuming the Ray Autoscaler starts a new node to satisfy the resource requirements.
If the autoscaler can't provide additional resources or if you're not using the autoscaler,
the placement group remains in the partially created state indefinitely.
Fault Tolerance of Actors and Tasks that Use the Bundle
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Ray reschedules Actors and tasks that use the bundle (reserved resources) based on their :ref:`fault tolerant policy <fault-tolerance>` once Ray recovers the bundle.
.. _pgroup-topology-strategy:
[Alpha] Topology aware scheduling
---------------------------------
.. warning::
Topology aware scheduling is an **alpha** feature. It's actively being iterated on and
the API surface may change. Ray currently only supports defining one topology label and
one node level strategy (described below). For topology labels, Ray currently supports
only ``STRICT_PACK``. Support for additional strategies and multi-level topologies is planned.
Why topology aware scheduling?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The placement strategies above (PACK, STRICT_PACK, SPREAD, STRICT_SPREAD) operate purely on a
per-node basis. For multi-node GPU domains such as GB200 or GB300 NVL racks where nodes share
fast interconnects, there's no native way to ensure all bundles land within the same GPU
domain.
For example, consider a cluster with 2 racks of 18 nodes each, where each node has
``{"GPU": 4, "CPU": 2}``. You want to schedule ``[{"GPU": 4, "CPU": 2}] * 18`` within a single rack:
- **STRICT_PACK** tries to place all 18 bundles onto a single *node*, which is infeasible because
a single node only has 4 GPUs and 2 CPUs.
- **PACK** spreads bundles across nodes but it has no concept of racks and bundles may land on nodes
across *both* racks.
You could work around this with static :ref:`label selectors <labels>` (such as
``bundle_label_selector=[{"my_custom_gpu_domain_label": "rack-1"}] * 18``), but that approach doesn't support
fault tolerance. If all nodes in ``rack-1`` go down, the placement group can't automatically move
to a different rack. Furthermore, you have to manually specify a domain when you really just want
any domain and this becomes cumbersome if you have many GPU domains.
Topology aware scheduling currently solves this by letting you express a topology strategy for the placement
group, which allows specifying a topology label within the cluster. Ray picks a value for this topology label
that can satisfy all bundles (for example, a specific rack) and then applies your node-level strategy
within that value.
How it works
~~~~~~~~~~~~
Pass ``topology_strategy=`` to :func:`ray.util.placement_group` to enable topology-aware
placement. The argument is a dict that maps a label key to the placement strategy used at that level.
Currently, ``topology_strategy`` is a dictionary that may contain up to two keys:
- The special key ``ray.io/node-id`` sets the **node-level** strategy and accepts any value
in ``{"PACK", "STRICT_PACK", "SPREAD", "STRICT_SPREAD"}``. If you omit it, the node-level
strategy defaults to ``PACK``.
- Any other key is a **topology label** that nodes have set (via ``ray start --labels`` or
your cluster configuration). Only ``STRICT_PACK`` is supported for these labels today.
.. code-block:: python
from ray.util.placement_group import placement_group
bundles = [{"GPU": 4, "CPU": 2}] * 18
pg = placement_group(
bundles=bundles,
topology_strategy={"ray.io/gpu-domain": "STRICT_PACK"},
)
ray.get(pg.ready())
With this, Ray accomplishes the following:
1. Groups candidate nodes by the value of the topology label you named
(``ray.io/gpu-domain`` in the example above).
2. Selects a value for that label that can satisfy all bundles.
3. Applies the node-level scheduling strategy within the selected value.
Here is a following example to STRICT_SPREAD bundles across distinct nodes while still
STRICT_PACKing them onto a single rack (``ray.io/gpu-domain`` is each rack's **topology label**):
.. code-block:: python
pg = placement_group(
bundles=[{"CPU": 1}] * 4,
topology_strategy={
"ray.io/node-id": "STRICT_SPREAD",
"ray.io/gpu-domain": "STRICT_PACK",
},
)
``topology_strategy`` is mutually exclusive with the ``strategy=`` parameter, and passing both
raises ``ValueError``. To override the default node-level strategy alongside a topology label,
put the node-level strategy under ``ray.io/node-id`` in the same dict, as shown above.
.. note::
Ray doesn't automatically set topology labels such as ``ray.io/gpu-domain`` on nodes.
Configure these labels through ``ray start --labels`` or your cluster configuration.
For example:
.. code-block:: bash
ray start --labels="ray.io/gpu-domain=rack-1"
Fault tolerance
~~~~~~~~~~~~~~~
Topology aware scheduling improves on static label selectors by providing automatic
fault tolerance at the topology-label level:
- **Partial failure** (some nodes within the selected value of the topology label die):
Ray reschedules the lost bundles onto surviving nodes **within the same value** (for
example, the same rack). Actors and tasks on the remaining bundles keep running. If
the selected value doesn't have enough resources to reschedule the lost bundles,
those bundles stay infeasible and queued until resources free up in the same value.
To force the placement group onto a different value, call
:func:`ray.util.remove_placement_group <ray.util.remove_placement_group>` and create
a new one. Removing the placement group forcefully kills every actor and task still
using its bundles and doesn't restart them, so you must re-create them yourself on
the new placement group.
- **Total failure** (all nodes with the selected value die): Ray clears the topology
assignment and reschedules the entire placement group onto a different value.
Observability
~~~~~~~~~~~~~
You can inspect topology-aware placement groups using the existing placement group
observability tools:
- **Dashboard**: The placement group table shows a ``Topology`` column, which displays the
strategy you requested and the value Ray selected for each topology label.
- **State API**: ``ray list placement-groups --detail`` returns the requested strategy in
``topology_strategy`` and the value Ray selected in ``topology_assignments``.
The following ``ray list placement-groups --detail`` output shows the two topology fields,
``topology_strategy`` and ``topology_assignments``, populated for a placement group that
packs onto a single ``ray.io/gpu-domain``:
.. code-block:: yaml
- placement_group_id: 237f47c3235ac1a96ad423c3f74501000000
name: gpu-domain-pg
state: CREATED
bundles:
- bundle_id:
placement_group_id: 237f47c3235ac1a96ad423c3f74501000000
bundle_index: 0
unit_resources:
CPU: 1.0
node_id: 0fd7eecf6335633ba39ab66f5a26b18eeb35c70c15a9563a29ee2bce
- bundle_id:
placement_group_id: 237f47c3235ac1a96ad423c3f74501000000
bundle_index: 1
unit_resources:
CPU: 1.0
node_id: 0fd7eecf6335633ba39ab66f5a26b18eeb35c70c15a9563a29ee2bce
is_detached: false
stats: ...
topology_strategy:
- entries:
ray.io/gpu-domain: STRICT_PACK
topology_assignments:
- assignments:
ray.io/gpu-domain: rack-2
For placement groups that don't use topology-aware scheduling, ``topology_strategy`` and
``topology_assignments`` are both empty lists. Both fields appear only when you pass
``--detail``.
API Reference
-------------
:ref:`Placement Group API reference <ray-placement-group-ref>`
doc/source/ray-core/scheduling/ray-oom-prevention.rst
+204
View File
@@ -0,0 +1,204 @@
.. _ray-oom-prevention:
Out-Of-Memory Prevention
========================
If application tasks or actors consume a large amount of heap space, it can cause the node to run out of memory (OOM). When that happens, the operating system will start killing worker or raylet processes, disrupting the application. OOM may also stall metrics and if this happens on the head node, it may stall the :ref:`dashboard <observability-getting-started>` or other control processes and cause the cluster to become unusable.
In this section we will go over:
- What is the memory monitor and how it works
- How to enable and configure it
Also view :ref:`Debugging Out of Memory <troubleshooting-out-of-memory>` to learn how to troubleshoot out-of-memory issues.
.. _ray-oom-monitor:
What is the memory monitor?
---------------------------
The memory monitor is a component that runs within the :ref:`raylet <whitepaper>` process on each node. It monitors the memory usage, which includes the worker heap, the object store, and the raylet as described in :ref:`memory management <memory>`. If the combined usage exceeds a configurable threshold the raylet will kill a task or actor process to free up memory and prevent Ray from failing.
It's available on Linux and is tested with Ray running inside a container that is using cgroup v1/v2. If you encounter issues when running the memory monitor outside of a container, :ref:`file an issue or post a question <oom-questions>`.
What to expect?
---------------
The default memory monitoring system protects the Ray node from node death due to memory contention and OOM.
Compared to the Linux OOM killer, it also aims to preserve as much application progress as possible by killing workers
based on the time since the task started executing. However, the default memory monitoring system makes no guarantees.
Starting in Ray 2.56, with resource isolation enabled, the memory monitoring system provides the following:
- Zero kernel OOM kills (work-preserving) when resource isolation is enabled and system-reserved memory is configured to cover the memory footprint of critical Ray system processes (raylet, gcs, agents) and other system overhead.
- Zero Ray OOM kills under the above configuration when tasks and actors specify accurate logical memory requests.
- Zero node deaths due to memory contention when resource isolation is enabled and system-reserved memory is configured correctly.
To enable resource isolation, see :ref:`How to Enable Cgroup v2 for Resource Isolation <enable-cgroupv2>`.
How do I disable the memory monitor?
--------------------------------------
The memory monitor is enabled by default and can only be disabled when resource isolation is disabled.
To disable the memory monitor when resource isolation is turned off, set the environment variable ``RAY_memory_monitor_refresh_ms`` to zero when Ray starts (e.g., ``RAY_memory_monitor_refresh_ms=0 ray start ...``).
How do I configure the memory monitor?
--------------------------------------
**Default memory monitor configuration:**
The memory monitor is controlled by the following environment variables:
- ``RAY_memory_monitor_refresh_ms (int, defaults to 250)`` is the interval to check memory usage and kill tasks or actors if needed. Task killing is disabled when this value is 0. The memory monitor selects and kills one task at a time and waits for it to be killed before choosing another one, regardless of how frequent the memory monitor runs.
- ``RAY_memory_usage_threshold (float, defaults to 0.95)`` is the threshold when the node is beyond the memory
capacity. If the memory usage is above this fraction it will start killing processes to free up memory. Ranges from [0, 1].
**Resource isolation memory monitor configuration:**
When resource isolation is enabled, the memory monitor is controlled by the following flag passed to ``ray start`` or ``ray.init``:
- ``--system-reserved-memory`` sets the amount of memory reserved for critical Ray system processes and other system processes outside of Ray's userspace.
By default, this value is 10% of the system's total memory, bounded by a minimum of 500MB and a maximum of 10GB. The memory monitor enforces that the
workload processes' memory footprint doesn't exceed ``total_memory - system_reserved_memory`` bytes.
Using the Memory Monitor
------------------------
.. _ray-oom-retry-policy:
Retry policy
~~~~~~~~~~~~
When a task or actor is killed by the memory monitor it will be retried with exponential backoff. There is a cap on the retry delay, which is 60 seconds.
If tasks are killed by the memory monitor, it retries infinitely (not respecting :ref:`max_retries <task-fault-tolerance>`) unless :ref:`max_retries <task-fault-tolerance>` is set to 0.
When :ref:`max_retries <task-fault-tolerance>` is set to 0, the task will not be retried.
If actors are killed by the memory monitor, it doesn't recreate the actor infinitely (It respects :ref:`max_restarts <actor-fault-tolerance>`, which is 0 by default).
.. _ray-oom-worker-killing-policy:
Worker killing policy
~~~~~~~~~~~~~~~~~~~~~
Worker killing policy since Ray 2.56
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. image:: ../images/time-based-killing-policy.png
:width: 1024
:alt: Time based worker killing policy
As shown in the diagram above, the worker killing policy prioritizes idle workers over active workers when selecting workers to kill.
**Idle worker policy:**
1. The memory monitor always consider all workers that have previously executed tasks or actors for killing regardless of the idle-worker killing memory threshold.
Workers that have never executed any tasks or actors (cold-start idle workers) are only considered for killing if their memory footprint exceeds the idle-worker killing memory threshold.
Cold-start idle workers should have a small memory footprint. In the unlikely case that the OOM logs show active workers being selected over idle workers while a large idle-worker memory footprint remains,
the dependencies inherited when starting a new process in Ray's userspace are likely too expensive. In that case, consider reducing the memory footprint of new processes in Ray's userspace, or
lowering the idle-worker killing memory threshold via the environment variable ``RAY_idle_worker_killing_memory_threshold_bytes`` (default is 1GiB).
2. Among the workers eligible for killing, the policy selects the worker with the largest memory footprint first.
**Active worker policy:**
1. For workers running tasks or actors (active workers), the policy prioritizes retriable tasks first to maximize retry opportunities.
2. Among the active workers with the same retriability, the policy selects the most recent workers next (newest granted lease time).
The policy continues to select workers until ``current_memory_usage - total_selected_workers_memory_footprint + kill_buffer <= available_memory_for_workload_processes``.
Where ``current_memory_usage`` is the current memory usage on the node,
``total_selected_workers_memory_footprint`` is the sum of the memory footprint of all selected workers to kill,
``kill_buffer`` is the amount of memory to leave as breathing room between the memory usage and the memory allocated to the workload processes.
``available_memory_for_workload_processes`` is the amount of memory available for the workload processes to use.
This is computed as ``total_system_memory - system_reserved_memory`` as mentioned above.
The ``kill_buffer`` defaults to 5% of the total system memory and caps at 3GiB (configurable via ``RAY_max_kill_memory_buffer_bytes``).
To revert to the legacy worker killing policy, set the environment variable ``RAY_worker_killing_policy_by_group`` to ``true`` before starting Ray.
Legacy worker killing policy
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The memory monitor avoids infinite loops of task retries by ensuring at least one task is able to run for each caller on each node. If it is unable to ensure this, the workload will fail with an OOM error. Note that this is only an issue for tasks, since the memory monitor will not indefinitely retry actors. If the workload fails, refer to :ref:`how to address memory issues <addressing-memory-issues>` on how to adjust the workload to make it pass. For code example, see the :ref:`last task <last-task-example>` example below.
When a worker needs to be killed, the policy first prioritizes tasks that are retriable, i.e. when :ref:`max_retries <task-fault-tolerance>` or :ref:`max_restarts <actor-fault-tolerance>` is > 0. This is done to minimize workload failure. Actors by default are not retriable since :ref:`max_restarts <actor-fault-tolerance>` defaults to 0. Therefore, by default, tasks are preferred to actors when it comes to what gets killed first.
When there are multiple callers that has created tasks, the policy will pick a task from the caller with the most number of running tasks. If two callers have the same number of tasks it picks the caller whose earliest task has a later start time. This is done to ensure fairness and allow each caller to make progress.
Amongst the tasks that share the same caller, the latest started task will be killed first.
Below is an example to demonstrate the policy. In the example we have a script that creates two tasks, which in turn creates four more tasks each. The tasks are colored such that each color forms a "group" of tasks where they belong to the same caller.
.. image:: ../images/oom_killer_example.svg
:width: 1024
:alt: Initial state of the task graph
If, at this point, the node runs out of memory, it will pick a task from the caller with the most number of tasks, and kill its task whose started the last:
.. image:: ../images/oom_killer_example_killed_one.svg
:width: 1024
:alt: Initial state of the task graph
If, at this point, the node still runs out of memory, the process will repeat:
.. image:: ../images/oom_killer_example_killed_two.svg
:width: 1024
:alt: Initial state of the task graph
.. _last-task-example:
.. dropdown:: Example: Workloads fails if the last task of the caller is killed
Let's create an application oom.py that runs a single task that requires more memory than what is available. It is set to infinite retry by setting ``max_retries`` to -1.
The worker killer policy sees that it is the last task of the caller, and will fail the workload when it kills the task as it is the last one for the caller, even when the task is set to retry forever.
.. literalinclude:: ../doc_code/ray_oom_prevention.py
:language: python
:start-after: __last_task_start__
:end-before: __last_task_end__
Set ``RAY_event_stats_print_interval_ms=1000`` so it prints the worker kill summary every second, since by default it prints every minute.
.. code-block:: bash
RAY_event_stats_print_interval_ms=1000 python oom.py
(raylet) node_manager.cc:3040: 1 Workers (tasks / actors) killed due to memory pressure (OOM), 0 Workers crashed due to other reasons at node (ID: 2c82620270df6b9dd7ae2791ef51ee4b5a9d5df9f795986c10dd219c, IP: 172.31.183.172) over the last time period. To see more information about the Workers killed on this node, use `ray logs raylet.out -ip 172.31.183.172`
(raylet)
(raylet) Refer to the documentation on how to address the out of memory issue: https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html. Consider provisioning more memory on this node or reducing task parallelism by requesting more CPUs per task. To adjust the kill threshold, set the environment variable `RAY_memory_usage_threshold` when starting Ray. To disable worker killing, set the environment variable `RAY_memory_monitor_refresh_ms` to zero.
task failed with OutOfMemoryError, which is expected
Verify the task was indeed executed twice via ``task_oom_retry``:
.. dropdown:: Example: memory monitor prefers to kill a retriable task
Let's first start Ray and specify the memory threshold.
.. code-block:: bash
RAY_memory_usage_threshold=0.4 ray start --head
Let's create an application two_actors.py that submits two actors, where the first one is retriable and the second one is non-retriable.
.. literalinclude:: ../doc_code/ray_oom_prevention.py
:language: python
:start-after: __two_actors_start__
:end-before: __two_actors_end__
Run the application to see that only the first actor was killed.
.. code-block:: bash
$ python two_actors.py
First started actor, which is retriable, was killed by the memory monitor.
Second started actor, which is not-retriable, finished.
.. _addressing-memory-issues:
.. _oom-questions:
Questions or Issues?
--------------------
.. include:: /_includes/_help.rst
doc/source/ray-core/scheduling/resources.rst
+207
View File
@@ -0,0 +1,207 @@
.. _core-resources:
Resources
=========
Ray allows you to seamlessly scale your applications from a laptop to a cluster without code change.
**Ray resources** are key to this capability.
They abstract away physical machines and let you express your computation in terms of resources,
while the system manages scheduling and autoscaling based on resource requests.
A resource in Ray is a key-value pair where the key denotes a resource name, and the value is a float quantity.
For convenience, Ray has native support for CPU, GPU, and memory resource types; CPU, GPU and memory are called **pre-defined resources**.
Besides those, Ray also supports :ref:`custom resources <custom-resources>`.
.. _logical-resources:
Physical Resources and Logical Resources
----------------------------------------
Physical resources are resources that a machine physically has such as physical CPUs and GPUs
and logical resources are virtual resources defined by a system.
Ray resources are **logical** and don’t need to have 1-to-1 mapping with physical resources.
For example, you can start a Ray head node with 0 logical CPUs via ``ray start --head --num-cpus=0``
even if it physically has eight
(This signals the Ray scheduler to not schedule any tasks or actors that require logical CPU resources
on the head node, mainly to reserve the head node for running Ray system processes.).
They are mainly used for admission control during scheduling.
The fact that resources are logical has several implications:
- Resource requirements of tasks or actors do NOT impose limits on actual physical resource usage.
For example, Ray doesn't prevent a ``num_cpus=1`` task from launching multiple threads and using multiple physical CPUs.
It's your responsibility to make sure tasks or actors use no more resources than specified via resource requirements.
- Ray doesn't provide CPU isolation for tasks or actors.
For example, Ray won't reserve a physical CPU exclusively and pin a ``num_cpus=1`` task to it.
Ray will let the operating system schedule and run the task instead.
If needed, you can use operating system APIs like ``sched_setaffinity`` to pin a task to a physical CPU.
- Ray does provide :ref:`GPU <gpu-support>` isolation in the form of *visible devices* by automatically setting the ``CUDA_VISIBLE_DEVICES`` environment variable,
which most ML frameworks will respect for purposes of GPU assignment.
.. _omp-num-thread-note:
.. note::
Ray sets the environment variable ``OMP_NUM_THREADS=<num_cpus>`` if ``num_cpus`` is set on
the task/actor via :func:`ray.remote() <ray.remote>` and :meth:`task.options() <ray.remote_function.RemoteFunction.options>`/:meth:`actor.options() <ray.actor.ActorClass.options>`.
Ray sets ``OMP_NUM_THREADS=1`` if ``num_cpus`` is not specified; this
is done to avoid performance degradation with many workers (issue #6998). You can
also override this by explicitly setting ``OMP_NUM_THREADS`` to override anything Ray sets by default.
``OMP_NUM_THREADS`` is commonly used in numpy, PyTorch, and Tensorflow to perform multi-threaded
linear algebra. In multi-worker setting, we want one thread per worker instead of many threads
per worker to avoid contention. Some other libraries may have their own way to configure
parallelism. For example, if you're using OpenCV, you should manually set the number of
threads using cv2.setNumThreads(num_threads) (set to 0 to disable multi-threading).
.. figure:: ../images/physical_resources_vs_logical_resources.svg
Physical resources vs logical resources
.. _custom-resources:
Custom Resources
----------------
You can specify custom resources for a Ray node and reference them to control scheduling for your tasks or actors.
Use custom resources when you need to manage scheduling using numeric values. If you need simple label-based scheduling, use labels instead. See :doc:`labels`.
.. _specify-node-resources:
Specifying Node Resources
-------------------------
By default, Ray nodes start with pre-defined CPU, GPU, and memory resources. The quantities of these logical resources on each node are set to the physical quantities auto detected by Ray.
By default, logical resources are configured by the following rule.
.. warning::
Ray **does not permit dynamic updates of resource capacities after Ray has been started on a node**.
- **Number of logical CPUs** (``num_cpus``): Set to the number of CPUs of the machine/container.
- **Number of logical GPUs** (``num_gpus``): Set to the number of GPUs of the machine/container.
- **Memory** (``memory``): Set to 70% of "available memory" when ray runtime starts.
- **Object Store Memory** (``object_store_memory``): Set to 30% of "available memory" when ray runtime starts. Note that the object store memory is not logical resource, and users cannot use it for scheduling.
However, you can always override that by manually specifying the quantities of pre-defined resources and adding custom resources.
There are several ways to do that depending on how you start the Ray cluster:
.. tab-set::
.. tab-item:: ray.init()
If you are using :func:`ray.init() <ray.init>` to start a single node Ray cluster, you can do the following to manually specify node resources:
.. literalinclude:: ../doc_code/resources.py
:language: python
:start-after: __specifying_node_resources_start__
:end-before: __specifying_node_resources_end__
.. tab-item:: ray start
If you are using :ref:`ray start <ray-start-doc>` to start a Ray node, you can run:
.. code-block:: shell
ray start --head --num-cpus=3 --num-gpus=4 --resources='{"special_hardware": 1, "custom_label": 1}'
.. tab-item:: ray up
If you are using :ref:`ray up <ray-up-doc>` to start a Ray cluster, you can set the :ref:`resources field <cluster-configuration-resources-type>` in the yaml file:
.. code-block:: yaml
available_node_types:
head:
...
resources:
CPU: 3
GPU: 4
special_hardware: 1
custom_label: 1
.. tab-item:: KubeRay
If you are using :ref:`KubeRay <kuberay-index>` to start a Ray cluster, you can set the :ref:`rayStartParams field <rayStartParams>` in the yaml file:
.. code-block:: yaml
headGroupSpec:
rayStartParams:
num-cpus: "3"
num-gpus: "4"
resources: '"{\"special_hardware\": 1, \"custom_label\": 1}"'
.. _resource-requirements:
Specifying Task or Actor Resource Requirements
----------------------------------------------
Ray allows specifying a task or actor's logical resource requirements (e.g., CPU, GPU, and custom resources).
The task or actor will only run on a node if there are enough required logical resources
available to execute the task or actor.
By default, Ray tasks use 1 logical CPU resource and Ray actors use 1 logical CPU for scheduling, and 0 logical CPU for running.
(This means, by default, actors cannot get scheduled on a zero-cpu node, but an infinite number of them can run on any non-zero cpu node.
The default resource requirements for actors was chosen for historical reasons.
It's recommended to always explicitly set ``num_cpus`` for actors to avoid any surprises.
If resources are specified explicitly, they are required both at schedule time and at execution time.)
You can also explicitly specify a task's or actor's logical resource requirements (for example, one task may require a GPU) instead of using default ones via :func:`ray.remote() <ray.remote>`
and :meth:`task.options() <ray.remote_function.RemoteFunction.options>`/:meth:`actor.options() <ray.actor.ActorClass.options>`.
.. tab-set::
.. tab-item:: Python
.. literalinclude:: ../doc_code/resources.py
:language: python
:start-after: __specifying_resource_requirements_start__
:end-before: __specifying_resource_requirements_end__
.. tab-item:: Java
.. code-block:: java
// Specify required resources.
Ray.task(MyRayApp::myFunction).setResource("CPU", 1.0).setResource("GPU", 1.0).setResource("special_hardware", 1.0).remote();
Ray.actor(Counter::new).setResource("CPU", 2.0).setResource("GPU", 1.0).remote();
.. tab-item:: C++
.. code-block:: c++
// Specify required resources.
ray::Task(MyFunction).SetResource("CPU", 1.0).SetResource("GPU", 1.0).SetResource("special_hardware", 1.0).Remote();
ray::Actor(CreateCounter).SetResource("CPU", 2.0).SetResource("GPU", 1.0).Remote();
Task and actor resource requirements have implications for the Ray's scheduling concurrency.
In particular, the sum of the logical resource requirements of all of the
concurrently executing tasks and actors on a given node cannot exceed the node's total logical resources.
This property can be used to :ref:`limit the number of concurrently running tasks or actors to avoid issues like OOM <core-patterns-limit-running-tasks>`.
.. _fractional-resource-requirements:
Fractional Resource Requirements
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Ray supports fractional resource requirements.
For example, if your task or actor is IO bound and has low CPU usage, you can specify fractional CPU ``num_cpus=0.5`` or even zero CPU ``num_cpus=0``.
The precision of the fractional resource requirement is 0.0001 so you should avoid specifying a double that's beyond that precision.
.. literalinclude:: ../doc_code/resources.py
:language: python
:start-after: __specifying_fractional_resource_requirements_start__
:end-before: __specifying_fractional_resource_requirements_end__
.. note::
GPU, TPU, and neuron_cores resource requirements that are greater than 1, need to be whole numbers. For example, ``num_gpus=1.5`` is invalid.
.. tip::
Besides resource requirements, you can also specify an environment for a task or actor to run in,
which can include Python packages, local files, environment variables, and more. See :ref:`Runtime Environments <runtime-environments>` for details.