wehub/ray-project--ray
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:17:40 +08:00
commit f1825c8ceb
10096 changed files with 2364182 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/ray-core/accelerator-types.rst
+11
View File
@@ -0,0 +1,11 @@
:orphan:
.. _accelerator_types:
Accelerator types
=================
Ray supports the following accelerator types:
.. literalinclude:: ../../../python/ray/util/accelerators/accelerators.py
:language: python
doc/source/ray-core/actors.rst
+557
View File
@@ -0,0 +1,557 @@
.. _ray-remote-classes:
.. _actor-guide:
Actors
======
Actors extend the Ray API from functions (tasks) to classes.
An actor is essentially a stateful worker (or a service).
When you instantiate a new actor, Ray creates a new worker and schedules methods of the actor on
that specific worker. The methods can access and mutate the state of that worker.
.. tab-set::
.. tab-item:: Python
The ``ray.remote`` decorator indicates that instances of the ``Counter`` class are actors. Each actor runs in its own Python process.
.. testcode::
import ray
@ray.remote
class Counter:
def __init__(self):
self.value = 0
def increment(self):
self.value += 1
return self.value
def get_counter(self):
return self.value
# Create an actor from this class.
counter = Counter.remote()
.. tab-item:: Java
``Ray.actor`` is used to create actors from regular Java classes.
.. code-block:: java
// A regular Java class.
public class Counter {
private int value = 0;
public int increment() {
this.value += 1;
return this.value;
}
}
// Create an actor from this class.
// `Ray.actor` takes a factory method that can produce
// a `Counter` object. Here, we pass `Counter`'s constructor
// as the argument.
ActorHandle<Counter> counter = Ray.actor(Counter::new).remote();
.. tab-item:: C++
``ray::Actor`` is used to create actors from regular C++ classes.
.. code-block:: c++
// A regular C++ class.
class Counter {
private:
int value = 0;
public:
int Increment() {
value += 1;
return value;
}
};
// Factory function of Counter class.
static Counter *CreateCounter() {
return new Counter();
};
RAY_REMOTE(&Counter::Increment, CreateCounter);
// Create an actor from this class.
// `ray::Actor` takes a factory method that can produce
// a `Counter` object. Here, we pass `Counter`'s factory function
// as the argument.
auto counter = ray::Actor(CreateCounter).Remote();
Use `ray list actors` from :ref:`State API <state-api-overview-ref>` to see actors states:
.. code-block:: bash
# This API is only available when you install Ray with `pip install "ray[default]"`.
ray list actors
.. code-block:: bash
======== List: 2023-05-25 10:10:50.095099 ========
Stats:
------------------------------
Total: 1
Table:
------------------------------
ACTOR_ID CLASS_NAME STATE JOB_ID NAME NODE_ID PID RAY_NAMESPACE
0 9e783840250840f87328c9f201000000 Counter ALIVE 01000000 13a475571662b784b4522847692893a823c78f1d3fd8fd32a2624923 38906 ef9de910-64fb-4575-8eb5-50573faa3ddf
Specifying required resources
-----------------------------
.. _actor-resource-guide:
Specify resource requirements in actors. See :ref:`resource-requirements` for more details.
.. tab-set::
.. tab-item:: Python
.. testcode::
# Specify required resources for an actor.
@ray.remote(num_cpus=2, num_gpus=0.5)
class Actor:
pass
.. tab-item:: Java
.. code-block:: java
// Specify required resources for an actor.
Ray.actor(Counter::new).setResource("CPU", 2.0).setResource("GPU", 0.5).remote();
.. tab-item:: C++
.. code-block:: c++
// Specify required resources for an actor.
ray::Actor(CreateCounter).SetResource("CPU", 2.0).SetResource("GPU", 0.5).Remote();
Calling the actor
-----------------
You can interact with the actor by calling its methods with the ``remote``
operator. You can then call ``get`` on the object ref to retrieve the actual
value.
.. tab-set::
.. tab-item:: Python
.. testcode::
# Call the actor.
obj_ref = counter.increment.remote()
print(ray.get(obj_ref))
.. testoutput::
1
.. tab-item:: Java
.. code-block:: java
// Call the actor.
ObjectRef<Integer> objectRef = counter.task(&Counter::increment).remote();
Assert.assertTrue(objectRef.get() == 1);
.. tab-item:: C++
.. code-block:: c++
// Call the actor.
auto object_ref = counter.Task(&Counter::increment).Remote();
assert(*object_ref.Get() == 1);
Methods called on different actors execute in parallel, and methods called on the same actor execute serially in the order you call them. Methods on the same actor share state with one another, as shown below.
.. tab-set::
.. tab-item:: Python
.. testcode::
# Create ten Counter actors.
counters = [Counter.remote() for _ in range(10)]
# Increment each Counter once and get the results. These tasks all happen in
# parallel.
results = ray.get([c.increment.remote() for c in counters])
print(results)
# Increment the first Counter five times. These tasks are executed serially
# and share state.
results = ray.get([counters[0].increment.remote() for _ in range(5)])
print(results)
.. testoutput::
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
[2, 3, 4, 5, 6]
.. tab-item:: Java
.. code-block:: java
// Create ten Counter actors.
List<ActorHandle<Counter>> counters = new ArrayList<>();
for (int i = 0; i < 10; i++) {
counters.add(Ray.actor(Counter::new).remote());
}
// Increment each Counter once and get the results. These tasks all happen in
// parallel.
List<ObjectRef<Integer>> objectRefs = new ArrayList<>();
for (ActorHandle<Counter> counterActor : counters) {
objectRefs.add(counterActor.task(Counter::increment).remote());
}
// prints [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
System.out.println(Ray.get(objectRefs));
// Increment the first Counter five times. These tasks are executed serially
// and share state.
objectRefs = new ArrayList<>();
for (int i = 0; i < 5; i++) {
objectRefs.add(counters.get(0).task(Counter::increment).remote());
}
// prints [2, 3, 4, 5, 6]
System.out.println(Ray.get(objectRefs));
.. tab-item:: C++
.. code-block:: c++
// Create ten Counter actors.
std::vector<ray::ActorHandle<Counter>> counters;
for (int i = 0; i < 10; i++) {
counters.emplace_back(ray::Actor(CreateCounter).Remote());
}
// Increment each Counter once and get the results. These tasks all happen in
// parallel.
std::vector<ray::ObjectRef<int>> object_refs;
for (ray::ActorHandle<Counter> counter_actor : counters) {
object_refs.emplace_back(counter_actor.Task(&Counter::Increment).Remote());
}
// prints 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
auto results = ray::Get(object_refs);
for (const auto &result : results) {
std::cout << *result;
}
// Increment the first Counter five times. These tasks are executed serially
// and share state.
object_refs.clear();
for (int i = 0; i < 5; i++) {
object_refs.emplace_back(counters[0].Task(&Counter::Increment).Remote());
}
// prints 2, 3, 4, 5, 6
results = ray::Get(object_refs);
for (const auto &result : results) {
std::cout << *result;
}
Passing around actor handles
----------------------------
You can pass actor handles into other tasks. You can also define remote functions or actor methods that use actor handles.
.. tab-set::
.. tab-item:: Python
.. testcode::
import time
@ray.remote
def f(counter):
for _ in range(10):
time.sleep(0.1)
counter.increment.remote()
.. tab-item:: Java
.. code-block:: java
public static class MyRayApp {
public static void foo(ActorHandle<Counter> counter) throws InterruptedException {
for (int i = 0; i < 1000; i++) {
TimeUnit.MILLISECONDS.sleep(100);
counter.task(Counter::increment).remote();
}
}
}
.. tab-item:: C++
.. code-block:: c++
void Foo(ray::ActorHandle<Counter> counter) {
for (int i = 0; i < 1000; i++) {
std::this_thread::sleep_for(std::chrono::milliseconds(100));
counter.Task(&Counter::Increment).Remote();
}
}
If you instantiate an actor, you can pass the handle around to various tasks.
.. tab-set::
.. tab-item:: Python
.. testcode::
counter = Counter.remote()
# Start some tasks that use the actor.
[f.remote(counter) for _ in range(3)]
# Print the counter value.
for _ in range(10):
time.sleep(0.1)
print(ray.get(counter.get_counter.remote()))
.. testoutput::
:options: +MOCK
0
3
8
10
15
18
20
25
30
30
.. tab-item:: Java
.. code-block:: java
ActorHandle<Counter> counter = Ray.actor(Counter::new).remote();
// Start some tasks that use the actor.
for (int i = 0; i < 3; i++) {
Ray.task(MyRayApp::foo, counter).remote();
}
// Print the counter value.
for (int i = 0; i < 10; i++) {
TimeUnit.SECONDS.sleep(1);
System.out.println(counter.task(Counter::getCounter).remote().get());
}
.. tab-item:: C++
.. code-block:: c++
auto counter = ray::Actor(CreateCounter).Remote();
// Start some tasks that use the actor.
for (int i = 0; i < 3; i++) {
ray::Task(Foo).Remote(counter);
}
// Print the counter value.
for (int i = 0; i < 10; i++) {
std::this_thread::sleep_for(std::chrono::seconds(1));
std::cout << *counter.Task(&Counter::GetCounter).Remote().Get() << std::endl;
}
Type hints and static typing for actors
---------------------------------------
Ray supports Python type hints for both remote functions and actors, enabling better IDE support and static type checking. To get the best type inference and pass type checkers when working with actors, follow these patterns:
- **Prefer** ``ray.remote(MyClass)`` **over** ``@ray.remote`` **for actors**: Instead of decorating your class with ``@ray.remote``, use ``ActorClass = ray.remote(MyClass)``. This preserves the original class type and allows type checkers and IDEs to infer the correct types.
- **Use** ``@ray.method`` **for actor methods**: Decorate actor methods with ``@ray.method`` to enable type hints for remote method calls on actor handles.
- **Use the** ``ActorClass`` **and** ``ActorProxy`` **types**: When you instantiate an actor, annotate the handle as ``ActorProxy[MyClass]`` to get type hints for remote methods.
**Example:**
.. testcode::
import ray
from ray.actor import ActorClass, ActorProxy
class Counter:
def __init__(self):
self.value = 0
@ray.method
def increment(self) -> int:
self.value += 1
return self.value
CounterActor: ActorClass[Counter] = ray.remote(Counter)
counter: ActorProxy[Counter] = CounterActor.remote()
# Type checkers and IDEs will now provide type hints for remote methods
obj_ref: ray.ObjectRef[int] = counter.increment.remote()
print(ray.get(obj_ref))
For more details and advanced patterns, see :ref:`Type hints in Ray <core-type-hint>`.
Generators
----------
Ray is compatible with Python generator syntax. See :ref:`Ray Generators <generators>` for more details.
Cancelling actor tasks
----------------------
Cancel Actor Tasks by calling :func:`ray.cancel() <ray.cancel>` on the returned `ObjectRef`.
.. tab-set::
.. tab-item:: Python
.. literalinclude:: doc_code/actors.py
:language: python
:start-after: __cancel_start__
:end-before: __cancel_end__
In Ray, Task cancellation behavior is contingent on the Task's current state:
**Unscheduled tasks**:
If Ray hasn't scheduled an Actor Task yet, Ray attempts to cancel the scheduling.
When Ray successfully cancels at this stage, it invokes ``ray.get(actor_task_ref)``
which produces a :class:`TaskCancelledError <ray.exceptions.TaskCancelledError>`.
**Running actor tasks (regular actor, threaded actor)**:
For tasks classified as a single-threaded Actor or a multi-threaded Actor,
Ray sets a cancellation flag that can be checked via ``ray.get_runtime_context().is_canceled()``.
This allows for graceful cancellation by periodically checking the cancellation status within the task.
**Running async actor tasks**:
For Tasks classified as :ref:`async Actors <async-actors>`, Ray seeks to cancel the associated `asyncio.Task`.
This cancellation approach aligns with the standards presented in
`asyncio task cancellation <https://docs.python.org/3/library/asyncio-task.html#task-cancellation>`__.
Note that `asyncio.Task` won't be interrupted in the middle of execution if you don't `await` within the async function.
Note: ``ray.get_runtime_context().is_canceled()`` is not supported for async actors and will raise a ``RuntimeError``.
**Cancellation guarantee**:
Ray attempts to cancel Tasks on a *best-effort* basis, meaning cancellation isn't always guaranteed.
For example, if the cancellation request doesn't get through to the executor,
the Task might not be cancelled.
You can check if a Task was successfully cancelled using ``ray.get(actor_task_ref)``.
**Recursive cancellation**:
Ray tracks all child and Actor Tasks. When the ``recursive=True`` argument is given,
it cancels all child and Actor Tasks.
Detecting cancellation in running actor tasks
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For non-async actor tasks, you can periodically check whether a cancellation has been requested
by calling ``ray.get_runtime_context().is_canceled()``. This allows tasks to detect cancellation
and perform cleanup operations before exiting gracefully.
.. tab-set::
.. tab-item:: Python
.. literalinclude:: doc_code/actors.py
:language: python
:start-after: __cancel_graceful_actor_start__
:end-before: __cancel_graceful_actor_end__
**Important notes:**
- For **non-async actor tasks**, direct interruption is not supported. You need to check ``is_canceled()`` periodically to detect cancellation requests.
- ``is_canceled()`` is **not supported** for async actor tasks and will raise a ``RuntimeError``.
Scheduling
----------
For each actor, Ray chooses a node to run it on,
and bases the scheduling decision on a few factors like
:ref:`the actor's resource requirements <ray-scheduling-resources>`
and :ref:`the specified scheduling strategy <ray-scheduling-strategies>`.
See :ref:`Ray scheduling <ray-scheduling>` for more details.
Fault Tolerance
---------------
By default, Ray actors won't be :ref:`restarted <fault-tolerance-actors>` and
actor tasks won't be retried when actors crash unexpectedly.
You can change this behavior by setting
``max_restarts`` and ``max_task_retries`` options
in :func:`ray.remote() <ray.remote>` and :meth:`.options() <ray.actor.ActorClass.options>`.
See :ref:`Ray fault tolerance <fault-tolerance>` for more details.
FAQ: Actors, Workers and Resources
----------------------------------
What's the difference between a worker and an actor?
Each "Ray worker" is a python process.
Ray treats a worker differently for tasks and actors. For tasks, Ray uses a "Ray worker" to execute multiple Ray tasks. For actors, Ray starts a "Ray worker" as a dedicated Ray actor.
* **Tasks**: When Ray starts on a machine, a number of Ray workers start automatically (1 per CPU by default). Ray uses them to execute tasks (like a process pool). If you execute 8 tasks with `num_cpus=2`, and total number of CPUs is 16 (`ray.cluster_resources()["CPU"] == 16`), you end up with 8 of your 16 workers idling.
* **Actor**: A Ray Actor is also a "Ray worker" but you instantiate it at runtime with `actor_cls.remote()`. All of its methods run on the same process, using the same resources Ray designates when you define the Actor. Note that unlike tasks, Ray doesn't reuse the Python processes that run Ray Actors. Ray terminates them when you delete the Actor.
To maximally utilize your resources, you want to maximize the time that
your workers work. You also want to allocate enough cluster resources
so Ray can run all of your needed actors and any other tasks you
define. This also implies that Ray schedules tasks more flexibly,
and that if you don't need the stateful part of an actor, it's better
to use tasks.
Task Events
-----------
By default, Ray traces the execution of actor tasks, reporting task status events and profiling events
that Ray Dashboard and :ref:`State API <state-api-overview-ref>` use.
You can disable task event reporting for the actor by setting the `enable_task_events` option to `False` in :func:`ray.remote() <ray.remote>` and :meth:`.options() <ray.actor.ActorClass.options>`. This setting reduces the overhead of task execution by reducing the amount of data Ray sends to the Ray Dashboard.
You can also disable task event reporting for some actor methods by setting the `enable_task_events` option to `False` in :func:`ray.remote() <ray.remote>` and :meth:`.options() <ray.remote_function.RemoteFunction.options>` on the actor method.
Method settings override the actor setting:
.. literalinclude:: doc_code/actors.py
:language: python
:start-after: __enable_task_events_start__
:end-before: __enable_task_events_end__
More about Ray Actors
---------------------
.. toctree::
:maxdepth: 1
actors/named-actors.rst
actors/terminating-actors.rst
actors/async_api.rst
actors/concurrency_group_api.rst
actors/actor-utils.rst
actors/out-of-band-communication.rst
actors/task-orders.rst
doc/source/ray-core/actors/actor-utils.rst
+34
View File
@@ -0,0 +1,34 @@
Utility Classes
===============
Actor Pool
~~~~~~~~~~
.. tab-set::
.. tab-item:: Python
The ``ray.util`` module contains a utility class, ``ActorPool``.
This class is similar to multiprocessing.Pool and lets you schedule Ray tasks over a fixed pool of actors.
.. literalinclude:: ../doc_code/actor-pool.py
See the :class:`package reference <ray.util.ActorPool>` for more information.
.. tab-item:: Java
Actor pool hasn't been implemented in Java yet.
.. tab-item:: C++
Actor pool hasn't been implemented in C++ yet.
Message passing using Ray Queue
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sometimes just using one signal to synchronize is not enough. If you need to send data among many tasks or
actors, you can use :class:`ray.util.queue.Queue <ray.util.queue.Queue>`.
.. literalinclude:: ../doc_code/actor-queue.py
Ray's Queue API has a similar API to Python's ``asyncio.Queue`` and ``queue.Queue``.
doc/source/ray-core/actors/async_api.rst
+335
View File
@@ -0,0 +1,335 @@
AsyncIO / Concurrency for Actors
================================
Within a single actor process, it is possible to execute concurrent threads.
Ray offers two types of concurrency within an actor:
* :ref:`async execution <async-actors>`
* :ref:`threading <threaded-actors>`
Keep in mind that the Python's `Global Interpreter Lock (GIL) <https://wiki.python.org/moin/GlobalInterpreterLock>`_ will only allow one thread of Python code running at once.
This means if you are just parallelizing Python code, you won't get true parallelism. If you call Numpy, Cython, Tensorflow, or PyTorch code, these libraries will release the GIL when calling into C/C++ functions.
**Neither the** :ref:`threaded-actors` nor :ref:`async-actors` **model will allow you to bypass the GIL.**
.. _async-actors:
AsyncIO for Actors
------------------
Since Python 3.5, it is possible to write concurrent code using the
``async/await`` `syntax <https://docs.python.org/3/library/asyncio.html>`__.
Ray natively integrates with asyncio. You can use Ray alongside popular
async frameworks like aiohttp, aioredis, etc.
.. testcode::
import ray
import asyncio
@ray.remote
class AsyncActor:
def __init__(self, expected_num_tasks: int):
self._event = asyncio.Event()
self._curr_num_tasks = 0
self._expected_num_tasks = expected_num_tasks
# Multiple invocations of this method can run concurrently on the same event loop.
async def run_concurrent(self):
self._curr_num_tasks += 1
if self._curr_num_tasks == self._expected_num_tasks:
print("All coroutines are executing concurrently, unblocking.")
self._event.set()
else:
print("Waiting for other coroutines to start.")
await self._event.wait()
print("All coroutines ran concurrently.")
actor = AsyncActor.remote(4)
refs = [actor.run_concurrent.remote() for _ in range(4)]
# Fetch results using regular `ray.get`.
ray.get(refs)
# Fetch results using `asyncio` APIs.
async def get_async():
return await asyncio.gather(*refs)
asyncio.run(get_async())
.. testoutput::
:options: +MOCK
(AsyncActor pid=9064) Waiting for other coroutines to start.
(AsyncActor pid=9064) Waiting for other coroutines to start.
(AsyncActor pid=9064) Waiting for other coroutines to start.
(AsyncActor pid=9064) All coroutines are executing concurrently, unblocking.
(AsyncActor pid=9064) All coroutines ran concurrently.
(AsyncActor pid=9064) All coroutines ran concurrently.
(AsyncActor pid=9064) All coroutines ran concurrently.
(AsyncActor pid=9064) All coroutines ran concurrently.
.. testcode::
:hide:
# NOTE: The outputs from the previous code block can show up in subsequent tests.
# To prevent flakiness, we wait for a grace period.
import time
print("Sleeping...")
time.sleep(1)
.. testoutput::
...
ObjectRefs as asyncio.Futures
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ObjectRefs can be translated to asyncio.Futures. This feature
make it possible to ``await`` on ray futures in existing concurrent
applications.
Instead of:
.. testcode::
import ray
@ray.remote
def some_task():
return 1
ray.get(some_task.remote())
ray.wait([some_task.remote()])
you can wait on the ref with Python 3.9 and Python 3.10:
.. testcode::
import ray
import asyncio
@ray.remote
def some_task():
return 1
async def await_obj_ref():
await some_task.remote()
await asyncio.wait([some_task.remote()])
asyncio.run(await_obj_ref())
or the Future object directly with Python 3.11+:
.. testcode::
import asyncio
async def convert_to_asyncio_future():
ref = some_task.remote()
fut: asyncio.Future = asyncio.wrap_future(ref.future())
print(await fut)
asyncio.run(convert_to_asyncio_future())
.. testoutput::
1
See the `asyncio doc <https://docs.python.org/3/library/asyncio-task.html>`__
for more `asyncio` patterns including timeouts and ``asyncio.gather``.
.. _async-ref-to-futures:
ObjectRefs as concurrent.futures.Futures
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ObjectRefs can also be wrapped into ``concurrent.futures.Future`` objects. This
is useful for interfacing with existing ``concurrent.futures`` APIs:
.. testcode::
import concurrent
refs = [some_task.remote() for _ in range(4)]
futs = [ref.future() for ref in refs]
for fut in concurrent.futures.as_completed(futs):
assert fut.done()
print(fut.result())
.. testoutput::
1
1
1
1
Defining an Async Actor
~~~~~~~~~~~~~~~~~~~~~~~
By using `async` method definitions, Ray will automatically detect whether an actor support `async` calls or not.
.. testcode::
import ray
import asyncio
@ray.remote
class AsyncActor:
def __init__(self, expected_num_tasks: int):
self._event = asyncio.Event()
self._curr_num_tasks = 0
self._expected_num_tasks = expected_num_tasks
async def run_task(self):
print("Started task")
self._curr_num_tasks += 1
if self._curr_num_tasks == self._expected_num_tasks:
self._event.set()
else:
# Yield the event loop for multiple coroutines to run concurrently.
await self._event.wait()
print("Finished task")
actor = AsyncActor.remote(5)
# All 5 tasks will start at once and run concurrently.
ray.get([actor.run_task.remote() for _ in range(5)])
.. testoutput::
:options: +MOCK
(AsyncActor pid=3456) Started task
(AsyncActor pid=3456) Started task
(AsyncActor pid=3456) Started task
(AsyncActor pid=3456) Started task
(AsyncActor pid=3456) Started task
(AsyncActor pid=3456) Finished task
(AsyncActor pid=3456) Finished task
(AsyncActor pid=3456) Finished task
(AsyncActor pid=3456) Finished task
(AsyncActor pid=3456) Finished task
Under the hood, Ray runs all of the methods inside a single python event loop.
Please note that running blocking ``ray.get`` or ``ray.wait`` inside async
actor method is not allowed, because ``ray.get`` will block the execution
of the event loop.
In async actors, only one task can be running at any point in time (though tasks can be multiplexed). There will be only one thread in AsyncActor! See :ref:`threaded-actors` if you want a threadpool.
Setting concurrency in Async Actors
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can set the number of "concurrent" task running at once using the
``max_concurrency`` flag. By default, 1000 tasks can be running concurrently.
.. testcode::
import asyncio
import ray
@ray.remote
class AsyncActor:
def __init__(self, batch_size: int):
self._event = asyncio.Event()
self._curr_tasks = 0
self._batch_size = batch_size
async def run_task(self):
print("Started task")
self._curr_tasks += 1
if self._curr_tasks == self._batch_size:
self._event.set()
else:
await self._event.wait()
self._event.clear()
self._curr_tasks = 0
print("Finished task")
actor = AsyncActor.options(max_concurrency=2).remote(2)
# Only 2 tasks will run concurrently.
# Once 2 finish, the next 2 should run.
ray.get([actor.run_task.remote() for _ in range(8)])
.. testoutput::
:options: +MOCK
(AsyncActor pid=5859) Started task
(AsyncActor pid=5859) Started task
(AsyncActor pid=5859) Finished task
(AsyncActor pid=5859) Finished task
(AsyncActor pid=5859) Started task
(AsyncActor pid=5859) Started task
(AsyncActor pid=5859) Finished task
(AsyncActor pid=5859) Finished task
(AsyncActor pid=5859) Started task
(AsyncActor pid=5859) Started task
(AsyncActor pid=5859) Finished task
(AsyncActor pid=5859) Finished task
(AsyncActor pid=5859) Started task
(AsyncActor pid=5859) Started task
(AsyncActor pid=5859) Finished task
(AsyncActor pid=5859) Finished task
.. _threaded-actors:
Threaded Actors
---------------
Sometimes, asyncio is not an ideal solution for your actor. For example, you may
have one method that performs some computation heavy task while blocking the event loop, not giving up control via ``await``. This would hurt the performance of an Async Actor because Async Actors can only execute 1 task at a time and rely on ``await`` to context switch.
Instead, you can use the ``max_concurrency`` Actor options without any async methods, allowing you to achieve threaded concurrency (like a thread pool).
.. warning::
When there is at least one ``async def`` method in actor definition, Ray
will recognize the actor as AsyncActor instead of ThreadedActor.
.. testcode::
@ray.remote
class ThreadedActor:
def task_1(self): print("I'm running in a thread!")
def task_2(self): print("I'm running in another thread!")
a = ThreadedActor.options(max_concurrency=2).remote()
ray.get([a.task_1.remote(), a.task_2.remote()])
.. testoutput::
:options: +MOCK
(ThreadedActor pid=4822) I'm running in a thread!
(ThreadedActor pid=4822) I'm running in another thread!
Each invocation of the threaded actor will be running in a thread pool. The size of the threadpool is limited by the ``max_concurrency`` value.
AsyncIO for Remote Tasks
------------------------
We don't support asyncio for remote tasks. The following snippet will fail:
.. testcode::
:skipif: True
@ray.remote
async def f():
pass
Instead, you can wrap the ``async`` function with a wrapper to run the task synchronously:
.. testcode::
async def f():
pass
@ray.remote
def wrapper():
import asyncio
asyncio.run(f())
doc/source/ray-core/actors/concurrency_group_api.rst
+199
View File
@@ -0,0 +1,199 @@
Limiting Concurrency Per-Method with Concurrency Groups
=======================================================
Besides setting the max concurrency overall for an actor, Ray allows methods to be separated into *concurrency groups*, each with its own thread(s). This allows you to limit the concurrency per-method, e.g., allow a health-check method to be given its own concurrency quota separate from request serving methods.
.. tip:: Concurrency groups work with both asyncio and threaded actors. The syntax is the same.
.. _defining-concurrency-groups:
Defining Concurrency Groups
---------------------------
This defines two concurrency groups, "io" with max concurrency = 2 and
"compute" with max concurrency = 4. The methods ``f1`` and ``f2`` are
placed in the "io" group, and the methods ``f3`` and ``f4`` are placed
into the "compute" group. Note that there is always a default
concurrency group for actors, which has a default concurrency of 1000 for
AsyncIO actors and 1 otherwise.
.. tab-set::
.. tab-item:: Python
You can define concurrency groups for actors using the ``concurrency_group`` decorator argument:
.. testcode::
import ray
@ray.remote(concurrency_groups={"io": 2, "compute": 4})
class AsyncIOActor:
def __init__(self):
pass
@ray.method(concurrency_group="io")
async def f1(self):
pass
@ray.method(concurrency_group="io")
async def f2(self):
pass
@ray.method(concurrency_group="compute")
async def f3(self):
pass
@ray.method(concurrency_group="compute")
async def f4(self):
pass
async def f5(self):
pass
a = AsyncIOActor.remote()
a.f1.remote() # executed in the "io" group.
a.f2.remote() # executed in the "io" group.
a.f3.remote() # executed in the "compute" group.
a.f4.remote() # executed in the "compute" group.
a.f5.remote() # executed in the default group.
.. tab-item:: Java
You can define concurrency groups for concurrent actors using the API ``setConcurrencyGroups()`` argument:
.. code-block:: java
class ConcurrentActor {
public long f1() {
return Thread.currentThread().getId();
}
public long f2() {
return Thread.currentThread().getId();
}
public long f3(int a, int b) {
return Thread.currentThread().getId();
}
public long f4() {
return Thread.currentThread().getId();
}
public long f5() {
return Thread.currentThread().getId();
}
}
ConcurrencyGroup group1 =
new ConcurrencyGroupBuilder<ConcurrentActor>()
.setName("io")
.setMaxConcurrency(1)
.addMethod(ConcurrentActor::f1)
.addMethod(ConcurrentActor::f2)
.build();
ConcurrencyGroup group2 =
new ConcurrencyGroupBuilder<ConcurrentActor>()
.setName("compute")
.setMaxConcurrency(1)
.addMethod(ConcurrentActor::f3)
.addMethod(ConcurrentActor::f4)
.build();
ActorHandle<ConcurrentActor> myActor = Ray.actor(ConcurrentActor::new)
.setConcurrencyGroups(group1, group2)
.remote();
myActor.task(ConcurrentActor::f1).remote(); // executed in the "io" group.
myActor.task(ConcurrentActor::f2).remote(); // executed in the "io" group.
myActor.task(ConcurrentActor::f3, 3, 5).remote(); // executed in the "compute" group.
myActor.task(ConcurrentActor::f4).remote(); // executed in the "compute" group.
myActor.task(ConcurrentActor::f5).remote(); // executed in the "default" group.
.. _default-concurrency-group:
Default Concurrency Group
-------------------------
By default, methods are placed in a default concurrency group which has a concurrency limit of 1000 for AsyncIO actors and 1 otherwise.
The concurrency of the default group can be changed by setting the ``max_concurrency`` actor option.
.. tab-set::
.. tab-item:: Python
The following actor has 2 concurrency groups: "io" and "default".
The max concurrency of "io" is 2, and the max concurrency of "default" is 10.
.. testcode::
@ray.remote(concurrency_groups={"io": 2})
class AsyncIOActor:
async def f1(self):
pass
actor = AsyncIOActor.options(max_concurrency=10).remote()
.. tab-item:: Java
The following concurrent actor has 2 concurrency groups: "io" and "default".
The max concurrency of "io" is 2, and the max concurrency of "default" is 10.
.. code-block:: java
class ConcurrentActor {
public long f1() {
return Thread.currentThread().getId();
}
}
ConcurrencyGroup group =
new ConcurrencyGroupBuilder<ConcurrentActor>()
.setName("io")
.setMaxConcurrency(2)
.addMethod(ConcurrentActor::f1)
.build();
ActorHandle<ConcurrentActor> myActor = Ray.actor(ConcurrentActor::new)
.setConcurrencyGroups(group)
.setMaxConcurrency(10)
.remote();
.. _setting-the-concurrency-group-at-runtime:
Setting the Concurrency Group at Runtime
----------------------------------------
You can also dispatch actor methods into a specific concurrency group at runtime.
The following snippet demonstrates setting the concurrency group of the
``f2`` method dynamically at runtime.
.. tab-set::
.. tab-item:: Python
You can use the ``.options`` method.
.. testcode::
# Executed in the "io" group (as defined in the actor class).
a.f2.options().remote()
# Executed in the "compute" group.
a.f2.options(concurrency_group="compute").remote()
.. tab-item:: Java
You can use ``setConcurrencyGroup`` method.
.. code-block:: java
// Executed in the "io" group (as defined in the actor creation).
myActor.task(ConcurrentActor::f2).remote();
// Executed in the "compute" group.
myActor.task(ConcurrentActor::f2).setConcurrencyGroup("compute").remote();
doc/source/ray-core/actors/named-actors.rst
+218
View File
@@ -0,0 +1,218 @@
Named Actors
============
An actor can be given a unique name within their :ref:`namespace <namespaces-guide>`.
This allows you to retrieve the actor from any job in the Ray cluster.
This can be useful if you cannot directly
pass the actor handle to the task that needs it, or if you are trying to
access an actor launched by another driver.
Note that the actor will still be garbage-collected if no handles to it
exist. See :ref:`actor-lifetimes` for more details.
.. tab-set::
.. tab-item:: Python
.. testcode::
import ray
@ray.remote
class Counter:
pass
# Create an actor with a name
counter = Counter.options(name="some_name").remote()
# Retrieve the actor later somewhere
counter = ray.get_actor("some_name")
.. tab-item:: Java
.. code-block:: java
// Create an actor with a name.
ActorHandle<Counter> counter = Ray.actor(Counter::new).setName("some_name").remote();
...
// Retrieve the actor later somewhere
Optional<ActorHandle<Counter>> counter = Ray.getActor("some_name");
Assert.assertTrue(counter.isPresent());
.. tab-item:: C++
.. code-block:: c++
// Create an actor with a globally unique name
ActorHandle<Counter> counter = ray::Actor(CreateCounter).SetGlobalName("some_name").Remote();
...
// Retrieve the actor later somewhere
boost::optional<ray::ActorHandle<Counter>> counter = ray::GetGlobalActor("some_name");
We also support non-global named actors in C++, which means that the actor name is only valid within the job and the actor cannot be accessed from another job.
.. code-block:: c++
// Create an actor with a job-scope-unique name
ActorHandle<Counter> counter = ray::Actor(CreateCounter).SetName("some_name").Remote();
...
// Retrieve the actor later somewhere in the same job
boost::optional<ray::ActorHandle<Counter>> counter = ray::GetActor("some_name");
.. note::
Named actors are scoped by namespace. If no namespace is assigned, they will
be placed in an anonymous namespace by default.
.. tab-set::
.. tab-item:: Python
.. testcode::
:skipif: True
import ray
@ray.remote
class Actor:
pass
# driver_1.py
# Job 1 creates an actor, "orange" in the "colors" namespace.
ray.init(address="auto", namespace="colors")
Actor.options(name="orange", lifetime="detached").remote()
# driver_2.py
# Job 2 is now connecting to a different namespace.
ray.init(address="auto", namespace="fruit")
# This fails because "orange" was defined in the "colors" namespace.
ray.get_actor("orange")
# You can also specify the namespace explicitly.
ray.get_actor("orange", namespace="colors")
# driver_3.py
# Job 3 connects to the original "colors" namespace
ray.init(address="auto", namespace="colors")
# This returns the "orange" actor we created in the first job.
ray.get_actor("orange")
.. tab-item:: Java
.. code-block:: java
import ray
class Actor {
}
// Driver1.java
// Job 1 creates an actor, "orange" in the "colors" namespace.
System.setProperty("ray.job.namespace", "colors");
Ray.init();
Ray.actor(Actor::new).setName("orange").remote();
// Driver2.java
// Job 2 is now connecting to a different namespace.
System.setProperty("ray.job.namespace", "fruits");
Ray.init();
// This fails because "orange" was defined in the "colors" namespace.
Optional<ActorHandle<Actor>> actor = Ray.getActor("orange");
Assert.assertFalse(actor.isPresent()); // actor.isPresent() is false.
// Driver3.java
System.setProperty("ray.job.namespace", "colors");
Ray.init();
// This returns the "orange" actor we created in the first job.
Optional<ActorHandle<Actor>> actor = Ray.getActor("orange");
Assert.assertTrue(actor.isPresent()); // actor.isPresent() is true.
Get-Or-Create a Named Actor
---------------------------
A common use case is to create an actor only if it doesn't exist.
Ray provides a ``get_if_exists`` option for actor creation that does this out of the box.
This method is available after you set a name for the actor via ``.options()``.
If the actor already exists, a handle to the actor will be returned
and the arguments will be ignored. Otherwise, a new actor will be
created with the specified arguments.
.. tab-set::
.. tab-item:: Python
.. literalinclude:: ../doc_code/get_or_create.py
.. tab-item:: Java
.. code-block:: java
// This feature is not yet available in Java.
.. tab-item:: C++
.. code-block:: c++
// This feature is not yet available in C++.
.. _actor-lifetimes:
Actor Lifetimes
---------------
Separately, actor lifetimes can be decoupled from the job, allowing an actor to persist even after the driver process of the job exits. We call these actors *detached*.
.. tab-set::
.. tab-item:: Python
.. testcode::
counter = Counter.options(name="CounterActor", lifetime="detached").remote()
The ``CounterActor`` will be kept alive even after the driver running above script
exits. Therefore it is possible to run the following script in a different
driver:
.. testcode::
counter = ray.get_actor("CounterActor")
Note that an actor can be named but not detached. If we only specified the
name without specifying ``lifetime="detached"``, then the CounterActor can
only be retrieved as long as the original driver is still running.
.. tab-item:: Java
.. code-block:: java
System.setProperty("ray.job.namespace", "lifetime");
Ray.init();
ActorHandle<Counter> counter = Ray.actor(Counter::new).setName("some_name").setLifetime(ActorLifetime.DETACHED).remote();
The CounterActor will be kept alive even after the driver running above process
exits. Therefore it is possible to run the following code in a different
driver:
.. code-block:: java
System.setProperty("ray.job.namespace", "lifetime");
Ray.init();
Optional<ActorHandle<Counter>> counter = Ray.getActor("some_name");
Assert.assertTrue(counter.isPresent());
.. tab-item:: C++
Customizing lifetime of an actor hasn't been implemented in C++ yet.
Unlike normal actors, detached actors are not automatically garbage-collected by Ray.
Detached actors must be manually destroyed once you are sure that they are no
longer needed. To do this, use ``ray.kill`` to :ref:`manually terminate <ray-kill-actors>` the actor.
After this call, the actor's name may be reused.
doc/source/ray-core/actors/out-of-band-communication.rst
+36
View File
@@ -0,0 +1,36 @@
Out-of-band Communication
=========================
Typically, Ray actor communication is done through actor method calls and data is shared through the distributed object store.
However, in some use cases out-of-band communication can be useful.
Wrapping Library Processes
--------------------------
Many libraries already have mature, high-performance internal communication stacks and
they leverage Ray as a language-integrated actor scheduler.
The actual communication between actors is mostly done out-of-band using existing communication stacks.
For example, Horovod-on-Ray uses NCCL or MPI-based collective communications, and RayDP uses Spark's internal RPC and object manager.
See `Ray Distributed Library Patterns <https://www.anyscale.com/blog/ray-distributed-library-patterns>`_ for more details.
Ray Collective
--------------
Ray's collective communication library (\ ``ray.util.collective``\ ) allows efficient out-of-band collective and point-to-point communication between distributed CPUs or GPUs.
See :ref:`Ray Collective <ray-collective>` for more details.
HTTP Server
-----------
You can start an HTTP server inside the actor and expose HTTP endpoints to clients
so users outside of the Ray cluster can communicate with the actor.
.. tab-set::
.. tab-item:: Python
.. literalinclude:: ../doc_code/actor-http-server.py
Similarly, you can expose other types of servers as well (e.g., gRPC servers).
Limitations
-----------
When using out-of-band communication with Ray actors, keep in mind that Ray does not manage the calls between actors. This means that functionality like distributed reference counting will not work with out-of-band communication, so you should take care not to pass object references in this way.
doc/source/ray-core/actors/task-orders.rst
+149
View File
@@ -0,0 +1,149 @@
.. _actor-task-order:
Actor Task Execution Order
==========================
Synchronous, Single-Threaded Actor
----------------------------------
In Ray, an actor receives tasks from multiple submitters (including driver and workers).
For tasks received from the same submitter, a synchronous, single-threaded actor executes
them in the order they were submitted, unless you set ``allow_out_of_order_execution``,
or Ray retries tasks. In other words, a given task will not be executed until previously
submitted tasks from the same submitter have finished execution.
For actors where `max_task_retries` is set to a non-zero number, the task
execution order is not guaranteed when task retries occur.
.. tab-set::
.. tab-item:: Python
.. testcode::
import ray
@ray.remote
class Counter:
def __init__(self):
self.value = 0
def add(self, addition):
self.value += addition
return self.value
counter = Counter.remote()
# For tasks from the same submitter,
# they are executed according to submission order.
value0 = counter.add.remote(1)
value1 = counter.add.remote(2)
# Output: 1. The first submitted task is executed first.
print(ray.get(value0))
# Output: 3. The later submitted task is executed later.
print(ray.get(value1))
.. testoutput::
1
3
However, the actor does not guarantee the execution order of the tasks from different
submitters. For example, suppose an unfulfilled argument blocks a previously submitted
task. In this case, the actor can still execute tasks submitted by a different worker.
.. tab-set::
.. tab-item:: Python
.. testcode::
import time
import ray
@ray.remote
class Counter:
def __init__(self):
self.value = 0
def add(self, addition):
self.value += addition
return self.value
counter = Counter.remote()
# Submit task from a worker
@ray.remote
def submitter(value):
return ray.get(counter.add.remote(value))
# Simulate delayed result resolution.
@ray.remote
def delayed_resolution(value):
time.sleep(1)
return value
# Submit tasks from different workers, with
# the first submitted task waiting for
# dependency resolution.
value0 = submitter.remote(delayed_resolution.remote(1))
value1 = submitter.remote(2)
# Output: 3. The first submitted task is executed later.
print(ray.get(value0))
# Output: 2. The later submitted task is executed first.
print(ray.get(value1))
.. testoutput::
3
2
Asynchronous or Threaded Actor
------------------------------
:ref:`Asynchronous or threaded actors <async-actors>` do not guarantee the
task execution order. This means the system might execute a task
even though previously submitted tasks are pending execution.
.. tab-set::
.. tab-item:: Python
.. testcode::
import time
import ray
@ray.remote
class AsyncCounter:
def __init__(self):
self.value = 0
async def add(self, addition):
self.value += addition
return self.value
counter = AsyncCounter.remote()
# Simulate delayed result resolution.
@ray.remote
def delayed_resolution(value):
time.sleep(1)
return value
# Submit tasks from the driver, with
# the first submitted task waiting for
# dependency resolution.
value0 = counter.add.remote(delayed_resolution.remote(1))
value1 = counter.add.remote(2)
# Output: 3. The first submitted task is executed later.
print(ray.get(value0))
# Output: 2. The later submitted task is executed first.
print(ray.get(value1))
.. testoutput::
3
2
doc/source/ray-core/actors/terminating-actors.rst
+250
View File
@@ -0,0 +1,250 @@
Terminating Actors
==================
Actor processes will be terminated automatically when all copies of the
actor handle have gone out of scope in Python, or if the original creator
process dies. When actors terminate gracefully, Ray calls the actor's
``__ray_shutdown__()`` method if defined, allowing for cleanup of resources
(see :ref:`actor-cleanup` for details).
Note that automatic termination of actors is not yet supported in Java or C++.
.. _ray-kill-actors:
Manual termination via an actor handle
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In most cases, Ray will automatically terminate actors that have gone out of
scope, but you may sometimes need to terminate an actor forcefully. This should
be reserved for cases where an actor is unexpectedly hanging or leaking
resources, and for :ref:`detached actors <actor-lifetimes>`, which must be
manually destroyed.
.. tab-set::
.. tab-item:: Python
.. testcode::
import ray
@ray.remote
class Actor:
pass
actor_handle = Actor.remote()
ray.kill(actor_handle)
# Force kill: the actor exits immediately without cleanup.
# This will NOT call __ray_shutdown__() or atexit handlers.
.. tab-item:: Java
.. code-block:: java
actorHandle.kill();
// This will not go through the normal Java System.exit teardown logic, so any
// shutdown hooks installed in the actor using ``Runtime.addShutdownHook(...)`` will
// not be called.
.. tab-item:: C++
.. code-block:: c++
actor_handle.Kill();
// This will not go through the normal C++ std::exit
// teardown logic, so any exit handlers installed in
// the actor using ``std::atexit`` will not be called.
This will cause the actor to immediately exit its process, causing any current,
pending, and future tasks to fail with a ``RayActorError``. If you would like
Ray to :ref:`automatically restart <fault-tolerance-actors>` the actor, make sure to set a nonzero
``max_restarts`` in the ``@ray.remote`` options for the actor, then pass the
flag ``no_restart=False`` to ``ray.kill``.
For :ref:`named and detached actors <actor-lifetimes>`, calling ``ray.kill`` on
an actor handle destroys the actor and allows the name to be reused.
Use `ray list actors --detail` from :ref:`State API <state-api-overview-ref>` to see the death cause of dead actors:
.. code-block:: bash
# This API is only available when you download Ray via `pip install "ray[default]"`
ray list actors --detail
.. code-block:: bash
---
- actor_id: e8702085880657b355bf7ef001000000
class_name: Actor
state: DEAD
job_id: '01000000'
name: ''
node_id: null
pid: 0
ray_namespace: dbab546b-7ce5-4cbb-96f1-d0f64588ae60
serialized_runtime_env: '{}'
required_resources: {}
death_cause:
actor_died_error_context: # <---- You could see the error message w.r.t why the actor exits.
error_message: The actor is dead because `ray.kill` killed it.
owner_id: 01000000ffffffffffffffffffffffffffffffffffffffffffffffff
owner_ip_address: 127.0.0.1
ray_namespace: dbab546b-7ce5-4cbb-96f1-d0f64588ae60
class_name: Actor
actor_id: e8702085880657b355bf7ef001000000
never_started: true
node_ip_address: ''
pid: 0
name: ''
is_detached: false
placement_group_id: null
repr_name: ''
Manual termination within the actor
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If necessary, you can manually terminate an actor from within one of the actor methods.
This will kill the actor process and release resources associated/assigned to the actor.
.. tab-set::
.. tab-item:: Python
.. testcode::
@ray.remote
class Actor:
def exit(self):
ray.actor.exit_actor()
actor = Actor.remote()
actor.exit.remote()
This approach should generally not be necessary as actors are automatically garbage
collected. The ``ObjectRef`` resulting from the task can be waited on to wait
for the actor to exit (calling ``ray.get()`` on it will raise a ``RayActorError``).
.. tab-item:: Java
.. code-block:: java
Ray.exitActor();
Garbage collection for actors hasn't been implemented yet, so this is currently the
only way to terminate an actor gracefully. The ``ObjectRef`` resulting from the task
can be waited on to wait for the actor to exit (calling ``ObjectRef::get`` on it will
throw a ``RayActorException``).
.. tab-item:: C++
.. code-block:: c++
ray::ExitActor();
Garbage collection for actors hasn't been implemented yet, so this is currently the
only way to terminate an actor gracefully. The ``ObjectRef`` resulting from the task
can be waited on to wait for the actor to exit (calling ``ObjectRef::Get`` on it will
throw a ``RayActorException``).
Note that this method of termination waits until any previously submitted
tasks finish executing and then exits the process gracefully with sys.exit.
You could see the actor is dead as a result of the user's `exit_actor()` call:
.. code-block:: bash
# This API is only available when you download Ray via `pip install "ray[default]"`
ray list actors --detail
.. code-block:: bash
---
- actor_id: 070eb5f0c9194b851bb1cf1602000000
class_name: Actor
state: DEAD
job_id: '02000000'
name: ''
node_id: 47ccba54e3ea71bac244c015d680e202f187fbbd2f60066174a11ced
pid: 47978
ray_namespace: 18898403-dda0-485a-9c11-e9f94dffcbed
serialized_runtime_env: '{}'
required_resources: {}
death_cause:
actor_died_error_context:
error_message: 'The actor is dead because its worker process has died.
Worker exit type: INTENDED_USER_EXIT Worker exit detail: Worker exits
by a user request. exit_actor() is called.'
owner_id: 02000000ffffffffffffffffffffffffffffffffffffffffffffffff
owner_ip_address: 127.0.0.1
node_ip_address: 127.0.0.1
pid: 47978
ray_namespace: 18898403-dda0-485a-9c11-e9f94dffcbed
class_name: Actor
actor_id: 070eb5f0c9194b851bb1cf1602000000
name: ''
never_started: false
is_detached: false
placement_group_id: null
repr_name: ''
.. _actor-cleanup:
Actor cleanup with `__ray_shutdown__`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When an actor terminates gracefully, Ray calls the ``__ray_shutdown__()`` method
if it exists, allowing cleanup of resources like database connections or file handles.
.. tab-set::
.. tab-item:: Python
.. testcode::
import ray
import tempfile
import os
@ray.remote
class FileProcessorActor:
def __init__(self):
self.temp_file = tempfile.NamedTemporaryFile(delete=False)
self.temp_file.write(b"processing data")
self.temp_file.flush()
def __ray_shutdown__(self):
# Clean up temporary file
if hasattr(self, 'temp_file'):
self.temp_file.close()
os.unlink(self.temp_file.name)
def process(self):
return "done"
actor = FileProcessorActor.remote()
ray.get(actor.process.remote())
del actor # __ray_shutdown__() is called automatically
When ``__ray_shutdown__()`` is called:
- **Automatic termination**: When all actor handles go out of scope (``del actor`` or natural scope exit)
- **Manual graceful termination**: When you call ``actor.__ray_terminate__.remote()``
When ``__ray_shutdown__()`` is **NOT** called:
- **Force kill**: When you use ``ray.kill(actor)`` - the actor is killed immediately without cleanup.
- **Unexpected termination**: When the actor process crashes or exits unexpectedly (such as a segfault or being killed by the OOM killer).
**Important notes:**
- ``__ray_shutdown__()`` runs after all actor tasks complete.
- By default, Ray waits 30 seconds for the graceful shutdown procedure (including ``__ray_shutdown__()``) to complete. If the actor doesn't exit within this timeout, it's force killed. Configure this with ``ray.init(_system_config={"actor_graceful_shutdown_timeout_ms": 60000})``.
- Exceptions in ``__ray_shutdown__()`` are caught and logged but don't prevent actor termination.
- ``__ray_shutdown__()`` must be a synchronous method, including for async actors.
doc/source/ray-core/advanced-topics.rst
+20
View File
@@ -0,0 +1,20 @@
Advanced topics
===============
This section covers extended topics on how to use Ray.
.. toctree::
:maxdepth: -1
tips-for-first-time
type-hint
starting-ray
ray-generator
namespaces
cross-language
using-ray-with-jupyter
ray-dag
miscellaneous
runtime_env_auth
user-spawn-processes
head-node-memory-management
doc/source/ray-core/api/cli.rst
+53
View File
@@ -0,0 +1,53 @@
Ray Core CLI
============
.. _ray-cli:
Debugging applications
----------------------
This section contains commands for inspecting and debugging the current cluster.
.. _ray-stack-doc:
.. click:: ray.scripts.scripts:stack
:prog: ray stack
:show-nested:
.. _ray-memory-doc:
.. click:: ray.scripts.scripts:memory
:prog: ray memory
:show-nested:
.. _ray-timeline-doc:
.. click:: ray.scripts.scripts:timeline
:prog: ray timeline
:show-nested:
.. _ray-status-doc:
.. click:: ray.scripts.scripts:status
:prog: ray status
:show-nested:
.. click:: ray.scripts.scripts:debug
:prog: ray debug
:show-nested:
Usage Stats
-----------
This section contains commands to enable/disable :ref:`Ray usage stats <ref-usage-stats>`.
.. _ray-disable-usage-stats-doc:
.. click:: ray.scripts.scripts:disable_usage_stats
:prog: ray disable-usage-stats
:show-nested:
.. _ray-enable-usage-stats-doc:
.. click:: ray.scripts.scripts:enable_usage_stats
:prog: ray enable-usage-stats
:show-nested:
doc/source/ray-core/api/core.rst
+75
View File
@@ -0,0 +1,75 @@
Core API
========
.. autosummary::
:nosignatures:
:toctree: doc/
ray.init
ray.shutdown
ray.is_initialized
ray.job_config.JobConfig
ray.LoggingConfig
Tasks
-----
.. autosummary::
:nosignatures:
:toctree: doc/
ray.remote
ray.remote_function.RemoteFunction.options
ray.cancel
Actors
------
.. autosummary::
:nosignatures:
:toctree: doc/
ray.remote
ray.actor.ActorClass
ray.actor.ActorClass.options
ray.actor.ActorMethod
ray.actor.ActorHandle
ray.actor.ActorClassInheritanceException
ray.actor.exit_actor
ray.method
ray.get_actor
ray.kill
Objects
-------
.. autosummary::
:nosignatures:
:toctree: doc/
ray.get
ray.wait
ray.put
ray.util.as_completed
ray.util.map_unordered
.. _runtime-context-apis:
Runtime Context
---------------
.. autosummary::
:nosignatures:
:toctree: doc/
ray.runtime_context.get_runtime_context
ray.runtime_context.RuntimeContext
ray.get_gpu_ids
Cross Language
--------------
.. autosummary::
:nosignatures:
:toctree: doc/
ray.cross_language.java_function
ray.cross_language.java_actor_class
doc/source/ray-core/api/direct-transport.rst
+43
View File
@@ -0,0 +1,43 @@
Ray Direct Transport (RDT) API
==============================
Usage with Core APIs
--------------------
Enable RDT for actor tasks with the :func:`@ray.method <ray.method>` decorator, or pass `_tensor_transport` to :func:`ray.put`. You can then pass the resulting `ray.ObjectRef` to other actor tasks, or use :func:`ray.get` to retrieve the result. See :ref:`Ray Direct Transport (RDT) <direct-transport>` for more details on usage.
.. autosummary::
:nosignatures:
:toctree: doc/
ray.method
ray.put
ray.get
Collective tensor transports
----------------------------
Collective tensor transports require a collective group to be created before RDT objects can be used. Use these methods to create and manage collective groups for the `gloo` and `nccl` tensor transports.
.. autosummary::
:nosignatures:
:toctree: doc/
ray.experimental.collective.create_collective_group
ray.experimental.collective.get_collective_groups
ray.experimental.collective.destroy_collective_group
Advanced APIs
-------------
.. autosummary::
:nosignatures:
:toctree: doc/
ray.experimental.register_nixl_memory
ray.experimental.deregister_nixl_memory
ray.experimental.register_nixl_memory_pool
ray.experimental.set_target_for_ref
ray.experimental.wait_tensor_freed
ray.experimental.register_tensor_transport
ray.experimental.TensorTransportManager
doc/source/ray-core/api/exceptions.rst
+42
View File
@@ -0,0 +1,42 @@
.. _ray-core-exceptions:
Exceptions
==========
.. autosummary::
:nosignatures:
:toctree: doc/
ray.exceptions.RayError
ray.exceptions.RayTaskError
ray.exceptions.RayActorError
ray.exceptions.TaskCancelledError
ray.exceptions.TaskUnschedulableError
ray.exceptions.ActorDiedError
ray.exceptions.ActorUnschedulableError
ray.exceptions.ActorUnavailableError
ray.exceptions.AsyncioActorExit
ray.exceptions.LocalRayletDiedError
ray.exceptions.WorkerCrashedError
ray.exceptions.TaskPlacementGroupRemoved
ray.exceptions.ActorPlacementGroupRemoved
ray.exceptions.ObjectStoreFullError
ray.exceptions.OutOfDiskError
ray.exceptions.OutOfMemoryError
ray.exceptions.ObjectLostError
ray.exceptions.ObjectFetchTimedOutError
ray.exceptions.GetTimeoutError
ray.exceptions.OwnerDiedError
ray.exceptions.PendingCallsLimitExceeded
ray.exceptions.PlasmaObjectNotAvailable
ray.exceptions.ObjectReconstructionFailedError
ray.exceptions.RayChannelError
ray.exceptions.RayChannelTimeoutError
ray.exceptions.RayCgraphCapacityExceeded
ray.exceptions.RayDirectTransportError
ray.exceptions.RuntimeEnvSetupError
ray.exceptions.CrossLanguageError
ray.exceptions.RaySystemError
ray.exceptions.NodeDiedError
ray.exceptions.UnserializableException
ray.exceptions.AuthenticationError
doc/source/ray-core/api/index.rst
+15
View File
@@ -0,0 +1,15 @@
Ray Core API
============
.. toctree::
:maxdepth: 2
core.rst
scheduling.rst
runtime-env.rst
utility.rst
exceptions.rst
cli.rst
../../ray-observability/reference/cli.rst
../../ray-observability/reference/api.rst
direct-transport.rst
doc/source/ray-core/api/runtime-env.rst
+9
View File
@@ -0,0 +1,9 @@
Runtime Env API
===============
.. autosummary::
:nosignatures:
:toctree: doc/
ray.runtime_env.RuntimeEnvConfig
ray.runtime_env.RuntimeEnv
doc/source/ray-core/api/scheduling.rst
+28
View File
@@ -0,0 +1,28 @@
Scheduling API
==============
Scheduling Strategy
-------------------
.. autosummary::
:nosignatures:
:toctree: doc/
ray.util.scheduling_strategies.PlacementGroupSchedulingStrategy
ray.util.scheduling_strategies.NodeAffinitySchedulingStrategy
.. _ray-placement-group-ref:
Placement Group
---------------
.. autosummary::
:nosignatures:
:toctree: doc/
ray.util.placement_group
ray.util.placement_group.get_placement_group
ray.util.placement_group.PlacementGroup
ray.util.placement_group_table
ray.util.remove_placement_group
ray.util.get_current_placement_group
doc/source/ray-core/api/utility.rst
+62
View File
@@ -0,0 +1,62 @@
Utility
=======
.. autosummary::
:nosignatures:
:toctree: doc/
ray.util.ActorPool
ray.util.queue.Queue
ray.util.list_named_actors
ray.util.serialization.register_serializer
ray.util.serialization.deregister_serializer
ray.util.tpu.get_current_pod_worker_count
ray.util.tpu.get_current_pod_name
ray.util.tpu.get_num_tpu_chips_on_node
ray.util.tpu.get_tpu_coordinator_env_vars
ray.util.tpu.get_tpu_slice_name_from_node
ray.util.tpu.get_tpu_nodes_for_slice
ray.util.tpu.get_num_ready_tpu_slices
ray.util.tpu.get_tpu_num_slices_for_workers
ray.util.tpu.get_tpu_version_from_type
ray.util.tpu.get_tpu_worker_resources
ray.util.tpu.init_jax_profiler
ray.util.tpu.SlicePlacementGroup
ray.util.tpu.slice_placement_group
ray.nodes
ray.cluster_resources
ray.available_resources
.. Other docs have references to these
ray.util.queue.Empty
ray.util.queue.Full
.. _custom-metric-api-ref:
Custom Metrics
--------------
.. autosummary::
:nosignatures:
:toctree: doc/
ray.util.metrics.Counter
ray.util.metrics.Gauge
ray.util.metrics.Histogram
.. _package-ref-debugging-apis:
Debugging
---------
.. autosummary::
:nosignatures:
:toctree: doc/
ray.util.rpdb.set_trace
ray.util.inspect_serializability
ray.timeline
doc/source/ray-core/compiled-graph/compiled-graph-api.rst
+43
View File
@@ -0,0 +1,43 @@
Compiled Graph API
==================
Input and Output Nodes
----------------------
.. autosummary::
:nosignatures:
:toctree: doc/
ray.dag.input_node.InputNode
ray.dag.output_node.MultiOutputNode
DAG Construction
----------------
.. autosummary::
:nosignatures:
:toctree: doc/
ray.actor.ActorMethod.bind
ray.dag.DAGNode.with_tensor_transport
ray.experimental.compiled_dag_ref.CompiledDAGRef
Compiled Graph Operations
-------------------------
.. autosummary::
:nosignatures:
:toctree: doc/
ray.dag.DAGNode.experimental_compile
ray.dag.compiled_dag_node.CompiledDAG.execute
ray.dag.compiled_dag_node.CompiledDAG.visualize
Configurations
--------------
.. autosummary::
:nosignatures:
:toctree: doc/
ray.dag.context.DAGContext
doc/source/ray-core/compiled-graph/overlap.rst
+25
View File
@@ -0,0 +1,25 @@
.. _compiled-graph-overlap:
Experimental: Overlapping communication and computation
=======================================================
Compiled Graph currently provides experimental support for GPU communication and computation overlap. When you turn this feature on, it automatically overlaps the GPU communication with computation operations, thereby hiding the communication overhead and improving performance.
To enable this feature, specify ``_overlap_gpu_communication=True`` when calling :func:`dag.experimental_compile() <ray.dag.DAGNode.experimental_compile>`.
The following code has GPU communication and computation operations that benefit
from overlapping.
.. literalinclude:: ../doc_code/cgraph_overlap.py
:language: python
:start-after: __cgraph_overlap_start__
:end-before: __cgraph_overlap_end__
The output of the preceding code includes the following two lines:
.. testoutput::
overlap_gpu_communication=False, duration=1.0670117866247892
overlap_gpu_communication=True, duration=0.9211348341777921
The actual performance numbers may vary on different hardware, but enabling ``_overlap_gpu_communication`` improves latency by about 14% for this example.
doc/source/ray-core/compiled-graph/profiling.rst
+87
View File
@@ -0,0 +1,87 @@
Profiling
=========
Ray Compiled Graph provides both PyTorch-based and Nsight-based profiling functionalities to better understand the performance
of individual tasks, system overhead, and performance bottlenecks. You can pick your favorite profiler based on your preference.
PyTorch profiler
----------------
To run PyTorch Profiling on Compiled Graph, simply set the environment variable ``RAY_CGRAPH_ENABLE_TORCH_PROFILING=1``
when running the script. For example, for a Compiled Graph script in ``example.py``, run the following command:
.. code-block:: bash
RAY_CGRAPH_ENABLE_TORCH_PROFILING=1 python3 example.py
After execution, Compiled Graph generates the profiling results in the `compiled_graph_torch_profiles` directory
under the current working directory. Compiled Graph generates one trace file per actor.
You can visualize traces by using https://ui.perfetto.dev/.
Nsight system profiler
----------------------
Compiled Graph builds on top of Ray's profiling capabilities, and leverages Nsight
system profiling.
To run Nsight Profiling on Compiled Graph, specify the runtime_env for the involved actors
as described in :ref:`Run Nsight on Ray <run-nsight-on-ray>`. For example,
.. literalinclude:: ../doc_code/cgraph_profiling.py
:language: python
:start-after: __profiling_setup_start__
:end-before: __profiling_setup_end__
Then, create a Compiled Graph as usual.
.. literalinclude:: ../doc_code/cgraph_profiling.py
:language: python
:start-after: __profiling_execution_start__
:end-before: __profiling_execution_end__
Finally, run the script as usual.
.. code-block:: bash
python3 example.py
After execution, Compiled Graph generates the profiling results under the `/tmp/ray/session_*/logs/{profiler_name}`
directory.
For fine-grained performance analysis of method calls and system overhead, set the environment variable
``RAY_CGRAPH_ENABLE_NVTX_PROFILING=1`` when running the script:
.. code-block:: bash
RAY_CGRAPH_ENABLE_NVTX_PROFILING=1 python3 example.py
This command leverages the `NVTX library <https://nvtx.readthedocs.io/en/latest/index.html#>`_ under the hood to automatically
annotate all methods called in the execution loops of compiled graph.
To visualize the profiling results, follow the same instructions as described in
:ref:`Nsight Profiling Result <profiling-result>`.
Visualization
-------------
To visualize the graph structure, call the :func:`visualize <ray.dag.compiled_dag_node.CompiledDAG.visualize>` method after calling :func:`experimental_compile <ray.dag.DAGNode.experimental_compile>`
on the graph.
.. literalinclude:: ../doc_code/cgraph_visualize.py
:language: python
:start-after: __cgraph_visualize_start__
:end-before: __cgraph_visualize_end__
By default, Ray generates a PNG image named ``compiled_graph.png`` and saves it in the current working directory.
Note that this requires ``graphviz``.
The following image shows the visualization for the preceding code.
Tasks that belong to the same actor are the same color.
.. image:: ../../images/compiled_graph_viz.png
:alt: Visualization of Graph Structure
:align: center
doc/source/ray-core/compiled-graph/quickstart.rst
+260
View File
@@ -0,0 +1,260 @@
Quickstart
==========
Hello World
-----------
This "hello world" example uses Ray Compiled Graph. First, install Ray.
.. code-block:: bash
pip install "ray[cgraph]"
# For a ray version before 2.41, use the following instead:
# pip install "ray[adag]"
First, define a simple actor that echoes its argument.
.. literalinclude:: ../doc_code/cgraph_quickstart.py
:language: python
:start-after: __simple_actor_start__
:end-before: __simple_actor_end__
Next instantiate the actor and use the classic Ray Core APIs ``remote`` and ``ray.get`` to execute tasks on the actor.
.. literalinclude:: ../doc_code/cgraph_quickstart.py
:language: python
:start-after: __ray_core_usage_start__
:end-before: __ray_core_usage_end__
.. code-block::
Execution takes 969.0364822745323 us
Now, create an equivalent program using Ray Compiled Graph.
First, define a graph to execute using classic Ray Core, without any compilation.
Later, compile this graph, to apply optimizations and prevent further modifications to the graph.
First, create a :ref:`Ray DAG <ray-dag-guide>` (directed acyclic graph), which is a lazily executed graph of Ray tasks.
Note 3 key differences with the classic Ray Core APIs:
1. Use the :class:`ray.dag.InputNode <ray.dag.input_node.InputNode>` context manager to indicate which inputs to the DAG should be provided at run time.
2. Use :func:`bind() <ray.actor.ActorMethod.bind>` instead of :func:`remote() <ray.remote>` to indicate lazily executed Ray tasks.
3. Use :func:`execute() <ray.dag.compiled_dag_node.CompiledDAG.execute>` to execute the DAG.
Here, define a graph and execute it.
Note that there is **no** compilation happening here. This uses the same execution backend as the preceding example:
.. literalinclude:: ../doc_code/cgraph_quickstart.py
:language: python
:start-after: __dag_usage_start__
:end-before: __dag_usage_end__
Next, compile the ``dag`` using the :func:`experimental_compile <ray.dag.DAGNode.experimental_compile>` API.
The graph uses the same APIs for execution:
.. literalinclude:: ../doc_code/cgraph_quickstart.py
:language: python
:start-after: __cgraph_usage_start__
:end-before: __cgraph_usage_end__
.. code-block::
Execution takes 86.72196418046951 us
The performance of the same task graph improved by 10X. This is because the function ``echo`` is cheap and thus highly affected by
the system overhead. Due to various bookkeeping and distributed protocols, the classic Ray Core APIs usually have 1 ms+ system overhead.
Because the system knows the task graph ahead of time, Ray Compiled Graphs can pre-allocate all necessary
resources ahead of time and greatly reduce the system overhead.
For example, if the actor ``a`` is on the same node as the driver, Ray Compiled Graphs uses shared memory instead of RPC to transfer data directly between the driver and the actor.
Currently, the DAG tasks run on a **background thread** of the involved actors.
An actor can only participate in one DAG at a time.
Normal tasks can still execute on the actors while the actors participate in a Compiled Graph, but these tasks execute on the main thread.
Once you're done, you can tear down the Compiled Graph by deleting it or explicitly calling ``dag.teardown()``.
This allows reuse of the actors in a new Compiled Graph.
.. literalinclude:: ../doc_code/cgraph_quickstart.py
:language: python
:start-after: __teardown_start__
:end-before: __teardown_end__
Specifying data dependencies
----------------------------
When creating the DAG, a ``ray.dag.DAGNode`` can be passed as an argument to other ``.bind`` calls to specify data dependencies.
For example, the following uses the preceding example to create a DAG that passes the same message from one actor to another:
.. literalinclude:: ../doc_code/cgraph_quickstart.py
:language: python
:start-after: __cgraph_bind_start__
:end-before: __cgraph_bind_end__
.. code-block::
hello
Here is another example that passes the same message to both actors, which can then execute in parallel.
It uses :class:`ray.dag.MultiOutputNode <ray.dag.output_node.MultiOutputNode>` to indicate that this DAG returns multiple outputs.
Then, :func:`dag.execute() <ray.dag.compiled_dag_node.CompiledDAG.execute>` returns multiple :class:`CompiledDAGRef <ray.experimental.compiled_dag_ref.CompiledDAGRef>` objects, one per node:
.. literalinclude:: ../doc_code/cgraph_quickstart.py
:language: python
:start-after: __cgraph_multi_output_start__
:end-before: __cgraph_multi_output_end__
.. code-block::
Execution takes 86.72196418046951 us
Be aware that:
* On the same actor, a Compiled Graph executes in order. If an actor has multiple tasks in the same Compiled Graph, it executes all of them to completion before executing on the next DAG input.
* Across actors in the same Compiled Graph, the execution may be pipelined. An actor may begin executing on the next DAG input while a downstream actor executes on the current one.
* Compiled Graphs currently only supports actor tasks. Non-actor tasks aren't supported.
``asyncio`` support
-------------------
If your Compiled Graph driver is running in an ``asyncio`` event loop, use the ``async`` APIs to ensure that executing
the Compiled Graph and getting the results doesn't block the event loop.
First, pass ``enable_async=True`` to the ``dag.experimental_compile()``:
.. literalinclude:: ../doc_code/cgraph_quickstart.py
:language: python
:start-after: __cgraph_async_compile_start__
:end-before: __cgraph_async_compile_end__
Next, use `execute_async` to invoke the Compiled Graph. Calling ``await`` on ``execute_async`` will return once
the input has been submitted, and it returns a future that can be used to get the result. Finally,
use `await` to get the result of the Compiled Graph.
.. literalinclude:: ../doc_code/cgraph_quickstart.py
:language: python
:start-after: __cgraph_async_execute_start__
:end-before: __cgraph_async_execute_end__
Execution and failure semantics
-------------------------------
Like classic Ray Core, Ray Compiled Graph propagates exceptions to the final output.
In particular:
- **Application exceptions**: If an application task throws an exception, Compiled Graph
wraps the exception in a :class:`RayTaskError <ray.exceptions.RayTaskError>` and
raises it when the caller calls :func:`ray.get() <ray.get>` on the result. The thrown
exception inherits from both :class:`RayTaskError <ray.exceptions.RayTaskError>`
and the original exception class.
- **System exceptions**: System exceptions include actor death or unexpected errors
such as network errors. For actor death, Compiled Graph raises a
:class:`ActorDiedError <ray.exceptions.ActorDiedError>`, and for other errors, it
raises a :class:`RayChannelError <ray.exceptions.RayChannelError>`.
The graph can still execute after application exceptions. However, the graph
automatically shuts down in the case of system exceptions. If an actor's death causes
the graph to shut down, the remaining actors stay alive.
For example, this example explicitly destroys an actor while it's participating in a Compiled Graph.
The remaining actors are reusable:
.. literalinclude:: ../doc_code/cgraph_quickstart.py
:language: python
:start-after: __cgraph_actor_death_start__
:end-before: __cgraph_actor_death_end__
Execution Timeouts
------------------
Some errors, such as NCCL network errors, require additional handling to avoid hanging.
In the future, Ray may attempt to detect such errors, but currently as a fallback, it allows
configurable timeouts for
:func:`compiled_dag.execute() <ray.dag.compiled_dag_node.CompiledDAG.execute>` and :func:`ray.get() <ray.get>`.
The default timeout is 10 seconds for both. Set the following environment variables
to change the default timeout:
- ``RAY_CGRAPH_submit_timeout``: Timeout for :func:`compiled_dag.execute() <ray.dag.compiled_dag_node.CompiledDAG.execute>`.
- ``RAY_CGRAPH_get_timeout``: Timeout for :func:`ray.get() <ray.get>`.
:func:`ray.get() <ray.get>` also has a timeout parameter to set timeout on a per-call basis.
CPU to GPU communication
------------------------
With classic Ray Core, passing ``torch.Tensors`` between actors can become expensive, especially
when transferring between devices. This is because Ray Core doesn't know the final destination device.
Therefore, you may see unnecessary copies across devices other than the source and destination devices.
Ray Compiled Graph ships with native support for passing ``torch.Tensors`` between actors executing on different
devices. Developers can now use type hint annotations in the Compiled Graph declaration to indicate the final
destination device of a ``torch.Tensor``.
.. literalinclude:: ../doc_code/cgraph_nccl.py
:language: python
:start-after: __cgraph_cpu_to_gpu_actor_start__
:end-before: __cgraph_cpu_to_gpu_actor_end__
In Ray Core, if you try to pass a CPU tensor from the driver,
the GPU actor receives a CPU tensor:
.. testcode::
:skipif: True
# This will fail because the driver passes a CPU copy of the tensor,
# and the GPU actor also receives a CPU copy.
ray.get(actor.process.remote(torch.zeros(10)))
With Ray Compiled Graph, you can annotate DAG nodes with type hints to indicate that there may be a ``torch.Tensor``
contained in the value:
.. literalinclude:: ../doc_code/cgraph_nccl.py
:language: python
:start-after: __cgraph_cpu_to_gpu_start__
:end-before: __cgraph_cpu_to_gpu_end__
Under the hood, the Ray Compiled Graph backend copies the ``torch.Tensor`` to the GPU assigned to the ``GPUActor`` by Ray Core.
Of course, you can also do this yourself, but there are advantages to using Compiled Graph instead:
- Ray Compiled Graph can minimize the number of data copies made. For example, passing from one CPU to
multiple GPUs requires one copy to a shared memory buffer, and then one host-to-device copy per
destination GPU.
- In the future, this can be further optimized through techniques such as
`memory pinning <https://pytorch.org/docs/stable/generated/torch.Tensor.pin_memory.html>`_,
using zero-copy deserialization when the CPU is the destination, etc.
GPU to GPU communication
------------------------
Ray Compiled Graphs supports NCCL-based transfers of CUDA ``torch.Tensor`` objects, avoiding any copies through Ray's CPU-based shared-memory object store.
With user-provided type hints, Ray prepares NCCL communicators and
operation scheduling ahead of time, avoiding deadlock and :ref:`overlapping compute and communication <compiled-graph-overlap>`.
Ray Compiled Graph uses `cupy <https://cupy.dev/>`_ under the hood to support NCCL operations.
The cupy version affects the NCCL version. The Ray team is also planning to support custom communicators in the future, for example to support collectives across CPUs or to reuse existing collective groups.
First, create sender and receiver actors. Note that this example requires at least 2 GPUs.
.. literalinclude:: ../doc_code/cgraph_nccl.py
:language: python
:start-after: __cgraph_nccl_setup_start__
:end-before: __cgraph_nccl_setup_end__
To support GPU-to-GPU communication with NCCL, wrap the DAG node that contains the ``torch.Tensor`` that you want to transmit using the ``with_tensor_transport`` API hint:
.. literalinclude:: ../doc_code/cgraph_nccl.py
:language: python
:start-after: __cgraph_nccl_exec_start__
:end-before: __cgraph_nccl_exec_end__
Current limitations include:
* ``torch.Tensor`` and NVIDIA NCCL only
* Support for peer-to-peer transfers. Collective communication operations are coming soon.
* Communication operations are currently done synchronously. :ref:`Overlapping compute and communication <compiled-graph-overlap>` is an experimental feature.
doc/source/ray-core/compiled-graph/ray-compiled-graph.rst
+84
View File
@@ -0,0 +1,84 @@
.. _ray-compiled-graph:
Ray Compiled Graph (beta)
=========================
.. warning::
Ray Compiled Graph is currently in beta (since Ray 2.44). The APIs are subject to change and expected to evolve.
The API is available from Ray 2.32, but it's recommended to use a version after 2.44.
As large language models (LLMs) become common, programming distributed systems with multiple GPUs is essential.
:ref:`Ray Core APIs <core-key-concepts>` facilitate using multiple GPUs but have limitations such as:
* System overhead of ~1 ms per task launch, which is unsuitable for high-performance tasks like LLM inference.
* Lack of support for direct GPU-to-GPU communication, requiring manual development with external libraries like NVIDIA Collective Communications Library (`NCCL <https://developer.nvidia.com/nccl>`_).
Ray Compiled Graph gives you a Ray Core-like API but with:
- **Less than 50us system overhead** for workloads that repeatedly execute the same task graph.
- **Native support for GPU-GPU communication** with NCCL.
For example, consider the following Ray Core code, which sends data to an actor
and gets the result:
.. testcode::
:skipif: True
# Ray Core API for remote execution.
# ~1ms overhead to invoke `recv`.
ref = receiver.recv.remote(data)
ray.get(ref)
This code shows how to compile and execute the same example as a Compiled Graph.
.. testcode::
:skipif: True
# Compiled Graph for remote execution.
# less than 50us overhead to invoke `recv` (during `graph.execute(data)`).
with InputNode() as inp:
graph = receiver.recv.bind(inp)
graph = graph.experimental_compile()
ref = graph.execute(data)
ray.get(ref)
Ray Compiled Graph has a static execution model. It's different from classic Ray APIs, which are eager. Because
of the static nature, Ray Compiled Graph can perform various optimizations such as:
- Pre-allocate resources so that it can reduce system overhead.
- Prepare NCCL communicators and apply deadlock-free scheduling.
- (experimental) Automatically overlap GPU compute and communication.
- Improve multi-node performance.
Use Cases
---------
Ray Compiled Graph APIs simplify development of high-performance multi-GPU workloads such as LLM inference or distributed training that require:
- Sub-millisecond level task orchestration.
- Direct GPU-GPU peer-to-peer or collective communication.
- `Heterogeneous <https://www.youtube.com/watch?v=Mg08QTBILWU>`_ or MPMD (Multiple Program Multiple Data) execution.
More Resources
--------------
- `Ray Compiled Graph blog <https://www.anyscale.com/blog/announcing-compiled-graphs>`_
- `Ray Compiled Graph talk at Ray Summit <https://www.youtube.com/watch?v=jv58Cpr6SAs>`_
- `Heterogeneous training with Ray Compiled Graph <https://www.youtube.com/watch?v=Mg08QTBILWU>`_
- `Distributed LLM inference with Ray Compiled Graph <https://www.youtube.com/watch?v=oMb_WiUwf5o>`_
Table of Contents
-----------------
Learn more details about Ray Compiled Graph from the following links.
.. toctree::
:maxdepth: 1
quickstart
profiling
overlap
troubleshooting
compiled-graph-api
doc/source/ray-core/compiled-graph/troubleshooting.rst
+99
View File
@@ -0,0 +1,99 @@
Troubleshooting
===============
This page contains common issues and solutions for Compiled Graph execution.
Limitations
-----------
Compiled Graph is a new feature and has some limitations:
- Invoking Compiled Graph
- Only the process that compiles the Compiled Graph may call it.
- A Compiled Graph has a maximum number of in-flight executions. When using the DAG API,
if there aren't enough resources at the time of ``dag.execute()``, Ray will queue the
tasks for later execution. Ray Compiled Graph currently doesn't support queuing past its
maximum capacity. Therefore, you may need to consume some results using ``ray.get()``
before submitting more executions. As a stopgap,
``dag.execute()`` throws a ``RayCgraphCapacityExceeded`` exception if the call takes too long.
In the future, Compiled Graph may have better error handling and queuing.
- Compiled Graph Execution
- Ideally, you should try not to execute other tasks on the actor while it is participating in a Compiled Graph.
Compiled Graph tasks will be executed on a **background thread**. Any concurrent tasks
submitted to the actor can still execute on the main thread, but you are responsible for
synchronization with the Compiled Graph background thread.
- For now, actors can only execute one Compiled Graph at a time. To execute a different Compiled Graph
on the same actor, you must teardown the current Compiled Graph.
See :ref:`Return NumPy arrays <troubleshoot-numpy>` for more details.
- Passing and getting Compiled Graph results (:class:`CompiledDAGRef <ray.experimental.compiled_dag_ref.CompiledDAGRef>`)
- Compiled Graph results can't be passed to another task or actor. This restriction may be loosened
in the future, but for now, it allows for better performance because the backend knows
exactly where to push the results.
- ``ray.get()`` can be called at most once on a :class:`CompiledDAGRef <ray.experimental.compiled_dag_ref.CompiledDAGRef>`. An exception will be raised if
it is called twice on the same :class:`CompiledDAGRef <ray.experimental.compiled_dag_ref.CompiledDAGRef>`. This is because the underlying memory for
the result may need to be reused for a future DAG execution. Restricting ``ray.get()`` to once
per reference simplifies the tracking of the memory buffers.
- If the value returned by ``ray.get()`` is zero-copy deserialized, then subsequent executions
of the same DAG will block until the value goes out of scope in Python. Thus, if you hold onto
zero-copy deserialized values returned by ``ray.get()``, and you try to execute the Compiled Graph above
its max concurrency, it may deadlock. This case will be detected in the future, but for now
you will receive a ``RayChannelTimeoutError``.
See :ref:`Explicitly teardown before reusing the same actors <troubleshoot-teardown>`
for more details.
- Collective operations
- For GPU to GPU communication, Compiled Graph only supports peer-to-peer transfers. Collective communication operations are coming soon.
Keep an eye out for additional features in future Ray releases:
- Support better queuing of DAG inputs, to enable more concurrent executions of the same DAG.
- Support for more collective operations with NCCL.
- Support for multiple DAGs executing on the same actor.
- General performance improvements.
If you run into additional issues, or have other feedback or questions, file an issue
on `GitHub <https://github.com/ray-project/ray/issues>`_.
For a full list of known issues, check the ``compiled-graphs`` label on Ray GitHub.
.. _troubleshoot-numpy:
Returning NumPy arrays
----------------------
Ray zero-copy deserializes NumPy arrays when possible. If you execute compiled graph with a NumPy array output multiple times,
you could possibly run into issues if a NumPy array output from a previous Compiled Graph execution isn't deleted before attempting to get the result
of a following execution of the same Compiled Graph. This is because the NumPy array stays in the buffer of the Compiled Graph until you or Python delete it.
It's recommended to explicitly delete the NumPy array as Python may not always garbage collect the NumPy array immediately as you may expect.
For example, the following code sample could result in a hang or RayChannelTimeoutError if the NumPy array isn't deleted:
.. literalinclude:: ../doc_code/cgraph_troubleshooting.py
:language: python
:start-after: __numpy_troubleshooting_start__
:end-before: __numpy_troubleshooting_end__
In the preceding code snippet, Python may not garbage collect the NumPy array in `result` on each iteration of the loop.
Therefore, you should explicitly delete the NumPy array before you try to get the result of subsequent Compiled Graph executions.
.. _troubleshoot-teardown:
Explicitly teardown before reusing the same actors
--------------------------------------------------
If you want to reuse the actors of a Compiled Graph, it's important to explicitly teardown the Compiled Graph before reusing the actors.
Without explicitly tearing down the Compiled Graph, the resources created for actors in a Compiled Graph may have conflicts with further usage of those actors.
For example, in the following code, Python could delay garbage collection, which triggers the implicit teardown of the first Compiled Graph. This could lead to a segfault due to the resource conflicts mentioned:
.. literalinclude:: ../doc_code/cgraph_troubleshooting.py
:language: python
:start-after: __teardown_troubleshooting_start__
:end-before: __teardown_troubleshooting_end__
doc/source/ray-core/configure.rst
+347
View File
@@ -0,0 +1,347 @@
.. _configuring-ray:
Configuring Ray
===============
.. note:: For running Java applications, see `Java Applications`_.
This page discusses the various ways to configure Ray, both from the Python API
and from the command line. Take a look at the ``ray.init`` `documentation
<package-ref.html#ray.init>`__ for a complete overview of the configurations.
.. important:: For the multi-node setting, you must first run ``ray start`` on the command line to start the Ray cluster services on the machine before ``ray.init`` in Python to connect to the cluster services. On a single machine, you can run ``ray.init()`` without ``ray start``, which both starts the Ray cluster services and connects to them.
.. _cluster-resources:
Cluster resources
-----------------
Ray by default detects available resources.
.. testcode::
:hide:
import ray
ray.shutdown()
.. testcode::
import ray
# This automatically detects available resources in the single machine.
ray.init()
If not running cluster mode, you can specify cluster resources overrides through ``ray.init`` as follows.
.. testcode::
:hide:
ray.shutdown()
.. testcode::
# If not connecting to an existing cluster, you can specify resources overrides:
ray.init(num_cpus=8, num_gpus=1)
.. testcode::
:hide:
ray.shutdown()
.. testcode::
# Specifying custom resources
ray.init(num_gpus=1, resources={'Resource1': 4, 'Resource2': 16})
When starting Ray from the command line, pass the ``--num-cpus`` and ``--num-gpus`` flags into ``ray start``. You can also specify custom resources.
.. code-block:: bash
# To start a head node.
$ ray start --head --num-cpus=<NUM_CPUS> --num-gpus=<NUM_GPUS>
# To start a non-head node.
$ ray start --address=<address> --num-cpus=<NUM_CPUS> --num-gpus=<NUM_GPUS>
# Specifying custom resources
ray start [--head] --num-cpus=<NUM_CPUS> --resources='{"Resource1": 4, "Resource2": 16}'
If using the command line, connect to the Ray cluster as follow:
.. testcode::
:skipif: True
# Connect to ray. Notice if connected to existing cluster, you don't specify resources.
ray.init(address=<address>)
.. _temp-dir-log-files:
Logging and debugging
---------------------
Each Ray session has a unique name. By default, the name is
``session_{timestamp}_{pid}``. The format of ``timestamp`` is
``%Y-%m-%d_%H-%M-%S_%f`` (See `Python time format <strftime.org>`__ for details);
the pid belongs to the startup process (the process calling ``ray.init()`` or
the Ray process executed by a shell in ``ray start``).
For each session, Ray places all its temporary files under the
*session directory*. A *session directory* is a subdirectory of the
*root temporary path* (``/tmp/ray`` by default),
so the default session directory is ``/tmp/ray/{ray_session_name}``.
You can sort by their names to find the latest session.
Change the *root temporary directory* by passing ``--temp-dir={your temp path}`` to ``ray start``.
There currently isn't a stable way to change the root temporary directory when calling ``ray.init()``, but if you need to, you can provide the ``_temp_dir`` argument to ``ray.init()``.
See :ref:`Logging Directory Structure <logging-directory-structure>` for more details.
.. _ray-ports:
Ports configurations
--------------------
Ray requires bi-directional communication among its nodes in a cluster. Each node opens specific ports to receive incoming network requests.
All Nodes
~~~~~~~~~
- ``--node-manager-port``: Raylet port for node manager. Default: Random value.
- ``--object-manager-port``: Raylet port for object manager. Default: Random value.
- ``--runtime-env-agent-port``: Raylet port for runtime env agent. Default: Random value.
The node manager and object manager run as separate processes with their own ports for communication.
The following options specify the ports used by dashboard agent process.
- ``--dashboard-agent-grpc-port``: The port to listen for grpc on. Default: Random value.
- ``--dashboard-agent-listen-port``: The port to listen for http on. Default: 52365.
- ``--metrics-export-port``: The port to use to expose Ray metrics. Default: Random value.
The following options specify the range of ports used by worker processes across machines. All ports in the range should be open.
- ``--min-worker-port``: Minimum port number for the worker to bind to. Default: 10002.
- ``--max-worker-port``: Maximum port number for the worker to bind to. Default: 19999.
Port numbers are how Ray differentiates input and output to and from multiple workers on a single node. Each worker takes input and gives output on a single port number. Therefore, by default, there's a maximum of 10,000 workers on each node, irrespective of number of CPUs.
In general, you should give Ray a wide range of possible worker ports, in case any of those ports happen to be in use by some other program on your machine. However, when debugging, it's useful to explicitly specify a short list of worker ports such as ``--worker-port-list=10000,10001,10002,10003,10004``
Note that this practice limits the number of workers, just like specifying a narrow range.
Head node
~~~~~~~~~
In addition to ports specified in the preceding section, the head node needs to open several more ports.
- ``--port``: Port of the Ray GCS server. The head node starts a GCS server listening on this port. Default: 6379.
- ``--ray-client-server-port``: Listening port for Ray Client Server. Default: 10001.
- ``--redis-shard-ports``: Comma-separated list of ports for non-primary Redis shards. Default: Random values.
- ``--dashboard-grpc-port``: (Deprecated) No longer used. Only kept for backward compatibility.
- If ``--include-dashboard`` is true (the default), then the head node must open ``--dashboard-port``. Default: 8265.
If ``--include-dashboard`` is true but the ``--dashboard-port`` isn't open on
the head node, you won't be able to access the dashboard, and you repeatedly get
.. code-block:: bash
WARNING worker.py:1114 -- The agent on node <hostname of node that tried to run a task> failed with the following error:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/grpc/aio/_call.py", line 285, in __await__
raise _create_rpc_error(self._cython_call._initial_metadata,
grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"
debug_error_string = "{"description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":4165,"referenced_errors":[{"description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":397,"grpc_status":14}]}"
If you see that error, check whether the ``--dashboard-port`` is accessible
through ``nc``, ``nmap``, or your hello browser.
.. code-block:: bash
$ nmap -sV --reason -p 8265 $HEAD_ADDRESS
Nmap scan report for compute04.berkeley.edu (123.456.78.910)
Host is up, received reset ttl 60 (0.00065s latency).
rDNS record for 123.456.78.910: compute04.berkeley.edu
PORT STATE SERVICE REASON VERSION
8265/tcp open http syn-ack ttl 60 aiohttp 3.7.2 (Python 3.8)
Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Note that the dashboard runs as a separate subprocess which can crash invisibly
in the background, so even if you checked port 8265 earlier, the port might be
closed *now* (for the prosaic reason that there's no longer a service running
on it). This also means that if you ``ray stop`` and ``ray start`` when the port is
unreachable, it may become reachable again due to the dashboard restarting.
If you don't want the dashboard, set ``--include-dashboard=false``.
TLS authentication
------------------
You can configure Ray to use TLS on its gRPC channels.
This means that connecting to the Ray head requires
an appropriate set of credentials and also that data exchanged between
various processes (client, head, workers) is encrypted.
TLS uses the private key and public key for encryption and decryption. The owner
keeps the private key secret and TLS shares the public key with the other party.
This pattern ensures that only the intended recipient can read the message.
A Certificate Authority (CA) is a trusted third party that certifies the identity of the
public key owner. The digital certificate issued by the CA contains the public key itself,
the identity of the public key owner, and the expiration date of the certificate. Note that
if the owner of the public key doesn't want to obtain a digital certificate from a CA,
they can generate a self-signed certificate with tools like OpenSSL.
To obtain a digital certificate, the owner of the public key must generate a Certificate Signing
Request (CSR). The CSR contains information about the owner of the public
key and the public key itself. Ray requires additional steps for achieving
a successful TLS encryption.
Here is a step-by-step guide for adding TLS Authentication to a static Kubernetes Ray cluster using
a self-signed certificates:
Step 1: Generate a private key and self-signed certificate for CA
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: bash
openssl req -x509 \
-sha256 -days 3650 \
-nodes \
-newkey rsa:2048 \
-subj "/CN=*.ray.io/C=US/L=San Francisco" \
-keyout ca.key -out ca.crt
Use the following command to encode the private key file and the self-signed certificate file,
then paste encoded strings to the secret.yaml.
.. code-block:: bash
cat ca.key | base64
cat ca.crt | base64
# Alternatively, this command automatically encode and create the secret for the CA key pair.
.. code-block:: bash
kubectl create secret generic ca-tls --from-file=ca.crt=<path-to-ca.crt> --from-file=ca.key=<path-to-ca.key>
Step 2: Generate individual private keys and self-signed certificates for the Ray head and workers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The `YAML file
<https://raw.githubusercontent.com/ray-project/ray/master/doc/source/cluster/kubernetes/configs/static-ray-cluster.tls.yaml>`__, has a ConfigMap named `tls` that
includes two shell scripts: `gencert_head.sh` and `gencert_worker.sh`. These scripts produce the private key
and self-signed certificate files (`tls.key` and `tls.crt`) for both head and worker Pods in the initContainer
of each deployment. By using the initContainer, we can dynamically retrieve the `POD_IP` to the `[alt_names]` section.
The scripts perform the following steps: first, it generates a 2048-bit RSA private key and saves the key as
`/etc/ray/tls/tls.key`. Then, a Certificate Signing Request (CSR) is generated using the `tls.key` file
and the `csr.conf` configuration file. Finally, a self-signed certificate (`tls.crt`) is created using
the Certificate Authority's (`ca.key and ca.crt`) keypair and the CSR (`ca.csr`).
Step 3: Set the environment variables for both Ray head and worker to enable TLS
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You enable TLS by setting environment variables.
- ``RAY_USE_TLS``: Either 1 or 0 to use/not-use TLS. If you set it to 1, you must set the environment variables below. Default: 0.
- ``RAY_TLS_SERVER_CERT``: Location of a `certificate file (tls.crt)`, which Ray presents to other endpoints to achieve mutual authentication.
- ``RAY_TLS_SERVER_KEY``: Location of a `private key file (tls.key)`, which is the cryptographic means to prove to other endpoints that you are the authorized user of a given certificate.
- ``RAY_TLS_CA_CERT``: Location of a `CA certificate file (ca.crt)`, which allows TLS to decide whether an the correct authority signed the endpoint's certificate.
Step 4: Verify TLS authentication
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: bash
# Log in to the worker Pod
kubectl exec -it ${WORKER_POD} -- bash
# Since the head Pod has the certificate of the full qualified DNS resolution for the Ray head service, the connection to the worker Pods
# is established successfully
ray health-check --address service-ray-head.default.svc.cluster.local:6379
# Since service-ray-head hasn't added to the alt_names section in the certificate, the connection fails and an error
# message similar to the following is displayed: "Peer name service-ray-head is not in peer certificate".
ray health-check --address service-ray-head:6379
# After you add `DNS.3 = service-ray-head` to the alt_names sections and deploy the YAML again, the connection is able to work.
Enabling TLS causes a performance hit due to the extra overhead of mutual
authentication and encryption.
Testing has shown that this overhead is large for small workloads and becomes
relatively smaller for large workloads.
The exact overhead depends on the nature of your workload.
Java applications
-----------------
.. important:: For the multi-node setting, you must first run ``ray start`` on the command line to start the Ray cluster services on the machine before ``ray.init()`` in Java to connect to the cluster services. On a single machine, you can run ``ray.init()`` without ``ray start``. It both starts the Ray cluster services and connects to them.
.. _code_search_path:
Code search path
~~~~~~~~~~~~~~~~
If you want to run a Java application in a multi-node cluster, you must specify the code search path in your driver. The code search path tells Ray where to load jars when starting Java workers. You must distribute your jar files to the same paths on all nodes of the Ray cluster before running your code.
.. code-block:: bash
$ java -classpath <classpath> \
-Dray.address=<address> \
-Dray.job.code-search-path=/path/to/jars/ \
<classname> <args>
The ``/path/to/jars/`` points to a directory which contains jars. Workers load all jars in the directory. You can also provide multiple directories for this parameter.
.. code-block:: bash
$ java -classpath <classpath> \
-Dray.address=<address> \
-Dray.job.code-search-path=/path/to/jars1:/path/to/jars2:/path/to/pys1:/path/to/pys2 \
<classname> <args>
You don't need to configure code search path if you run a Java application in a single-node cluster.
See ``ray.job.code-search-path`` under :ref:`Driver Options <java-driver-options>` for more information.
.. note:: Currently there's no way to configure Ray when running a Java application in single machine mode. If you need to configure Ray, run ``ray start`` to start the Ray cluster first.
.. _java-driver-options:
Driver options
~~~~~~~~~~~~~~
There's a limited set of options for Java drivers. They're not for configuring the Ray cluster, but only for configuring the driver.
Ray uses `Typesafe Config <https://lightbend.github.io/config/>`__ to read options. There are several ways to set options:
- System properties. You can configure system properties either by adding options in the format of ``-Dkey=value`` in the driver command line, or by invoking ``System.setProperty("key", "value");`` before ``Ray.init()``.
- A `HOCON format <https://github.com/lightbend/config/blob/master/HOCON.md>`__ configuration file. By default, Ray will try to read the file named ``ray.conf`` in the root of the classpath. You can customize the location of the file by setting system property ``ray.config-file`` to the path of the file.
.. note:: Options configured by system properties have higher priority than options configured in the configuration file.
The list of available driver options:
- ``ray.address``
- The cluster address if the driver connects to an existing Ray cluster. If it's empty, Ray creates a new Ray cluster.
- Type: ``String``
- Default: empty string.
- ``ray.job.code-search-path``
- The paths for Java workers to load code from. Currently, Ray only supports directories. You can specify one or more directories split by a ``:``. You don't need to configure code search path if you run a Java application in single machine mode or local mode. Ray also uses the code search path to load Python code, if specified. This parameter is required for :ref:`cross_language`. If you specify a code search path, you can only run Python remote functions which you can find in the code search path.
- Type: ``String``
- Default: empty string.
- Example: ``/path/to/jars1:/path/to/jars2:/path/to/pys1:/path/to/pys2``
- ``ray.job.namespace``
- The namespace of this job. Ray uses it for isolation between jobs. Jobs in different namespaces can't access each other. If it's not specified, Ray uses a randomized value.
- Type: ``String``
- Default: A random UUID string value.
.. _`Apache Arrow`: https://arrow.apache.org/
doc/source/ray-core/cross-language.rst
+293
View File
@@ -0,0 +1,293 @@
.. _cross_language:
Cross-language programming
==========================
This page shows you how to use Ray's cross-language programming feature.
Setup the driver
-----------------
You need to set :ref:`code_search_path` in your driver.
.. tab-set::
.. tab-item:: Python
.. literalinclude:: ./doc_code/cross_language.py
:language: python
:start-after: __crosslang_init_start__
:end-before: __crosslang_init_end__
.. tab-item:: Java
.. code-block:: bash
java -classpath <classpath> \
-Dray.address=<address> \
-Dray.job.code-search-path=/path/to/code/ \
<classname> <args>
You may want to include multiple directories to load both Python and Java code for workers, if you place them in different directories.
.. tab-set::
.. tab-item:: Python
.. literalinclude:: ./doc_code/cross_language.py
:language: python
:start-after: __crosslang_multidir_start__
:end-before: __crosslang_multidir_end__
.. tab-item:: Java
.. code-block:: bash
java -classpath <classpath> \
-Dray.address=<address> \
-Dray.job.code-search-path=/path/to/jars:/path/to/pys \
<classname> <args>
Python calling Java
-------------------
Suppose you have a Java static method and a Java class as follows:
.. code-block:: java
package io.ray.demo;
public class Math {
public static int add(int a, int b) {
return a + b;
}
}
.. code-block:: java
package io.ray.demo;
// A regular Java class.
public class Counter {
private int value = 0;
public int increment() {
this.value += 1;
return this.value;
}
}
Then, in Python, you can call the preceding Java remote function, or create an actor
from the preceding Java class.
.. literalinclude:: ./doc_code/cross_language.py
:language: python
:start-after: __python_call_java_start__
:end-before: __python_call_java_end__
Java calling Python
-------------------
Suppose you have a Python module as follows:
.. literalinclude:: ./doc_code/cross_language.py
:language: python
:start-after: __python_module_start__
:end-before: __python_module_end__
.. note::
* You should decorate the function or class with `@ray.remote`.
Then, in Java, you can call the preceding Python remote function, or create an actor
from the preceding Python class.
.. code-block:: java
package io.ray.demo;
import io.ray.api.ObjectRef;
import io.ray.api.PyActorHandle;
import io.ray.api.Ray;
import io.ray.api.function.PyActorClass;
import io.ray.api.function.PyActorMethod;
import io.ray.api.function.PyFunction;
import org.testng.Assert;
public class JavaCallPythonDemo {
public static void main(String[] args) {
// Set the code-search-path to the directory of your `ray_demo.py` file.
System.setProperty("ray.job.code-search-path", "/path/to/the_dir/");
Ray.init();
// Define a Python class.
PyActorClass actorClass = PyActorClass.of(
"ray_demo", "Counter");
// Create a Python actor and call actor method.
PyActorHandle actor = Ray.actor(actorClass).remote();
ObjectRef objRef1 = actor.task(
PyActorMethod.of("increment", int.class)).remote();
Assert.assertEquals(objRef1.get(), 1);
ObjectRef objRef2 = actor.task(
PyActorMethod.of("increment", int.class)).remote();
Assert.assertEquals(objRef2.get(), 2);
// Call the Python remote function.
ObjectRef objRef3 = Ray.task(PyFunction.of(
"ray_demo", "add", int.class), 1, 2).remote();
Assert.assertEquals(objRef3.get(), 3);
Ray.shutdown();
}
}
Cross-language data serialization
---------------------------------
Ray automatically serializes and deserializes the arguments and return values of Ray calls
if their types are the following:
- Primitive data types
=========== ======= =======
MessagePack Python Java
=========== ======= =======
nil None null
bool bool Boolean
int int Short / Integer / Long / BigInteger
float float Float / Double
str str String
bin bytes byte[]
=========== ======= =======
- Basic container types
=========== ======= =======
MessagePack Python Java
=========== ======= =======
array list Array
=========== ======= =======
- Ray builtin types
- ActorHandle
.. note::
* Be aware of float / double precision between Python and Java. If Java is using a
float type to receive the input argument, the double precision Python data
reduces to float precision in Java.
* BigInteger can support a max value of 2^64-1. See:
https://github.com/msgpack/msgpack/blob/master/spec.md#int-format-family.
If the value is larger than 2^64-1, then sending the value to Python raises an exception.
The following example shows how to pass these types as parameters and how to
return these types.
You can write a Python function which returns the input data:
.. literalinclude:: ./doc_code/cross_language.py
:language: python
:start-after: __serialization_start__
:end-before: __serialization_end__
Then you can transfer the object from Java to Python, and back from Python
to Java:
.. code-block:: java
package io.ray.demo;
import io.ray.api.ObjectRef;
import io.ray.api.Ray;
import io.ray.api.function.PyFunction;
import java.math.BigInteger;
import org.testng.Assert;
public class SerializationDemo {
public static void main(String[] args) {
Ray.init();
Object[] inputs = new Object[]{
true, // Boolean
Byte.MAX_VALUE, // Byte
Short.MAX_VALUE, // Short
Integer.MAX_VALUE, // Integer
Long.MAX_VALUE, // Long
BigInteger.valueOf(Long.MAX_VALUE), // BigInteger
"Hello World!", // String
1.234f, // Float
1.234, // Double
"example binary".getBytes()}; // byte[]
for (Object o : inputs) {
ObjectRef res = Ray.task(
PyFunction.of("ray_serialization", "py_return_input", o.getClass()),
o).remote();
Assert.assertEquals(res.get(), o);
}
Ray.shutdown();
}
}
Cross-language exception stacks
-------------------------------
Suppose you have a Java package as follows:
.. code-block:: java
package io.ray.demo;
import io.ray.api.ObjectRef;
import io.ray.api.Ray;
import io.ray.api.function.PyFunction;
public class MyRayClass {
public static int raiseExceptionFromPython() {
PyFunction<Integer> raiseException = PyFunction.of(
"ray_exception", "raise_exception", Integer.class);
ObjectRef<Integer> refObj = Ray.task(raiseException).remote();
return refObj.get();
}
}
and a Python module as follows:
.. literalinclude:: ./doc_code/cross_language.py
:language: python
:start-after: __raise_exception_start__
:end-before: __raise_exception_end__
Then, run the following code:
.. literalinclude:: ./doc_code/cross_language.py
:language: python
:start-after: __raise_exception_demo_start__
:end-before: __raise_exception_demo_end__
The exception stack will be:
.. code-block:: text
Traceback (most recent call last):
File "ray_exception_demo.py", line 9, in <module>
ray.get(obj_ref) # <-- raise exception from here.
File "ray/python/ray/_private/client_mode_hook.py", line 105, in wrapper
return func(*args, **kwargs)
File "ray/python/ray/_private/worker.py", line 2247, in get
raise value
ray.exceptions.CrossLanguageError: An exception raised from JAVA:
io.ray.api.exception.RayTaskException: (pid=61894, ip=172.17.0.2) Error executing task c8ef45ccd0112571ffffffffffffffffffffffff01000000
at io.ray.runtime.task.TaskExecutor.execute(TaskExecutor.java:186)
at io.ray.runtime.RayNativeRuntime.nativeRunTaskExecutor(Native Method)
at io.ray.runtime.RayNativeRuntime.run(RayNativeRuntime.java:231)
at io.ray.runtime.runner.worker.DefaultWorker.main(DefaultWorker.java:15)
Caused by: io.ray.api.exception.CrossLanguageException: An exception raised from PYTHON:
ray.exceptions.RayTaskError: ray::raise_exception() (pid=62041, ip=172.17.0.2)
File "ray_exception.py", line 7, in raise_exception
1 / 0
ZeroDivisionError: division by zero
doc/source/ray-core/direct-transport/custom-tensor-transport.rst
+207
View File
@@ -0,0 +1,207 @@
.. _custom-tensor-transport:
*************************************************
Implementing a custom tensor transport (Advanced)
*************************************************
Ray Direct Transport (RDT) allows you to register custom tensor transports at runtime.
This page explains how to implement a custom tensor transport by implementing the :class:`TensorTransportManager <ray.experimental.TensorTransportManager>` abstract interface.
Overview
========
To create a custom tensor transport:
1. Implement the abstract interface :class:`ray.experimental.TensorTransportManager <ray.experimental.TensorTransportManager>`.
2. Define custom metadata classes by extending :class:`TensorTransportMetadata <ray.experimental.TensorTransportMetadata>` and :class:`CommunicatorMetadata <ray.experimental.CommunicatorMetadata>`.
3. Register your transport using :func:`ray.experimental.register_tensor_transport <ray.experimental.register_tensor_transport>`.
When Ray needs to transfer a tensor between actors using your transport, it calls specific methods on your ``TensorTransportManager`` implementation at different stages of the transfer lifecycle.
Implementing TensorTransportManager
===================================
The :class:`TensorTransportManager <ray.experimental.TensorTransportManager>` abstract class defines the interface for custom tensor transports. You must implement all abstract methods.
The following diagram shows when each method is called during a tensor transfer:
.. code-block:: text
Source Actor Owner Process Destination Actor
============ ============= =================
| | |
1. Task returns tensor | |
``extract_tensor_transport_metadata`` |
| | |
| ---- transport_metadata ----> | |
| | |
| 2. Prepare communicator |
| ``get_communicator_metadata`` |
| | |
| <---- comm metadata --------- | ---- comm metadata --------> |
| | |
3. ``send_multiple_tensors`` | 3. ``recv_multiple_tensors``
| |
| ------------ tensors ---------------------------------------> |
| | |
| (transfer complete) |
| | |
| 5. Ref goes out of scope |
| <---------------------------- | |
5. Clean up resources | |
``garbage_collect`` | |
Note that Ray will not call `send_multiple_tensors` for one-sided transports.
The following diagram shows where each method is called in the ray.put / ray.get case supported by one-sided transports.
.. code-block:: text
Source Actor Destination Actor
============ =================
| |
1. User ``ray.put``'s tensor |
``extract_tensor_transport_metadata`` |
| |
| |
2. User passes ref to another actor |
| ---- transport_metadata ----------------------------------> |
| |
| |
| 3. User ``ray.get``'s on object ref
``get_communicator_metadata``
| ``recv_multiple_tensors``
| ------------ tensors --------- -----------------------------> |
| |
| (transfer complete) |
| |
4. Clean up resources |
``garbage_collect`` |
(when ref goes out of scope) |
The API reference page for :class:`TensorTransportManager <ray.experimental.TensorTransportManager>` has more details on what each method does and how to implement them.
See implementations of Ray's default transports (NCCL, NIXL, etc.) in the `python/ray/experimental/rdt/ <https://github.com/ray-project/ray/tree/master/python/ray/experimental/rdt>`_ directory.
The following is an walk-through for implementing and using a custom tensor transport.
Example: Shared memory tensor transport
========================================
The following walks through a complete custom tensor transport that transfers ``numpy`` arrays through shared memory.
Note that because shared memory is one-sided (the receiver directly reads the memory block the sender wrote to),
``is_one_sided`` returns ``True`` and Ray never calls ``send_multiple_tensors``.
Define metadata classes
-----------------------
Your transport uses two metadata classes that flow through different stages of the transfer:
- :class:`TensorTransportMetadata <ray.experimental.TensorTransportMetadata>` is created on the **source actor** during ``extract_tensor_transport_metadata``. It carries per-tensor information (shapes, dtypes, devices) plus any transport-specific identifiers (e.g., shared memory block names, RDMA keys) that the receiver needs to locate and read the data.
- :class:`CommunicatorMetadata <ray.experimental.CommunicatorMetadata>` is created on the **owner/driver process** during ``get_communicator_metadata``. It carries any coordination information both actors need, such as ranks in a collective group. For one-sided transports (where the receiver can directly read the sender's memory), an empty metadata object is typically sufficient.
Start by extending these classes to carry any transport-specific state.
``ShmTransportMetadata`` stores the shared memory block name and size so the receiver can locate and read the data.
This transport doesn't need any communicator metadata, so ``ShmCommunicatorMetadata`` is empty.
.. literalinclude:: ../doc_code/direct_transport_custom.py
:language: python
:start-after: __custom_metadata_start__
:end-before: __custom_metadata_end__
Extract tensor transport metadata
---------------------------------
Ray calls ``extract_tensor_transport_metadata`` on the source actor right after the task produces its result tensors.
Record shapes and dtypes, then perform any transport-specific registration. Here, the implementation serializes the tensors
into a shared memory block and records the block name and size in the metadata so the receiver can find it.
.. literalinclude:: ../doc_code/direct_transport_custom.py
:language: python
:start-after: __custom_extract_start__
:end-before: __custom_extract_end__
Get communicator metadata
-------------------------
Ray calls ``get_communicator_metadata`` on the owner/driver process before orchestrating the transfer.
Return any information both actors need to coordinate, such as ranks in a collective group.
For one-sided transports such as shared memory, an empty metadata object is fine.
.. literalinclude:: ../doc_code/direct_transport_custom.py
:language: python
:start-after: __custom_communicator_start__
:end-before: __custom_communicator_end__
Transport properties
--------------------
Define your ``TensorTransportManager`` subclass and implement the property methods.
``tensor_transport_backend`` returns the name that users pass to ``@ray.method(tensor_transport=...)``.
``is_one_sided`` and ``can_abort_transport`` tell Ray how to orchestrate transfers and handle errors.
``actor_has_tensor_transport`` lets Ray check whether a given actor can use this transport.
.. literalinclude:: ../doc_code/direct_transport_custom.py
:language: python
:start-after: __custom_properties_start__
:end-before: __custom_properties_end__
Send and receive
----------------
``recv_multiple_tensors`` runs on the destination actor. For this shared memory transport, it opens the
shared memory block by name and deserializes the tensors.
``send_multiple_tensors`` runs on the source actor for two-sided transports. Since shared memory is one-sided,
Ray never calls this method, so it raises ``NotImplementedError`` as a safety guard.
.. literalinclude:: ../doc_code/direct_transport_custom.py
:language: python
:start-after: __custom_send_recv_start__
:end-before: __custom_send_recv_end__
Cleanup
-------
``garbage_collect`` runs on the source actor when Ray's reference counting determines the object is out of scope.
Release any transport resources here, in this case closing and unlinking the shared memory block.
``abort_transport`` runs on both actors when a system error occurs during transfer, if ``can_abort_transport`` returns ``True``.
Since this transport returns ``False`` for ``can_abort_transport``, Ray kills the involved actors instead,
so ``abort_transport`` is a no-op.
.. literalinclude:: ../doc_code/direct_transport_custom.py
:language: python
:start-after: __custom_cleanup_start__
:end-before: __custom_cleanup_end__
Registering your transport
==========================
After implementing your transport, the **driver process** must register it with :func:`ray.experimental.register_tensor_transport <ray.experimental.register_tensor_transport>` before creating any actors that use it:
.. literalinclude:: ../doc_code/direct_transport_custom.py
:language: python
:start-after: __custom_usage_start__
:end-before: __custom_usage_end__
Limitations
===========
Custom tensor transports have the following limitations:
- **Actor restarts aren't supported.** Your actor doesn't have access to the custom transport after a restart.
- **Register transports before actor creation.** If you register a transport after creating an actor, that actor can't use the new transport.
- **Out-of-order actors** If you have an out-of-order actor (such as an async actor) and the process where you submit the actor task is different from where you created the actor, Ray can't guarantee it has registered your custom transport on the actor at task execution time.
- **Actor creation and task submission from different processes** If the process where you submit an actor task is different from where you created the actor, Ray can't guarantee it has registered your custom transport on the actor at task execution time.
For general RDT limitations, see :ref:`limitations <limitations>`.
Also feel free to reach out through `GitHub issues <https://github.com/ray-project/ray/issues>`_ or the `Ray Slack <https://docs.google.com/forms/d/e/1FAIpQLSfAcoiLCHOguOm8e7Jnn-JJdZaCxPGjgVCvFijHB5PLaQLeig/viewform>`_ to ask any questions.
doc/source/ray-core/direct-transport/direct-transport.rst
+367
View File
@@ -0,0 +1,367 @@
.. _direct-transport:
**************************
Ray Direct Transport (RDT)
**************************
Ray objects are normally stored in Ray's CPU-based object store and copied and deserialized when accessed by a Ray task or actor.
For GPU data specifically, this can lead to unnecessary and expensive data transfers.
For example, passing a CUDA ``torch.Tensor`` from one Ray task to another would require a copy from GPU to CPU memory, then back again to GPU memory.
*Ray Direct Transport (RDT)* is a new feature that allows Ray to store and pass objects directly between Ray actors.
This feature augments the familiar Ray :class:`ObjectRef <ray.ObjectRef>` API by:
- Keeping GPU data in GPU memory until a transfer is necessary
- Avoiding expensive serialization and copies to and from the Ray object store
- Using efficient data transports like collective communication libraries (`Gloo <https://github.com/pytorch/gloo>`__ or `NCCL <https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/index.html>`__) or point-to-point RDMA (via `NVIDIA's NIXL <https://github.com/ai-dynamo/nixl>`__) to transfer data directly between devices, including both CPU and GPUs
.. note::
RDT is currently in **alpha** and doesn't support all Ray Core APIs yet. Future releases may introduce breaking API changes. See the :ref:`limitations <limitations>` section for more details.
Getting started
===============
.. tip::
RDT currently supports ``torch.Tensor`` objects created by Ray actor tasks. Other datatypes and Ray non-actor tasks may be supported in future releases.
This walkthrough will show how to create and use RDT with different *tensor transports*, i.e. the mechanism used to transfer the tensor between actors.
Currently, RDT supports the following tensor transports:
1. `Gloo <https://github.com/pytorch/gloo>`__: A collective communication library for PyTorch and CPUs.
2. `NVIDIA NCCL <https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/index.html>`__: A collective communication library for NVIDIA GPUs.
3. `NVIDIA NIXL <https://github.com/ai-dynamo/nixl>`__: A library for accelerating point-to-point transfers via RDMA, especially between various types of memory and NVIDIA GPUs. Backed by `LIBFABRIC <https://ofiwg.github.io/libfabric/>`__ on AWS instances with `EFA <https://aws.amazon.com/hpc/efa/>`__, and `UCX <https://github.com/openucx/ucx>`__ everywhere else.
For ease of following along, we'll start with the `Gloo <https://github.com/pytorch/gloo>`__ transport, which can be used without any physical GPUs.
.. _direct-transport-gloo:
Usage with Gloo (CPUs only)
---------------------------
Installation
^^^^^^^^^^^^
.. note::
Under construction.
Walkthrough
^^^^^^^^^^^
To get started, define an actor class and a task that returns a ``torch.Tensor``:
.. literalinclude:: ../doc_code/direct_transport_gloo.py
:language: python
:start-after: __normal_example_start__
:end-before: __normal_example_end__
As written, when the ``torch.Tensor`` is returned, it will be copied into Ray's CPU-based object store.
For CPU-based tensors, this can require an expensive step to copy and serialize the object, while GPU-based tensors additionally require a copy to and from CPU memory.
To enable RDT, use the ``tensor_transport`` option in the :func:`@ray.method <ray.method>` decorator.
.. literalinclude:: ../doc_code/direct_transport_gloo.py
:language: python
:start-after: __gloo_example_start__
:end-before: __gloo_example_end__
This decorator can be added to any actor tasks that return a ``torch.Tensor``, or that return ``torch.Tensors`` nested inside other Python objects.
Adding this decorator will change Ray's behavior in the following ways:
1. When returning the tensor, Ray will store a *reference* to the tensor instead of copying it to CPU memory.
2. When the :class:`ray.ObjectRef` is passed to another task, Ray will use Gloo to transfer the tensor to the destination task.
Note that for (2) to work, the :func:`@ray.method(tensor_transport) <ray.method>` decorator only needs to be added to the actor task that *returns* the tensor. It should not be added to actor tasks that *consume* the tensor (unless those tasks also return tensors).
Also, for (2) to work, we must first create a *collective group* of actors.
Creating a collective group
^^^^^^^^^^^^^^^^^^^^^^^^^^^
To create a collective group for use with RDT:
1. Create multiple Ray actors.
2. Create a collective group on the actors using the :func:`ray.experimental.collective.create_collective_group <ray.experimental.collective.create_collective_group>` function. The `backend` specified must match the `tensor_transport` used in the :func:`@ray.method <ray.method>` decorator.
Here is an example:
.. literalinclude:: ../doc_code/direct_transport_gloo.py
:language: python
:start-after: __gloo_group_start__
:end-before: __gloo_group_end__
The actors can now communicate directly via gloo.
The group can also be destroyed using the :func:`ray.experimental.collective.destroy_collective_group <ray.experimental.collective.destroy_collective_group>` function.
After calling this function, a new collective group can be created on the same actors.
Passing objects to other actors
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Now that we have a collective group, we can create and pass RDT objects between the actors.
Here is a full example:
.. literalinclude:: ../doc_code/direct_transport_gloo.py
:language: python
:start-after: __gloo_full_example_start__
:end-before: __gloo_full_example_end__
When the :class:`ray.ObjectRef` is passed to another task, Ray will use Gloo to transfer the tensor directly from the source actor to the destination actor instead of the default object store.
Note that the :func:`@ray.method(tensor_transport) <ray.method>` decorator is only added to the actor task that *returns* the tensor; once this hint has been added, the receiving actor task `receiver.sum` will automatically use Gloo to receive the tensor.
In this example, because `MyActor.sum` does not have the :func:`@ray.method(tensor_transport) <ray.method>` decorator, it will use the default Ray object store transport to return `torch.sum(tensor)`.
RDT also supports passing tensors nested inside Python data structures, as well as actor tasks that return multiple tensors, like in this example:
.. literalinclude:: ../doc_code/direct_transport_gloo.py
:language: python
:start-after: __gloo_multiple_tensors_example_start__
:end-before: __gloo_multiple_tensors_example_end__
Passing RDT objects to the actor that produced them
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RDT :class:`ray.ObjectRefs <ray.ObjectRef>` can also be passed to the actor that produced them.
This avoids any copies and just provides a reference to the same ``torch.Tensor`` that was previously created.
For example:
.. literalinclude:: ../doc_code/direct_transport_gloo.py
:language: python
:start-after: __gloo_intra_actor_start__
:end-before: __gloo_intra_actor_end__
.. note::
Ray only keeps a reference to the tensor created by the user, so the tensor objects are *mutable*.
If ``sender.sum`` were to modify the tensor in the above example, the changes would also be seen by ``receiver.sum``.
This differs from the normal Ray Core API, which always makes an immutable copy of data returned by actors.
``ray.get``
^^^^^^^^^^^
The :func:`ray.get <ray.get>` function can also be used as usual to retrieve the result of an RDT object. However, :func:`ray.get <ray.get>` will by default use the same tensor transport as the one specified in the :func:`@ray.method <ray.method>` decorator. For collective-based transports, this will not work if the caller is not part of the collective group.
Therefore, users need to specify the Ray object store as the tensor transport explicitly by setting ``_use_object_store`` in :func:`ray.get <ray.get>`.
.. literalinclude:: ../doc_code/direct_transport_gloo.py
:language: python
:start-after: __gloo_get_start__
:end-before: __gloo_get_end__
Object mutability
^^^^^^^^^^^^^^^^^
Unlike objects in the Ray object store, RDT objects are *mutable*, meaning that Ray only holds a reference to the tensor and will not copy it until a transfer is requested.
This means that if the actor that returns a tensor also keeps a reference to the tensor, and the actor later modifies it in place while Ray is still storing the tensor reference, it's possible that some or all of the changes may be seen by receiving actors.
Here is an example of what can go wrong:
.. literalinclude:: ../doc_code/direct_transport_gloo.py
:language: python
:start-after: __gloo_wait_tensor_freed_bad_start__
:end-before: __gloo_wait_tensor_freed_bad_end__
In this example, the sender actor returns a tensor to Ray, but it also keeps a reference to the tensor in its local state.
Then, in `sender.increment_and_sum_stored_tensor`, the sender actor modifies the tensor in place while Ray is still holding the tensor reference.
Then, the `receiver.increment_and_sum` task receives the modified tensor instead of the original, so the assertion fails.
To fix this kind of error, use the :func:`ray.experimental.wait_tensor_freed <ray.experimental.wait_tensor_freed>` function to wait for Ray to release all references to the tensor, so that the actor can safely write to the tensor again.
:func:`wait_tensor_freed <ray.experimental.wait_tensor_freed>` will unblock once all tasks that depend on the tensor have finished executing and all corresponding `ObjectRefs` have gone out of scope.
Ray tracks tasks that depend on the tensor by keeping track of which tasks take the `ObjectRef` corresponding to the tensor as an argument.
Here's a fixed version of the earlier example.
.. literalinclude:: ../doc_code/direct_transport_gloo.py
:language: python
:start-after: __gloo_wait_tensor_freed_start__
:end-before: __gloo_wait_tensor_freed_end__
The main changes are:
1. `sender` calls :func:`wait_tensor_freed <ray.experimental.wait_tensor_freed>` before modifying the tensor in place.
2. The driver skips :func:`ray.get <ray.get>` because :func:`wait_tensor_freed <ray.experimental.wait_tensor_freed>` blocks until all `ObjectRefs` pointing to the tensor are freed, so calling :func:`ray.get <ray.get>` here would cause a deadlock.
3. The driver calls `del tensor` to release its reference to the tensor. Again, this is necessary because :func:`wait_tensor_freed <ray.experimental.wait_tensor_freed>` blocks until all `ObjectRefs` pointing to the tensor are freed.
When an RDT `ObjectRef` is passed back to the same actor that produced it, Ray passes back a *reference* to the tensor instead of a copy. Therefore, the same kind of bug can occur.
To help catch such cases, Ray will print a warning if an RDT object is passed to the actor that produced it and a different actor, like so:
.. literalinclude:: ../doc_code/direct_transport_gloo.py
:language: python
:start-after: __gloo_object_mutability_warning_start__
:end-before: __gloo_object_mutability_warning_end__
Usage with NCCL (NVIDIA GPUs only)
----------------------------------
RDT requires just a few lines of code change to switch tensor transports. Here is the :ref:`Gloo example <direct-transport-gloo>`, modified to use NVIDIA GPUs and the `NCCL <https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/index.html>`__ library for collective GPU communication.
.. literalinclude:: ../doc_code/direct_transport_nccl.py
:language: python
:start-after: __nccl_full_example_start__
:end-before: __nccl_full_example_end__
The main code differences are:
1. The :func:`@ray.method <ray.method>` uses ``tensor_transport="nccl"`` instead of ``tensor_transport="gloo"``.
2. The :func:`ray.experimental.collective.create_collective_group <ray.experimental.collective.create_collective_group>` function is used to create a collective group.
3. The tensor is created on the GPU using the ``.cuda()`` method.
Usage with NIXL (CPUs or NVIDIA GPUs)
-------------------------------------
Installation
^^^^^^^^^^^^
First, install NIXL with a plain ``pip install nixl``.
For maximum performance, run the `install_gdrcopy.sh <https://github.com/ray-project/ray/blob/master/doc/tools/install_gdrcopy.sh>`__ script (e.g., ``install_gdrcopy.sh "${GDRCOPY_OS_VERSION}" "12.8" "x64"``). You can find available OS versions `here <https://developer.download.nvidia.com/compute/redist/gdrcopy/CUDA%2012.8/>`__.
When Ray selects the ``UCX`` backend (see :ref:`NIXL backend selection <nixl-backend-selection>` below), set these UCX environment variables to either let UCX choose the right transport from all options, or so that you can set your preferred transport option yourself. These variables apply only to the ``UCX`` backend and have no effect when Ray runs the ``LIBFABRIC`` backend.
.. code-block:: bash
# Example UCX configuration, adjust according to your environment
$ export UCX_TLS=all # or specify specific transports like "rc,ud,sm,^cuda_ipc" ..etc
$ export UCX_NET_DEVICES=all # or specify network devices like "mlx5_0:1,mlx5_1:1"
On the ``LIBFABRIC`` backend, use libfabric environment variables instead, such as ``FI_PROVIDER=efa`` to pin the provider and ``FI_LOG_LEVEL=Debug`` for diagnostics. See the `NIXL libfabric plugin documentation <https://github.com/ai-dynamo/nixl/blob/main/src/plugins/libfabric/README.md>`__ for additional configuration and troubleshooting.
.. _nixl-backend-selection:
NIXL backend selection
^^^^^^^^^^^^^^^^^^^^^^
Ray automatically selects the NIXL transport backend based on the available hardware:
- **AWS instances with EFA**: Ray detects EFA devices and validates that a realistic-size CUDA memory registration succeeds before it selects the ``LIBFABRIC`` backend. If validation fails, GPUDirect is usually misconfigured (for example, missing ``nvidia-peermem`` or dmabuf support). To detect EFA both on bare hosts and inside containers, Ray checks for the EFA network device (``/sys/class/net/efa*``) and for rdma-verbs devices bound to the kernel ``efa`` driver (under ``/sys/class/infiniband``). The host network device isn't visible inside a pod, so the verbs check is what makes detection work under Kubernetes. Ordinary InfiniBand or RoCE NICs also expose verbs devices, so Ray confirms the ``efa`` driver binding to avoid treating them as EFA.
- **All other environments**: Ray uses the ``UCX`` backend.
No configuration is required. On AWS EFA instances, make sure the `EFA installer <https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-start.html>`__ has been run, which installs both the EFA driver and libfabric. If LIBFABRIC validation fails at startup, see the `NIXL libfabric plugin documentation <https://github.com/ai-dynamo/nixl/blob/main/src/plugins/libfabric/README.md>`__ for troubleshooting.
Walkthrough
^^^^^^^^^^^
NIXL can transfer data between different devices, including CPUs and NVIDIA GPUs, but doesn't require a collective group to be created ahead of time.
This means that any actor that has NIXL installed in its environment can be used to create and pass an RDT object.
Otherwise, the usage is the same as in the :ref:`Gloo example <direct-transport-gloo>`.
Here is an example showing how to use NIXL to transfer an RDT object between two actors:
.. literalinclude:: ../doc_code/direct_transport_nixl.py
:language: python
:start-after: __nixl_full_example_start__
:end-before: __nixl_full_example_end__
Compared to the :ref:`Gloo example <direct-transport-gloo>`, the main code differences are:
1. The :func:`@ray.method <ray.method>` uses ``tensor_transport="nixl"`` instead of ``tensor_transport="gloo"``.
2. No collective group is needed.
ray.put and ray.get with NIXL
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Unlike the collective-based tensor transports (Gloo and NCCL), the :func:`ray.get <ray.get>` function can use NIXL to retrieve a copy of the result.
By default, the tensor transport for :func:`ray.get <ray.get>` will be the one specified in the :func:`@ray.method <ray.method>` decorator.
.. literalinclude:: ../doc_code/direct_transport_nixl.py
:language: python
:start-after: __nixl_get_start__
:end-before: __nixl_get_end__
You can also use NIXL to retrieve the result from references created by :func:`ray.put <ray.put>`.
.. literalinclude:: ../doc_code/direct_transport_nixl.py
:language: python
:start-after: __nixl_put__and_get_start__
:end-before: __nixl_put__and_get_end__
Summary
-------
RDT allows Ray to store and pass objects directly between Ray actors, using accelerated transports like GLOO, NCCL, and NIXL.
Here are the main points to keep in mind:
* If using a collective-based tensor transport (Gloo or NCCL), a collective group must be created ahead of time. NIXL just requires all involved actors to have NIXL installed.
* Unlike objects in the Ray object store, RDT objects are *mutable*, meaning that Ray only holds a reference, not a copy, to the stored tensor(s).
* Otherwise, actors can be used as normal.
For a full list of limitations, see the :ref:`limitations <limitations>` section.
Microbenchmarks
===============
.. note::
Under construction.
.. _limitations:
Limitations
===========
RDT is currently in alpha and currently has the following limitations, which may be addressed in future releases:
* Support for ``torch.Tensor`` objects only.
* Support for Ray actors only, not Ray tasks.
* Support for the following transports: GLOO, NCCL, and NIXL.
* Support for CPUs and NVIDIA GPUs only.
* RDT objects are *mutable*. This means that Ray only holds a reference to the tensor, and will not copy it until a transfer is requested. Thus, if the application code also keeps a reference to a tensor before returning it, and modifies the tensor in place, then some or all of the changes may be seen by the receiving actor.
* `await` on an RDT ref is temporarily not supported.
For collective-based / two-sided tensor transports (Gloo and NCCL):
* Only the process that created the collective group can submit actor tasks that return and pass RDT objects. If the creating process passes the actor handles to other processes, those processes can submit actor tasks as usual, but will not be able to use RDT objects.
* Similarly, the process that created the collective group cannot serialize and pass RDT :class:`ray.ObjectRefs <ray.ObjectRef>` to other Ray tasks or actors. Instead, the :class:`ray.ObjectRef`\s can only be passed as direct arguments to other actor tasks, and those actors must be in the same collective group.
* Each actor can only be in one collective group per tensor transport at a time.
* No support for :func:`ray.put <ray.put>`.
* No support for out-of-order actors such as async actors or actors with ``max_concurrency`` > 1.
Due to a known issue, for NIXL, we currently do not support storing different GPU objects at the same actor, where the objects contain an overlapping but not equal set of tensors. To support this pattern, ensure that the first `ObjectRef` has gone out of scope before storing the same tensor(s) again in a second object.
.. literalinclude:: ../doc_code/direct_transport_nixl.py
:language: python
:start-after: __nixl_limitations_start__
:end-before: __nixl_limitations_end__
Error handling
==============
* Application-level errors, i.e. exceptions raised by user code, will not destroy the collective group and will instead be propagated to any dependent task(s), as for non-RDT Ray objects.
* If a system-level error occurs during a GLOO or NCCL collective operation, the collective group will be destroyed and the actors will be killed to prevent any hanging.
* If a system-level error occurs during a NIXL transfer, Ray or NIXL will abort the transfer with an exception and Ray will raise the exception in the dependent task or on the ray.get on the NIXL ref.
* System-level errors include:
* Errors internal to the third-party transport, e.g., NCCL network errors
* Actor or node failures
* Transport errors due to tensor device / transport mismatches, e.g., a CPU tensor when using NCCL
* Ray RDT object fetch timeouts (can be overridden by setting the ``RAY_rdt_fetch_fail_timeout_milliseconds`` environment variable)
* Any unexpected system bugs
Advanced: Registering a custom tensor transport
===============================================
Ray allows you to register custom tensor transports at runtime for use with RDT.
To implement a custom tensor transport, you can implement the abstract interface :class:`ray.experimental.TensorTransportManager <ray.experimental.TensorTransportManager>` and register it using :func:`ray.experimental.register_tensor_transport <ray.experimental.register_tensor_transport>`.
For a complete guide on implementing custom tensor transports, including detailed documentation of all required methods, see :ref:`custom-tensor-transport`.
Advanced: RDT Internals
=======================
.. note::
Under construction.
Table of Contents
-----------------
Learn more details about Ray Direct Transport from the following links.
.. toctree::
:maxdepth: 1
custom-tensor-transport
doc/source/ray-core/doc_code/actor-http-server.py
+32
View File
@@ -0,0 +1,32 @@
import ray
import asyncio
import requests
from aiohttp import web
@ray.remote
class Counter:
async def __init__(self):
self.counter = 0
asyncio.get_running_loop().create_task(self.run_http_server())
async def run_http_server(self):
app = web.Application()
app.add_routes([web.get("/", self.get)])
runner = web.AppRunner(app)
await runner.setup()
site = web.TCPSite(runner, "127.0.0.1", 25001)
await site.start()
async def get(self, request):
return web.Response(text=str(self.counter))
async def increment(self):
self.counter = self.counter + 1
ray.init()
counter = Counter.remote()
[ray.get(counter.increment.remote()) for i in range(5)]
r = requests.get("http://127.0.0.1:25001/")
assert r.text == "5"
doc/source/ray-core/doc_code/actor-pool.py
+17
View File
@@ -0,0 +1,17 @@
import ray
from ray.util import ActorPool
@ray.remote
class Actor:
def double(self, n):
return n * 2
a1, a2 = Actor.remote(), Actor.remote()
pool = ActorPool([a1, a2])
# pool.map(..) returns a Python generator object ActorPool.map
gen = pool.map(lambda a, v: a.double.remote(v), [1, 2, 3, 4])
print(list(gen))
# [2, 4, 6, 8]
doc/source/ray-core/doc_code/actor-queue.py
+23
View File
@@ -0,0 +1,23 @@
import ray
from ray.util.queue import Queue, Empty
ray.init()
# You can pass this object around to different tasks/actors
queue = Queue(maxsize=100)
@ray.remote
def consumer(id, queue):
try:
while True:
next_item = queue.get(block=True, timeout=1)
print(f"consumer {id} got work {next_item}")
except Empty:
pass
[queue.put(i) for i in range(10)]
print("Put work 1 - 10 to queue...")
consumers = [consumer.remote(id, queue) for id in range(2)]
ray.get(consumers)
doc/source/ray-core/doc_code/actor-repr.py
+19
View File
@@ -0,0 +1,19 @@
import ray
@ray.remote
class MyActor:
def __init__(self, index):
self.index = index
def foo(self):
print("hello there")
def __repr__(self):
return f"MyActor(index={self.index})"
a = MyActor.remote(1)
b = MyActor.remote(2)
ray.get(a.foo.remote())
ray.get(b.foo.remote())
doc/source/ray-core/doc_code/actor-sync.py
+46
View File
@@ -0,0 +1,46 @@
import asyncio
import ray
# We set num_cpus to zero because this actor will mostly just block on I/O.
@ray.remote(num_cpus=0)
class SignalActor:
def __init__(self):
self.ready_event = asyncio.Event()
def send(self, clear=False):
self.ready_event.set()
if clear:
self.ready_event.clear()
async def wait(self, should_wait=True):
if should_wait:
await self.ready_event.wait()
@ray.remote
def wait_and_go(signal):
ray.get(signal.wait.remote())
print("go!")
signal = SignalActor.remote()
tasks = [wait_and_go.remote(signal) for _ in range(4)]
print("ready...")
# Tasks will all be waiting for the signals.
print("set..")
ray.get(signal.send.remote())
# Tasks are unblocked.
ray.get(tasks)
# Output is:
# ready...
# set..
# (wait_and_go pid=77366) go!
# (wait_and_go pid=77372) go!
# (wait_and_go pid=77367) go!
# (wait_and_go pid=77358) go!
doc/source/ray-core/doc_code/actor_checkpointing.py
+95
View File
@@ -0,0 +1,95 @@
# __actor_checkpointing_manual_restart_begin__
import os
import sys
import ray
import json
import tempfile
import shutil
@ray.remote(num_cpus=1)
class Worker:
def __init__(self):
self.state = {"num_tasks_executed": 0}
def execute_task(self, crash=False):
if crash:
sys.exit(1)
# Execute the task
# ...
# Update the internal state
self.state["num_tasks_executed"] = self.state["num_tasks_executed"] + 1
def checkpoint(self):
return self.state
def restore(self, state):
self.state = state
class Controller:
def __init__(self):
self.worker = Worker.remote()
self.worker_state = ray.get(self.worker.checkpoint.remote())
def execute_task_with_fault_tolerance(self):
i = 0
while True:
i = i + 1
try:
ray.get(self.worker.execute_task.remote(crash=(i % 2 == 1)))
# Checkpoint the latest worker state
self.worker_state = ray.get(self.worker.checkpoint.remote())
return
except ray.exceptions.RayActorError:
print("Actor crashes, restarting...")
# Restart the actor and restore the state
self.worker = Worker.remote()
ray.get(self.worker.restore.remote(self.worker_state))
controller = Controller()
controller.execute_task_with_fault_tolerance()
controller.execute_task_with_fault_tolerance()
assert ray.get(controller.worker.checkpoint.remote())["num_tasks_executed"] == 2
# __actor_checkpointing_manual_restart_end__
# __actor_checkpointing_auto_restart_begin__
@ray.remote(max_restarts=-1, max_task_retries=-1)
class ImmortalActor:
def __init__(self, checkpoint_file):
self.checkpoint_file = checkpoint_file
if os.path.exists(self.checkpoint_file):
# Restore from a checkpoint
with open(self.checkpoint_file, "r") as f:
self.state = json.load(f)
else:
self.state = {}
def update(self, key, value):
import random
if random.randrange(10) < 5:
sys.exit(1)
self.state[key] = value
# Checkpoint the latest state
with open(self.checkpoint_file, "w") as f:
json.dump(self.state, f)
def get(self, key):
return self.state[key]
checkpoint_dir = tempfile.mkdtemp()
actor = ImmortalActor.remote(os.path.join(checkpoint_dir, "checkpoint.json"))
ray.get(actor.update.remote("1", 1))
ray.get(actor.update.remote("2", 2))
assert ray.get(actor.get.remote("1")) == 1
shutil.rmtree(checkpoint_dir)
# __actor_checkpointing_auto_restart_end__
doc/source/ray-core/doc_code/actor_creator_failure.py
+44
View File
@@ -0,0 +1,44 @@
# flake8: noqa
# fmt: off
# __actor_creator_failure_begin__
import ray
import os
import signal
ray.init()
@ray.remote(max_restarts=-1)
class Actor:
def ping(self):
return "hello"
@ray.remote
class Parent:
def generate_actors(self):
self.child = Actor.remote()
self.detached_actor = Actor.options(name="actor", lifetime="detached").remote()
return self.child, self.detached_actor, os.getpid()
parent = Parent.remote()
actor, detached_actor, pid = ray.get(parent.generate_actors.remote())
os.kill(pid, signal.SIGKILL)
try:
print("actor.ping:", ray.get(actor.ping.remote()))
except ray.exceptions.RayActorError as e:
print("Failed to submit actor call", e)
# Failed to submit actor call The actor died unexpectedly before finishing this task.
# class_name: Actor
# actor_id: 56f541b178ff78470f79c3b601000000
# namespace: ea8b3596-7426-4aa8-98cc-9f77161c4d5f
# The actor is dead because because all references to the actor were removed.
try:
print("detached_actor.ping:", ray.get(detached_actor.ping.remote()))
except ray.exceptions.RayActorError as e:
print("Failed to submit detached actor call", e)
# detached_actor.ping: hello
# __actor_creator_failure_end__
# fmt: on
doc/source/ray-core/doc_code/actor_restart.py
+44
View File
@@ -0,0 +1,44 @@
# flake8: noqa
# fmt: off
# __actor_restart_begin__
import os
import ray
ray.init()
# This actor kills itself after executing 10 tasks.
@ray.remote(max_restarts=4, max_task_retries=-1)
class Actor:
def __init__(self):
self.counter = 0
def increment_and_possibly_fail(self):
# Exit after every 10 tasks.
if self.counter == 10:
os._exit(0)
self.counter += 1
return self.counter
actor = Actor.remote()
# The actor will be reconstructed up to 4 times, so we can execute up to 50
# tasks successfully. The actor is reconstructed by rerunning its constructor.
# Methods that were executing when the actor died will be retried and will not
# raise a `RayActorError`. Retried methods may execute twice, once on the
# failed actor and a second time on the restarted actor.
for _ in range(50):
counter = ray.get(actor.increment_and_possibly_fail.remote())
print(counter) # Prints the sequence 1-10 5 times.
# After the actor has been restarted 4 times, all subsequent methods will
# raise a `RayActorError`.
for _ in range(10):
try:
counter = ray.get(actor.increment_and_possibly_fail.remote())
print(counter) # Unreachable.
except ray.exceptions.RayActorError:
print("FAILURE") # Prints 10 times.
# __actor_restart_end__
# fmt: on
doc/source/ray-core/doc_code/actors.py
+91
View File
@@ -0,0 +1,91 @@
# flake8: noqa
# __cancel_start__
import ray
import asyncio
import time
@ray.remote
class Actor:
async def f(self):
try:
await asyncio.sleep(5)
except asyncio.CancelledError:
print("Actor task canceled.")
actor = Actor.remote()
ref = actor.f.remote()
# Wait until task is scheduled.
time.sleep(1)
ray.cancel(ref)
try:
ray.get(ref)
except ray.exceptions.RayTaskError:
print("Object reference was cancelled.")
# __cancel_end__
# __cancel_graceful_actor_start__
import ray
import time
@ray.remote
class SyncActor:
def __init__(self):
self.is_canceled = False
def long_running_method(self):
"""A sync actor method that checks for cancellation periodically."""
for i in range(100):
# For sync actor tasks, is_canceled() can be checked in the task body
if ray.get_runtime_context().is_canceled():
self.is_canceled = True
print("Actor task canceled, cleaning up...")
return "canceled"
time.sleep(0.1)
return "completed"
def get_cancel_status(self):
return self.is_canceled
# Sync actor task cancellation with periodic checking
actor = SyncActor.remote()
actor_task_ref = actor.long_running_method.remote()
# Wait until task is scheduled.
time.sleep(1)
ray.cancel(actor_task_ref)
# The TaskCancelledError will be raised when calling ray.get
try:
result = ray.get(actor_task_ref)
except ray.exceptions.TaskCancelledError:
print("Actor task was cancelled")
# The get_cancel_status will return True after cancellation
cancel_status = ray.get(actor.get_cancel_status.remote())
print(f"Actor detected cancellation: {cancel_status}")
# __cancel_graceful_actor_end__
# __enable_task_events_start__
@ray.remote
class FooActor:
# Disable task events reporting for this method.
@ray.method(enable_task_events=False)
def foo(self):
pass
foo_actor = FooActor.remote()
ray.get(foo_actor.foo.remote())
# __enable_task_events_end__
doc/source/ray-core/doc_code/anti_pattern_closure_capture_large_objects.py
+44
View File
@@ -0,0 +1,44 @@
# __anti_pattern_start__
import ray
import numpy as np
ray.init()
large_object = np.zeros(10 * 1024 * 1024)
@ray.remote
def f1():
return len(large_object) # large_object is serialized along with f1!
ray.get(f1.remote())
# __anti_pattern_end__
# __better_approach_1_start__
large_object_ref = ray.put(np.zeros(10 * 1024 * 1024))
@ray.remote
def f2(large_object):
return len(large_object)
# Large object is passed through object store.
ray.get(f2.remote(large_object_ref))
# __better_approach_1_end__
# __better_approach_2_start__
large_object_creator = lambda: np.zeros(10 * 1024 * 1024) # noqa E731
@ray.remote
def f3():
large_object = (
large_object_creator()
) # Lambda is small compared with the large object.
return len(large_object)
ray.get(f3.remote())
# __better_approach_2_end__
doc/source/ray-core/doc_code/anti_pattern_fork_new_processes.py
+44
View File
@@ -0,0 +1,44 @@
import os
os.environ["RAY_DEDUP_LOGS"] = "0"
import ray
from concurrent.futures import ProcessPoolExecutor, as_completed
import multiprocessing
import numpy as np
@ray.remote
def generate_response(request):
print(request)
array = np.ones(100000)
return array
def process_response(response, idx):
print(f"Processing response {idx}")
return response
def main():
ray.init()
responses = ray.get([generate_response.remote(f"request {i}") for i in range(4)])
# Better approach: Set the start method to "spawn"
multiprocessing.set_start_method("spawn", force=True)
with ProcessPoolExecutor(max_workers=4) as executor:
future_to_task = {}
for idx, response in enumerate(responses):
future_to_task[executor.submit(process_response, response, idx)] = idx
for future in as_completed(future_to_task):
idx = future_to_task[future]
response_entry = future.result()
print(f"Response {idx} processed: {response_entry}")
ray.shutdown()
if __name__ == "__main__":
main()
doc/source/ray-core/doc_code/anti_pattern_global_variables.py
+51
View File
@@ -0,0 +1,51 @@
# __anti_pattern_start__
import ray
ray.init()
global_var = 3
@ray.remote
class Actor:
def f(self):
return global_var + 3
actor = Actor.remote()
global_var = 4
# This returns 6, not 7. It is because the value change of global_var
# inside a driver is not reflected to the actor
# because they are running in different processes.
assert ray.get(actor.f.remote()) == 6
# __anti_pattern_end__
# __better_approach_start__
@ray.remote
class GlobalVarActor:
def __init__(self):
self.global_var = 3
def set_global_var(self, var):
self.global_var = var
def get_global_var(self):
return self.global_var
@ray.remote
class Actor:
def __init__(self, global_var_actor):
self.global_var_actor = global_var_actor
def f(self):
return ray.get(self.global_var_actor.get_global_var.remote()) + 3
global_var_actor = GlobalVarActor.remote()
actor = Actor.remote(global_var_actor)
ray.get(global_var_actor.set_global_var.remote(4))
# This returns 7 correctly.
assert ray.get(actor.f.remote()) == 7
# __better_approach_end__
doc/source/ray-core/doc_code/anti_pattern_nested_ray_get.py
+27
View File
@@ -0,0 +1,27 @@
# __anti_pattern_start__
import ray
import time
@ray.remote
def f():
return 1
@ray.remote
def pass_via_nested_ref(refs):
print(sum(ray.get(refs)))
@ray.remote
def pass_via_direct_arg(*args):
print(sum(args))
# Anti-pattern: Passing nested refs requires `ray.get` in a nested task.
ray.get(pass_via_nested_ref.remote([f.remote() for _ in range(3)]))
# Better approach: Pass refs as direct arguments. Use *args syntax to unpack
# multiple arguments.
ray.get(pass_via_direct_arg.remote(*[f.remote() for _ in range(3)]))
# __anti_pattern_end__
doc/source/ray-core/doc_code/anti_pattern_out_of_band_object_ref_serialization.py
+64
View File
@@ -0,0 +1,64 @@
# __anti_pattern_start__
import ray
import pickle
from ray._private.internal_api import memory_summary
import ray.exceptions
ray.init()
@ray.remote
def out_of_band_serialization_pickle():
obj_ref = ray.put(1)
import pickle
# object_ref is serialized from user code using a regular pickle.
# Ray can't keep track of the reference, so the underlying object
# can be GC'ed unexpectedly, which can cause unexpected hangs.
return pickle.dumps(obj_ref)
@ray.remote
def out_of_band_serialization_ray_cloudpickle():
obj_ref = ray.put(1)
from ray import cloudpickle
# ray.cloudpickle can serialize only when
# RAY_allow_out_of_band_object_ref_serialization=1 env var is set.
# However, the object_ref is pinned for the lifetime of the worker,
# which can cause Ray object leaks that can cause spilling.
return cloudpickle.dumps(obj_ref)
print("==== serialize object ref with pickle ====")
result = ray.get(out_of_band_serialization_pickle.remote())
try:
ray.get(pickle.loads(result), timeout=5)
except ray.exceptions.GetTimeoutError:
print("Underlying object is unexpectedly GC'ed!\n\n")
print("==== serialize object ref with ray.cloudpickle ====")
# By default, it's allowed to serialize ray.ObjectRef using
# ray.cloudpickle.
ray.get(out_of_band_serialization_ray_cloudpickle.options().remote())
# you can see objects are still pinned although it's GC'ed and not used anymore.
print(memory_summary())
print(
"==== serialize object ref with ray.cloudpickle with env var "
"RAY_allow_out_of_band_object_ref_serialization=0 for debugging ===="
)
try:
ray.get(
out_of_band_serialization_ray_cloudpickle.options(
runtime_env={
"env_vars": {
"RAY_allow_out_of_band_object_ref_serialization": "0",
}
}
).remote()
)
except Exception as e:
print(f"Exception raised from out_of_band_serialization_ray_cloudpickle {e}\n\n")
# __anti_pattern_end__
doc/source/ray-core/doc_code/anti_pattern_pass_large_arg_by_value.py
+23
View File
@@ -0,0 +1,23 @@
# __anti_pattern_start__
import ray
import numpy as np
ray.init()
@ray.remote
def func(large_arg, i):
return len(large_arg) + i
large_arg = np.zeros(1024 * 1024)
# 10 copies of large_arg are stored in the object store.
outputs = ray.get([func.remote(large_arg, i) for i in range(10)])
# __anti_pattern_end__
# __better_approach_start__
# 1 copy of large_arg is stored in the object store.
large_arg_ref = ray.put(large_arg)
outputs = ray.get([func.remote(large_arg_ref, i) for i in range(10)])
# __better_approach_end__
doc/source/ray-core/doc_code/anti_pattern_ray_get_loop.py
+25
View File
@@ -0,0 +1,25 @@
# __anti_pattern_start__
import ray
ray.init()
@ray.remote
def f(i):
return i
# Anti-pattern: no parallelism due to calling ray.get inside of the loop.
sequential_returns = []
for i in range(100):
sequential_returns.append(ray.get(f.remote(i)))
# Better approach: parallelism because the tasks are executed in parallel.
refs = []
for i in range(100):
refs.append(f.remote(i))
parallel_returns = ray.get(refs)
# __anti_pattern_end__
assert sequential_returns == parallel_returns
doc/source/ray-core/doc_code/anti_pattern_ray_get_submission_order.py
+36
View File
@@ -0,0 +1,36 @@
# __anti_pattern_start__
import random
import time
import ray
ray.init()
@ray.remote
def f(i):
time.sleep(random.random())
return i
# Anti-pattern: process results in the submission order.
sum_in_submission_order = 0
refs = [f.remote(i) for i in range(100)]
for ref in refs:
# Blocks until this ObjectRef is ready.
result = ray.get(ref)
# process result
sum_in_submission_order = sum_in_submission_order + result
# Better approach: process results in the completion order.
sum_in_completion_order = 0
refs = [f.remote(i) for i in range(100)]
unfinished = refs
while unfinished:
# Returns the first ObjectRef that is ready.
finished, unfinished = ray.wait(unfinished, num_returns=1)
result = ray.get(finished[0])
# process result
sum_in_completion_order = sum_in_completion_order + result
# __anti_pattern_end__
assert sum_in_submission_order == sum_in_completion_order
doc/source/ray-core/doc_code/anti_pattern_ray_get_too_many_objects.py
+37
View File
@@ -0,0 +1,37 @@
# __anti_pattern_start__
import ray
import numpy as np
ray.init()
def process_results(results):
# custom process logic
pass
@ray.remote
def return_big_object():
return np.zeros(1024 * 10)
NUM_TASKS = 1000
object_refs = [return_big_object.remote() for _ in range(NUM_TASKS)]
# This will fail with heap out-of-memory
# or object store out-of-space if NUM_TASKS is large enough.
results = ray.get(object_refs)
process_results(results)
# __anti_pattern_end__
# __better_approach_start__
BATCH_SIZE = 100
while object_refs:
# Process results in the finish order instead of the submission order.
ready_object_refs, object_refs = ray.wait(object_refs, num_returns=BATCH_SIZE)
# The node only needs enough space to store
# a batch of objects instead of all objects.
results = ray.get(ready_object_refs)
process_results(results)
# __better_approach_end__
doc/source/ray-core/doc_code/anti_pattern_redefine_task_actor_loop.py
+34
View File
@@ -0,0 +1,34 @@
# __anti_pattern_start__
import ray
ray.init()
outputs = []
for i in range(10):
@ray.remote
def double(i):
return i * 2
outputs.append(double.remote(i))
outputs = ray.get(outputs)
# The double remote function is pickled and uploaded 10 times.
# __anti_pattern_end__
assert outputs == [i * 2 for i in range(10)]
# __better_approach_start__
@ray.remote
def double(i):
return i * 2
outputs = []
for i in range(10):
outputs.append(double.remote(i))
outputs = ray.get(outputs)
# The double remote function is pickled and uploaded 1 time.
# __better_approach_end__
assert outputs == [i * 2 for i in range(10)]
doc/source/ray-core/doc_code/anti_pattern_return_ray_put.py
+151
View File
@@ -0,0 +1,151 @@
# __return_single_value_start__
import ray
import numpy as np
@ray.remote
def task_with_single_small_return_value_bad():
small_return_value = 1
# The value will be stored in the object store
# and the reference is returned to the caller.
small_return_value_ref = ray.put(small_return_value)
return small_return_value_ref
@ray.remote
def task_with_single_small_return_value_good():
small_return_value = 1
# Ray will return the value inline to the caller
# which is faster than the previous approach.
return small_return_value
assert ray.get(ray.get(task_with_single_small_return_value_bad.remote())) == ray.get(
task_with_single_small_return_value_good.remote()
)
@ray.remote
def task_with_single_large_return_value_bad():
large_return_value = np.zeros(10 * 1024 * 1024)
large_return_value_ref = ray.put(large_return_value)
return large_return_value_ref
@ray.remote
def task_with_single_large_return_value_good():
# Both approaches will store the large array to the object store
# but this is better since it's faster and more fault tolerant.
large_return_value = np.zeros(10 * 1024 * 1024)
return large_return_value
assert np.array_equal(
ray.get(ray.get(task_with_single_large_return_value_bad.remote())),
ray.get(task_with_single_large_return_value_good.remote()),
)
# Same thing applies for actor tasks as well.
@ray.remote
class Actor:
def task_with_single_return_value_bad(self):
single_return_value = np.zeros(9 * 1024 * 1024)
return ray.put(single_return_value)
def task_with_single_return_value_good(self):
return np.zeros(9 * 1024 * 1024)
actor = Actor.remote()
assert np.array_equal(
ray.get(ray.get(actor.task_with_single_return_value_bad.remote())),
ray.get(actor.task_with_single_return_value_good.remote()),
)
# __return_single_value_end__
# __return_static_multi_values_start__
# This will return a single object
# which is a tuple of two ObjectRefs to the actual values.
@ray.remote(num_returns=1)
def task_with_static_multiple_returns_bad1():
return_value_1_ref = ray.put(1)
return_value_2_ref = ray.put(2)
return (return_value_1_ref, return_value_2_ref)
# This will return two objects each of which is an ObjectRef to the actual value.
@ray.remote(num_returns=2)
def task_with_static_multiple_returns_bad2():
return_value_1_ref = ray.put(1)
return_value_2_ref = ray.put(2)
return (return_value_1_ref, return_value_2_ref)
# This will return two objects each of which is the actual value.
@ray.remote(num_returns=2)
def task_with_static_multiple_returns_good():
return_value_1 = 1
return_value_2 = 2
return (return_value_1, return_value_2)
assert (
ray.get(ray.get(task_with_static_multiple_returns_bad1.remote())[0])
== ray.get(ray.get(task_with_static_multiple_returns_bad2.remote()[0]))
== ray.get(task_with_static_multiple_returns_good.remote()[0])
)
@ray.remote
class Actor:
@ray.method(num_returns=1)
def task_with_static_multiple_returns_bad1(self):
return_value_1_ref = ray.put(1)
return_value_2_ref = ray.put(2)
return (return_value_1_ref, return_value_2_ref)
@ray.method(num_returns=2)
def task_with_static_multiple_returns_bad2(self):
return_value_1_ref = ray.put(1)
return_value_2_ref = ray.put(2)
return (return_value_1_ref, return_value_2_ref)
@ray.method(num_returns=2)
def task_with_static_multiple_returns_good(self):
# This is faster and more fault tolerant.
return_value_1 = 1
return_value_2 = 2
return (return_value_1, return_value_2)
actor = Actor.remote()
assert (
ray.get(ray.get(actor.task_with_static_multiple_returns_bad1.remote())[0])
== ray.get(ray.get(actor.task_with_static_multiple_returns_bad2.remote()[0]))
== ray.get(actor.task_with_static_multiple_returns_good.remote()[0])
)
# __return_static_multi_values_end__
# __return_dynamic_multi_values_start__
@ray.remote(num_returns=1)
def task_with_dynamic_returns_bad(n):
return_value_refs = []
for i in range(n):
return_value_refs.append(ray.put(np.zeros(i * 1024 * 1024)))
return return_value_refs
@ray.remote(num_returns="dynamic")
def task_with_dynamic_returns_good(n):
for i in range(n):
yield np.zeros(i * 1024 * 1024)
assert np.array_equal(
ray.get(ray.get(task_with_dynamic_returns_bad.remote(2))[0]),
ray.get(next(iter(ray.get(task_with_dynamic_returns_good.remote(2))))),
)
# __return_dynamic_multi_values_end__
doc/source/ray-core/doc_code/anti_pattern_too_fine_grained_tasks.py
+59
View File
@@ -0,0 +1,59 @@
# __anti_pattern_start__
import ray
import time
import itertools
ray.init()
numbers = list(range(10000))
def double(number):
time.sleep(0.00001)
return number * 2
start_time = time.time()
serial_doubled_numbers = [double(number) for number in numbers]
end_time = time.time()
print(f"Ordinary function call takes {end_time - start_time} seconds")
# Ordinary function call takes 0.16506004333496094 seconds
@ray.remote
def remote_double(number):
return double(number)
start_time = time.time()
doubled_number_refs = [remote_double.remote(number) for number in numbers]
parallel_doubled_numbers = ray.get(doubled_number_refs)
end_time = time.time()
print(f"Parallelizing tasks takes {end_time - start_time} seconds")
# Parallelizing tasks takes 1.6061789989471436 seconds
# __anti_pattern_end__
assert serial_doubled_numbers == parallel_doubled_numbers
# __batching_start__
@ray.remote
def remote_double_batch(numbers):
return [double(number) for number in numbers]
BATCH_SIZE = 1000
start_time = time.time()
doubled_batch_refs = []
for i in range(0, len(numbers), BATCH_SIZE):
batch = numbers[i : i + BATCH_SIZE]
doubled_batch_refs.append(remote_double_batch.remote(batch))
parallel_doubled_numbers_with_batching = list(
itertools.chain(*ray.get(doubled_batch_refs))
)
end_time = time.time()
print(f"Parallelizing tasks with batching takes {end_time - start_time} seconds")
# Parallelizing tasks with batching takes 0.030150890350341797 seconds
# __batching_end__
assert serial_doubled_numbers == parallel_doubled_numbers_with_batching
doc/source/ray-core/doc_code/anti_pattern_unnecessary_ray_get.py
+33
View File
@@ -0,0 +1,33 @@
# __anti_pattern_start__
import ray
import numpy as np
ray.init()
@ray.remote
def generate_rollout():
return np.ones((10000, 10000))
@ray.remote
def reduce(rollout):
return np.sum(rollout)
# `ray.get()` downloads the result here.
rollout = ray.get(generate_rollout.remote())
# Now we have to reupload `rollout`
reduced = ray.get(reduce.remote(rollout))
# __anti_pattern_end__
assert reduced == 100000000
# __better_approach_start__
# Don't need ray.get here.
rollout_obj_ref = generate_rollout.remote()
# Rollout object is passed by reference.
reduced = ray.get(reduce.remote(rollout_obj_ref))
# __better_approach_end__
assert reduced == 100000000
doc/source/ray-core/doc_code/cgraph_nccl.py
+75
View File
@@ -0,0 +1,75 @@
import os
# CI environment is slow, set the timeout to 60 seconds
os.environ["RAY_CGRAPH_get_timeout"] = "60"
os.environ["RAY_CGRAPH_submit_timeout"] = "60"
# __cgraph_cpu_to_gpu_actor_start__
import torch
import ray
import ray.dag
@ray.remote(num_gpus=1)
class GPUActor:
def process(self, tensor: torch.Tensor):
assert tensor.device.type == "cuda"
return tensor.shape
actor = GPUActor.remote()
# __cgraph_cpu_to_gpu_actor_end__
# __cgraph_cpu_to_gpu_start__
with ray.dag.InputNode() as inp:
inp = inp.with_tensor_transport(device="cuda")
dag = actor.process.bind(inp)
cdag = dag.experimental_compile()
print(ray.get(cdag.execute(torch.zeros(10))))
# __cgraph_cpu_to_gpu_end__
# __cgraph_nccl_setup_start__
import torch
import ray
import ray.dag
from ray.experimental.channel.torch_tensor_type import TorchTensorType
# Note that the following example requires at least 2 GPUs.
assert (
ray.available_resources().get("GPU") >= 2
), "At least 2 GPUs are required to run this example."
@ray.remote(num_gpus=1)
class GPUSender:
def send(self, shape):
return torch.zeros(shape, device="cuda")
@ray.remote(num_gpus=1)
class GPUReceiver:
def recv(self, tensor: torch.Tensor):
assert tensor.device.type == "cuda"
return tensor.shape
sender = GPUSender.remote()
receiver = GPUReceiver.remote()
# __cgraph_nccl_setup_end__
# __cgraph_nccl_exec_start__
with ray.dag.InputNode() as inp:
dag = sender.send.bind(inp)
# Add a type hint that the return value of `send` should use NCCL.
dag = dag.with_tensor_transport("nccl")
# NOTE: With ray<2.42, use `with_type_hint()` instead.
# dag = dag.with_type_hint(TorchTensorType(transport="nccl"))
dag = receiver.recv.bind(dag)
# Compile API prepares the NCCL communicator across all workers and schedule operations
# accordingly.
dag = dag.experimental_compile()
assert ray.get(dag.execute((10,))) == (10,)
# __cgraph_nccl_exec_end__
doc/source/ray-core/doc_code/cgraph_overlap.py
+70
View File
@@ -0,0 +1,70 @@
# __cgraph_overlap_start__
import ray
import time
import torch
from ray.dag import InputNode, MultiOutputNode
@ray.remote(num_cpus=0, num_gpus=1)
class TorchTensorWorker:
def send(self, shape, dtype, value: int, send_tensor=True):
if not send_tensor:
return 1
return torch.ones(shape, dtype=dtype, device="cuda") * value
def recv_and_matmul(self, two_d_tensor):
"""
Receive the tensor and do some expensive computation (matmul).
Args:
two_d_tensor: a 2D tensor that has the same size for its dimensions
"""
# Check that tensor got loaded to the correct device.
assert two_d_tensor.dim() == 2
assert two_d_tensor.size(0) == two_d_tensor.size(1)
torch.matmul(two_d_tensor, two_d_tensor)
return (two_d_tensor[0][0].item(), two_d_tensor.shape, two_d_tensor.dtype)
def test(overlap_gpu_communication):
num_senders = 3
senders = [TorchTensorWorker.remote() for _ in range(num_senders)]
receiver = TorchTensorWorker.remote()
shape = (10000, 10000)
dtype = torch.float16
with InputNode() as inp:
branches = [sender.send.bind(shape, dtype, inp) for sender in senders]
branches = [
branch.with_tensor_transport(
transport="nccl", _static_shape=True, _direct_return=True
)
# For a ray version before 2.42, use `with_type_hint()` instead.
# branch.with_type_hint(
# TorchTensorType(
# transport="nccl", _static_shape=True, _direct_return=True
# )
# )
for branch in branches
]
branches = [receiver.recv_and_matmul.bind(branch) for branch in branches]
dag = MultiOutputNode(branches)
compiled_dag = dag.experimental_compile(
_overlap_gpu_communication=overlap_gpu_communication
)
start = time.monotonic()
for i in range(5):
ref = compiled_dag.execute(i)
result = ray.get(ref)
assert result == [(i, shape, dtype)] * num_senders
duration = time.monotonic() - start
print(f"{overlap_gpu_communication=}, {duration=}")
compiled_dag.teardown(kill_actors=True)
for overlap_gpu_communication in [False, True]:
test(overlap_gpu_communication)
# __cgraph_overlap_end__
doc/source/ray-core/doc_code/cgraph_profiling.py
+63
View File
@@ -0,0 +1,63 @@
import os
import subprocess
from pathlib import Path
NSIGHT_FAKE_DIR = str(
Path(os.path.realpath(__file__)).parents[4]
/ "python"
/ "ray"
/ "tests"
/ "nsight_fake"
)
subprocess.check_call(
["pip", "install", NSIGHT_FAKE_DIR],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
# __profiling_setup_start__
import ray
import torch
from ray.dag import InputNode
@ray.remote(num_gpus=1, runtime_env={"nsight": "default"})
class RayActor:
def send(self, shape, dtype, value: int):
return torch.ones(shape, dtype=dtype, device="cuda") * value
def recv(self, tensor):
return (tensor[0].item(), tensor.shape, tensor.dtype)
sender = RayActor.remote()
receiver = RayActor.remote()
# __profiling_setup_end__
# __profiling_execution_start__
shape = (10,)
dtype = torch.float16
# Test normal execution.
with InputNode() as inp:
dag = sender.send.bind(inp.shape, inp.dtype, inp[0])
dag = dag.with_tensor_transport(transport="nccl")
dag = receiver.recv.bind(dag)
compiled_dag = dag.experimental_compile()
for i in range(3):
shape = (10 * (i + 1),)
ref = compiled_dag.execute(i, shape=shape, dtype=dtype)
assert ray.get(ref) == (i, shape, dtype)
# __profiling_execution_end__
subprocess.check_call(
["pip", "uninstall", "nsys", "--y"],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
doc/source/ray-core/doc_code/cgraph_quickstart.py
+174
View File
@@ -0,0 +1,174 @@
# __simple_actor_start__
import ray
@ray.remote
class SimpleActor:
def echo(self, msg):
return msg
# __simple_actor_end__
# __ray_core_usage_start__
import time
a = SimpleActor.remote()
# warmup
for _ in range(5):
msg_ref = a.echo.remote("hello")
ray.get(msg_ref)
start = time.perf_counter()
msg_ref = a.echo.remote("hello")
ray.get(msg_ref)
end = time.perf_counter()
print(f"Execution takes {(end - start) * 1000 * 1000} us")
# __ray_core_usage_end__
# __dag_usage_start__
import ray.dag
with ray.dag.InputNode() as inp:
# Note that it uses `bind` instead of `remote`.
# This returns a ray.dag.DAGNode, instead of the usual ray.ObjectRef.
dag = a.echo.bind(inp)
# warmup
for _ in range(5):
msg_ref = dag.execute("hello")
ray.get(msg_ref)
start = time.perf_counter()
# `dag.execute` runs the DAG and returns an ObjectRef. You can use `ray.get` API.
msg_ref = dag.execute("hello")
ray.get(msg_ref)
end = time.perf_counter()
print(f"Execution takes {(end - start) * 1000 * 1000} us")
# __dag_usage_end__
# __cgraph_usage_start__
dag = dag.experimental_compile()
# warmup
for _ in range(5):
msg_ref = dag.execute("hello")
ray.get(msg_ref)
start = time.perf_counter()
# `dag.execute` runs the DAG and returns CompiledDAGRef. Similar to
# ObjectRefs, you can use the ray.get API.
msg_ref = dag.execute("hello")
ray.get(msg_ref)
end = time.perf_counter()
print(f"Execution takes {(end - start) * 1000 * 1000} us")
# __cgraph_usage_end__
# __teardown_start__
dag.teardown()
# __teardown_end__
# __cgraph_bind_start__
a = SimpleActor.remote()
b = SimpleActor.remote()
with ray.dag.InputNode() as inp:
# Note that it uses `bind` instead of `remote`.
# This returns a ray.dag.DAGNode, instead of the usual ray.ObjectRef.
dag = a.echo.bind(inp)
dag = b.echo.bind(dag)
dag = dag.experimental_compile()
print(ray.get(dag.execute("hello")))
# __cgraph_bind_end__
dag.teardown()
# __cgraph_multi_output_start__
import ray.dag
a = SimpleActor.remote()
b = SimpleActor.remote()
with ray.dag.InputNode() as inp:
# Note that it uses `bind` instead of `remote`.
# This returns a ray.dag.DAGNode, instead of the usual ray.ObjectRef.
dag = ray.dag.MultiOutputNode([a.echo.bind(inp), b.echo.bind(inp)])
dag = dag.experimental_compile()
print(ray.get(dag.execute("hello")))
# __cgraph_multi_output_end__
dag.teardown()
# __cgraph_async_compile_start__
import ray
@ray.remote
class EchoActor:
def echo(self, msg):
return msg
actor = EchoActor.remote()
with ray.dag.InputNode() as inp:
dag = actor.echo.bind(inp)
cdag = dag.experimental_compile(enable_asyncio=True)
# __cgraph_async_compile_end__
# __cgraph_async_execute_start__
import asyncio
async def async_method(i):
fut = await cdag.execute_async(i)
result = await fut
assert result == i
loop = asyncio.get_event_loop()
loop.run_until_complete(async_method(42))
# __cgraph_async_execute_end__
cdag.teardown()
# __cgraph_actor_death_start__
from ray.dag import InputNode, MultiOutputNode
@ray.remote
class EchoActor:
def echo(self, msg):
return msg
actors = [EchoActor.remote() for _ in range(4)]
with InputNode() as inp:
outputs = [actor.echo.bind(inp) for actor in actors]
dag = MultiOutputNode(outputs)
compiled_dag = dag.experimental_compile()
# Kill one of the actors to simulate unexpected actor death.
ray.kill(actors[0])
ref = compiled_dag.execute(1)
live_actors = []
try:
ray.get(ref)
except ray.exceptions.ActorDiedError:
# At this point, the Compiled Graph is shutting down.
for actor in actors:
try:
# Check for live actors.
ray.get(actor.echo.remote("ping"))
live_actors.append(actor)
except ray.exceptions.RayActorError:
pass
# Optionally, use the live actors to create a new Compiled Graph.
assert live_actors == actors[1:]
# __cgraph_actor_death_end__
doc/source/ray-core/doc_code/cgraph_troubleshooting.py
+43
View File
@@ -0,0 +1,43 @@
# __numpy_troubleshooting_start__
import ray
import numpy as np
from ray.dag import InputNode
@ray.remote
class NumPyActor:
def get_arr(self, _):
numpy_arr = np.ones((5, 1024))
return numpy_arr
actor = NumPyActor.remote()
with InputNode() as inp:
dag = actor.get_arr.bind(inp)
cgraph = dag.experimental_compile()
for _ in range(5):
ref = cgraph.execute(0)
result = ray.get(ref)
# Adding this explicit del would fix any issues
# del result
# __numpy_troubleshooting_end__
# __teardown_troubleshooting_start__
@ray.remote
class SendActor:
def send(self, x):
return x
actor = SendActor.remote()
with InputNode() as inp:
dag = actor.send.bind(inp)
cgraph = dag.experimental_compile()
# Not adding this explicit teardown before reusing `actor` could cause problems
# cgraph.teardown()
with InputNode() as inp:
dag = actor.send.bind(inp)
cgraph = dag.experimental_compile()
# __teardown_troubleshooting_end__
doc/source/ray-core/doc_code/cgraph_visualize.py
+31
View File
@@ -0,0 +1,31 @@
# __cgraph_visualize_start__
import ray
from ray.dag import InputNode, MultiOutputNode
@ray.remote
class Worker:
def inc(self, x):
return x + 1
def double(self, x):
return x * 2
def echo(self, x):
return x
sender1 = Worker.remote()
sender2 = Worker.remote()
receiver = Worker.remote()
with InputNode() as inp:
w1 = sender1.inc.bind(inp)
w1 = receiver.echo.bind(w1)
w2 = sender2.double.bind(inp)
w2 = receiver.echo.bind(w2)
dag = MultiOutputNode([w1, w2])
compiled_dag = dag.experimental_compile()
compiled_dag.visualize()
# __cgraph_visualize_end__
doc/source/ray-core/doc_code/cross_language.py
+102
View File
@@ -0,0 +1,102 @@
# flake8: noqa
# fmt: off
# __crosslang_init_start__
import ray
ray.init(job_config=ray.job_config.JobConfig(code_search_path=["/path/to/code"]))
# __crosslang_init_end__
# fmt: on
# fmt: off
# __crosslang_multidir_start__
import ray
ray.init(job_config=ray.job_config.JobConfig(code_search_path="/path/to/jars:/path/to/pys"))
# __crosslang_multidir_end__
# fmt: on
# fmt: off
# __python_call_java_start__
import ray
with ray.init(job_config=ray.job_config.JobConfig(code_search_path=["/path/to/code"])):
# Define a Java class.
counter_class = ray.cross_language.java_actor_class(
"io.ray.demo.Counter")
# Create a Java actor and call actor method.
counter = counter_class.remote()
obj_ref1 = counter.increment.remote()
assert ray.get(obj_ref1) == 1
obj_ref2 = counter.increment.remote()
assert ray.get(obj_ref2) == 2
# Define a Java function.
add_function = ray.cross_language.java_function(
"io.ray.demo.Math", "add")
# Call the Java remote function.
obj_ref3 = add_function.remote(1, 2)
assert ray.get(obj_ref3) == 3
# __python_call_java_end__
# fmt: on
# fmt: off
# __python_module_start__
# /path/to/the_dir/ray_demo.py
import ray
@ray.remote
class Counter(object):
def __init__(self):
self.value = 0
def increment(self):
self.value += 1
return self.value
@ray.remote
def add(a, b):
return a + b
# __python_module_end__
# fmt: on
# fmt: off
# __serialization_start__
# ray_serialization.py
import ray
@ray.remote
def py_return_input(v):
return v
# __serialization_end__
# fmt: on
# fmt: off
# __raise_exception_start__
# ray_exception.py
import ray
@ray.remote
def raise_exception():
1 / 0
# __raise_exception_end__
# fmt: on
# fmt: off
# __raise_exception_demo_start__
# ray_exception_demo.py
import ray
with ray.init(job_config=ray.job_config.JobConfig(code_search_path=["/path/to/ray_exception"])):
obj_ref = ray.cross_language.java_function(
"io.ray.demo.MyRayClass",
"raiseExceptionFromPython").remote()
ray.get(obj_ref) # <-- raise exception from here.
# __raise_exception_demo_end__
# fmt: on
doc/source/ray-core/doc_code/deser.py
+18
View File
@@ -0,0 +1,18 @@
import ray
import numpy as np
@ray.remote
def f(arr):
# arr = arr.copy() # Adding a copy will fix the error.
arr[0] = 1
try:
ray.get(f.remote(np.zeros(100)))
except ray.exceptions.RayTaskError as e:
print(e)
# ray.exceptions.RayTaskError(ValueError): ray::f()
# File "test.py", line 6, in f
# arr[0] = 1
# ValueError: assignment destination is read-only
doc/source/ray-core/doc_code/direct_transport_custom.py
+169
View File
@@ -0,0 +1,169 @@
# flake8: noqa
# __custom_metadata_start__
import multiprocessing.shared_memory as shm
import pickle
from dataclasses import dataclass
from typing import Any, Dict, List, Optional
import numpy
import ray
from ray.experimental import (
CommunicatorMetadata,
TensorTransportManager,
TensorTransportMetadata,
register_tensor_transport,
)
@dataclass
class ShmTransportMetadata(TensorTransportMetadata):
"""Custom metadata that stores the shared memory name and size."""
shm_name: Optional[str] = None
shm_size: Optional[int] = None
@dataclass
class ShmCommunicatorMetadata(CommunicatorMetadata):
"""No extra communicator metadata needed for shared memory."""
pass
# __custom_metadata_end__
# __custom_properties_start__
class SharedMemoryTransport(TensorTransportManager):
"""A one-sided tensor transport that transfers numpy arrays through shared memory."""
def __init__(self):
self.shared_memory_objects: Dict[str, shm.SharedMemory] = {}
def tensor_transport_backend(self) -> str:
return "shared_memory"
@staticmethod
def is_one_sided() -> bool:
return True
@staticmethod
def can_abort_transport() -> bool:
return False
def actor_has_tensor_transport(self, actor: "ray.actor.ActorHandle") -> bool:
return True
# __custom_properties_end__
# __custom_extract_start__
def extract_tensor_transport_metadata(
self,
obj_id: str,
rdt_object: List[numpy.ndarray],
) -> TensorTransportMetadata:
# Record shapes and dtypes.
tensor_meta = []
if rdt_object:
for tensor in rdt_object:
tensor_meta.append((tensor.shape, tensor.dtype))
# Serialize the tensors and store them in shared memory.
serialized_rdt_object = pickle.dumps(rdt_object)
size = len(serialized_rdt_object)
name = obj_id[:20]
shm_obj = shm.SharedMemory(name=name, create=True, size=size)
shm_obj.buf[:size] = serialized_rdt_object
self.shared_memory_objects[obj_id] = shm_obj
return ShmTransportMetadata(
tensor_meta=tensor_meta, tensor_device="cpu", shm_name=name, shm_size=size
)
# __custom_extract_end__
# __custom_communicator_start__
def get_communicator_metadata(
self,
src_actor: "ray.actor.ActorHandle",
dst_actor: "ray.actor.ActorHandle",
backend: Optional[str] = None,
) -> CommunicatorMetadata:
return ShmCommunicatorMetadata()
# __custom_communicator_end__
# __custom_send_recv_start__
def recv_multiple_tensors(
self,
obj_id: str,
tensor_transport_metadata: TensorTransportMetadata,
communicator_metadata: CommunicatorMetadata,
target_buffers: Optional[List[Any]] = None,
):
# Open the shared memory block and deserialize.
shm_name = tensor_transport_metadata.shm_name
size = tensor_transport_metadata.shm_size
shm_block = shm.SharedMemory(name=shm_name)
recv_tensors = pickle.loads(shm_block.buf[:size])
shm_block.close()
return recv_tensors
def send_multiple_tensors(
self,
tensors: List[numpy.ndarray],
tensor_transport_metadata: TensorTransportMetadata,
communicator_metadata: CommunicatorMetadata,
):
raise NotImplementedError("One-sided transport doesn't use send.")
# __custom_send_recv_end__
# __custom_cleanup_start__
def garbage_collect(
self,
obj_id: str,
tensor_transport_meta: TensorTransportMetadata,
tensors: List[numpy.ndarray],
):
self.shared_memory_objects[obj_id].close()
self.shared_memory_objects[obj_id].unlink()
del self.shared_memory_objects[obj_id]
def abort_transport(
self,
obj_id: str,
communicator_metadata: CommunicatorMetadata,
):
pass
# __custom_cleanup_end__
# __custom_usage_start__
register_tensor_transport(
"shared_memory", # Transport name
["cpu"], # Supported device types
SharedMemoryTransport, # TensorTransportManager class
numpy.ndarray, # Data type for this transport
)
@ray.remote
class MyActor:
@ray.method(tensor_transport="shared_memory")
def echo(self, data):
return data
def sum(self, data):
return data.sum().item()
actors = [MyActor.remote() for _ in range(2)]
ref = actors[0].echo.remote(numpy.array([1, 2, 3]))
result = actors[1].sum.remote(ref)
print(ray.get(result))
# 6
# __custom_usage_end__
doc/source/ray-core/doc_code/direct_transport_gloo.py
+277
View File
@@ -0,0 +1,277 @@
# flake8: noqa
# __normal_example_start__
import torch
import ray
@ray.remote
class MyActor:
def random_tensor(self):
return torch.randn(1000, 1000)
# __normal_example_end__
# __gloo_example_start__
@ray.remote
class MyActor:
@ray.method(tensor_transport="gloo")
def random_tensor(self):
return torch.randn(1000, 1000)
# __gloo_example_end__
# __gloo_group_start__
import torch
import ray
from ray.experimental.collective import create_collective_group
@ray.remote
class MyActor:
@ray.method(tensor_transport="gloo")
def random_tensor(self):
return torch.randn(1000, 1000)
def sum(self, tensor: torch.Tensor):
return torch.sum(tensor)
sender, receiver = MyActor.remote(), MyActor.remote()
# The tensor_transport specified here must match the one used in the @ray.method
# decorator.
group = create_collective_group([sender, receiver], backend="torch_gloo")
# __gloo_group_end__
# __gloo_group_destroy_start__
from ray.experimental.collective import destroy_collective_group
destroy_collective_group(group)
# __gloo_group_destroy_end__
# __gloo_full_example_start__
import torch
import ray
from ray.experimental.collective import create_collective_group
@ray.remote
class MyActor:
@ray.method(tensor_transport="gloo")
def random_tensor(self):
return torch.randn(1000, 1000)
def sum(self, tensor: torch.Tensor):
return torch.sum(tensor)
sender, receiver = MyActor.remote(), MyActor.remote()
group = create_collective_group([sender, receiver], backend="torch_gloo")
# The tensor will be stored by the `sender` actor instead of in Ray's object
# store.
tensor = sender.random_tensor.remote()
result = receiver.sum.remote(tensor)
print(ray.get(result))
# __gloo_full_example_end__
# __gloo_multiple_tensors_example_start__
import torch
import ray
from ray.experimental.collective import create_collective_group
@ray.remote
class MyActor:
@ray.method(tensor_transport="gloo")
def random_tensor_dict(self):
return {"tensor1": torch.randn(1000, 1000), "tensor2": torch.randn(1000, 1000)}
def sum(self, tensor_dict: dict):
return torch.sum(tensor_dict["tensor1"]) + torch.sum(tensor_dict["tensor2"])
sender, receiver = MyActor.remote(), MyActor.remote()
group = create_collective_group([sender, receiver], backend="torch_gloo")
# Both tensor values in the dictionary will be stored by the `sender` actor
# instead of in Ray's object store.
tensor_dict = sender.random_tensor_dict.remote()
result = receiver.sum.remote(tensor_dict)
print(ray.get(result))
# __gloo_multiple_tensors_example_end__
# __gloo_intra_actor_start__
import torch
import ray
import pytest
from ray.experimental.collective import create_collective_group
@ray.remote
class MyActor:
@ray.method(tensor_transport="gloo")
def random_tensor(self):
return torch.randn(1000, 1000)
def sum(self, tensor: torch.Tensor):
return torch.sum(tensor)
sender, receiver = MyActor.remote(), MyActor.remote()
group = create_collective_group([sender, receiver], backend="torch_gloo")
tensor = sender.random_tensor.remote()
# Pass the ObjectRef back to the actor that produced it. The tensor will be
# passed back to the same actor without copying.
sum1 = sender.sum.remote(tensor)
sum2 = receiver.sum.remote(tensor)
assert torch.allclose(*ray.get([sum1, sum2]))
# __gloo_intra_actor_end__
# __gloo_get_start__
# Wrong example of ray.get(). Since the tensor transport in the @ray.method decorator is Gloo,
# ray.get() will try to use Gloo to fetch the tensor, which is not supported
# because the caller is not part of the collective group.
with pytest.raises(ValueError) as e:
ray.get(tensor)
assert (
"ray.get is not allowed on RDT objects using the two-sided transport"
in str(e.value)
)
# Correct example of ray.get(), using the object store to fetch the RDT object because the caller
# is not part of the collective group.
print(ray.get(tensor, _use_object_store=True))
# torch.Tensor(...)
# __gloo_get_end__
# __gloo_object_mutability_warning_start__
import torch
import ray
from ray.experimental.collective import create_collective_group
@ray.remote
class MyActor:
@ray.method(tensor_transport="gloo")
def random_tensor(self):
return torch.randn(1000, 1000)
def increment_and_sum(self, tensor: torch.Tensor):
# In-place update.
tensor += 1
return torch.sum(tensor)
sender, receiver = MyActor.remote(), MyActor.remote()
group = create_collective_group([sender, receiver], backend="torch_gloo")
tensor = sender.random_tensor.remote()
tensor1 = sender.increment_and_sum.remote(tensor)
tensor2 = receiver.increment_and_sum.remote(tensor)
# A warning will be printed:
# UserWarning: GPU ObjectRef(...) is being passed back to the actor that created it Actor(MyActor, ...). Note that GPU objects are mutable. If the tensor is modified, Ray's internal copy will also be updated, and subsequent passes to other actors will receive the updated version instead of the original.
try:
# This assertion may fail because the tensor returned by sender.random_tensor
# is modified in-place by sender.increment_and_sum while being sent to
# receiver.increment_and_sum.
assert torch.allclose(ray.get(tensor1), ray.get(tensor2))
except AssertionError:
print("AssertionError: sender and receiver returned different sums.")
# __gloo_object_mutability_warning_end__
# __gloo_wait_tensor_freed_bad_start__
import torch
import ray
from ray.experimental.collective import create_collective_group
@ray.remote
class MyActor:
@ray.method(tensor_transport="gloo")
def random_tensor(self):
self.tensor = torch.randn(1000, 1000)
# After this function returns, Ray and this actor will both hold a
# reference to the same tensor.
return self.tensor
def increment_and_sum_stored_tensor(self):
# NOTE: In-place update, while Ray still holds a reference to the same tensor.
self.tensor += 1
return torch.sum(self.tensor)
def increment_and_sum(self, tensor: torch.Tensor):
return torch.sum(tensor + 1)
sender, receiver = MyActor.remote(), MyActor.remote()
group = create_collective_group([sender, receiver], backend="torch_gloo")
tensor = sender.random_tensor.remote()
tensor1 = sender.increment_and_sum_stored_tensor.remote()
# Wait for sender.increment_and_sum_stored_tensor task to finish.
tensor1 = ray.get(tensor1)
# Receiver will now receive the updated value instead of the original.
tensor2 = receiver.increment_and_sum.remote(tensor)
try:
# This assertion will fail because sender.increment_and_sum_stored_tensor
# modified the tensor in place before sending it to
# receiver.increment_and_sum.
assert torch.allclose(tensor1, ray.get(tensor2))
except AssertionError:
print("AssertionError: sender and receiver returned different sums.")
# __gloo_wait_tensor_freed_bad_end__
# __gloo_wait_tensor_freed_start__
import torch
import ray
from ray.experimental.collective import create_collective_group
@ray.remote
class MyActor:
@ray.method(tensor_transport="gloo")
def random_tensor(self):
self.tensor = torch.randn(1000, 1000)
return self.tensor
def increment_and_sum_stored_tensor(self):
# 1. Sender actor waits for Ray to release all references to the tensor
# before modifying the tensor in place.
ray.experimental.wait_tensor_freed(self.tensor)
# NOTE: In-place update, but Ray guarantees that it has already released
# its references to this tensor.
self.tensor += 1
return torch.sum(self.tensor)
def increment_and_sum(self, tensor: torch.Tensor):
# Receiver task remains the same.
return torch.sum(tensor + 1)
sender, receiver = MyActor.remote(), MyActor.remote()
group = create_collective_group([sender, receiver], backend="torch_gloo")
tensor = sender.random_tensor.remote()
tensor1 = sender.increment_and_sum_stored_tensor.remote()
# 2. Skip `ray.get`` because `wait_tensor_freed`` will block until all
# references to `tensor` are freed, so calling `ray.get` here would cause a
# deadlock.
# tensor1 = ray.get(tensor1)
tensor2 = receiver.increment_and_sum.remote(tensor)
# 3. Delete all references to `tensor`, to unblock wait_tensor_freed.
del tensor
# This assertion will now pass.
assert torch.allclose(ray.get(tensor1), ray.get(tensor2))
# __gloo_wait_tensor_freed_end__
doc/source/ray-core/doc_code/direct_transport_nccl.py
+27
View File
@@ -0,0 +1,27 @@
# flake8: noqa
# __nccl_full_example_start__
import torch
import ray
from ray.experimental.collective import create_collective_group
@ray.remote(num_gpus=1)
class MyActor:
@ray.method(tensor_transport="nccl")
def random_tensor(self):
return torch.randn(1000, 1000).cuda()
def sum(self, tensor: torch.Tensor):
return torch.sum(tensor)
sender, receiver = MyActor.remote(), MyActor.remote()
group = create_collective_group([sender, receiver], backend="nccl")
# The tensor will be stored by the `sender` actor instead of in Ray's object
# store.
tensor = sender.random_tensor.remote()
result = receiver.sum.remote(tensor)
ray.get(result)
# __nccl_full_example_end__
doc/source/ray-core/doc_code/direct_transport_nixl.py
+91
View File
@@ -0,0 +1,91 @@
# flake8: noqa
# __nixl_full_example_start__
import torch
import ray
@ray.remote(num_gpus=1)
class MyActor:
@ray.method(tensor_transport="nixl")
def random_tensor(self):
return torch.randn(1000, 1000).cuda()
def sum(self, tensor: torch.Tensor):
return torch.sum(tensor)
def produce(self, tensors):
refs = []
for t in tensors:
refs.append(ray.put(t, _tensor_transport="nixl"))
return refs
def consume_with_nixl(self, refs):
# ray.get will also use NIXL to retrieve the
# result.
tensors = [ray.get(ref) for ref in refs]
sum = 0
for t in tensors:
assert t.device.type == "cuda"
sum += t.sum().item()
return sum
# No collective group is needed. The two actors just need to have NIXL
# installed.
sender, receiver = MyActor.remote(), MyActor.remote()
# The tensor will be stored by the `sender` actor instead of in Ray's object
# store.
tensor = sender.random_tensor.remote()
result = receiver.sum.remote(tensor)
ray.get(result)
# __nixl_full_example_end__
# __nixl_get_start__
# ray.get will also use NIXL to retrieve the
# result.
print(ray.get(tensor))
# torch.Tensor(...)
# __nixl_get_end__
# __nixl_put__and_get_start__
tensor1 = torch.randn(1000, 1000).cuda()
tensor2 = torch.randn(1000, 1000).cuda()
refs = sender.produce.remote([tensor1, tensor2])
ref1 = receiver.consume_with_nixl.remote(refs)
print(ray.get(ref1))
# __nixl_put__and_get_end__
# __nixl_limitations_start__
@ray.remote(num_gpus=1)
class Actor:
def __init__(self):
self.tensor1 = torch.tensor([1, 2, 3])
self.tensor2 = torch.tensor([4, 5, 6])
self.tensor3 = torch.tensor([7, 8, 9])
@ray.method(tensor_transport="nixl")
def send_dict1(self):
return {"round1-1": self.tensor1, "round1-2": self.tensor2}
@ray.method(tensor_transport="nixl")
def send_dict2(self):
return {"round2-1": self.tensor1, "round2-3": self.tensor3}
def sum_dict(self, dict):
return sum(v.sum().item() for v in dict.values())
sender, receiver = Actor.remote(), Actor.remote()
ref1 = sender.send_dict1.remote()
result1 = receiver.sum_dict.remote(ref1)
print(ray.get(result1))
ref2 = sender.send_dict2.remote()
result2 = receiver.sum_dict.remote(ref2)
try:
print(ray.get(result2))
except ValueError as e:
print("Error caught:", e)
# __nixl_limitations_end__
doc/source/ray-core/doc_code/fault_tolerance_tips.py
+88
View File
@@ -0,0 +1,88 @@
# __return_ray_put_start__
import ray
# Non-fault tolerant version:
@ray.remote
def a():
x_ref = ray.put(1)
return x_ref
x_ref = ray.get(a.remote())
# Object x outlives its owner task A.
try:
# If owner of x (i.e. the worker process running task A) dies,
# the application can no longer get value of x.
print(ray.get(x_ref))
except ray.exceptions.OwnerDiedError:
pass
# __return_ray_put_end__
# __return_directly_start__
# Fault tolerant version:
@ray.remote
def a():
# Here we return the value directly instead of calling ray.put() first.
return 1
# The owner of x is the driver
# so x is accessible and can be auto recovered
# during the entire lifetime of the driver.
x_ref = a.remote()
print(ray.get(x_ref))
# __return_directly_end__
# __node_ip_resource_start__
@ray.remote
def b():
return 1
# If the node with ip 127.0.0.3 fails while task b is running,
# Ray cannot retry the task on other nodes.
b.options(resources={"node:127.0.0.3": 1}).remote()
# __node_ip_resource_end__
# __node_affinity_scheduling_strategy_start__
# Prefer running on the particular node specified by node id
# but can also run on other nodes if the target node fails.
b.options(
scheduling_strategy=ray.util.scheduling_strategies.NodeAffinitySchedulingStrategy(
node_id=ray.get_runtime_context().get_node_id(), soft=True
)
).remote()
# __node_affinity_scheduling_strategy_end__
# __manual_retry_start__
@ray.remote
class Actor:
def read_only(self):
import sys
import random
rand = random.random()
if rand < 0.2:
return 2 / 0
elif rand < 0.3:
sys.exit(1)
return 2
actor = Actor.remote()
# Manually retry the actor task.
while True:
try:
print(ray.get(actor.read_only.remote()))
break
except ZeroDivisionError:
pass
except ray.exceptions.RayActorError:
# Manually restart the actor
actor = Actor.remote()
# __manual_retry_end__
doc/source/ray-core/doc_code/generator.py
+120
View File
@@ -0,0 +1,120 @@
# __program_start__
import ray
from ray import DynamicObjectRefGenerator
# fmt: off
# __dynamic_generator_start__
import numpy as np
@ray.remote(num_returns="dynamic")
def split(array, chunk_size):
while len(array) > 0:
yield array[:chunk_size]
array = array[chunk_size:]
array_ref = ray.put(np.zeros(np.random.randint(1000_000)))
block_size = 1000
# Returns an ObjectRef[DynamicObjectRefGenerator].
dynamic_ref = split.remote(array_ref, block_size)
print(dynamic_ref)
# ObjectRef(c8ef45ccd0112571ffffffffffffffffffffffff0100000001000000)
i = -1
ref_generator = ray.get(dynamic_ref)
print(ref_generator)
# <ray._raylet.DynamicObjectRefGenerator object at 0x7f7e2116b290>
for i, ref in enumerate(ref_generator):
# Each DynamicObjectRefGenerator iteration returns an ObjectRef.
assert len(ray.get(ref)) <= block_size
num_blocks_generated = i + 1
array_size = len(ray.get(array_ref))
assert array_size <= num_blocks_generated * block_size
print(f"Split array of size {array_size} into {num_blocks_generated} blocks of "
f"size {block_size} each.")
# Split array of size 63153 into 64 blocks of size 1000 each.
# NOTE: The dynamic_ref points to the generated ObjectRefs. Make sure that this
# ObjectRef goes out of scope so that Ray can garbage-collect the internal
# ObjectRefs.
del dynamic_ref
# __dynamic_generator_end__
# fmt: on
# fmt: off
# __dynamic_generator_pass_start__
@ray.remote
def get_size(ref_generator : DynamicObjectRefGenerator):
print(ref_generator)
num_elements = 0
for ref in ref_generator:
array = ray.get(ref)
assert len(array) <= block_size
num_elements += len(array)
return num_elements
# Returns an ObjectRef[DynamicObjectRefGenerator].
dynamic_ref = split.remote(array_ref, block_size)
assert array_size == ray.get(get_size.remote(dynamic_ref))
# (get_size pid=1504184)
# <ray._raylet.DynamicObjectRefGenerator object at 0x7f81c4250ad0>
# This also works, but should be avoided because you have to call an additional
# `ray.get`, which blocks the driver.
ref_generator = ray.get(dynamic_ref)
assert array_size == ray.get(get_size.remote(ref_generator))
# (get_size pid=1504184)
# <ray._raylet.DynamicObjectRefGenerator object at 0x7f81c4251b50>
# __dynamic_generator_pass_end__
# fmt: on
# fmt: off
# __generator_errors_start__
@ray.remote
def generator():
for i in range(2):
yield i
raise Exception("error")
ref1, ref2, ref3, ref4 = generator.options(num_returns=4).remote()
assert ray.get([ref1, ref2]) == [0, 1]
# All remaining ObjectRefs will contain the error.
try:
ray.get([ref3, ref4])
except Exception as error:
print(error)
dynamic_ref = generator.options(num_returns="dynamic").remote()
ref_generator = ray.get(dynamic_ref)
ref1, ref2, ref3 = ref_generator
assert ray.get([ref1, ref2]) == [0, 1]
# Generators with num_returns="dynamic" will store the exception in the final
# ObjectRef.
try:
ray.get(ref3)
except Exception as error:
print(error)
# __generator_errors_end__
# fmt: on
# fmt: off
# __generator_errors_unsupported_start__
# Generators that yield more values than expected currently do not throw an
# exception (the error is only logged).
# See https://github.com/ray-project/ray/issues/28689.
ref1, ref2 = generator.options(num_returns=2).remote()
assert ray.get([ref1, ref2]) == [0, 1]
"""
(generator pid=2375938) 2022-09-28 11:08:51,386 ERROR worker.py:755 --
Unhandled error: Task threw exception, but all return values already
created. This should only occur when using generator tasks.
...
"""
# __generator_errors_unsupported_end__
# fmt: on
doc/source/ray-core/doc_code/get_or_create.py
+19
View File
@@ -0,0 +1,19 @@
import ray
@ray.remote
class Greeter:
def __init__(self, value):
self.value = value
def say_hello(self):
return self.value
# Actor `g1` doesn't yet exist, so it is created with the given args.
a = Greeter.options(name="g1", get_if_exists=True).remote("Old Greeting")
assert ray.get(a.say_hello.remote()) == "Old Greeting"
# Actor `g1` already exists, so it is returned (new args are ignored).
b = Greeter.options(name="g1", get_if_exists=True).remote("New Greeting")
assert ray.get(b.say_hello.remote()) == "Old Greeting"
doc/source/ray-core/doc_code/getting_started.py
+75
View File
@@ -0,0 +1,75 @@
# flake8: noqa
# __starting_ray_start__
import ray
ray.init()
# __starting_ray_end__
# fmt: off
# __running_task_start__
# Define the square task.
@ray.remote
def square(x):
return x * x
# Launch four parallel square tasks.
futures = [square.remote(i) for i in range(4)]
# Retrieve results.
print(ray.get(futures))
# -> [0, 1, 4, 9]
# __running_task_end__
# fmt: on
# fmt: off
# __calling_actor_start__
# Define the Counter actor.
@ray.remote
class Counter:
def __init__(self):
self.i = 0
def get(self):
return self.i
def incr(self, value):
self.i += value
# Create a Counter actor.
c = Counter.remote()
# Submit calls to the actor. These calls run asynchronously but in
# submission order on the remote actor process.
for _ in range(10):
c.incr.remote(1)
# Retrieve final actor state.
print(ray.get(c.get.remote()))
# -> 10
# __calling_actor_end__
# fmt: on
# fmt: off
# __passing_object_start__
import numpy as np
# Define a task that sums the values in a matrix.
@ray.remote
def sum_matrix(matrix):
return np.sum(matrix)
# Call the task with a literal argument value.
print(ray.get(sum_matrix.remote(np.ones((100, 100)))))
# -> 10000.0
# Put a large array into the object store.
matrix_ref = ray.put(np.ones((1000, 1000)))
# Call the task with the object reference as an argument.
print(ray.get(sum_matrix.remote(matrix_ref)))
# -> 1000000.0
# __passing_object_end__
# fmt: on
doc/source/ray-core/doc_code/limit_pending_tasks.py
+38
View File
@@ -0,0 +1,38 @@
# __without_backpressure_start__
import ray
ray.init()
@ray.remote
class Actor:
async def heavy_compute(self):
# taking a long time...
# await asyncio.sleep(5)
return
actor = Actor.remote()
NUM_TASKS = 1000
result_refs = []
# When NUM_TASKS is large enough, this will eventually OOM.
for _ in range(NUM_TASKS):
result_refs.append(actor.heavy_compute.remote())
ray.get(result_refs)
# __without_backpressure_end__
# __with_backpressure_start__
MAX_NUM_PENDING_TASKS = 100
result_refs = []
for _ in range(NUM_TASKS):
if len(result_refs) > MAX_NUM_PENDING_TASKS:
# update result_refs to only
# track the remaining tasks.
ready_refs, result_refs = ray.wait(result_refs, num_returns=1)
ray.get(ready_refs)
result_refs.append(actor.heavy_compute.remote())
ray.get(result_refs)
# __with_backpressure_end__
doc/source/ray-core/doc_code/limit_running_tasks.py
+35
View File
@@ -0,0 +1,35 @@
# __without_limit_start__
import ray
# Assume this Ray node has 16 CPUs and 16G memory.
ray.init()
@ray.remote
def process(file):
# Actual work is reading the file and process the data.
# Assume it needs to use 2G memory.
pass
NUM_FILES = 1000
result_refs = []
for i in range(NUM_FILES):
# By default, process task will use 1 CPU resource and no other resources.
# This means 16 tasks can run concurrently
# and will OOM since 32G memory is needed while the node only has 16G.
result_refs.append(process.remote(f"{i}.csv"))
ray.get(result_refs)
# __without_limit_end__
# __with_limit_start__
result_refs = []
for i in range(NUM_FILES):
# Now each task will use 2G memory resource
# and the number of concurrently running tasks is limited to 8.
# In this case, setting num_cpus to 2 has the same effect.
result_refs.append(
process.options(memory=2 * 1024 * 1024 * 1024).remote(f"{i}.csv")
)
ray.get(result_refs)
# __with_limit_end__
doc/source/ray-core/doc_code/monte_carlo_pi.py
+90
View File
@@ -0,0 +1,90 @@
# __starting_ray_start__
import ray
import math
import time
import random
ray.init()
# __starting_ray_end__
# fmt: off
# __defining_actor_start__
@ray.remote
class ProgressActor:
def __init__(self, total_num_samples: int):
self.total_num_samples = total_num_samples
self.num_samples_completed_per_task = {}
def report_progress(self, task_id: int, num_samples_completed: int) -> None:
self.num_samples_completed_per_task[task_id] = num_samples_completed
def get_progress(self) -> float:
return (
sum(self.num_samples_completed_per_task.values()) / self.total_num_samples
)
# __defining_actor_end__
# fmt: on
# fmt: off
# __defining_task_start__
@ray.remote
def sampling_task(num_samples: int, task_id: int,
progress_actor: ray.actor.ActorHandle) -> int:
num_inside = 0
for i in range(num_samples):
x, y = random.uniform(-1, 1), random.uniform(-1, 1)
if math.hypot(x, y) <= 1:
num_inside += 1
# Report progress every 1 million samples.
if (i + 1) % 1_000_000 == 0:
# This is async.
progress_actor.report_progress.remote(task_id, i + 1)
# Report the final progress.
progress_actor.report_progress.remote(task_id, num_samples)
return num_inside
# __defining_task_end__
# fmt: on
# __creating_actor_start__
# Change this to match your cluster scale.
NUM_SAMPLING_TASKS = 10
NUM_SAMPLES_PER_TASK = 10_000_000
TOTAL_NUM_SAMPLES = NUM_SAMPLING_TASKS * NUM_SAMPLES_PER_TASK
# Create the progress actor.
progress_actor = ProgressActor.remote(TOTAL_NUM_SAMPLES)
# __creating_actor_end__
# __executing_task_start__
# Create and execute all sampling tasks in parallel.
results = [
sampling_task.remote(NUM_SAMPLES_PER_TASK, i, progress_actor)
for i in range(NUM_SAMPLING_TASKS)
]
# __executing_task_end__
# __calling_actor_start__
# Query progress periodically.
while True:
progress = ray.get(progress_actor.get_progress.remote())
print(f"Progress: {int(progress * 100)}%")
if progress == 1:
break
time.sleep(1)
# __calling_actor_end__
# __calculating_pi_start__
# Get all the sampling tasks results.
total_num_inside = sum(ray.get(results))
pi = (total_num_inside * 4) / TOTAL_NUM_SAMPLES
print(f"Estimated value of π is: {pi}")
# __calculating_pi_end__
assert str(pi).startswith("3.14")
doc/source/ray-core/doc_code/namespaces.py
+127
View File
@@ -0,0 +1,127 @@
# flake8: noqa
# fmt: off
# __init_namespace_start__
import ray
ray.init(namespace="hello")
# __init_namespace_end__
# fmt: on
ray.shutdown()
# fmt: off
# __actor_namespace_start__
import subprocess
import ray
try:
subprocess.check_output(["ray", "start", "--head"])
@ray.remote
class Actor:
pass
# Job 1 creates two actors, "orange" and "purple" in the "colors" namespace.
with ray.init("ray://localhost:10001", namespace="colors"):
Actor.options(name="orange", lifetime="detached").remote()
Actor.options(name="purple", lifetime="detached").remote()
# Job 2 is now connecting to a different namespace.
with ray.init("ray://localhost:10001", namespace="fruits"):
# This fails because "orange" was defined in the "colors" namespace.
try:
ray.get_actor("orange")
except ValueError:
pass
# This succeeds because the name "orange" is unused in this namespace.
Actor.options(name="orange", lifetime="detached").remote()
Actor.options(name="watermelon", lifetime="detached").remote()
# Job 3 connects to the original "colors" namespace
context = ray.init("ray://localhost:10001", namespace="colors")
# This fails because "watermelon" was in the fruits namespace.
try:
ray.get_actor("watermelon")
except ValueError:
pass
# This returns the "orange" actor we created in the first job, not the second.
ray.get_actor("orange")
# We are manually managing the scope of the connection in this example.
context.disconnect()
finally:
subprocess.check_output(["ray", "stop", "--force"])
# __actor_namespace_end__
# fmt: on
# fmt: off
# __specify_actor_namespace_start__
import subprocess
import ray
try:
subprocess.check_output(["ray", "start", "--head"])
@ray.remote
class Actor:
pass
ctx = ray.init("ray://localhost:10001")
# Create an actor with specified namespace.
Actor.options(name="my_actor", namespace="actor_namespace", lifetime="detached").remote()
# It is accessible in its namespace.
ray.get_actor("my_actor", namespace="actor_namespace")
ctx.disconnect()
finally:
subprocess.check_output(["ray", "stop", "--force"])
# __specify_actor_namespace_end__
# fmt: on
# fmt: off
# __anonymous_namespace_start__
import subprocess
import ray
try:
subprocess.check_output(["ray", "start", "--head"])
@ray.remote
class Actor:
pass
# Job 1 connects to an anonymous namespace by default
with ray.init("ray://localhost:10001"):
Actor.options(name="my_actor", lifetime="detached").remote()
# Job 2 connects to a _different_ anonymous namespace by default
with ray.init("ray://localhost:10001"):
# This succeeds because the second job is in its own namespace.
Actor.options(name="my_actor", lifetime="detached").remote()
finally:
subprocess.check_output(["ray", "stop", "--force"])
# __anonymous_namespace_end__
# fmt: on
# fmt: off
# __get_namespace_start__
import subprocess
import ray
try:
subprocess.check_output(["ray", "start", "--head"])
ray.init(address="auto", namespace="colors")
# Will print namespace name "colors".
print(ray.get_runtime_context().namespace)
finally:
subprocess.check_output(["ray", "stop", "--force"])
# __get_namespace_end__
# fmt: on
doc/source/ray-core/doc_code/nested-tasks.py
+42
View File
@@ -0,0 +1,42 @@
# flake8: noqa
# __nested_start__
import ray
@ray.remote
def f():
return 1
@ray.remote
def g():
# Call f 4 times and return the resulting object refs.
return [f.remote() for _ in range(4)]
@ray.remote
def h():
# Call f 4 times, block until those 4 tasks finish,
# retrieve the results, and return the values.
return ray.get([f.remote() for _ in range(4)])
# __nested_end__
ray.init(num_cpus=4, num_gpus=1)
obj_refs = ray.get(g.remote())
assert len(obj_refs) == 4
assert all(isinstance(o, ray.ObjectRef) for o in obj_refs)
# __yield_start__
@ray.remote(num_cpus=1, num_gpus=1)
def g():
return ray.get(f.remote())
# __yield_end__
assert ray.get(g.remote()) == 1
doc/source/ray-core/doc_code/obj_capture.py
+16
View File
@@ -0,0 +1,16 @@
import ray
# Put the values (1, 2, 3) into Ray's object store.
a, b, c = ray.put(1), ray.put(2), ray.put(3)
@ray.remote
def print_via_capture():
"""This function prints the values of (a, b, c) to stdout."""
print(ray.get([a, b, c]))
# Passing object references via closure-capture. Inside the `print_via_capture`
# function, the global object refs (a, b, c) can be retrieved and printed.
print_via_capture.remote()
# -> prints [1, 2, 3]
doc/source/ray-core/doc_code/obj_ref.py
+18
View File
@@ -0,0 +1,18 @@
import ray
@ray.remote
def echo_and_get(x_list): # List[ObjectRef]
"""This function prints its input values to stdout."""
print("args:", x_list)
print("values:", ray.get(x_list))
# Put the values (1, 2, 3) into Ray's object store.
a, b, c = ray.put(1), ray.put(2), ray.put(3)
# Passing an object as a nested argument to `echo_and_get`. Ray does not
# de-reference nested args, so `echo_and_get` sees the references.
echo_and_get.remote([a, b, c])
# -> prints args: [ObjectRef(...), ObjectRef(...), ObjectRef(...)]
# values: [1, 2, 3]
doc/source/ray-core/doc_code/obj_val.py
+20
View File
@@ -0,0 +1,20 @@
import ray
@ray.remote
def echo(a: int, b: int, c: int):
"""This function prints its input values to stdout."""
print(a, b, c)
# Passing the literal values (1, 2, 3) to `echo`.
echo.remote(1, 2, 3)
# -> prints "1 2 3"
# Put the values (1, 2, 3) into Ray's object store.
a, b, c = ray.put(1), ray.put(2), ray.put(3)
# Passing an object as a top-level argument to `echo`. Ray will de-reference top-level
# arguments, so `echo` will see the literal values (1, 2, 3) in this case as well.
echo.remote(a, b, c)
# -> prints "1 2 3"
doc/source/ray-core/doc_code/object_ref_serialization.py
+25
View File
@@ -0,0 +1,25 @@
import ray
from ray import cloudpickle
FILE = "external_store.pickle"
ray.init()
my_dict = {"hello": "world"}
obj_ref = ray.put(my_dict)
with open(FILE, "wb+") as f:
cloudpickle.dump(obj_ref, f)
# ObjectRef remains pinned in memory because
# it was serialized with ray.cloudpickle.
del obj_ref
with open(FILE, "rb") as f:
new_obj_ref = cloudpickle.load(f)
# The deserialized ObjectRef works as expected.
assert ray.get(new_obj_ref) == my_dict
# Explicitly free the object.
ray._private.internal_api.free(new_obj_ref)
doc/source/ray-core/doc_code/original_resource_unavailable_example.py
+28
View File
@@ -0,0 +1,28 @@
import ray
from ray.util.placement_group import (
placement_group,
)
from ray.util.scheduling_strategies import PlacementGroupSchedulingStrategy
# Two "CPU"s are available.
ray.init(num_cpus=2)
# Create a placement group.
pg = placement_group([{"CPU": 2}])
ray.get(pg.ready())
# Now, 2 CPUs are not available anymore because
# they are pre-reserved by the placement group.
@ray.remote(num_cpus=2)
def f():
return True
# Won't be scheduled because there are no 2 cpus.
f.remote()
# Will be scheduled because 2 cpus are reserved by the placement group.
f.options(
scheduling_strategy=PlacementGroupSchedulingStrategy(placement_group=pg)
).remote()
doc/source/ray-core/doc_code/owners.py
+19
View File
@@ -0,0 +1,19 @@
# flake8: noqa
# __owners_begin__
import ray
import numpy as np
@ray.remote
def large_array():
return np.zeros(int(1e5))
x = ray.put(1) # The driver owns x and also creates the value of x.
y = large_array.remote()
# The driver is the owner of y, even though the value may be stored somewhere else.
# If the node that stores the value of y dies, Ray will automatically recover
# it by re-executing the large_array task.
# If the driver dies, anyone still using y will receive an OwnerDiedError.
# __owners_end__
doc/source/ray-core/doc_code/pattern_async_actor.py
+70
View File
@@ -0,0 +1,70 @@
# __sync_actor_start__
import ray
@ray.remote
class TaskStore:
def get_next_task(self):
return "task"
@ray.remote
class TaskExecutor:
def __init__(self, task_store):
self.task_store = task_store
self.num_executed_tasks = 0
def run(self):
while True:
task = ray.get(self.task_store.get_next_task.remote())
self._execute_task(task)
def _execute_task(self, task):
# Executing the task
self.num_executed_tasks = self.num_executed_tasks + 1
def get_num_executed_tasks(self):
return self.num_executed_tasks
task_store = TaskStore.remote()
task_executor = TaskExecutor.remote(task_store)
task_executor.run.remote()
try:
# This will timeout since task_executor.run occupies the entire actor thread
# and get_num_executed_tasks cannot run.
ray.get(task_executor.get_num_executed_tasks.remote(), timeout=5)
except ray.exceptions.GetTimeoutError:
print("get_num_executed_tasks didn't finish in 5 seconds")
# __sync_actor_end__
# __async_actor_start__
@ray.remote
class AsyncTaskExecutor:
def __init__(self, task_store):
self.task_store = task_store
self.num_executed_tasks = 0
async def run(self):
while True:
# Here we use await instead of ray.get() to
# wait for the next task and it will yield
# the control while waiting.
task = await self.task_store.get_next_task.remote()
self._execute_task(task)
def _execute_task(self, task):
# Executing the task
self.num_executed_tasks = self.num_executed_tasks + 1
def get_num_executed_tasks(self):
return self.num_executed_tasks
async_task_executor = AsyncTaskExecutor.remote(task_store)
async_task_executor.run.remote()
# We are able to run get_num_executed_tasks while run method is running.
num_executed_tasks = ray.get(async_task_executor.get_num_executed_tasks.remote())
print(f"num of executed tasks so far: {num_executed_tasks}")
# __async_actor_end__
doc/source/ray-core/doc_code/pattern_generators.py
+61
View File
@@ -0,0 +1,61 @@
import sys
if len(sys.argv) == 1:
# A small number for CI test.
sys.argv.append("5")
# __program_start__
import sys
import ray
# fmt: off
# __large_values_start__
import numpy as np
@ray.remote
def large_values(num_returns):
return [
np.random.randint(np.iinfo(np.int8).max, size=(100_000_000, 1), dtype=np.int8)
for _ in range(num_returns)
]
# __large_values_end__
# fmt: on
# fmt: off
# __large_values_generator_start__
@ray.remote
def large_values_generator(num_returns):
for i in range(num_returns):
yield np.random.randint(
np.iinfo(np.int8).max, size=(100_000_000, 1), dtype=np.int8
)
print(f"yielded return value {i}")
# __large_values_generator_end__
# fmt: on
# A large enough value (e.g. 100).
num_returns = int(sys.argv[1])
# Worker will likely OOM using normal returns.
print("Using normal functions...")
try:
ray.get(
large_values.options(num_returns=num_returns, max_retries=0).remote(
num_returns
)[0]
)
except ray.exceptions.WorkerCrashedError:
print("Worker failed with normal function")
# Using a generator will allow the worker to finish.
# Note that this will block until the full task is complete, i.e. the
# last yield finishes.
print("Using generators...")
ray.get(
large_values_generator.options(num_returns=num_returns, max_retries=0).remote(
num_returns
)[0]
)
print("Success!")
doc/source/ray-core/doc_code/pattern_nested_tasks.py
+72
View File
@@ -0,0 +1,72 @@
# __pattern_start__
import ray
import time
from numpy import random
def partition(collection):
# Use the last element as the pivot
pivot = collection.pop()
greater, lesser = [], []
for element in collection:
if element > pivot:
greater.append(element)
else:
lesser.append(element)
return lesser, pivot, greater
def quick_sort(collection):
if len(collection) <= 200000: # magic number
return sorted(collection)
else:
lesser, pivot, greater = partition(collection)
lesser = quick_sort(lesser)
greater = quick_sort(greater)
return lesser + [pivot] + greater
@ray.remote
def quick_sort_distributed(collection):
# Tiny tasks are an antipattern.
# Thus, in our example we have a "magic number" to
# toggle when distributed recursion should be used vs
# when the sorting should be done in place. The rule
# of thumb is that the duration of an individual task
# should be at least 1 second.
if len(collection) <= 200000: # magic number
return sorted(collection)
else:
lesser, pivot, greater = partition(collection)
lesser = quick_sort_distributed.remote(lesser)
greater = quick_sort_distributed.remote(greater)
return ray.get(lesser) + [pivot] + ray.get(greater)
for size in [200000, 4000000, 8000000]:
print(f"Array size: {size}")
unsorted = random.randint(1000000, size=(size)).tolist()
s = time.time()
quick_sort(unsorted)
print(f"Sequential execution: {(time.time() - s):.3f}")
s = time.time()
ray.get(quick_sort_distributed.remote(unsorted))
print(f"Distributed execution: {(time.time() - s):.3f}")
print("--" * 10)
# Outputs:
# Array size: 200000
# Sequential execution: 0.040
# Distributed execution: 0.152
# --------------------
# Array size: 4000000
# Sequential execution: 6.161
# Distributed execution: 5.779
# --------------------
# Array size: 8000000
# Sequential execution: 15.459
# Distributed execution: 11.282
# --------------------
# __pattern_end__
doc/source/ray-core/doc_code/pattern_pipelining.py
+66
View File
@@ -0,0 +1,66 @@
import ray
@ray.remote
class WorkQueue:
def __init__(self):
self.queue = list(range(10))
def get_work_item(self):
if self.queue:
return self.queue.pop(0)
else:
return None
@ray.remote
class WorkerWithoutPipelining:
def __init__(self, work_queue):
self.work_queue = work_queue
def process(self, work_item):
print(work_item)
def run(self):
while True:
# Get work from the remote queue.
work_item = ray.get(self.work_queue.get_work_item.remote())
if work_item is None:
break
# Do work.
self.process(work_item)
@ray.remote
class WorkerWithPipelining:
def __init__(self, work_queue):
self.work_queue = work_queue
def process(self, work_item):
print(work_item)
def run(self):
self.work_item_ref = self.work_queue.get_work_item.remote()
while True:
# Get work from the remote queue.
work_item = ray.get(self.work_item_ref)
if work_item is None:
break
self.work_item_ref = self.work_queue.get_work_item.remote()
# Do work while we are fetching the next work item.
self.process(work_item)
work_queue = WorkQueue.remote()
worker_without_pipelining = WorkerWithoutPipelining.remote(work_queue)
ray.get(worker_without_pipelining.run.remote())
work_queue = WorkQueue.remote()
worker_with_pipelining = WorkerWithPipelining.remote(work_queue)
ray.get(worker_with_pipelining.run.remote())
doc/source/ray-core/doc_code/pattern_tree_of_actors.py
+32
View File
@@ -0,0 +1,32 @@
import ray
@ray.remote(num_cpus=1)
class Trainer:
def __init__(self, hyperparameter, data):
self.hyperparameter = hyperparameter
self.data = data
# Train the model on the given training data shard.
def fit(self):
return self.data * self.hyperparameter
@ray.remote(num_cpus=1)
class Supervisor:
def __init__(self, hyperparameter, data):
self.trainers = [Trainer.remote(hyperparameter, d) for d in data]
def fit(self):
# Train with different data shard in parallel.
return ray.get([trainer.fit.remote() for trainer in self.trainers])
data = [1, 2, 3]
supervisor1 = Supervisor.remote(1, data)
supervisor2 = Supervisor.remote(2, data)
# Train with different hyperparameters in parallel.
model1 = supervisor1.fit.remote()
model2 = supervisor2.fit.remote()
assert ray.get(model1) == [1, 2, 3]
assert ray.get(model2) == [2, 4, 6]
doc/source/ray-core/doc_code/placement_group_capture_child_tasks_example.py
+68
View File
@@ -0,0 +1,68 @@
# __child_capture_pg_start__
import ray
from ray.util.placement_group import placement_group
from ray.util.scheduling_strategies import PlacementGroupSchedulingStrategy
ray.init(num_cpus=2)
# Create a placement group.
pg = placement_group([{"CPU": 2}])
ray.get(pg.ready())
@ray.remote(num_cpus=1)
def child():
import time
time.sleep(5)
@ray.remote(num_cpus=1)
def parent():
# The child task is scheduled to the same placement group as its parent,
# although it didn't specify the PlacementGroupSchedulingStrategy.
ray.get(child.remote())
# Since the child and parent use 1 CPU each, the placement group
# bundle {"CPU": 2} is fully occupied.
ray.get(
parent.options(
scheduling_strategy=PlacementGroupSchedulingStrategy(
placement_group=pg, placement_group_capture_child_tasks=True
)
).remote()
)
# __child_capture_pg_end__
# __child_capture_disable_pg_start__
@ray.remote
def parent():
# In this case, the child task isn't
# scheduled with the parent's placement group.
ray.get(
child.options(
scheduling_strategy=PlacementGroupSchedulingStrategy(placement_group=None)
).remote()
)
# This times out because we cannot schedule the child task.
# The cluster has {"CPU": 2}, and both of them are reserved by
# the placement group with a bundle {"CPU": 2}. Since the child shouldn't
# be scheduled within this placement group, it cannot be scheduled because
# there's no available CPU resources.
try:
ray.get(
parent.options(
scheduling_strategy=PlacementGroupSchedulingStrategy(
placement_group=pg, placement_group_capture_child_tasks=True
)
).remote(),
timeout=5,
)
except Exception as e:
print("Couldn't create a child task!")
print(e)
# __child_capture_disable_pg_end__
doc/source/ray-core/doc_code/placement_group_example.py
+141
View File
@@ -0,0 +1,141 @@
# __create_pg_start__
from pprint import pprint
import time
# Import placement group APIs.
from ray.util.placement_group import (
placement_group,
placement_group_table,
remove_placement_group,
)
from ray.util.scheduling_strategies import PlacementGroupSchedulingStrategy
# Initialize Ray.
import ray
# Create a single node Ray cluster with 2 CPUs and 2 GPUs.
ray.init(num_cpus=2, num_gpus=2)
# Reserve a placement group of 1 bundle that reserves 1 CPU and 1 GPU.
pg = placement_group([{"CPU": 1, "GPU": 1}])
# __create_pg_end__
# __ready_pg_start__
# Wait until placement group is created.
ray.get(pg.ready(), timeout=10)
# You can also use ray.wait.
ready, unready = ray.wait([pg.ready()], timeout=10)
# You can look at placement group states using this API.
print(placement_group_table(pg))
# __ready_pg_end__
# __create_pg_failed_start__
# Cannot create this placement group because we
# cannot create a {"GPU": 2} bundle.
pending_pg = placement_group([{"CPU": 1}, {"GPU": 2}])
# This raises the timeout exception!
try:
ray.get(pending_pg.ready(), timeout=5)
except Exception as e:
print(
"Cannot create a placement group because "
"{'GPU': 2} bundle cannot be created."
)
print(e)
# __create_pg_failed_end__
# __schedule_pg_start__
@ray.remote(num_cpus=1)
class Actor:
def __init__(self):
pass
def ready(self):
pass
# Create an actor to a placement group.
actor = Actor.options(
scheduling_strategy=PlacementGroupSchedulingStrategy(
placement_group=pg,
)
).remote()
# Verify the actor is scheduled.
ray.get(actor.ready.remote(), timeout=10)
# __schedule_pg_end__
# __schedule_pg_3_start__
@ray.remote(num_cpus=0, num_gpus=1)
class Actor:
def __init__(self):
pass
def ready(self):
pass
# Create a GPU actor on the first bundle of index 0.
actor2 = Actor.options(
scheduling_strategy=PlacementGroupSchedulingStrategy(
placement_group=pg,
placement_group_bundle_index=0,
)
).remote()
# Verify that the GPU actor is scheduled.
ray.get(actor2.ready.remote(), timeout=10)
# __schedule_pg_3_end__
# __remove_pg_start__
# This API is asynchronous.
remove_placement_group(pg)
# Wait until placement group is killed.
time.sleep(1)
# Check that the placement group has died.
pprint(placement_group_table(pg))
"""
{'bundles': {0: {'GPU': 1.0}, 1: {'CPU': 1.0}},
'name': 'unnamed_group',
'placement_group_id': '40816b6ad474a6942b0edb45809b39c3',
'state': 'REMOVED',
'strategy': 'PACK'}
"""
# __remove_pg_end__
# __strategy_pg_start__
# Reserve a placement group of 2 bundles
# that have to be packed on the same node.
pg = placement_group([{"CPU": 1}, {"GPU": 1}], strategy="PACK")
# __strategy_pg_end__
remove_placement_group(pg)
# __detached_pg_start__
# driver_1.py
# Create a detached placement group that survives even after
# the job terminates.
pg = placement_group([{"CPU": 1}], lifetime="detached", name="global_name")
ray.get(pg.ready())
# __detached_pg_end__
remove_placement_group(pg)
# __get_pg_start__
# first_driver.py
# Create a placement group with a unique name within a namespace.
# Start Ray or connect to a Ray cluster using: ray.init(namespace="pg_namespace")
pg = placement_group([{"CPU": 1}], name="pg_name")
ray.get(pg.ready())
# second_driver.py
# Retrieve a placement group with a unique name within a namespace.
# Start Ray or connect to a Ray cluster using: ray.init(namespace="pg_namespace")
pg = ray.util.get_placement_group("pg_name")
# __get_pg_end__
doc/source/ray-core/doc_code/ray-dag.py
+192
View File
@@ -0,0 +1,192 @@
# flake8: noqa
# fmt: off
# __dag_tasks_begin__
import ray
ray.init()
@ray.remote
def func(src, inc=1):
return src + inc
a_ref = func.bind(1, inc=2)
assert ray.get(a_ref.execute()) == 3 # 1 + 2 = 3
b_ref = func.bind(a_ref, inc=3)
assert ray.get(b_ref.execute()) == 6 # (1 + 2) + 3 = 6
c_ref = func.bind(b_ref, inc=a_ref)
assert ray.get(c_ref.execute()) == 9 # ((1 + 2) + 3) + (1 + 2) = 9
# __dag_tasks_end__
# fmt: on
ray.shutdown()
# fmt: off
# __dag_actors_begin__
import ray
ray.init()
@ray.remote
class Actor:
def __init__(self, init_value):
self.i = init_value
def inc(self, x):
self.i += x
def get(self):
return self.i
a1 = Actor.bind(10) # Instantiate Actor with init_value 10.
val = a1.get.bind() # ClassMethod that returns value from get() from
# the actor created.
assert ray.get(val.execute()) == 10
@ray.remote
def combine(x, y):
return x + y
a2 = Actor.bind(10) # Instantiate another Actor with init_value 10.
a1.inc.bind(2) # Call inc() on the actor created with increment of 2.
a1.inc.bind(4) # Call inc() on the actor created with increment of 4.
a2.inc.bind(6) # Call inc() on the actor created with increment of 6.
# Combine outputs from a1.get() and a2.get()
dag = combine.bind(a1.get.bind(), a2.get.bind())
# a1 + a2 + inc(2) + inc(4) + inc(6)
# 10 + (10 + ( 2 + 4 + 6)) = 32
assert ray.get(dag.execute()) == 32
# __dag_actors_end__
# fmt: on
ray.shutdown()
# fmt: off
# __dag_input_node_begin__
import ray
ray.init()
from ray.dag.input_node import InputNode
@ray.remote
def a(user_input):
return user_input * 2
@ray.remote
def b(user_input):
return user_input + 1
@ray.remote
def c(x, y):
return x + y
with InputNode() as dag_input:
a_ref = a.bind(dag_input)
b_ref = b.bind(dag_input)
dag = c.bind(a_ref, b_ref)
# a(2) + b(2) = c
# (2 * 2) + (2 + 1)
assert ray.get(dag.execute(2)) == 7
# a(3) + b(3) = c
# (3 * 2) + (3 + 1)
assert ray.get(dag.execute(3)) == 10
# __dag_input_node_end__
# fmt: on
# fmt: off
# __dag_multi_output_node_begin__
import ray
from ray.dag.input_node import InputNode
from ray.dag.output_node import MultiOutputNode
@ray.remote
def f(input):
return input + 1
with InputNode() as input_data:
dag = MultiOutputNode([f.bind(input_data["x"]), f.bind(input_data["y"])])
refs = dag.execute({"x": 1, "y": 2})
assert ray.get(refs) == [2, 3]
# __dag_multi_output_node_end__
# fmt: on
# fmt: off
# __dag_multi_output_node_begin__
import ray
from ray.dag.input_node import InputNode
from ray.dag.output_node import MultiOutputNode
@ray.remote
def f(input):
return input + 1
with InputNode() as input_data:
dag = MultiOutputNode([f.bind(input_data["x"]), f.bind(input_data["y"])])
refs = dag.execute({"x": 1, "y": 2})
assert ray.get(refs) == [2, 3]
# __dag_multi_output_node_end__
# fmt: on
# fmt: off
# __dag_multi_output_node_begin__
import ray
from ray.dag.input_node import InputNode
from ray.dag.output_node import MultiOutputNode
@ray.remote
def f(input):
return input + 1
with InputNode() as input_data:
dag = MultiOutputNode([f.bind(input_data["x"]), f.bind(input_data["y"])])
refs = dag.execute({"x": 1, "y": 2})
assert ray.get(refs) == [2, 3]
# __dag_multi_output_node_end__
# fmt: on
# fmt: off
# __dag_actor_reuse_begin__
import ray
from ray.dag.input_node import InputNode
from ray.dag.output_node import MultiOutputNode
@ray.remote
class Worker:
def __init__(self):
self.forwarded = 0
def forward(self, input_data: int):
self.forwarded += 1
return input_data + 1
def num_forwarded(self):
return self.forwarded
# Create an actor via ``remote`` API not ``bind`` API to avoid
# killing actors when a DAG is finished.
worker = Worker.remote()
with InputNode() as input_data:
dag = MultiOutputNode([worker.forward.bind(input_data)])
# Actors are reused. The DAG definition doesn't include
# actor creation.
assert ray.get(dag.execute(1)) == [2]
assert ray.get(dag.execute(2)) == [3]
assert ray.get(dag.execute(3)) == [4]
# You can still use other actor methods via `remote` API.
assert ray.get(worker.num_forwarded.remote()) == 3
# __dag_actor_reuse_end__
# fmt: on
doc/source/ray-core/doc_code/ray_oom_prevention.py
+87
View File
@@ -0,0 +1,87 @@
# flake8: noqa
import ray
ray.init(
_system_config={
"memory_usage_threshold": 0.4,
},
)
# fmt: off
# __last_task_start__
import ray
@ray.remote(max_retries=0)
def leaks_memory():
chunks = []
bits_to_allocate = 8 * 100 * 1024 * 1024 # ~100 MiB
while True:
chunks.append([0] * bits_to_allocate)
try:
ray.get(leaks_memory.remote())
except ray.exceptions.OutOfMemoryError as ex:
print("task failed with OutOfMemoryError, which is expected")
# __last_task_end__
# fmt: on
# fmt: off
# __two_actors_start__
from math import ceil
import ray
from ray._private.utils import (
get_system_memory,
) # do not use outside of this example as these are private methods.
from ray._private.utils import (
get_used_memory,
) # do not use outside of this example as these are private methods.
# estimates the number of bytes to allocate to reach the desired memory usage percentage.
def get_additional_bytes_to_reach_memory_usage_pct(pct: float) -> int:
used = get_used_memory()
total = get_system_memory()
bytes_needed = int(total * pct) - used
assert (
bytes_needed > 0
), "memory usage is already above the target. Increase the target percentage."
return bytes_needed
@ray.remote
class MemoryHogger:
def __init__(self):
self.allocations = []
def allocate(self, bytes_to_allocate: float) -> None:
# divide by 8 as each element in the array occupies 8 bytes
new_list = [0] * ceil(bytes_to_allocate / 8)
self.allocations.append(new_list)
first_actor = MemoryHogger.options(
max_restarts=1, max_task_retries=1, name="first_actor"
).remote()
second_actor = MemoryHogger.options(
max_restarts=0, max_task_retries=0, name="second_actor"
).remote()
# We want total bytes to exceed 40% threshold to trigger the memory monitor.
allocate_bytes = get_additional_bytes_to_reach_memory_usage_pct(0.5)
first_actor_task = first_actor.allocate.remote(0.9 * allocate_bytes)
second_actor_task = second_actor.allocate.remote(0.1* allocate_bytes)
error_thrown = False
try:
ray.get(first_actor_task)
except ray.exceptions.OutOfMemoryError as ex:
error_thrown = True
print("First started actor, which is retriable, was killed by the memory monitor.")
assert error_thrown
ray.get(second_actor_task)
print("Second started actor, which is not-retriable, finished.")
# __two_actors_end__
# fmt: on
doc/source/ray-core/doc_code/resources.py
+59
View File
@@ -0,0 +1,59 @@
import ray
# __specifying_node_resources_start__
# This will start a Ray node with 3 logical cpus, 4 logical gpus,
# 1 special_hardware resource and 1 custom_label resource.
ray.init(num_cpus=3, num_gpus=4, resources={"special_hardware": 1, "custom_label": 1})
# __specifying_node_resources_end__
# __specifying_resource_requirements_start__
# Specify the default resource requirements for this remote function.
@ray.remote(num_cpus=2, num_gpus=2, resources={"special_hardware": 1})
def func():
return 1
# You can override the default resource requirements.
func.options(num_cpus=3, num_gpus=1, resources={"special_hardware": 0}).remote()
@ray.remote(num_cpus=0, num_gpus=1)
class Actor:
pass
# You can override the default resource requirements for actors as well.
actor = Actor.options(num_cpus=1, num_gpus=0).remote()
# __specifying_resource_requirements_end__
# __specifying_fractional_resource_requirements_start__
@ray.remote(num_cpus=0.5)
def io_bound_task():
import time
time.sleep(1)
return 2
io_bound_task.remote()
@ray.remote(num_gpus=0.5)
class IOActor:
def ping(self):
import os
print(f"CUDA_VISIBLE_DEVICES: {os.environ['CUDA_VISIBLE_DEVICES']}")
# Two actors can share the same GPU.
io_actor1 = IOActor.remote()
io_actor2 = IOActor.remote()
ray.get(io_actor1.ping.remote())
ray.get(io_actor2.ping.remote())
# Output:
# (IOActor pid=96328) CUDA_VISIBLE_DEVICES: 1
# (IOActor pid=96329) CUDA_VISIBLE_DEVICES: 1
# __specifying_fractional_resource_requirements_end__
doc/source/ray-core/doc_code/runtime_env_example.py
+81
View File
@@ -0,0 +1,81 @@
# flake8: noqa
"""
This file holds code for the runtime envs documentation.
FIXME: We switched our code formatter from YAPF to Black. Check if we can enable code
formatting on this module and update the paragraph below. See issue #21318.
It ignores yapf because yapf doesn't allow comments right after code blocks,
but we put comments right after code blocks to prevent large white spaces
in the documentation.
"""
import ray
# fmt: off
# __runtime_env_pip_def_start__
runtime_env = {
"pip": ["emoji"],
"env_vars": {"TF_WARNINGS": "none"}
}
# __runtime_env_pip_def_end__
# __strong_typed_api_runtime_env_pip_def_start__
from ray.runtime_env import RuntimeEnv
runtime_env = RuntimeEnv(
pip=["emoji"],
env_vars={"TF_WARNINGS": "none"}
)
# __strong_typed_api_runtime_env_pip_def_end__
# __ray_init_start__
# Option 1: Starting a single-node local Ray cluster or connecting to existing local cluster
ray.init(runtime_env=runtime_env)
# __ray_init_end__
@ray.remote
def f_job():
pass
@ray.remote
class Actor_job:
def g(self):
pass
ray.get(f_job.remote())
a = Actor_job.remote()
ray.get(a.g.remote())
ray.shutdown()
ray.init()
@ray.remote
def f():
pass
@ray.remote
class SomeClass:
pass
# __per_task_per_actor_start__
# Invoke a remote task that will run in a specified runtime environment.
f.options(runtime_env=runtime_env).remote()
# Instantiate an actor that will run in a specified runtime environment.
actor = SomeClass.options(runtime_env=runtime_env).remote()
# Specify a runtime environment in the task definition. Future invocations via
# `g.remote()` will use this runtime environment unless overridden by using
# `.options()` as above.
@ray.remote(runtime_env=runtime_env)
def g():
pass
# Specify a runtime environment in the actor definition. Future instantiations
# via `MyClass.remote()` will use this runtime environment unless overridden by
# using `.options()` as above.
@ray.remote(runtime_env=runtime_env)
class MyClass:
pass
# __per_task_per_actor_end__
doc/source/ray-core/doc_code/scheduling.py
+122
View File
@@ -0,0 +1,122 @@
import ray
ray.init(num_cpus=64)
# __default_scheduling_strategy_start__
@ray.remote
def func():
return 1
@ray.remote(num_cpus=1)
class Actor:
pass
# If unspecified, "DEFAULT" scheduling strategy is used.
func.remote()
actor = Actor.remote()
# Explicitly set scheduling strategy to "DEFAULT".
func.options(scheduling_strategy="DEFAULT").remote()
actor = Actor.options(scheduling_strategy="DEFAULT").remote()
# Zero-CPU (and no other resources) actors are randomly assigned to nodes.
actor = Actor.options(num_cpus=0).remote()
# __default_scheduling_strategy_end__
# __spread_scheduling_strategy_start__
@ray.remote(scheduling_strategy="SPREAD")
def spread_func():
return 2
@ray.remote(num_cpus=1)
class SpreadActor:
pass
# Spread tasks across the cluster.
[spread_func.remote() for _ in range(10)]
# Spread actors across the cluster.
actors = [SpreadActor.options(scheduling_strategy="SPREAD").remote() for _ in range(10)]
# __spread_scheduling_strategy_end__
# __node_affinity_scheduling_strategy_start__
@ray.remote
def node_affinity_func():
return ray.get_runtime_context().get_node_id()
@ray.remote(num_cpus=1)
class NodeAffinityActor:
pass
# Only run the task on the local node.
node_affinity_func.options(
scheduling_strategy=ray.util.scheduling_strategies.NodeAffinitySchedulingStrategy(
node_id=ray.get_runtime_context().get_node_id(),
soft=False,
)
).remote()
# Run the two node_affinity_func tasks on the same node if possible.
node_affinity_func.options(
scheduling_strategy=ray.util.scheduling_strategies.NodeAffinitySchedulingStrategy(
node_id=ray.get(node_affinity_func.remote()),
soft=True,
)
).remote()
# Only run the actor on the local node.
actor = NodeAffinityActor.options(
scheduling_strategy=ray.util.scheduling_strategies.NodeAffinitySchedulingStrategy(
node_id=ray.get_runtime_context().get_node_id(),
soft=False,
)
).remote()
# __node_affinity_scheduling_strategy_end__
# __locality_aware_scheduling_start__
@ray.remote
def large_object_func():
# Large object is stored in the local object store
# and available in the distributed memory,
# instead of returning inline directly to the caller.
return [1] * (1024 * 1024)
@ray.remote
def small_object_func():
# Small object is returned inline directly to the caller,
# instead of storing in the distributed memory.
return [1]
@ray.remote
def consume_func(data):
return len(data)
large_object = large_object_func.remote()
small_object = small_object_func.remote()
# Ray will try to run consume_func on the same node
# where large_object_func runs.
consume_func.remote(large_object)
# Ray will try to spread consume_func across the entire cluster
# instead of only running on the node where large_object_func runs.
[
consume_func.options(scheduling_strategy="SPREAD").remote(large_object)
for i in range(10)
]
# Ray won't consider locality for scheduling consume_func
# since the argument is small and will be sent to the worker node inline directly.
consume_func.remote(small_object)
# __locality_aware_scheduling_end__
doc/source/ray-core/doc_code/streaming_generator.py
+209
View File
@@ -0,0 +1,209 @@
# flake8: noqa
# fmt: off
# __streaming_generator_define_start__
import ray
import time
@ray.remote
def task():
for i in range(5):
time.sleep(5)
yield i
# __streaming_generator_define_end__
# __streaming_generator_execute_start__
gen = task.remote()
# Blocks for 5 seconds.
ref = next(gen)
# return 0
ray.get(ref)
# Blocks for 5 seconds.
ref = next(gen)
# Return 1
ray.get(ref)
# Returns 2~4 every 5 seconds.
for ref in gen:
print(ray.get(ref))
# __streaming_generator_execute_end__
# __streaming_generator_exception_start__
@ray.remote
def task():
for i in range(5):
time.sleep(1)
if i == 1:
raise ValueError
yield i
gen = task.remote()
# it's okay.
ray.get(next(gen))
# Raises an exception
try:
ray.get(next(gen))
except ValueError as e:
print(f"Exception is raised when i == 1 as expected {e}")
# __streaming_generator_exception_end__
# __streaming_generator_actor_model_start__
@ray.remote
class Actor:
def f(self):
for i in range(5):
yield i
@ray.remote
class AsyncActor:
async def f(self):
for i in range(5):
yield i
@ray.remote(max_concurrency=5)
class ThreadedActor:
def f(self):
for i in range(5):
yield i
actor = Actor.remote()
for ref in actor.f.remote():
print(ray.get(ref))
actor = AsyncActor.remote()
for ref in actor.f.remote():
print(ray.get(ref))
actor = ThreadedActor.remote()
for ref in actor.f.remote():
print(ray.get(ref))
# __streaming_generator_actor_model_end__
# __streaming_generator_asyncio_start__
import asyncio
@ray.remote
def task():
for i in range(5):
time.sleep(1)
yield i
async def main():
async for ref in task.remote():
print(await ref)
asyncio.run(main())
# __streaming_generator_asyncio_end__
# __streaming_generator_gc_start__
@ray.remote
def task():
for i in range(5):
time.sleep(1)
yield i
gen = task.remote()
ref1 = next(gen)
del gen
# __streaming_generator_gc_end__
# __streaming_generator_concurrency_asyncio_start__
import asyncio
@ray.remote
def task():
for i in range(5):
time.sleep(1)
yield i
async def async_task():
async for ref in task.remote():
print(await ref)
async def main():
t1 = async_task()
t2 = async_task()
await asyncio.gather(t1, t2)
asyncio.run(main())
# __streaming_generator_concurrency_asyncio_end__
# __streaming_generator_wait_simple_start__
@ray.remote
def task():
for i in range(5):
time.sleep(5)
yield i
gen = task.remote()
# Because it takes 5 seconds to make the first yield,
# with 0 timeout, the generator is unready.
ready, unready = ray.wait([gen], timeout=0)
print("timeout 0, nothing is ready.")
print(ready)
assert len(ready) == 0
assert len(unready) == 1
# Without a timeout argument, ray.wait waits until the given argument
# is ready. When a next item is ready, it returns.
ready, unready = ray.wait([gen])
print("Wait for 5 seconds. The next item is ready.")
assert len(ready) == 1
assert len(unready) == 0
next(gen)
# Because the second yield hasn't happened yet,
ready, unready = ray.wait([gen], timeout=0)
print("Wait for 0 seconds. The next item is not ready.")
print(ready, unready)
assert len(ready) == 0
assert len(unready) == 1
# __streaming_generator_wait_simple_end__
# __streaming_generator_wait_complex_start__
from ray._raylet import ObjectRefGenerator
@ray.remote
def generator_task():
for i in range(5):
time.sleep(5)
yield i
@ray.remote
def regular_task():
for i in range(5):
time.sleep(5)
return
gen = [generator_task.remote()]
ref = [regular_task.remote()]
ready, unready = [], [*gen, *ref]
result = []
while unready:
ready, unready = ray.wait(unready)
for r in ready:
if isinstance(r, ObjectRefGenerator):
try:
ref = next(r)
result.append(ray.get(ref))
except StopIteration:
pass
else:
unready.append(r)
else:
result.append(ray.get(r))
# __streaming_generator_wait_complex_end__
doc/source/ray-core/doc_code/task_exceptions.py
+105
View File
@@ -0,0 +1,105 @@
# flake8: noqa
# fmt: off
# __task_exceptions_begin__
import ray
@ray.remote
def f():
raise Exception("the real error")
@ray.remote
def g(x):
return
try:
ray.get(f.remote())
except ray.exceptions.RayTaskError as e:
print(e)
# ray::f() (pid=71867, ip=XXX.XX.XXX.XX)
# File "errors.py", line 5, in f
# raise Exception("the real error")
# Exception: the real error
try:
ray.get(g.remote(f.remote()))
except ray.exceptions.RayTaskError as e:
print(e)
# ray::g() (pid=73085, ip=128.32.132.47)
# At least one of the input arguments for this task could not be computed:
# ray.exceptions.RayTaskError: ray::f() (pid=73085, ip=XXX.XX.XXX.XX)
# File "errors.py", line 5, in f
# raise Exception("the real error")
# Exception: the real error
# __task_exceptions_end__
# __unserializable_exceptions_begin__
import threading
class UnserializableException(Exception):
def __init__(self):
self.lock = threading.Lock()
@ray.remote
def raise_unserializable_error():
raise UnserializableException
try:
ray.get(raise_unserializable_error.remote())
except ray.exceptions.RayTaskError as e:
print(e)
# ray::raise_unserializable_error() (pid=328577, ip=172.31.5.154)
# File "/home/ubuntu/ray/tmp~/main.py", line 25, in raise_unserializable_error
# raise UnserializableException
# UnserializableException
print(type(e.cause))
# <class 'ray.exceptions.RayError'>
print(e.cause)
# The original cause of the RayTaskError (<class '__main__.UnserializableException'>) isn't serializable: cannot pickle '_thread.lock' object. Overwriting the cause to a RayError.
# __unserializable_exceptions_end__
# __catch_user_exceptions_begin__
class MyException(Exception):
...
@ray.remote
def raises_my_exc():
raise MyException("a user exception")
try:
ray.get(raises_my_exc.remote())
except MyException as e:
print(e)
# ray::raises_my_exc() (pid=15329, ip=127.0.0.1)
# File "<$PWD>/task_exceptions.py", line 45, in raises_my_exc
# raise MyException("a user exception")
# MyException: a user exception
# __catch_user_exceptions_end__
# __catch_user_final_exceptions_begin__
class MyFinalException(Exception):
def __init_subclass__(cls, /, *args, **kwargs):
raise TypeError("Cannot subclass this little exception class.")
@ray.remote
def raises_my_final_exc():
raise MyFinalException("a *final* user exception")
try:
ray.get(raises_my_final_exc.remote())
except ray.exceptions.RayTaskError as e:
assert isinstance(e.cause, MyFinalException)
print(e)
# 2024-04-08 21:11:47,417 WARNING exceptions.py:177 -- User exception type <class '__main__.MyFinalException'> in RayTaskError can not be subclassed! This exception will be raised as RayTaskError only. You can use `ray_task_error.cause` to access the user exception. Failure in subclassing: Cannot subclass this little exception class.
# ray::raises_my_final_exc() (pid=88226, ip=127.0.0.1)
# File "<$PWD>/task_exceptions.py", line 66, in raises_my_final_exc
# raise MyFinalException("a *final* user exception")
# MyFinalException: a *final* user exception
print(type(e.cause))
# <class '__main__.MyFinalException'>
print(e.cause)
# a *final* user exception
# __catch_user_final_exceptions_end__
# fmt: on
doc/source/ray-core/doc_code/tasks.py
+137
View File
@@ -0,0 +1,137 @@
# flake8: noqa
# __tasks_start__
import ray
import time
# A regular Python function.
def normal_function():
return 1
# By adding the `@ray.remote` decorator, a regular Python function
# becomes a Ray remote function.
@ray.remote
def my_function():
return 1
# To invoke this remote function, use the `remote` method.
# This will immediately return an object ref (a future) and then create
# a task that will be executed on a worker process.
obj_ref = my_function.remote()
# The result can be retrieved with ``ray.get``.
assert ray.get(obj_ref) == 1
@ray.remote
def slow_function():
time.sleep(10)
return 1
# Ray tasks are executed in parallel.
# All computation is performed in the background, driven by Ray's internal event loop.
for _ in range(4):
# This doesn't block.
slow_function.remote()
# __tasks_end__
# __pass_by_ref_start__
@ray.remote
def function_with_an_argument(value):
return value + 1
obj_ref1 = my_function.remote()
assert ray.get(obj_ref1) == 1
# You can pass an object ref as an argument to another Ray task.
obj_ref2 = function_with_an_argument.remote(obj_ref1)
assert ray.get(obj_ref2) == 2
# __pass_by_ref_end__
# __wait_start__
object_refs = [slow_function.remote() for _ in range(2)]
# Return as soon as one of the tasks finished execution.
ready_refs, remaining_refs = ray.wait(object_refs, num_returns=1, timeout=None)
# __wait_end__
# __multiple_returns_start__
# By default, a Ray task only returns a single Object Ref.
@ray.remote
def return_single():
return 0, 1, 2
object_ref = return_single.remote()
assert ray.get(object_ref) == (0, 1, 2)
# However, you can configure Ray tasks to return multiple Object Refs.
@ray.remote(num_returns=3)
def return_multiple():
return 0, 1, 2
object_ref0, object_ref1, object_ref2 = return_multiple.remote()
assert ray.get(object_ref0) == 0
assert ray.get(object_ref1) == 1
assert ray.get(object_ref2) == 2
# __multiple_returns_end__
# __generator_start__
@ray.remote(num_returns=3)
def return_multiple_as_generator():
for i in range(3):
yield i
# NOTE: Similar to normal functions, these objects will not be available
# until the full task is complete and all returns have been generated.
a, b, c = return_multiple_as_generator.remote()
# __generator_end__
# __cancel_start__
@ray.remote
def blocking_operation():
time.sleep(10e6)
obj_ref = blocking_operation.remote()
ray.cancel(obj_ref)
try:
ray.get(obj_ref)
except ray.exceptions.TaskCancelledError:
print("Object reference was cancelled.")
# __cancel_end__
# __resource_start__
# Specify required resources.
@ray.remote(num_cpus=4, num_gpus=2)
def my_function():
return 1
# Override the default resource requirements.
my_function.options(num_cpus=3).remote()
# __resource_end__
# __fraction_resource_start__
# Ray also supports fractional resource requirements.
@ray.remote(num_gpus=0.5)
def h():
return 1
# Ray support custom resources too.
@ray.remote(resources={"Custom": 1})
def f():
return 1
# __fraction_resource_end__
doc/source/ray-core/doc_code/tasks_fault_tolerance.py
+91
View File
@@ -0,0 +1,91 @@
# flake8: noqa
# fmt: off
# __tasks_fault_tolerance_retries_begin__
import numpy as np
import os
import ray
import time
ray.init(ignore_reinit_error=True)
@ray.remote(max_retries=1)
def potentially_fail(failure_probability):
time.sleep(0.2)
if np.random.random() < failure_probability:
os._exit(0)
return 0
for _ in range(3):
try:
# If this task crashes, Ray will retry it up to one additional
# time. If either of the attempts succeeds, the call to ray.get
# below will return normally. Otherwise, it will raise an
# exception.
ray.get(potentially_fail.remote(0.5))
print('SUCCESS')
except ray.exceptions.WorkerCrashedError:
print('FAILURE')
# __tasks_fault_tolerance_retries_end__
# fmt: on
# fmt: off
# __tasks_fault_tolerance_retries_exception_begin__
import numpy as np
import os
import ray
import time
ray.init(ignore_reinit_error=True)
class RandomError(Exception):
pass
@ray.remote(max_retries=1, retry_exceptions=True)
def potentially_fail(failure_probability):
if failure_probability < 0 or failure_probability > 1:
raise ValueError(
"failure_probability must be between 0 and 1, but got: "
f"{failure_probability}"
)
time.sleep(0.2)
if np.random.random() < failure_probability:
raise RandomError("Failed!")
return 0
for _ in range(3):
try:
# If this task crashes, Ray will retry it up to one additional
# time. If either of the attempts succeeds, the call to ray.get
# below will return normally. Otherwise, it will raise an
# exception.
ray.get(potentially_fail.remote(0.5))
print('SUCCESS')
except RandomError:
print('FAILURE')
# Provide the exceptions that we want to retry as an allowlist.
retry_on_exception = potentially_fail.options(retry_exceptions=[RandomError])
try:
# This will fail since we're passing in -1 for the failure_probability,
# which will raise a ValueError in the task and does not match the RandomError
# exception that we provided.
ray.get(retry_on_exception.remote(-1))
except ValueError:
print("FAILED AS EXPECTED")
else:
raise RuntimeError("An exception should be raised so this shouldn't be reached.")
# These will retry on the RandomError exception.
for _ in range(3):
try:
# If this task crashes, Ray will retry it up to one additional
# time. If either of the attempts succeeds, the call to ray.get
# below will return normally. Otherwise, it will raise an
# exception.
ray.get(retry_on_exception.remote(0.5))
print('SUCCESS')
except RandomError:
print('FAILURE AFTER RETRIES')
# __tasks_fault_tolerance_retries_exception_end__
# fmt: on
doc/source/ray-core/doc_code/tqdm.py
+14
View File
@@ -0,0 +1,14 @@
import time
import ray
# Instead of "from tqdm import tqdm", use:
from ray.experimental.tqdm_ray import tqdm
@ray.remote
def f(name):
for x in tqdm(range(100), desc=name):
time.sleep(0.1)
ray.get([f.remote("task 1"), f.remote("task 2")])
doc/source/ray-core/examples/BUILD.bazel
+40
View File
@@ -0,0 +1,40 @@
load("//bazel:python.bzl", "py_test_run_all_notebooks")
filegroup(
name = "core_examples",
srcs = glob(["*.ipynb"]),
visibility = ["//doc:__subpackages__"]
)
# --------------------------------------------------------------------
# Test the lightweight, CPU-only Ray Core example notebooks. The
# excluded notebooks need GPU, long-running training, or live network
# access and aren't runnable in the standard CPU CI job.
# --------------------------------------------------------------------
py_test_run_all_notebooks(
size = "large",
include = ["*.ipynb"],
exclude = [
"batch_prediction.ipynb", # Requires GPU (num_gpus=1) and torch.
"highly_parallel.ipynb", # Needs an existing multi-node cluster and belongs in a release test.
"plot_hyperparameter.ipynb", # Model-training loop.
"plot_parameter_server.ipynb", # Torch training loop; slow/non-deterministic.
"plot_pong_example.ipynb", # Long-running RL (Pong) training.
"web_crawler.ipynb", # Makes live network requests.
],
data = ["//doc/source/ray-core/examples:core_examples"],
tags = [
"exclusive",
"team:core",
],
)
filegroup(
name = "core_examples_ci_configs",
srcs = glob([
"**/ci/aws.yaml",
"**/ci/gce.yaml",
]),
visibility = ["//doc:__pkg__"],
)
doc/source/ray-core/examples/batch_prediction.ipynb
+345
View File
@@ -0,0 +1,345 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# Batch Prediction with Ray Core\n",
"\n",
"<a id=\"try-anyscale-quickstart-batch_prediction\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=batch_prediction\">\n",
" <img src=\"../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
"</a>\n",
"<br></br>\n",
"\n",
"```{note}\n",
"For a higher level API for batch inference on large datasets, see [batch inference with Ray Data](batch_inference_home). This example is for users who want more control over data sharding and execution.\n",
"```\n",
"\n",
"The batch prediction is the process of using a trained model to generate predictions for a collection of observations. It has the following elements:\n",
"* Input dataset: this is a collection of observations to generate predictions for. The data is usually stored in an external storage system like S3, HDFS or database, and can be large.\n",
"* ML model: this is a trained ML model which is usually also stored in an external storage system.\n",
"* Predictions: these are the outputs when applying the ML model on observations. The predictions are usually written back to the storage system.\n",
"\n",
"With Ray, you can build scalable batch prediction for large datasets at high prediction throughput. Ray Data provides a [higher-level API for offline batch inference](batch_inference_home), with built-in optimizations. However, for more control, you can use the lower-level Ray Core APIs. This example demonstrates batch inference with Ray Core by splitting the dataset into disjoint shards and executing them in parallel, with either Ray Tasks or Ray Actors across a Ray Cluster."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Task-based batch prediction\n",
"\n",
"With Ray tasks, you can build a batch prediction program in this way:\n",
"1. Loads the model\n",
"2. Launches Ray tasks, with each taking in the model and a shard of input dataset\n",
"3. Each worker executes predictions on the assigned shard, and writes out results\n",
"\n",
"Let’s take NYC taxi data in 2009 for example. Suppose we have this simple model:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import numpy as np\n",
"\n",
"def load_model():\n",
" # A dummy model.\n",
" def model(batch: pd.DataFrame) -> pd.DataFrame:\n",
" # Dummy payload so copying the model will actually copy some data\n",
" # across nodes.\n",
" model.payload = np.zeros(100_000_000)\n",
" return pd.DataFrame({\"score\": batch[\"passenger_count\"] % 2 == 0})\n",
" \n",
" return model"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The dataset has 12 files (one for each month) so we can naturally have each Ray task to take one file. By taking the model and a shard of input dataset (i.e. a single file), we can define a Ray remote task for prediction:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pyarrow.parquet as pq\n",
"import ray\n",
"\n",
"@ray.remote\n",
"def make_prediction(model, shard_path):\n",
" df = pq.read_table(shard_path).to_pandas()\n",
" result = model(df)\n",
"\n",
" # Write out the prediction result.\n",
" # NOTE: unless the driver will have to further process the\n",
" # result (other than simply writing out to storage system),\n",
" # writing out at remote task is recommended, as it can avoid\n",
" # congesting or overloading the driver.\n",
" # ...\n",
"\n",
" # Here we just return the size about the result in this example.\n",
" return len(result)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The driver launches all tasks for the entire input dataset. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 12 files, one for each remote task.\n",
"input_files = [\n",
" f\"s3://anonymous@air-example-data/ursa-labs-taxi-data/downsampled_2009_full_year_data.parquet\"\n",
" f\"/fe41422b01c04169af2a65a83b753e0f_{i:06d}.parquet\"\n",
" for i in range(12)\n",
"]\n",
"\n",
"# ray.put() the model just once to local object store, and then pass the\n",
"# reference to the remote tasks.\n",
"model = load_model()\n",
"model_ref = ray.put(model)\n",
"\n",
"result_refs = []\n",
"\n",
"# Launch all prediction tasks.\n",
"for file in input_files:\n",
" # Launch a prediction task by passing model reference and shard file to it.\n",
" # NOTE: it would be highly inefficient if you are passing the model itself\n",
" # like make_prediction.remote(model, file), which in order to pass the model\n",
" # to remote node will ray.put(model) for each task, potentially overwhelming\n",
" # the local object store and causing out-of-disk error.\n",
" result_refs.append(make_prediction.remote(model_ref, file))\n",
"\n",
"results = ray.get(result_refs)\n",
"\n",
"# Let's check prediction output size.\n",
"for r in results:\n",
" print(\"Prediction output size:\", r)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In order to not overload the cluster and cause OOM, we can control the parallelism by setting the proper resource requirement for tasks, see details about this design pattern in {doc}`/ray-core/patterns/limit-running-tasks`.\n",
"For example, if it's easy for your to get a good estimate of the in-memory size for data loaded from external storage, you can control the parallelism by specifying the amount of memory needed for each task, e.g. launching tasks with ``make_prediction.options(memory=100*1023*1025).remote(model_ref, file)``. Ray will then do the right thing and make sure tasks scheduled to a node will not exceed its total memory."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```{tip}\n",
"To avoid repeatedly storing the same model into object store (this can cause Out-of-disk for driver node), use ray.put() to store the model once, and then pass the reference around.\n",
"```\n",
"```{tip}\n",
"To avoid congest or overload the driver node, it’s preferable to have each task to write out the predictions (instead of returning results back to driver which actualy does nothing but write out to storage system).\n",
"```\n",
"\n",
"## Actor-based batch prediction\n",
"In the above solution, each Ray task will have to fetch the model from the driver node before it can start performing the prediction. This is an overhead cost that can be significant if the model size is large. We can optimize it by using Ray actors, which will fetch the model just once and reuse it for all tasks assigned to the actor.\n",
"\n",
"First, we define a callable class that’s structured with an interface (i.e. constructor) to load/cache the model, and the other to take in a file and perform prediction."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import pyarrow.parquet as pq\n",
"import ray\n",
"\n",
"@ray.remote\n",
"class BatchPredictor:\n",
" def __init__(self, model):\n",
" self.model = model\n",
" \n",
" def predict(self, shard_path):\n",
" df = pq.read_table(shard_path).to_pandas()\n",
" result =self.model(df)\n",
"\n",
" # Write out the prediction result.\n",
" # NOTE: unless the driver will have to further process the\n",
" # result (other than simply writing out to storage system),\n",
" # writing out at remote task is recommended, as it can avoid\n",
" # congesting or overloading the driver.\n",
" # ...\n",
"\n",
" # Here we just return the size about the result in this example.\n",
" return len(result)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The constructor is called only once per actor worker. We use ActorPool to manage a set of actors that can receive prediction requests."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from ray.util.actor_pool import ActorPool\n",
"\n",
"model = load_model()\n",
"model_ref = ray.put(model)\n",
"num_actors = 4\n",
"actors = [BatchPredictor.remote(model_ref) for _ in range(num_actors)]\n",
"pool = ActorPool(actors)\n",
"input_files = [\n",
" f\"s3://anonymous@air-example-data/ursa-labs-taxi-data/downsampled_2009_full_year_data.parquet\"\n",
" f\"/fe41422b01c04169af2a65a83b753e0f_{i:06d}.parquet\"\n",
" for i in range(12)\n",
"]\n",
"for file in input_files:\n",
" pool.submit(lambda a, v: a.predict.remote(v), file)\n",
"while pool.has_next():\n",
" print(\"Prediction output size:\", pool.get_next())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that the ActorPool is fixed in size, unlike task-based approach where the number of parallel tasks can be dynamic (as long as it's not exceeding max_in_flight_tasks). To have autoscaling actor pool, you will need to use the [Ray Data batch prediction](batch_inference_home)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Batch prediction with GPUs\n",
"\n",
"If your cluster has GPU nodes and your predictor can utilize the GPUs, you can direct the tasks or actors to those GPU nodes by specifying num_gpus. Ray will schedule them onto GPU nodes accordingly. On the node, you will need to move the model to GPU. The following is an example for Torch model."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import torch\n",
"\n",
"@ray.remote(num_gpus=1)\n",
"def make_torch_prediction(model: torch.nn.Module, shard_path):\n",
" # Move model to GPU.\n",
" model.to(torch.device(\"cuda\"))\n",
" inputs = pq.read_table(shard_path).to_pandas().to_numpy()\n",
"\n",
" results = []\n",
" # for each tensor in inputs:\n",
" # results.append(model(tensor))\n",
" #\n",
" # Write out the results right in task instead of returning back\n",
" # to the driver node (unless you have to), to avoid congest/overload\n",
" # driver node.\n",
" # ...\n",
"\n",
" # Here we just return simple/light meta information.\n",
" return len(results)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## FAQs\n",
"\n",
"### How to load and pass model efficiently in Ray cluster if the model is large?\n",
"The recommended way is to (taking task-based batch prediction for example, the actor-based is the same):\n",
"1. let the driver load the model (e.g. from storage system)\n",
"2. let the driver ray.put(model) to store the model into object store; and\n",
"3. pass the same reference of the model to each remote tasks when launching them.\n",
"The remote task will fetch the model (from driver's object store) to its local object store before start performing prediction.\n",
"\n",
"Note it's highly inefficient if you skip the step 2 and pass the model (instead of reference) to remote tasks. If the model is large and there are many tasks, it'll likely cause out-of-disk crash for the driver node."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# GOOD: the model will be stored to driver's object store only once\n",
"model = load_model()\n",
"model_ref = ray.put(model)\n",
"for file in input_files:\n",
" make_prediction.remote(model_ref, file)\n",
"\n",
"# BAD: the same model will be stored to driver's object store repeatedly for each task\n",
"model = load_model()\n",
"for file in input_files:\n",
" make_prediction.remote(model, file)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For more details, check out {doc}`/ray-core/patterns/pass-large-arg-by-value`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### How to improve the GPU utilization rate?\n",
"To keep GPUs busy, there are following things to look at:\n",
"- **Schedule multiple tasks on the same GPU node if it has multiple GPUs**: If there are multiple GPUs on same node and a single task cannot use them all, you can direct multiple tasks to the node. This is automatically handled by Ray, e.g. if you specify num_gpus=1 and there are 4 GPUs, Ray will schedule 4 tasks to the node, provided there are enough tasks and no other resource constraints.\n",
"- **Use actor-based approach**: as mentioned above, actor-based approach is more efficient because it reuses model initialization for many tasks, so the node will spend more time on the actual workload."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.8.10 ('venv': venv)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
},
"orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "3c0d54d489a08ae47a06eae2fd00ff032d6cddb527c382959b7b2575f6a8167f"
}
}
},
"nbformat": 4,
"nbformat_minor": 2
}
doc/source/ray-core/examples/gentle_walkthrough.ipynb
+781
View File
@@ -0,0 +1,781 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "9cdddbae",
"metadata": {},
"source": [
"# A Gentle Introduction to Ray Core by Example\n",
"\n",
"<a id=\"try-anyscale-quickstart-gentle_walkthrough\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=gentle_walkthrough\">\n",
" <img src=\"../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
"</a>\n",
"<br></br>"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "9d4d0ecd",
"metadata": {},
"source": [
"Implement a function in Ray Core to understand how Ray works and its basic concepts.\n",
"Python programmers from those with less experience to those who are interested in advanced tasks,\n",
"can start working with distributed computing using Python by learning the Ray Core API.\n",
"\n",
"## Install Ray\n",
"Install Ray with the following command:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6115afbb",
"metadata": {},
"outputs": [],
"source": [
"! pip install ray"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "be9d3c98",
"metadata": {},
"source": [
"## Ray Core\n",
"\n",
"Start a local cluster by running the following commands:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import ray\n",
"ray.init()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"collapsed": false
},
"source": [
"Note the following lines in the output:"
]
},
{
"cell_type": "markdown",
"id": "e75826f4",
"metadata": {},
"source": [
"```\n",
"... INFO services.py:1263 -- View the Ray dashboard at http://127.0.0.1:8265\n",
"{'node_ip_address': '192.168.1.41',\n",
"...\n",
"'node_id': '...'}\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "4c837608",
"metadata": {},
"source": [
"These messages indicate that the Ray cluster is working as expected. In this example output, the address of the Ray dashboard is `http://127.0.0.1:8265`. Access the Ray dashboard at the address on the first line of your output. The Ray dashboard displays information such as the number of CPU cores available and the total utilization of the current Ray application.\n",
"This is a typical output for a laptop:\n",
"\n",
"```\n",
"{'CPU': 12.0,\n",
"'memory': 14203886388.0,\n",
"'node:127.0.0.1': 1.0,\n",
"'object_store_memory': 2147483648.0}\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "cf542ed9",
"metadata": {},
"source": [
"Next, is a brief introduction to the Ray Core API, which we refer to as the Ray API.\n",
"The Ray API builds on concepts such as decorators, functions, and classes, that are familiar to Python programmers.\n",
"It is a universal programming interface for distributed computing. \n",
"The engine handles the complicated work, allowing developers to use Ray with existing Python libraries and systems."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "83d1a3bf",
"metadata": {},
"source": [
"## Your First Ray API Example\n",
"\n",
"The following function retrieves and processes\n",
"data from a database. The dummy `database` is a plain Python list containing the\n",
"words of the title of the [\"Learning Ray\" book](https://www.amazon.com/Learning-Ray-Flexible-Distributed-Machine/dp/1098117220/).\n",
"The `sleep` function pauses for a certain amount of time to simulate the cost of accessing and processing data from the database. "
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "e053331e",
"metadata": {},
"outputs": [],
"source": [
"import time\n",
"\n",
"database = [\n",
" \"Learning\", \"Ray\",\n",
" \"Flexible\", \"Distributed\", \"Python\", \"for\", \"Machine\", \"Learning\"\n",
"]\n",
"\n",
"\n",
"def retrieve(item):\n",
" time.sleep(item / 10.)\n",
" return item, database[item]"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "2b518f4f",
"metadata": {},
"source": [
"If the item with index 5 takes half a second `(5 / 10.)`, an estimate of the total runtime to retrieve all eight items sequentially is `(0+1+2+3+4+5+6+7)/10. = 2.8` seconds.\n",
"Run the following code to get the actual time:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "b0091149",
"metadata": {
"lines_to_next_cell": 2
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Runtime: 2.82 seconds, data:\n",
"(0, 'Learning')\n",
"(1, 'Ray')\n",
"(2, 'Flexible')\n",
"(3, 'Distributed')\n",
"(4, 'Python')\n",
"(5, 'for')\n",
"(6, 'Machine')\n",
"(7, 'Learning')\n"
]
}
],
"source": [
"def print_runtime(input_data, start_time):\n",
" print(f'Runtime: {time.time() - start_time:.2f} seconds, data:')\n",
" print(*input_data, sep=\"\\n\")\n",
"\n",
"\n",
"start = time.time()\n",
"data = [retrieve(item) for item in range(8)]\n",
"print_runtime(data, start)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "97aa047d",
"metadata": {},
"source": [
"The total time to run the function is 2.82 seconds in this example, but time may be different for your computer.\n",
"Note that this basic Python version cannot run the function simultaneously."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "30291db3",
"metadata": {},
"source": [
"You may expect that Python list comprehensions are more efficient. The measured runtime of 2.8 seconds is actually the worst case scenario.\n",
"Although this program \"sleeps\" for most of its runtime, it is slow because of the Global Interpreter Lock (GIL)."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "2ebb32d4",
"metadata": {},
"source": [
"### Ray Tasks\n",
"\n",
"This task can benefit from parallelization. If it is perfectly distributed, the runtime should not take much longer than the slowest subtask,\n",
"that is, `7/10. = 0.7` seconds.\n",
"To extend this example to run in parallel on Ray, start by using the @ray.remote decorator:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "1e21e7c7",
"metadata": {},
"outputs": [],
"source": [
"import ray \n",
"\n",
"\n",
"@ray.remote\n",
"def retrieve_task(item):\n",
" return retrieve(item)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "935a4062",
"metadata": {},
"source": [
"With the decorator, the function `retrieve_task` becomes a :ref:`ray-remote-functions<Ray task>`_.\n",
"A Ray task is a function that Ray executes on a different process from where\n",
"it was called, and possibly on a different machine."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "78afc80a",
"metadata": {},
"source": [
"Ray is convenient to use because you can continue writing Python code,\n",
"without having to significantly change your approach or programming style.\n",
"Using the :func:`ray.remote()<@ray.remote>` decorator on the retrieve function is the intended use of decorators,\n",
"and you did not modify the original code in this example."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "c5dc2e18",
"metadata": {},
"source": [
"To retrieve database entries and measure performance, you do not need to make many changes to the code. Here's an overview of the process:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "a34697da",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2022-12-20 13:52:34,632\tINFO worker.py:1529 -- Started a local Ray instance. View the dashboard at \u001b[1m\u001b[32m127.0.0.1:8265 \u001b[39m\u001b[22m\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Runtime: 0.71 seconds, data:\n",
"(0, 'Learning')\n",
"(1, 'Ray')\n",
"(2, 'Flexible')\n",
"(3, 'Distributed')\n",
"(4, 'Python')\n",
"(5, 'for')\n",
"(6, 'Machine')\n",
"(7, 'Learning')\n"
]
}
],
"source": [
"start = time.time()\n",
"object_references = [\n",
" retrieve_task.remote(item) for item in range(8)\n",
"]\n",
"data = ray.get(object_references)\n",
"print_runtime(data, start)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "8df4087c",
"metadata": {},
"source": [
"Running the task in parallel requires two minor code modifications.\n",
"To execute your Ray task remotely, you must use a `.remote()` call.\n",
"Ray executes remote tasks asynchronously, even on a local cluster.\n",
"The items in the `object_references` list in the code snippet do not directly contain the results.\n",
"If you check the Python type of the first item using `type(object_references[0])`,\n",
"you see that it is actually an `ObjectRef`.\n",
"These object references correspond to _futures_ for which you need to request the result.\n",
"The call :func:`ray.get()<ray.get(...)>` is for requesting the result. Whenever you call remote on a Ray task,\n",
"it immediately returns one or more object references.\n",
"Consider Ray tasks as the primary way of creating objects.\n",
"The following section is an example that links multiple tasks together and allows\n",
"Ray to pass and resolve the objects between them."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "2373ddd9",
"metadata": {},
"source": [
"Let's review the previous steps.\n",
"You started with a Python function, then decorated it with `@ray.remote`, making the function a Ray task.\n",
"Instead of directly calling the original function in the code, you called `.remote(...)` on the Ray task.\n",
"Finally, you retrieved the results from the Ray cluster using `.get(...)`.\n",
"Consider creating a Ray task from one of your own functions as an additional exercise."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": []
},
{
"attachments": {},
"cell_type": "markdown",
"id": "e008a500",
"metadata": {},
"source": [
"Let's review the performance gain from using Ray tasks.\n",
"On most laptops the runtime is around 0.71 seconds,\n",
"which is slightly more than the slowest subtask, which is 0.7 seconds.\n",
"You can further improve the program by leveraging more of Ray’s API."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "54f53644",
"metadata": {},
"source": [
"### Object Stores\n",
"\n",
"The retrieve definition directly accesses items from the `database`. While this works well on a local Ray cluster, consider how it functions on an actual cluster with multiple computers.\n",
"A Ray cluster has a head node with a driver process and multiple worker nodes with worker processes executing tasks.\n",
"In this scenario the database is only defined on the driver, but the worker processes need access to it to run the retrieve task.\n",
"Ray's solution for sharing objects between the driver and workers or between workers is to use\n",
"the `ray.put` function to place the data into Ray's distributed object store.\n",
"In the `retrieve_task` definition, you can add a `db` argument to pass later as the `db_object_ref` object."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "da66a836",
"metadata": {},
"outputs": [],
"source": [
"db_object_ref = ray.put(database)\n",
"\n",
"\n",
"@ray.remote\n",
"def retrieve_task(item, db):\n",
" time.sleep(item / 10.)\n",
" return item, db[item]"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "72f37eb4",
"metadata": {},
"source": [
"By using the object store, you allow Ray to manage data access throughout the entire cluster.\n",
"Although the object store involves some overhead, it improves performance for larger datasets.\n",
"This step is crucial for a truly distributed environment.\n",
"Rerun the example with the `retrieve_task` function to confirm that it executes as you expect."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "453e312f",
"metadata": {},
"source": [
"### Non-blocking calls\n",
"\n",
"In the previous section, you used `ray.get(object_references)` to retrieve results.\n",
"This call blocks the driver process until all results are available.\n",
"This dependency can cause problems if each database item takes several minutes to process.\n",
"More efficiency gains are possible if you allow the driver process to perform other tasks while waiting for results,\n",
"and to process results as they are completed rather than waiting for all items to finish.\n",
"Additionally, if one of the database items cannot be retrieved due to an issue like a deadlock in the database connection,\n",
"the driver hangs indefinitely.\n",
"To prevent indefinite hangs, set reasonable `timeout` values when using the `wait` function.\n",
"For example, if you want to wait less than ten times the time of the slowest data retrieval task,\n",
"use the `wait` function to stop the task after that time has passed."
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "75da06ec",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Runtime: 0.11 seconds, data:\n",
"(0, 'Learning')\n",
"(1, 'Ray')\n",
"Runtime: 0.31 seconds, data:\n",
"(2, 'Flexible')\n",
"(3, 'Distributed')\n",
"Runtime: 0.51 seconds, data:\n",
"(4, 'Python')\n",
"(5, 'for')\n",
"Runtime: 0.71 seconds, data:\n",
"(6, 'Machine')\n",
"(7, 'Learning')\n",
"Runtime: 0.71 seconds, data:\n",
"(0, 'Learning')\n",
"(1, 'Ray')\n",
"(2, 'Flexible')\n",
"(3, 'Distributed')\n",
"(4, 'Python')\n",
"(5, 'for')\n",
"(6, 'Machine')\n",
"(7, 'Learning')\n"
]
}
],
"source": [
"start = time.time()\n",
"object_references = [\n",
" retrieve_task.remote(item, db_object_ref) for item in range(8)\n",
"]\n",
"all_data = []\n",
"\n",
"while len(object_references) > 0:\n",
" finished, object_references = ray.wait(\n",
" object_references, timeout=7.0\n",
" )\n",
" data = ray.get(finished)\n",
" print_runtime(data, start)\n",
" all_data.extend(data)\n",
"\n",
"print_runtime(all_data, start)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "cf6f00c3",
"metadata": {},
"source": [
"Instead of printing the results, you can use the retrieved values\n",
"within the `while` loop to initiate new tasks on other workers."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "1a9f6be5",
"metadata": {},
"source": [
"### Task dependencies\n",
"\n",
"You may want to perform an additional processing task on the retrieved data. For example, \n",
"use the results from the first retrieval task to query other related data from the same database (perhaps from a different table).\n",
"The code below sets up this follow-up task and executes both the `retrieve_task` and `follow_up_task` in sequence."
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "f5734bb1",
"metadata": {
"lines_to_next_cell": 2
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"((0, 'Learning'), (1, 'Ray'))\n",
"((2, 'Flexible'), (3, 'Distributed'))\n",
"((4, 'Python'), (5, 'for'))\n",
"((6, 'Machine'), (7, 'Learning'))\n"
]
}
],
"source": [
"@ray.remote\n",
"def follow_up_task(retrieve_result):\n",
" original_item, _ = retrieve_result\n",
" follow_up_result = retrieve(original_item + 1)\n",
" return retrieve_result, follow_up_result\n",
"\n",
"\n",
"retrieve_refs = [retrieve_task.remote(item, db_object_ref) for item in [0, 2, 4, 6]]\n",
"follow_up_refs = [follow_up_task.remote(ref) for ref in retrieve_refs]\n",
"\n",
"result = [print(data) for data in ray.get(follow_up_refs)]"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "6e4a6945",
"metadata": {},
"source": [
"If you're unfamiliar with asynchronous programming, this example may not be particularly impressive.\n",
"However, at second glance it might be surprising that the code runs at all.\n",
"The code appears to be a regular Python function with a few list comprehensions.\n",
"\n",
"The function body of `follow_up_task` expects a Python tuple for its input argument `retrieve_result`.\n",
"However, when you use the `[follow_up_task.remote(ref) for ref in retrieve_refs]` command,\n",
"you are not passing tuples to the follow-up task.\n",
"Instead, you are using the `retrieve_refs` to pass in Ray object references.\n",
"\n",
"Behind the scenes, Ray recognizes that the `follow_up_task` needs actual values,\n",
"so it _automatically_ uses the `ray.get` function to resolve these futures.\n",
"Additionally, Ray creates a dependency graph for all the tasks and executes them in a way that respects their dependencies.\n",
"You don't have to explicitly tell Ray when to wait for a previous task to be completed––it infers the order of execution.\n",
"This feature of the Ray object store is useful because you avoid copying large intermediate values\n",
"back to the driver by passing the object references to the next task and letting Ray handle the rest."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "f8722673",
"metadata": {},
"source": [
"The next steps in the process are only scheduled once the tasks specifically designed to retrieve information are completed.\n",
"In fact, if `retrieve_refs` was called `retrieve_result`, you might not have noticed this crucial and intentional naming nuance. Ray allows you to concentrate on your work rather than the technicalities of cluster computing.\n",
"The dependency graph for the two tasks looks like this:"
]
},
{
"cell_type": "markdown",
"id": "d03b8e46",
"metadata": {
"lines_to_next_cell": 2
},
"source": [
"![Task dependency](https://raw.githubusercontent.com/maxpumperla/learning_ray/main/notebooks/images/chapter_02/task_dependency.png)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "e4001edb",
"metadata": {},
"source": [
"### Ray Actors\n",
"\n",
"This example covers one more significant aspect of Ray Core.\n",
"Up until this step, everything is essentially a function.\n",
"You used the `@ray.remote` decorator to make certain functions remote, but aside from that, you only used standard Python."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "2a7a6e69",
"metadata": {},
"source": [
"If you want to keep track of how often the database is being queried, you could count the results of the retrieve tasks.\n",
"However, is there a more efficient way to do this? Ideally, you want to track this in a distributed manner that can handle a large amount of data.\n",
"Ray provides a solution with actors, which run stateful computations on a cluster and can also communicate with each other.\n",
"Similar to how you create Ray tasks using decorated functions, create Ray actors using decorated Python classes.\n",
"Therefore, you can create a simple counter using a Ray actor to track the number of database calls."
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "717df7d0",
"metadata": {},
"outputs": [],
"source": [
"@ray.remote\n",
"class DataTracker:\n",
" def __init__(self):\n",
" self._counts = 0\n",
"\n",
" def increment(self):\n",
" self._counts += 1\n",
"\n",
" def counts(self):\n",
" return self._counts"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "2e003cc6",
"metadata": {},
"source": [
"The DataTracker class becomes an actor when you give it the `ray.remote` decorator. This actor is capable of tracking state,\n",
"such as a count, and its methods are Ray actor tasks that you can invoke in the same way as functions using `.remote()`.\n",
"Modify the retrieve_task to incorporate this actor."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "6843b8d9",
"metadata": {
"lines_to_next_cell": 2
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[(0, 'Learning'), (1, 'Ray'), (2, 'Flexible'), (3, 'Distributed'), (4, 'Python'), (5, 'for'), (6, 'Machine'), (7, 'Learning')]\n",
"8\n"
]
}
],
"source": [
"@ray.remote\n",
"def retrieve_tracker_task(item, tracker, db):\n",
" time.sleep(item / 10.)\n",
" tracker.increment.remote()\n",
" return item, db[item]\n",
"\n",
"\n",
"tracker = DataTracker.remote()\n",
"\n",
"object_references = [\n",
" retrieve_tracker_task.remote(item, tracker, db_object_ref) for item in range(8)\n",
"]\n",
"data = ray.get(object_references)\n",
"\n",
"print(data)\n",
"print(ray.get(tracker.counts.remote()))"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "ae886162",
"metadata": {},
"source": [
"As expected, the outcome of this calculation is 8.\n",
"Although you don't need actors to perform this calculation, this demonstrates a way to maintain state across the cluster, possibly involving multiple tasks.\n",
"In fact, you could pass the actor into any related task or even into the constructor of a different actor.\n",
"The Ray API is flexible, allowing for limitless possibilities.\n",
"It's rare for distributed Python tools to allow for stateful computations,\n",
"which is especially useful for running complex distributed algorithms such as reinforcement learning."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "fb8bd0f5",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"In this example, you only used six API methods.\n",
"These included `ray.init()` to initiate the cluster, `@ray.remote` to transform functions and classes into tasks and actors,\n",
"`ray.put()` to transfer values into Ray's object store, and `ray.get()` to retrieve objects from the cluster.\n",
"Additionally, you used `.remote()` on actor methods or tasks to execute code on the cluster, and `ray.wait` to prevent blocking calls."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0d936caa",
"metadata": {},
"outputs": [],
"source": []
},
{
"attachments": {},
"cell_type": "markdown",
"id": "0d8693e9",
"metadata": {},
"source": [
"The Ray API consists of more than these six calls, but these six are powerful, if you're just starting out.\n",
"To summarize more generally, the methods are as follows:"
]
},
{
"cell_type": "markdown",
"id": "f2fb14bd",
"metadata": {},
"source": [
"- `ray.init()`: Initializes your Ray cluster. Pass in an address to connect to an existing cluster.\n",
"- `@ray.remote`: Turns functions into tasks and classes into actors.\n",
"- `ray.put()`: Puts values into Ray’s object store.\n",
"- `ray.get()`: Gets values from the object store. Returns the values you’ve put there or that were computed by a task or actor.\n",
"- `.remote()`: Runs actor methods or tasks on your Ray cluster and is used to instantiate actors.\n",
"- `ray.wait()`: Returns two lists of object references, one with finished tasks we’re waiting for and one with unfinished tasks."
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": false
},
"source": [
"## Want to learn more?\n",
"\n",
"This example is a simplified version of the Ray Core walkthrough of [our \"Learning Ray\" book](https://maxpumperla.com/learning_ray/).\n",
"If you liked it, check out the [Ray Core Examples Gallery](./overview.rst) or some of the ML workloads in our [Use Case Gallery](../../ray-overview/use-cases.rst)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": []
}
],
"metadata": {
"jupytext": {
"cell_metadata_filter": "-all",
"main_language": "python",
"notebook_metadata_filter": "-all"
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.13"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
doc/source/ray-core/examples/highly_parallel.ipynb
+338
View File
@@ -0,0 +1,338 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "515dffba",
"metadata": {},
"source": [
"# Using Ray for Highly Parallelizable Tasks\n",
"\n",
"<a id=\"try-anyscale-quickstart-highly_parallel\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=highly_parallel\">\n",
" <img src=\"../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
"</a>\n",
"<br></br>\n",
"\n",
"While Ray can be used for very complex parallelization tasks,\n",
"often we just want to do something simple in parallel.\n",
"For example, we may have 100,000 time series to process with exactly the same algorithm,\n",
"and each one takes a minute of processing.\n",
"\n",
"Clearly running it on a single processor is prohibitive: this would take 70 days.\n",
"Even if we managed to use 8 processors on a single machine,\n",
"that would bring it down to 9 days. But if we can use 8 machines, each with 16 cores,\n",
"it can be done in about 12 hours.\n",
"\n",
"How can we use Ray for these types of task? \n",
"\n",
"We take the simple example of computing the digits of pi.\n",
"The algorithm is simple: generate random x and y, and if ``x^2 + y^2 < 1``, it's\n",
"inside the circle, we count as in. This actually turns out to be pi/4\n",
"(remembering your high school math).\n",
"\n",
"The following code (and this notebook) assumes you have already set up your Ray cluster and that you are running on the head node. For more details on how to set up a Ray cluster please see [Ray Clusters Getting Started](https://docs.ray.io/en/master/cluster/getting-started.html). \n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "8e3e7c4f",
"metadata": {},
"outputs": [],
"source": [
"import ray\n",
"import random\n",
"import time\n",
"import math\n",
"from fractions import Fraction"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "92d2461b",
"metadata": {
"scrolled": true,
"tags": [
"remove-output"
]
},
"outputs": [],
"source": [
"# Let's start Ray\n",
"ray.init(address='auto')"
]
},
{
"cell_type": "markdown",
"id": "b96f2eb9",
"metadata": {},
"source": [
"We use the ``@ray.remote`` decorator to create a Ray task.\n",
"A task is like a function, except the result is returned asynchronously.\n",
"\n",
"It also may not run on the local machine, it may run elsewhere in the cluster.\n",
"This way you can run multiple tasks in parallel,\n",
"beyond the limit of the number of processors you can have in a single machine."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "ece9887c",
"metadata": {},
"outputs": [],
"source": [
"@ray.remote\n",
"def pi4_sample(sample_count):\n",
" \"\"\"pi4_sample runs sample_count experiments, and returns the \n",
" fraction of time it was inside the circle. \n",
" \"\"\"\n",
" in_count = 0\n",
" for i in range(sample_count):\n",
" x = random.random()\n",
" y = random.random()\n",
" if x*x + y*y <= 1:\n",
" in_count += 1\n",
" return Fraction(in_count, sample_count)\n"
]
},
{
"cell_type": "markdown",
"id": "05bf8675",
"metadata": {},
"source": [
"To get the result of a future, we use ray.get() which \n",
"blocks until the result is complete. "
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "9d9a3509",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running 1000000 tests took 1.4935967922210693 seconds\n"
]
}
],
"source": [
"SAMPLE_COUNT = 1000 * 1000\n",
"start = time.time() \n",
"future = pi4_sample.remote(sample_count = SAMPLE_COUNT)\n",
"pi4 = ray.get(future)\n",
"end = time.time()\n",
"dur = end - start\n",
"print(f'Running {SAMPLE_COUNT} tests took {dur} seconds')"
]
},
{
"cell_type": "markdown",
"id": "cc17429d",
"metadata": {},
"source": [
"Now let's see how good our approximation is."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "42d4c464",
"metadata": {},
"outputs": [],
"source": [
"pi = pi4 * 4"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "4009bee0",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"3.143024"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"float(pi)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "d19155d6",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.0004554042254233261"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"abs(pi-math.pi)/pi"
]
},
{
"cell_type": "markdown",
"id": "ddb3b095",
"metadata": {},
"source": [
"Meh. A little off -- that's barely 4 decimal places.\n",
"Why don't we do it a 100,000 times as much? Let's do 100 billion!"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "b7b9cff9",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Doing 100000 batches\n"
]
}
],
"source": [
"FULL_SAMPLE_COUNT = 100 * 1000 * 1000 * 1000 # 100 billion samples! \n",
"BATCHES = int(FULL_SAMPLE_COUNT / SAMPLE_COUNT)\n",
"print(f'Doing {BATCHES} batches')\n",
"results = []\n",
"for _ in range(BATCHES):\n",
" results.append(pi4_sample.remote(sample_count = SAMPLE_COUNT))\n",
"output = ray.get(results)"
]
},
{
"cell_type": "markdown",
"id": "94264de4",
"metadata": {},
"source": [
"Notice that in the above, we generated a list with 100,000 futures.\n",
"Now all we do is have to do is wait for the result.\n",
"\n",
"Depending on your ray cluster's size, this might take a few minutes.\n",
"But to give you some idea, if we were to do it on a single machine,\n",
"when I ran this it took 0.4 seconds.\n",
"\n",
"On a single core, that means we're looking at 0.4 * 100000 = about 11 hours. \n",
"\n",
"Here's what the Dashboard looks like: \n",
"\n",
"![View of the dashboard](../images/dashboard.png)\n",
"\n",
"So now, rather than just a single core working on this,\n",
"I have 168 working on the task together. And its ~80% efficient."
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "76eba02d",
"metadata": {},
"outputs": [],
"source": [
"pi = sum(output)*4/len(output)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "ede2bd8c",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"3.14159518188"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"float(pi)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "bb62cb27",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"8.047791203506436e-07"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"abs(pi-math.pi)/pi"
]
},
{
"cell_type": "markdown",
"id": "30d12e50",
"metadata": {},
"source": [
"Not bad at all -- we're off by a millionth. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1b36747b",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"celltoolbar": "Tags",
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.13"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
doc/source/ray-core/examples/images/a3c.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 24 KiB

doc/source/ray-core/examples/images/automl.svg
+4
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 9.7 KiB

Some files were not shown because too many files have changed in this diff Show More