chore: import upstream snapshot with attribution

doc/source/ray-core/patterns/actor-sync.rst

@@ -0,0 +1,16 @@
+Pattern: Using an actor to synchronize other tasks and actors
+=============================================================
+
+When you have multiple tasks that need to wait on some condition or otherwise
+need to synchronize across tasks & actors on a cluster, you can use a central
+actor to coordinate among them.
+
+Example use case
+----------------
+
+You can use an actor to implement a distributed ``asyncio.Event`` that multiple tasks can wait on.
+
+Code example
+------------
+
+.. literalinclude:: ../doc_code/actor-sync.py

doc/source/ray-core/patterns/closure-capture-large-objects.rst

@@ -0,0 +1,38 @@
+Anti-pattern: Closure capturing large objects harms performance
+===============================================================
+
+**TLDR:** Avoid closure capturing large objects in remote functions or classes, use object store instead.
+
+When you define a :func:`ray.remote <ray.remote>` function or class,
+it is easy to accidentally capture large (more than a few MB) objects implicitly in the definition.
+This can lead to slow performance or even OOM since Ray is not designed to handle serialized functions or classes that are very large.
+
+For such large objects, there are two options to resolve this problem:
+
+- Use :func:`ray.put() <ray.put>` to put the large objects in the Ray object store, and then pass object references as arguments to the remote functions or classes (*"better approach #1"* below)
+- Create the large objects inside the remote functions or classes by passing a lambda method (*"better approach #2"*). This is also the only option for using unserializable objects.
+
+
+Code example
+------------
+
+**Anti-pattern:**
+
+.. literalinclude:: ../doc_code/anti_pattern_closure_capture_large_objects.py
+    :language: python
+    :start-after: __anti_pattern_start__
+    :end-before: __anti_pattern_end__
+
+**Better approach #1:**
+
+.. literalinclude:: ../doc_code/anti_pattern_closure_capture_large_objects.py
+    :language: python
+    :start-after: __better_approach_1_start__
+    :end-before: __better_approach_1_end__
+
+**Better approach #2:**
+
+.. literalinclude:: ../doc_code/anti_pattern_closure_capture_large_objects.py
+    :language: python
+    :start-after: __better_approach_2_start__
+    :end-before: __better_approach_2_end__

doc/source/ray-core/patterns/concurrent-operations-async-actor.rst

@@ -0,0 +1,33 @@
+Pattern: Using asyncio to run actor methods concurrently
+========================================================
+
+By default, a Ray :ref:`actor <ray-remote-classes>` runs in a single thread and
+actor method calls are executed sequentially. This means that a long running method call blocks all the following ones.
+In this pattern, we use ``await`` to yield control from the long running method call so other method calls can run concurrently.
+Normally the control is yielded when the method is doing IO operations but you can also use ``await asyncio.sleep(0)`` to yield control explicitly.
+
+.. note::
+   You can also use :ref:`threaded actors <threaded-actors>` to achieve concurrency.
+
+Example use case
+----------------
+
+You have an actor with a long polling method that continuously fetches tasks from the remote store and executes them.
+You also want to query the number of tasks executed while the long polling method is running.
+
+With the default actor, the code will look like this:
+
+.. literalinclude:: ../doc_code/pattern_async_actor.py
+    :language: python
+    :start-after: __sync_actor_start__
+    :end-before: __sync_actor_end__
+
+This is problematic because ``TaskExecutor.run`` method runs forever and never yields control to run other methods.
+We can solve this problem by using :ref:`async actors <async-actors>` and use ``await`` to yield control:
+
+.. literalinclude:: ../doc_code/pattern_async_actor.py
+    :language: python
+    :start-after: __async_actor_start__
+    :end-before: __async_actor_end__
+
+Here, instead of using the blocking :func:`ray.get() <ray.get>` to get the value of an ObjectRef, we use ``await`` so it can yield control while we are waiting for the object to be fetched.

doc/source/ray-core/patterns/fork-new-processes.rst

@@ -0,0 +1,27 @@
+.. _forking-ray-processes-antipattern:
+
+Anti-pattern: Forking new processes in application code
+========================================================
+
+**Summary:** Don't fork new processes in Ray application code—for example, in 
+driver, tasks or actors. Instead, use the "spawn" method to start new processes or use Ray 
+tasks and actors to parallelize your workload
+
+Ray manages the lifecycle of processes for you. Ray Objects, Tasks, and 
+Actors manage sockets to communicate with the Raylet and the GCS. If you fork new 
+processes in your application code, the processes could share the same sockets without 
+any synchronization. This can lead to corrupted messages and unexpected
+behavior.
+
+The solution is to:
+1. use the "spawn" method to start new processes so that the parent process's 
+memory space is not copied to the child processes or
+2. use Ray tasks and 
+actors to parallelize your workload and let Ray manage the lifecycle of the 
+processes for you.
+
+Code example
+------------
+.. literalinclude:: ../doc_code/anti_pattern_fork_new_processes.py
+    :language: python
+

doc/source/ray-core/patterns/generators.rst

@@ -0,0 +1,48 @@
+.. _generator-pattern:
+
+Pattern: Using generators to reduce heap memory usage
+=====================================================
+
+In this pattern, we use **generators** in Python to reduce the total heap memory usage during a task. The key idea is that for tasks that return multiple objects, we can return them one at a time instead of all at once. This allows a worker to free the heap memory used by a previous return value before returning the next one.
+
+Example use case
+----------------
+
+You have a task that returns multiple large values. Another possibility is a task that returns a single large value, but you want to stream this value through Ray's object store by breaking it up into smaller chunks.
+
+Using normal Python functions, we can write such a task like this. Here's an example that returns numpy arrays of size 100MB each:
+
+.. literalinclude:: ../doc_code/pattern_generators.py
+    :language: python
+    :start-after: __large_values_start__
+    :end-before: __large_values_end__
+
+However, this will require the task to hold all ``num_returns`` arrays in heap memory at the same time at the end of the task. If there are many return values, this can lead to high heap memory usage and potentially an out-of-memory error.
+
+We can fix the above example by rewriting ``large_values`` as a **generator**. Instead of returning all values at once as a tuple or list, we can ``yield`` one value at a time.
+
+.. literalinclude:: ../doc_code/pattern_generators.py
+    :language: python
+    :start-after: __large_values_generator_start__
+    :end-before: __large_values_generator_end__
+
+Code example
+------------
+
+.. literalinclude:: ../doc_code/pattern_generators.py
+    :language: python
+    :start-after: __program_start__
+
+.. code-block:: text
+
+    $ RAY_IGNORE_UNHANDLED_ERRORS=1 python test.py 100
+
+    Using normal functions...
+    ... -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker...
+    Worker failed
+    Using generators...
+    (large_values_generator pid=373609) yielded return value 0
+    (large_values_generator pid=373609) yielded return value 1
+    (large_values_generator pid=373609) yielded return value 2
+    ...
+    Success!

doc/source/ray-core/patterns/global-variables.rst

@@ -0,0 +1,30 @@
+Anti-pattern: Using global variables to share state between tasks and actors
+============================================================================
+
+**TLDR:** Don't use global variables to share state with tasks and actors. Instead, encapsulate the global variables in an actor and pass the actor handle to other tasks and actors.
+
+Ray drivers, tasks and actors are running in
+different processes, so they don’t share the same address space.
+This means that if you modify global variables
+in one process, changes are not reflected in other processes.
+
+The solution is to use an actor's instance variables to hold the global state and pass the actor handle to places where the state needs to be modified or accessed.
+Note that using class variables to manage state between instances of the same class is not supported.
+Each actor instance is instantiated in its own process, so each actor will have its own copy of the class variables.
+
+Code example
+------------
+
+**Anti-pattern:**
+
+.. literalinclude:: ../doc_code/anti_pattern_global_variables.py
+    :language: python
+    :start-after: __anti_pattern_start__
+    :end-before: __anti_pattern_end__
+
+**Better approach:**
+
+.. literalinclude:: ../doc_code/anti_pattern_global_variables.py
+    :language: python
+    :start-after: __better_approach_start__
+    :end-before: __better_approach_end__

doc/source/ray-core/patterns/index.rst

@@ -0,0 +1,31 @@
+.. _core-patterns:
+
+Design Patterns & Anti-patterns
+===============================
+
+This section is a collection of common design patterns and anti-patterns for writing Ray applications.
+
+.. toctree::
+    :maxdepth: 1
+
+    nested-tasks
+    generators
+    limit-pending-tasks
+    limit-running-tasks
+    concurrent-operations-async-actor
+    actor-sync
+    tree-of-actors
+    pipelining
+    return-ray-put
+    nested-ray-get
+    ray-get-loop
+    unnecessary-ray-get
+    ray-get-submission-order
+    ray-get-too-many-objects
+    too-fine-grained-tasks
+    redefine-task-actor-loop
+    pass-large-arg-by-value
+    closure-capture-large-objects
+    global-variables
+    out-of-band-object-ref-serialization
+    fork-new-processes

doc/source/ray-core/patterns/limit-pending-tasks.rst

@@ -0,0 +1,49 @@
+.. _core-patterns-limit-pending-tasks:
+
+Pattern: Using ray.wait to limit the number of pending tasks
+============================================================
+
+In this pattern, we use :func:`ray.wait() <ray.wait>` to limit the number of pending tasks.
+
+If we continuously submit tasks faster than their process time, we will accumulate tasks in the pending task queue, which can eventually cause OOM.
+With ``ray.wait()``, we can apply backpressure and limit the number of pending tasks so that the pending task queue won't grow indefinitely and cause OOM.
+
+.. note::
+
+   If we submit a finite number of tasks, it's unlikely that we will hit the issue mentioned above since each task only uses a small amount of memory for bookkeeping in the queue.
+   It's more likely to happen when we have an infinite stream of tasks to run.
+
+.. note::
+
+   This method is meant primarily to limit how many tasks should be in flight at the same time.
+   It can also be used to limit how many tasks can run *concurrently*, but it is not recommended, as it can hurt scheduling performance.
+   Ray automatically decides task parallelism based on resource availability, so the recommended method for adjusting how many tasks can run concurrently is to :ref:`modify each task's resource requirements <core-patterns-limit-running-tasks>` instead.
+
+Example use case
+----------------
+
+You have a worker actor that processes tasks at a rate of X tasks per second and you want to submit tasks to it at a rate lower than X to avoid OOM.
+
+For example, Ray Serve uses this pattern to limit the number of pending queries for each worker.
+
+.. figure:: ../images/limit-pending-tasks.svg
+
+    Limit number of pending tasks
+
+
+Code example
+------------
+
+**Without backpressure:**
+
+.. literalinclude:: ../doc_code/limit_pending_tasks.py
+    :language: python
+    :start-after: __without_backpressure_start__
+    :end-before: __without_backpressure_end__
+
+**With backpressure:**
+
+.. literalinclude:: ../doc_code/limit_pending_tasks.py
+    :language: python
+    :start-after: __with_backpressure_start__
+    :end-before: __with_backpressure_end__

doc/source/ray-core/patterns/limit-running-tasks.rst

@@ -0,0 +1,42 @@
+.. _core-patterns-limit-running-tasks:
+
+Pattern: Using resources to limit the number of concurrently running tasks
+==========================================================================
+
+In this pattern, we use :ref:`resources <resource-requirements>` to limit the number of concurrently running tasks.
+
+By default, Ray tasks require 1 CPU each and Ray actors require 0 CPU each, so the scheduler limits task concurrency to the available CPUs and actor concurrency to infinite.
+Tasks that use more than 1 CPU (e.g., via multithreading) may experience slowdown due to interference from concurrent ones, but otherwise are safe to run.
+
+However, tasks or actors that use more than their proportionate share of memory may overload a node and cause issues like OOM.
+If that is the case, we can reduce the number of concurrently running tasks or actors on each node by increasing the amount of resources requested by them.
+This works because Ray makes sure that the sum of the resource requirements of all of the concurrently running tasks and actors on a given node does not exceed the node's total resources.
+
+.. note::
+
+   For actor tasks, the number of running actors limits the number of concurrently running actor tasks we can have.
+
+Example use case
+----------------
+
+You have a data processing workload that processes each input file independently using Ray :ref:`remote functions <ray-remote-functions>`.
+Since each task needs to load the input data into heap memory and do the processing, running too many of them can cause OOM.
+In this case, you can use the ``memory`` resource to limit the number of concurrently running tasks (usage of other resources like ``num_cpus`` can achieve the same goal as well).
+Note that similar to ``num_cpus``, the ``memory`` resource requirement is *logical*, meaning that Ray will not enforce the physical memory usage of each task if it exceeds this amount.
+
+Code example
+------------
+
+**Without limit:**
+
+.. literalinclude:: ../doc_code/limit_running_tasks.py
+    :language: python
+    :start-after: __without_limit_start__
+    :end-before: __without_limit_end__
+
+**With limit:**
+
+.. literalinclude:: ../doc_code/limit_running_tasks.py
+    :language: python
+    :start-after: __with_limit_start__
+    :end-before: __with_limit_end__

doc/source/ray-core/patterns/nested-ray-get.rst

@@ -0,0 +1,26 @@
+.. _nested-ray-get:
+
+Anti-pattern: Calling ray.get on task arguments harms performance
+=================================================================
+
+
+**TLDR:** If possible, pass ``ObjectRefs`` as direct task arguments, instead of passing a list as the task argument and then calling :func:`ray.get() <ray.get>` inside the task.
+
+When a task calls ``ray.get()``, it must block until the value of the ``ObjectRef`` is ready.
+If all cores are already occupied, this situation can lead to a deadlock, as the task that produces the ``ObjectRef``'s value may need the caller task's resources in order to run.
+To handle this issue, if the caller task would block in ``ray.get()``, Ray temporarily releases the caller's CPU resources to allow the pending task to run.
+This behavior can harm performance and stability because the caller continues to use a process and memory to hold its stack while other tasks run.
+
+Therefore, it is always better to pass ``ObjectRefs`` as direct arguments to a task and avoid calling ``ray.get`` inside of the task, if possible.
+
+For example, in the following code, prefer the latter method of invoking the dependent task.
+
+.. literalinclude:: ../doc_code/anti_pattern_nested_ray_get.py
+    :language: python
+    :start-after: __anti_pattern_start__
+    :end-before: __anti_pattern_end__
+
+Avoiding ``ray.get`` in nested tasks may not always be possible. Some valid reasons to call ``ray.get`` include:
+
+- :doc:`nested-tasks`
+- If the nested task has multiple ``ObjectRefs`` to ``ray.get``, and it wants to choose the order and number to get.

doc/source/ray-core/patterns/nested-tasks.rst

@@ -0,0 +1,35 @@
+.. _nested-tasks:
+
+Pattern: Using nested tasks to achieve nested parallelism
+=========================================================
+
+In this pattern, a remote task can dynamically call other remote tasks (including itself) for nested parallelism.
+This is useful when sub-tasks can be parallelized.
+
+Keep in mind, though, that nested tasks come with their own cost: extra worker processes, scheduling overhead, bookkeeping overhead, etc.
+To achieve speedup with nested parallelism, make sure each of your nested tasks does significant work. See :doc:`too-fine-grained-tasks` for more details.
+
+Example use case
+----------------
+
+You want to quick-sort a large list of numbers.
+By using nested tasks, we can sort the list in a distributed and parallel fashion.
+
+.. figure:: ../images/tree-of-tasks.svg
+
+    Tree of tasks
+
+
+Code example
+------------
+
+.. literalinclude:: ../doc_code/pattern_nested_tasks.py
+    :language: python
+    :start-after: __pattern_start__
+    :end-before: __pattern_end__
+
+We call :func:`ray.get() <ray.get>` after both ``quick_sort_distributed`` function invocations take place.
+This allows you to maximize parallelism in the workload. See :doc:`ray-get-loop` for more details.
+
+Notice in the execution times above that with smaller tasks, the non-distributed version is faster. However, as the task execution
+time increases, i.e. because the lists to sort are larger, the distributed version is faster.

doc/source/ray-core/patterns/out-of-band-object-ref-serialization.rst

@@ -0,0 +1,26 @@
+.. _ray-out-of-band-object-ref-serialization:
+
+Anti-pattern: Serialize ray.ObjectRef out of band
+=================================================
+
+**TLDR:** Avoid serializing ``ray.ObjectRef`` because Ray can't know when to garbage collect the underlying object.
+
+Ray's ``ray.ObjectRef`` is distributed reference counted. Ray pins the underlying object until the reference isn't used by the system anymore.
+When all references to the pinned object are gone, Ray garbage collects the pinned object and cleans it up from the system.
+However, if user code serializes ``ray.ObjectRef``, Ray can't keep track of the reference.
+
+To avoid incorrect behavior, if ``ray.cloudpickle`` serializes ``ray.ObjectRef``, Ray pins the object for the lifetime of a worker. "Pin" means that object can't be evicted from the object store
+until the corresponding owner worker dies. It's prone to Ray object leaks, which can lead to disk spilling. See :ref:`this page <serialize-object-ref>` for more details.
+
+To detect if this pattern exists in your code, you can set an environment variable ``RAY_allow_out_of_band_object_ref_serialization=0``. If Ray detects
+that ``ray.cloudpickle`` serialized ``ray.ObjectRef``, it raises an exception with helpful messages.
+
+Code example
+------------
+
+**Anti-pattern:**
+
+.. literalinclude:: ../doc_code/anti_pattern_out_of_band_object_ref_serialization.py
+    :language: python
+    :start-after: __anti_pattern_start__
+    :end-before: __anti_pattern_end__

doc/source/ray-core/patterns/pass-large-arg-by-value.rst

@@ -0,0 +1,31 @@
+.. _ray-pass-large-arg-by-value:
+
+Anti-pattern: Passing the same large argument by value repeatedly harms performance
+===================================================================================
+
+**TLDR:** Avoid passing the same large argument by value to multiple tasks, use :func:`ray.put() <ray.put>` and pass by reference instead.
+
+When passing a large argument (>100KB) by value to a task,
+Ray will implicitly store the argument in the object store and the worker process will fetch the argument to the local object store from the caller's object store before running the task.
+If we pass the same large argument to multiple tasks, Ray will end up storing multiple copies of the argument in the object store since Ray doesn't do deduplication.
+
+Instead of passing the large argument by value to multiple tasks,
+we should use ``ray.put()`` to store the argument to the object store once and get an ``ObjectRef``,
+then pass the argument reference to tasks. This way, we make sure all tasks use the same copy of the argument, which is faster and uses less object store memory.
+
+Code example
+------------
+
+**Anti-pattern:**
+
+.. literalinclude:: ../doc_code/anti_pattern_pass_large_arg_by_value.py
+    :language: python
+    :start-after: __anti_pattern_start__
+    :end-before: __anti_pattern_end__
+
+**Better approach:**
+
+.. literalinclude:: ../doc_code/anti_pattern_pass_large_arg_by_value.py
+    :language: python
+    :start-after: __better_approach_start__
+    :end-before: __better_approach_end__

doc/source/ray-core/patterns/pipelining.rst

@@ -0,0 +1,27 @@
+Pattern: Using pipelining to increase throughput
+================================================
+
+If you have multiple work items and each requires several steps to complete,
+you can use the `pipelining <https://en.wikipedia.org/wiki/Pipeline_(computing)>`__ technique to improve the cluster utilization and increase the throughput of your system.
+
+.. note::
+
+  Pipelining is an important technique to improve the performance and is heavily used by Ray libraries.
+  See :ref:`Ray Data <data>` as an example.
+
+.. figure:: ../images/pipelining.svg
+
+Example use case
+----------------
+
+A component of your application needs to do both compute-intensive work and communicate with other processes.
+Ideally, you want to overlap computation and communication to saturate the CPU and increase the overall throughput.
+
+Code example
+------------
+
+.. literalinclude:: ../doc_code/pattern_pipelining.py
+
+In the example above, a worker actor pulls work off of a queue and then does some computation on it.
+Without pipelining, we call :func:`ray.get() <ray.get>` immediately after requesting a work item, so we block while that RPC is in flight, causing idle CPU time.
+With pipelining, we instead preemptively request the next work item before processing the current one, so we can use the CPU while the RPC is in flight which increases the CPU utilization.

doc/source/ray-core/patterns/ray-get-loop.rst

@@ -0,0 +1,34 @@
+.. _ray-get-loop:
+
+Anti-pattern: Calling ray.get in a loop harms parallelism
+=========================================================
+
+**TLDR:** Avoid calling :func:`ray.get() <ray.get>` in a loop since it's a blocking call; use ``ray.get()`` only for the final result.
+
+A call to ``ray.get()`` fetches the results of remotely executed functions. However, it is a blocking call, which means that it always waits until the requested result is available.
+If you call ``ray.get()`` in a loop, the loop will not continue to run until the call to ``ray.get()`` is resolved.
+
+If you also spawn the remote function calls in the same loop, you end up with no parallelism at all, as you wait for the previous function call to finish (because of ``ray.get()``) and only spawn the next call in the next iteration of the loop.
+The solution here is to separate the call to ``ray.get()`` from the call to the remote functions. That way all remote functions are spawned before we wait for the results and can run in parallel in the background. Additionally, you can pass a list of object references to ``ray.get()`` instead of calling it one by one to wait for all of the tasks to finish.
+
+Code example
+------------
+
+.. literalinclude:: ../doc_code/anti_pattern_ray_get_loop.py
+    :language: python
+    :start-after: __anti_pattern_start__
+    :end-before: __anti_pattern_end__
+
+.. figure:: ../images/ray-get-loop.svg
+
+    Calling ``ray.get()`` in a loop
+
+When calling ``ray.get()`` right after scheduling the remote work, the loop blocks until the result is received. We thus end up with sequential processing.
+Instead, we should first schedule all remote calls, which are then processed in parallel. After scheduling the work, we can then request all the results at once.
+
+Other ``ray.get()`` related anti-patterns are:
+
+- :doc:`nested-ray-get`
+- :doc:`unnecessary-ray-get`
+- :doc:`ray-get-submission-order`
+- :doc:`ray-get-too-many-objects`

doc/source/ray-core/patterns/ray-get-submission-order.rst

@@ -0,0 +1,27 @@
+Anti-pattern: Processing results in submission order using ray.get increases runtime
+====================================================================================
+
+**TLDR:** Avoid processing independent results in submission order using :func:`ray.get() <ray.get>` since results may be ready in a different order than the submission order.
+
+A batch of tasks is submitted, and we need to process their results individually once they’re done.
+If each task takes a different amount of time to finish and we process results in submission order, we may waste time waiting for all of the slower (straggler) tasks that were submitted earlier to finish while later faster tasks have already finished.
+
+Instead, we want to process the tasks in the order that they finish using :func:`ray.wait() <ray.wait>` to speed up total time to completion.
+
+.. figure:: ../images/ray-get-submission-order.svg
+
+    Processing results in submission order vs completion order
+
+
+Code example
+------------
+
+.. literalinclude:: ../doc_code/anti_pattern_ray_get_submission_order.py
+    :language: python
+    :start-after: __anti_pattern_start__
+    :end-before: __anti_pattern_end__
+
+Other ``ray.get()`` related anti-patterns are:
+
+- :doc:`unnecessary-ray-get`
+- :doc:`ray-get-loop`

doc/source/ray-core/patterns/ray-get-too-many-objects.rst

@@ -0,0 +1,32 @@
+.. _ray-get-too-many-objects:
+
+Anti-pattern: Fetching too many objects at once with ray.get causes failure
+===========================================================================
+
+**TLDR:** Avoid calling :func:`ray.get() <ray.get>` on too many objects since this will lead to heap out-of-memory or object store out-of-space. Instead fetch and process one batch at a time.
+
+If you have a large number of tasks that you want to run in parallel, trying to do ``ray.get()`` on all of them at once could lead to failure with heap out-of-memory or object store out-of-space since Ray needs to fetch all the objects to the caller at the same time.
+Instead you should get and process the results one batch at a time. Once a batch is processed, Ray will evict objects in that batch to make space for future batches.
+
+.. figure:: ../images/ray-get-too-many-objects.svg
+
+    Fetching too many objects at once with ``ray.get()``
+
+Code example
+------------
+
+**Anti-pattern:**
+
+.. literalinclude:: ../doc_code/anti_pattern_ray_get_too_many_objects.py
+    :language: python
+    :start-after: __anti_pattern_start__
+    :end-before: __anti_pattern_end__
+
+**Better approach:**
+
+.. literalinclude:: ../doc_code/anti_pattern_ray_get_too_many_objects.py
+    :language: python
+    :start-after: __better_approach_start__
+    :end-before: __better_approach_end__
+
+Here besides getting one batch at a time to avoid failure, we are also using ``ray.wait()`` to process results in the finish order instead of the submission order to reduce the runtime. See :doc:`ray-get-submission-order` for more details.

doc/source/ray-core/patterns/redefine-task-actor-loop.rst

@@ -0,0 +1,29 @@
+Anti-pattern: Redefining the same remote function or class harms performance
+============================================================================
+
+**TLDR:** Avoid redefining the same remote function or class.
+
+Decorating the same function or class multiple times using the :func:`ray.remote <ray.remote>` decorator leads to slow performance in Ray.
+For each Ray remote function or class, Ray will pickle it and upload to GCS.
+Later on, the worker that runs the task or actor will download and unpickle it.
+Each decoration of the same function or class generates a new remote function or class from Ray's perspective.
+As a result, the pickle, upload, download and unpickle work will happen every time we redefine and run the remote function or class.
+
+Code example
+------------
+
+**Anti-pattern:**
+
+.. literalinclude:: ../doc_code/anti_pattern_redefine_task_actor_loop.py
+    :language: python
+    :start-after: __anti_pattern_start__
+    :end-before: __anti_pattern_end__
+
+**Better approach:**
+
+.. literalinclude:: ../doc_code/anti_pattern_redefine_task_actor_loop.py
+    :language: python
+    :start-after: __better_approach_start__
+    :end-before: __better_approach_end__
+
+We should define the same remote function or class outside of the loop instead of multiple times inside a loop so that it's pickled and uploaded only once.

doc/source/ray-core/patterns/return-ray-put.rst

@@ -0,0 +1,37 @@
+Anti-pattern: Returning ray.put() ObjectRefs from a task harms performance and fault tolerance
+==============================================================================================
+
+**TLDR:** Avoid calling :func:`ray.put() <ray.put>` on task return values and returning the resulting ObjectRefs.
+Instead, return these values directly if possible.
+
+Returning ray.put() ObjectRefs are considered anti-patterns for the following reasons:
+
+- It disallows inlining small return values: Ray has a performance optimization to return small (<= 100KB) values inline directly to the caller, avoiding going through the distributed object store.
+  On the other hand, ``ray.put()`` will unconditionally store the value to the object store which makes the optimization for small return values impossible.
+- Returning ObjectRefs involves extra distributed reference counting protocol which is slower than returning the values directly.
+- It's less :ref:`fault tolerant <fault-tolerance>`: the worker process that calls ``ray.put()`` is the "owner" of the returned ``ObjectRef`` and the return value fate shares with the owner. If the worker process dies, the return value is lost.
+  In contrast, the caller process (often the driver) is the owner of the return value if it's returned directly.
+
+Code example
+------------
+
+If you want to return a single value regardless if it's small or large, you should return it directly.
+
+.. literalinclude:: ../doc_code/anti_pattern_return_ray_put.py
+    :language: python
+    :start-after: __return_single_value_start__
+    :end-before: __return_single_value_end__
+
+If you want to return multiple values and you know the number of returns before calling the task, you should use the :ref:`num_returns <ray-task-returns>` option.
+
+.. literalinclude:: ../doc_code/anti_pattern_return_ray_put.py
+    :language: python
+    :start-after: __return_static_multi_values_start__
+    :end-before: __return_static_multi_values_end__
+
+If you don't know the number of returns before calling the task, you should use the :ref:`dynamic generator <dynamic-generators>` pattern if possible.
+
+.. literalinclude:: ../doc_code/anti_pattern_return_ray_put.py
+    :language: python
+    :start-after: __return_dynamic_multi_values_start__
+    :end-before: __return_dynamic_multi_values_end__

doc/source/ray-core/patterns/too-fine-grained-tasks.rst

@@ -0,0 +1,29 @@
+Anti-pattern: Over-parallelizing with too fine-grained tasks harms speedup
+==========================================================================
+
+**TLDR:** Avoid over-parallelizing. Parallelizing tasks has higher overhead than using normal functions.
+
+Parallelizing or distributing tasks usually comes with higher overhead than an ordinary function call. Therefore, if you parallelize a function that executes very quickly, the overhead could take longer than the actual function call!
+
+To handle this problem, we should be careful about parallelizing too much. If you have a function or task that’s too small, you can use a technique called **batching** to make your tasks do more meaningful work in a single call.
+
+
+Code example
+------------
+
+**Anti-pattern:**
+
+.. literalinclude:: ../doc_code/anti_pattern_too_fine_grained_tasks.py
+    :language: python
+    :start-after: __anti_pattern_start__
+    :end-before: __anti_pattern_end__
+
+**Better approach:** Use batching.
+
+.. literalinclude:: ../doc_code/anti_pattern_too_fine_grained_tasks.py
+    :language: python
+    :start-after: __batching_start__
+    :end-before: __batching_end__
+
+As we can see from the example above, over-parallelizing has higher overhead and the program runs slower than the serial version.
+Through batching with a proper batch size, we are able to amortize the overhead and achieve the expected speedup.

doc/source/ray-core/patterns/tree-of-actors.rst

@@ -0,0 +1,32 @@
+Pattern: Using a supervisor actor to manage a tree of actors
+============================================================
+
+Actor supervision is a pattern in which a supervising actor manages a collection of worker actors.
+The supervisor delegates tasks to subordinates and handles their failures.
+This pattern simplifies the driver since it manages only a few supervisors and does not deal with failures from worker actors directly.
+Furthermore, multiple supervisors can act in parallel to parallelize more work.
+
+.. figure:: ../images/tree-of-actors.svg
+
+    Tree of actors
+
+.. note::
+
+    - If the supervisor dies (or the driver), the worker actors are automatically terminated thanks to actor reference counting.
+    - Actors can be nested to multiple levels to form a tree.
+
+Example use case
+----------------
+
+You want to do data parallel training and train the same model with different hyperparameters in parallel.
+For each hyperparameter, you can launch a supervisor actor to do the orchestration and it will create worker actors to do the actual training per data shard.
+
+.. note::
+    For data parallel training and hyperparameter tuning, it's recommended to use :ref:`Ray Train <train-key-concepts>` (:py:class:`~ray.train.data_parallel_trainer.DataParallelTrainer` and :ref:`Ray Tune's Tuner <tune-main>`)
+    which applies this pattern under the hood.
+
+Code example
+------------
+
+.. literalinclude:: ../doc_code/pattern_tree_of_actors.py
+    :language: python

doc/source/ray-core/patterns/unnecessary-ray-get.rst

@@ -0,0 +1,41 @@
+.. _unnecessary-ray-get:
+
+Anti-pattern: Calling ray.get unnecessarily harms performance
+=============================================================
+
+**TLDR:** Avoid calling :func:`ray.get() <ray.get>` unnecessarily for intermediate steps. Work with object references directly, and only call ``ray.get()`` at the end to get the final result.
+
+When ``ray.get()`` is called, objects must be transferred to the worker/node that calls ``ray.get()``. If you don't need to manipulate the object, you probably don't need to call ``ray.get()`` on it!
+
+Typically, it’s best practice to wait as long as possible before calling ``ray.get()``, or even design your program to avoid having to call ``ray.get()`` at all.
+
+Code example
+------------
+
+**Anti-pattern:**
+
+.. literalinclude:: ../doc_code/anti_pattern_unnecessary_ray_get.py
+    :language: python
+    :start-after: __anti_pattern_start__
+    :end-before: __anti_pattern_end__
+
+.. figure:: ../images/unnecessary-ray-get-anti.svg
+
+**Better approach:**
+
+.. literalinclude:: ../doc_code/anti_pattern_unnecessary_ray_get.py
+    :language: python
+    :start-after: __better_approach_start__
+    :end-before: __better_approach_end__
+
+.. figure:: ../images/unnecessary-ray-get-better.svg
+
+Notice in the anti-pattern example, we call ``ray.get()`` which forces us to transfer the large rollout to the driver, then again to the *reduce* worker.
+
+In the fixed version, we only pass the reference to the object to the *reduce* task.
+The ``reduce`` worker will implicitly call ``ray.get()`` to fetch the actual rollout data directly from the ``generate_rollout`` worker, avoiding the extra copy to the driver.
+
+Other ``ray.get()`` related anti-patterns are:
+
+- :doc:`ray-get-loop`
+- :doc:`ray-get-submission-order`