chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,43 @@
|
||||
Compiled Graph API
|
||||
==================
|
||||
|
||||
Input and Output Nodes
|
||||
----------------------
|
||||
|
||||
.. autosummary::
|
||||
:nosignatures:
|
||||
:toctree: doc/
|
||||
|
||||
ray.dag.input_node.InputNode
|
||||
ray.dag.output_node.MultiOutputNode
|
||||
|
||||
DAG Construction
|
||||
----------------
|
||||
|
||||
.. autosummary::
|
||||
:nosignatures:
|
||||
:toctree: doc/
|
||||
|
||||
ray.actor.ActorMethod.bind
|
||||
ray.dag.DAGNode.with_tensor_transport
|
||||
ray.experimental.compiled_dag_ref.CompiledDAGRef
|
||||
|
||||
Compiled Graph Operations
|
||||
-------------------------
|
||||
|
||||
.. autosummary::
|
||||
:nosignatures:
|
||||
:toctree: doc/
|
||||
|
||||
ray.dag.DAGNode.experimental_compile
|
||||
ray.dag.compiled_dag_node.CompiledDAG.execute
|
||||
ray.dag.compiled_dag_node.CompiledDAG.visualize
|
||||
|
||||
Configurations
|
||||
--------------
|
||||
|
||||
.. autosummary::
|
||||
:nosignatures:
|
||||
:toctree: doc/
|
||||
|
||||
ray.dag.context.DAGContext
|
||||
@@ -0,0 +1,25 @@
|
||||
.. _compiled-graph-overlap:
|
||||
|
||||
Experimental: Overlapping communication and computation
|
||||
=======================================================
|
||||
|
||||
Compiled Graph currently provides experimental support for GPU communication and computation overlap. When you turn this feature on, it automatically overlaps the GPU communication with computation operations, thereby hiding the communication overhead and improving performance.
|
||||
|
||||
To enable this feature, specify ``_overlap_gpu_communication=True`` when calling :func:`dag.experimental_compile() <ray.dag.DAGNode.experimental_compile>`.
|
||||
|
||||
The following code has GPU communication and computation operations that benefit
|
||||
from overlapping.
|
||||
|
||||
.. literalinclude:: ../doc_code/cgraph_overlap.py
|
||||
:language: python
|
||||
:start-after: __cgraph_overlap_start__
|
||||
:end-before: __cgraph_overlap_end__
|
||||
|
||||
The output of the preceding code includes the following two lines:
|
||||
|
||||
.. testoutput::
|
||||
|
||||
overlap_gpu_communication=False, duration=1.0670117866247892
|
||||
overlap_gpu_communication=True, duration=0.9211348341777921
|
||||
|
||||
The actual performance numbers may vary on different hardware, but enabling ``_overlap_gpu_communication`` improves latency by about 14% for this example.
|
||||
@@ -0,0 +1,87 @@
|
||||
Profiling
|
||||
=========
|
||||
|
||||
Ray Compiled Graph provides both PyTorch-based and Nsight-based profiling functionalities to better understand the performance
|
||||
of individual tasks, system overhead, and performance bottlenecks. You can pick your favorite profiler based on your preference.
|
||||
|
||||
PyTorch profiler
|
||||
----------------
|
||||
|
||||
To run PyTorch Profiling on Compiled Graph, simply set the environment variable ``RAY_CGRAPH_ENABLE_TORCH_PROFILING=1``
|
||||
when running the script. For example, for a Compiled Graph script in ``example.py``, run the following command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
RAY_CGRAPH_ENABLE_TORCH_PROFILING=1 python3 example.py
|
||||
|
||||
After execution, Compiled Graph generates the profiling results in the `compiled_graph_torch_profiles` directory
|
||||
under the current working directory. Compiled Graph generates one trace file per actor.
|
||||
|
||||
You can visualize traces by using https://ui.perfetto.dev/.
|
||||
|
||||
|
||||
Nsight system profiler
|
||||
----------------------
|
||||
|
||||
Compiled Graph builds on top of Ray's profiling capabilities, and leverages Nsight
|
||||
system profiling.
|
||||
|
||||
To run Nsight Profiling on Compiled Graph, specify the runtime_env for the involved actors
|
||||
as described in :ref:`Run Nsight on Ray <run-nsight-on-ray>`. For example,
|
||||
|
||||
.. literalinclude:: ../doc_code/cgraph_profiling.py
|
||||
:language: python
|
||||
:start-after: __profiling_setup_start__
|
||||
:end-before: __profiling_setup_end__
|
||||
|
||||
Then, create a Compiled Graph as usual.
|
||||
|
||||
.. literalinclude:: ../doc_code/cgraph_profiling.py
|
||||
:language: python
|
||||
:start-after: __profiling_execution_start__
|
||||
:end-before: __profiling_execution_end__
|
||||
|
||||
Finally, run the script as usual.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python3 example.py
|
||||
|
||||
After execution, Compiled Graph generates the profiling results under the `/tmp/ray/session_*/logs/{profiler_name}`
|
||||
directory.
|
||||
|
||||
For fine-grained performance analysis of method calls and system overhead, set the environment variable
|
||||
``RAY_CGRAPH_ENABLE_NVTX_PROFILING=1`` when running the script:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
RAY_CGRAPH_ENABLE_NVTX_PROFILING=1 python3 example.py
|
||||
|
||||
|
||||
This command leverages the `NVTX library <https://nvtx.readthedocs.io/en/latest/index.html#>`_ under the hood to automatically
|
||||
annotate all methods called in the execution loops of compiled graph.
|
||||
|
||||
To visualize the profiling results, follow the same instructions as described in
|
||||
:ref:`Nsight Profiling Result <profiling-result>`.
|
||||
|
||||
Visualization
|
||||
-------------
|
||||
To visualize the graph structure, call the :func:`visualize <ray.dag.compiled_dag_node.CompiledDAG.visualize>` method after calling :func:`experimental_compile <ray.dag.DAGNode.experimental_compile>`
|
||||
on the graph.
|
||||
|
||||
.. literalinclude:: ../doc_code/cgraph_visualize.py
|
||||
:language: python
|
||||
:start-after: __cgraph_visualize_start__
|
||||
:end-before: __cgraph_visualize_end__
|
||||
|
||||
By default, Ray generates a PNG image named ``compiled_graph.png`` and saves it in the current working directory.
|
||||
Note that this requires ``graphviz``.
|
||||
|
||||
The following image shows the visualization for the preceding code.
|
||||
Tasks that belong to the same actor are the same color.
|
||||
|
||||
.. image:: ../../images/compiled_graph_viz.png
|
||||
:alt: Visualization of Graph Structure
|
||||
:align: center
|
||||
|
||||
|
||||
@@ -0,0 +1,260 @@
|
||||
Quickstart
|
||||
==========
|
||||
|
||||
Hello World
|
||||
-----------
|
||||
This "hello world" example uses Ray Compiled Graph. First, install Ray.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pip install "ray[cgraph]"
|
||||
|
||||
# For a ray version before 2.41, use the following instead:
|
||||
# pip install "ray[adag]"
|
||||
|
||||
|
||||
First, define a simple actor that echoes its argument.
|
||||
|
||||
.. literalinclude:: ../doc_code/cgraph_quickstart.py
|
||||
:language: python
|
||||
:start-after: __simple_actor_start__
|
||||
:end-before: __simple_actor_end__
|
||||
|
||||
Next instantiate the actor and use the classic Ray Core APIs ``remote`` and ``ray.get`` to execute tasks on the actor.
|
||||
|
||||
.. literalinclude:: ../doc_code/cgraph_quickstart.py
|
||||
:language: python
|
||||
:start-after: __ray_core_usage_start__
|
||||
:end-before: __ray_core_usage_end__
|
||||
|
||||
.. code-block::
|
||||
|
||||
Execution takes 969.0364822745323 us
|
||||
|
||||
Now, create an equivalent program using Ray Compiled Graph.
|
||||
First, define a graph to execute using classic Ray Core, without any compilation.
|
||||
Later, compile this graph, to apply optimizations and prevent further modifications to the graph.
|
||||
|
||||
First, create a :ref:`Ray DAG <ray-dag-guide>` (directed acyclic graph), which is a lazily executed graph of Ray tasks.
|
||||
Note 3 key differences with the classic Ray Core APIs:
|
||||
|
||||
1. Use the :class:`ray.dag.InputNode <ray.dag.input_node.InputNode>` context manager to indicate which inputs to the DAG should be provided at run time.
|
||||
2. Use :func:`bind() <ray.actor.ActorMethod.bind>` instead of :func:`remote() <ray.remote>` to indicate lazily executed Ray tasks.
|
||||
3. Use :func:`execute() <ray.dag.compiled_dag_node.CompiledDAG.execute>` to execute the DAG.
|
||||
|
||||
Here, define a graph and execute it.
|
||||
Note that there is **no** compilation happening here. This uses the same execution backend as the preceding example:
|
||||
|
||||
.. literalinclude:: ../doc_code/cgraph_quickstart.py
|
||||
:language: python
|
||||
:start-after: __dag_usage_start__
|
||||
:end-before: __dag_usage_end__
|
||||
|
||||
Next, compile the ``dag`` using the :func:`experimental_compile <ray.dag.DAGNode.experimental_compile>` API.
|
||||
The graph uses the same APIs for execution:
|
||||
|
||||
.. literalinclude:: ../doc_code/cgraph_quickstart.py
|
||||
:language: python
|
||||
:start-after: __cgraph_usage_start__
|
||||
:end-before: __cgraph_usage_end__
|
||||
|
||||
.. code-block::
|
||||
|
||||
Execution takes 86.72196418046951 us
|
||||
|
||||
The performance of the same task graph improved by 10X. This is because the function ``echo`` is cheap and thus highly affected by
|
||||
the system overhead. Due to various bookkeeping and distributed protocols, the classic Ray Core APIs usually have 1 ms+ system overhead.
|
||||
|
||||
Because the system knows the task graph ahead of time, Ray Compiled Graphs can pre-allocate all necessary
|
||||
resources ahead of time and greatly reduce the system overhead.
|
||||
For example, if the actor ``a`` is on the same node as the driver, Ray Compiled Graphs uses shared memory instead of RPC to transfer data directly between the driver and the actor.
|
||||
|
||||
Currently, the DAG tasks run on a **background thread** of the involved actors.
|
||||
An actor can only participate in one DAG at a time.
|
||||
Normal tasks can still execute on the actors while the actors participate in a Compiled Graph, but these tasks execute on the main thread.
|
||||
|
||||
Once you're done, you can tear down the Compiled Graph by deleting it or explicitly calling ``dag.teardown()``.
|
||||
This allows reuse of the actors in a new Compiled Graph.
|
||||
|
||||
.. literalinclude:: ../doc_code/cgraph_quickstart.py
|
||||
:language: python
|
||||
:start-after: __teardown_start__
|
||||
:end-before: __teardown_end__
|
||||
|
||||
|
||||
Specifying data dependencies
|
||||
----------------------------
|
||||
|
||||
When creating the DAG, a ``ray.dag.DAGNode`` can be passed as an argument to other ``.bind`` calls to specify data dependencies.
|
||||
For example, the following uses the preceding example to create a DAG that passes the same message from one actor to another:
|
||||
|
||||
.. literalinclude:: ../doc_code/cgraph_quickstart.py
|
||||
:language: python
|
||||
:start-after: __cgraph_bind_start__
|
||||
:end-before: __cgraph_bind_end__
|
||||
|
||||
.. code-block::
|
||||
|
||||
hello
|
||||
|
||||
Here is another example that passes the same message to both actors, which can then execute in parallel.
|
||||
It uses :class:`ray.dag.MultiOutputNode <ray.dag.output_node.MultiOutputNode>` to indicate that this DAG returns multiple outputs.
|
||||
Then, :func:`dag.execute() <ray.dag.compiled_dag_node.CompiledDAG.execute>` returns multiple :class:`CompiledDAGRef <ray.experimental.compiled_dag_ref.CompiledDAGRef>` objects, one per node:
|
||||
|
||||
.. literalinclude:: ../doc_code/cgraph_quickstart.py
|
||||
:language: python
|
||||
:start-after: __cgraph_multi_output_start__
|
||||
:end-before: __cgraph_multi_output_end__
|
||||
|
||||
.. code-block::
|
||||
|
||||
Execution takes 86.72196418046951 us
|
||||
|
||||
|
||||
Be aware that:
|
||||
* On the same actor, a Compiled Graph executes in order. If an actor has multiple tasks in the same Compiled Graph, it executes all of them to completion before executing on the next DAG input.
|
||||
* Across actors in the same Compiled Graph, the execution may be pipelined. An actor may begin executing on the next DAG input while a downstream actor executes on the current one.
|
||||
* Compiled Graphs currently only supports actor tasks. Non-actor tasks aren't supported.
|
||||
|
||||
``asyncio`` support
|
||||
-------------------
|
||||
|
||||
If your Compiled Graph driver is running in an ``asyncio`` event loop, use the ``async`` APIs to ensure that executing
|
||||
the Compiled Graph and getting the results doesn't block the event loop.
|
||||
First, pass ``enable_async=True`` to the ``dag.experimental_compile()``:
|
||||
|
||||
.. literalinclude:: ../doc_code/cgraph_quickstart.py
|
||||
:language: python
|
||||
:start-after: __cgraph_async_compile_start__
|
||||
:end-before: __cgraph_async_compile_end__
|
||||
|
||||
Next, use `execute_async` to invoke the Compiled Graph. Calling ``await`` on ``execute_async`` will return once
|
||||
the input has been submitted, and it returns a future that can be used to get the result. Finally,
|
||||
use `await` to get the result of the Compiled Graph.
|
||||
|
||||
.. literalinclude:: ../doc_code/cgraph_quickstart.py
|
||||
:language: python
|
||||
:start-after: __cgraph_async_execute_start__
|
||||
:end-before: __cgraph_async_execute_end__
|
||||
|
||||
|
||||
Execution and failure semantics
|
||||
-------------------------------
|
||||
Like classic Ray Core, Ray Compiled Graph propagates exceptions to the final output.
|
||||
In particular:
|
||||
|
||||
- **Application exceptions**: If an application task throws an exception, Compiled Graph
|
||||
wraps the exception in a :class:`RayTaskError <ray.exceptions.RayTaskError>` and
|
||||
raises it when the caller calls :func:`ray.get() <ray.get>` on the result. The thrown
|
||||
exception inherits from both :class:`RayTaskError <ray.exceptions.RayTaskError>`
|
||||
and the original exception class.
|
||||
|
||||
- **System exceptions**: System exceptions include actor death or unexpected errors
|
||||
such as network errors. For actor death, Compiled Graph raises a
|
||||
:class:`ActorDiedError <ray.exceptions.ActorDiedError>`, and for other errors, it
|
||||
raises a :class:`RayChannelError <ray.exceptions.RayChannelError>`.
|
||||
|
||||
The graph can still execute after application exceptions. However, the graph
|
||||
automatically shuts down in the case of system exceptions. If an actor's death causes
|
||||
the graph to shut down, the remaining actors stay alive.
|
||||
|
||||
For example, this example explicitly destroys an actor while it's participating in a Compiled Graph.
|
||||
The remaining actors are reusable:
|
||||
|
||||
.. literalinclude:: ../doc_code/cgraph_quickstart.py
|
||||
:language: python
|
||||
:start-after: __cgraph_actor_death_start__
|
||||
:end-before: __cgraph_actor_death_end__
|
||||
|
||||
|
||||
Execution Timeouts
|
||||
------------------
|
||||
|
||||
Some errors, such as NCCL network errors, require additional handling to avoid hanging.
|
||||
In the future, Ray may attempt to detect such errors, but currently as a fallback, it allows
|
||||
configurable timeouts for
|
||||
:func:`compiled_dag.execute() <ray.dag.compiled_dag_node.CompiledDAG.execute>` and :func:`ray.get() <ray.get>`.
|
||||
|
||||
The default timeout is 10 seconds for both. Set the following environment variables
|
||||
to change the default timeout:
|
||||
|
||||
- ``RAY_CGRAPH_submit_timeout``: Timeout for :func:`compiled_dag.execute() <ray.dag.compiled_dag_node.CompiledDAG.execute>`.
|
||||
- ``RAY_CGRAPH_get_timeout``: Timeout for :func:`ray.get() <ray.get>`.
|
||||
|
||||
:func:`ray.get() <ray.get>` also has a timeout parameter to set timeout on a per-call basis.
|
||||
|
||||
CPU to GPU communication
|
||||
------------------------
|
||||
|
||||
With classic Ray Core, passing ``torch.Tensors`` between actors can become expensive, especially
|
||||
when transferring between devices. This is because Ray Core doesn't know the final destination device.
|
||||
Therefore, you may see unnecessary copies across devices other than the source and destination devices.
|
||||
|
||||
Ray Compiled Graph ships with native support for passing ``torch.Tensors`` between actors executing on different
|
||||
devices. Developers can now use type hint annotations in the Compiled Graph declaration to indicate the final
|
||||
destination device of a ``torch.Tensor``.
|
||||
|
||||
.. literalinclude:: ../doc_code/cgraph_nccl.py
|
||||
:language: python
|
||||
:start-after: __cgraph_cpu_to_gpu_actor_start__
|
||||
:end-before: __cgraph_cpu_to_gpu_actor_end__
|
||||
|
||||
In Ray Core, if you try to pass a CPU tensor from the driver,
|
||||
the GPU actor receives a CPU tensor:
|
||||
|
||||
.. testcode::
|
||||
:skipif: True
|
||||
|
||||
# This will fail because the driver passes a CPU copy of the tensor,
|
||||
# and the GPU actor also receives a CPU copy.
|
||||
ray.get(actor.process.remote(torch.zeros(10)))
|
||||
|
||||
With Ray Compiled Graph, you can annotate DAG nodes with type hints to indicate that there may be a ``torch.Tensor``
|
||||
contained in the value:
|
||||
|
||||
.. literalinclude:: ../doc_code/cgraph_nccl.py
|
||||
:language: python
|
||||
:start-after: __cgraph_cpu_to_gpu_start__
|
||||
:end-before: __cgraph_cpu_to_gpu_end__
|
||||
|
||||
Under the hood, the Ray Compiled Graph backend copies the ``torch.Tensor`` to the GPU assigned to the ``GPUActor`` by Ray Core.
|
||||
|
||||
Of course, you can also do this yourself, but there are advantages to using Compiled Graph instead:
|
||||
|
||||
- Ray Compiled Graph can minimize the number of data copies made. For example, passing from one CPU to
|
||||
multiple GPUs requires one copy to a shared memory buffer, and then one host-to-device copy per
|
||||
destination GPU.
|
||||
|
||||
- In the future, this can be further optimized through techniques such as
|
||||
`memory pinning <https://pytorch.org/docs/stable/generated/torch.Tensor.pin_memory.html>`_,
|
||||
using zero-copy deserialization when the CPU is the destination, etc.
|
||||
|
||||
|
||||
GPU to GPU communication
|
||||
------------------------
|
||||
Ray Compiled Graphs supports NCCL-based transfers of CUDA ``torch.Tensor`` objects, avoiding any copies through Ray's CPU-based shared-memory object store.
|
||||
With user-provided type hints, Ray prepares NCCL communicators and
|
||||
operation scheduling ahead of time, avoiding deadlock and :ref:`overlapping compute and communication <compiled-graph-overlap>`.
|
||||
|
||||
Ray Compiled Graph uses `cupy <https://cupy.dev/>`_ under the hood to support NCCL operations.
|
||||
The cupy version affects the NCCL version. The Ray team is also planning to support custom communicators in the future, for example to support collectives across CPUs or to reuse existing collective groups.
|
||||
|
||||
First, create sender and receiver actors. Note that this example requires at least 2 GPUs.
|
||||
|
||||
.. literalinclude:: ../doc_code/cgraph_nccl.py
|
||||
:language: python
|
||||
:start-after: __cgraph_nccl_setup_start__
|
||||
:end-before: __cgraph_nccl_setup_end__
|
||||
|
||||
To support GPU-to-GPU communication with NCCL, wrap the DAG node that contains the ``torch.Tensor`` that you want to transmit using the ``with_tensor_transport`` API hint:
|
||||
|
||||
.. literalinclude:: ../doc_code/cgraph_nccl.py
|
||||
:language: python
|
||||
:start-after: __cgraph_nccl_exec_start__
|
||||
:end-before: __cgraph_nccl_exec_end__
|
||||
|
||||
Current limitations include:
|
||||
|
||||
* ``torch.Tensor`` and NVIDIA NCCL only
|
||||
* Support for peer-to-peer transfers. Collective communication operations are coming soon.
|
||||
* Communication operations are currently done synchronously. :ref:`Overlapping compute and communication <compiled-graph-overlap>` is an experimental feature.
|
||||
@@ -0,0 +1,84 @@
|
||||
.. _ray-compiled-graph:
|
||||
|
||||
Ray Compiled Graph (beta)
|
||||
=========================
|
||||
|
||||
.. warning::
|
||||
|
||||
Ray Compiled Graph is currently in beta (since Ray 2.44). The APIs are subject to change and expected to evolve.
|
||||
The API is available from Ray 2.32, but it's recommended to use a version after 2.44.
|
||||
|
||||
As large language models (LLMs) become common, programming distributed systems with multiple GPUs is essential.
|
||||
:ref:`Ray Core APIs <core-key-concepts>` facilitate using multiple GPUs but have limitations such as:
|
||||
|
||||
* System overhead of ~1 ms per task launch, which is unsuitable for high-performance tasks like LLM inference.
|
||||
* Lack of support for direct GPU-to-GPU communication, requiring manual development with external libraries like NVIDIA Collective Communications Library (`NCCL <https://developer.nvidia.com/nccl>`_).
|
||||
|
||||
Ray Compiled Graph gives you a Ray Core-like API but with:
|
||||
|
||||
- **Less than 50us system overhead** for workloads that repeatedly execute the same task graph.
|
||||
- **Native support for GPU-GPU communication** with NCCL.
|
||||
|
||||
For example, consider the following Ray Core code, which sends data to an actor
|
||||
and gets the result:
|
||||
|
||||
.. testcode::
|
||||
:skipif: True
|
||||
|
||||
# Ray Core API for remote execution.
|
||||
# ~1ms overhead to invoke `recv`.
|
||||
ref = receiver.recv.remote(data)
|
||||
ray.get(ref)
|
||||
|
||||
|
||||
This code shows how to compile and execute the same example as a Compiled Graph.
|
||||
|
||||
.. testcode::
|
||||
:skipif: True
|
||||
|
||||
# Compiled Graph for remote execution.
|
||||
# less than 50us overhead to invoke `recv` (during `graph.execute(data)`).
|
||||
with InputNode() as inp:
|
||||
graph = receiver.recv.bind(inp)
|
||||
|
||||
graph = graph.experimental_compile()
|
||||
ref = graph.execute(data)
|
||||
ray.get(ref)
|
||||
|
||||
Ray Compiled Graph has a static execution model. It's different from classic Ray APIs, which are eager. Because
|
||||
of the static nature, Ray Compiled Graph can perform various optimizations such as:
|
||||
|
||||
- Pre-allocate resources so that it can reduce system overhead.
|
||||
- Prepare NCCL communicators and apply deadlock-free scheduling.
|
||||
- (experimental) Automatically overlap GPU compute and communication.
|
||||
- Improve multi-node performance.
|
||||
|
||||
Use Cases
|
||||
---------
|
||||
Ray Compiled Graph APIs simplify development of high-performance multi-GPU workloads such as LLM inference or distributed training that require:
|
||||
|
||||
- Sub-millisecond level task orchestration.
|
||||
- Direct GPU-GPU peer-to-peer or collective communication.
|
||||
- `Heterogeneous <https://www.youtube.com/watch?v=Mg08QTBILWU>`_ or MPMD (Multiple Program Multiple Data) execution.
|
||||
|
||||
More Resources
|
||||
--------------
|
||||
|
||||
- `Ray Compiled Graph blog <https://www.anyscale.com/blog/announcing-compiled-graphs>`_
|
||||
- `Ray Compiled Graph talk at Ray Summit <https://www.youtube.com/watch?v=jv58Cpr6SAs>`_
|
||||
- `Heterogeneous training with Ray Compiled Graph <https://www.youtube.com/watch?v=Mg08QTBILWU>`_
|
||||
- `Distributed LLM inference with Ray Compiled Graph <https://www.youtube.com/watch?v=oMb_WiUwf5o>`_
|
||||
|
||||
Table of Contents
|
||||
-----------------
|
||||
|
||||
Learn more details about Ray Compiled Graph from the following links.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
quickstart
|
||||
profiling
|
||||
overlap
|
||||
troubleshooting
|
||||
compiled-graph-api
|
||||
@@ -0,0 +1,99 @@
|
||||
Troubleshooting
|
||||
===============
|
||||
|
||||
This page contains common issues and solutions for Compiled Graph execution.
|
||||
|
||||
Limitations
|
||||
-----------
|
||||
|
||||
Compiled Graph is a new feature and has some limitations:
|
||||
|
||||
- Invoking Compiled Graph
|
||||
|
||||
- Only the process that compiles the Compiled Graph may call it.
|
||||
|
||||
- A Compiled Graph has a maximum number of in-flight executions. When using the DAG API,
|
||||
if there aren't enough resources at the time of ``dag.execute()``, Ray will queue the
|
||||
tasks for later execution. Ray Compiled Graph currently doesn't support queuing past its
|
||||
maximum capacity. Therefore, you may need to consume some results using ``ray.get()``
|
||||
before submitting more executions. As a stopgap,
|
||||
``dag.execute()`` throws a ``RayCgraphCapacityExceeded`` exception if the call takes too long.
|
||||
In the future, Compiled Graph may have better error handling and queuing.
|
||||
|
||||
- Compiled Graph Execution
|
||||
|
||||
- Ideally, you should try not to execute other tasks on the actor while it is participating in a Compiled Graph.
|
||||
Compiled Graph tasks will be executed on a **background thread**. Any concurrent tasks
|
||||
submitted to the actor can still execute on the main thread, but you are responsible for
|
||||
synchronization with the Compiled Graph background thread.
|
||||
|
||||
- For now, actors can only execute one Compiled Graph at a time. To execute a different Compiled Graph
|
||||
on the same actor, you must teardown the current Compiled Graph.
|
||||
See :ref:`Return NumPy arrays <troubleshoot-numpy>` for more details.
|
||||
|
||||
- Passing and getting Compiled Graph results (:class:`CompiledDAGRef <ray.experimental.compiled_dag_ref.CompiledDAGRef>`)
|
||||
|
||||
- Compiled Graph results can't be passed to another task or actor. This restriction may be loosened
|
||||
in the future, but for now, it allows for better performance because the backend knows
|
||||
exactly where to push the results.
|
||||
|
||||
- ``ray.get()`` can be called at most once on a :class:`CompiledDAGRef <ray.experimental.compiled_dag_ref.CompiledDAGRef>`. An exception will be raised if
|
||||
it is called twice on the same :class:`CompiledDAGRef <ray.experimental.compiled_dag_ref.CompiledDAGRef>`. This is because the underlying memory for
|
||||
the result may need to be reused for a future DAG execution. Restricting ``ray.get()`` to once
|
||||
per reference simplifies the tracking of the memory buffers.
|
||||
|
||||
- If the value returned by ``ray.get()`` is zero-copy deserialized, then subsequent executions
|
||||
of the same DAG will block until the value goes out of scope in Python. Thus, if you hold onto
|
||||
zero-copy deserialized values returned by ``ray.get()``, and you try to execute the Compiled Graph above
|
||||
its max concurrency, it may deadlock. This case will be detected in the future, but for now
|
||||
you will receive a ``RayChannelTimeoutError``.
|
||||
See :ref:`Explicitly teardown before reusing the same actors <troubleshoot-teardown>`
|
||||
for more details.
|
||||
|
||||
- Collective operations
|
||||
|
||||
- For GPU to GPU communication, Compiled Graph only supports peer-to-peer transfers. Collective communication operations are coming soon.
|
||||
|
||||
Keep an eye out for additional features in future Ray releases:
|
||||
|
||||
- Support better queuing of DAG inputs, to enable more concurrent executions of the same DAG.
|
||||
- Support for more collective operations with NCCL.
|
||||
- Support for multiple DAGs executing on the same actor.
|
||||
- General performance improvements.
|
||||
|
||||
If you run into additional issues, or have other feedback or questions, file an issue
|
||||
on `GitHub <https://github.com/ray-project/ray/issues>`_.
|
||||
For a full list of known issues, check the ``compiled-graphs`` label on Ray GitHub.
|
||||
|
||||
.. _troubleshoot-numpy:
|
||||
|
||||
Returning NumPy arrays
|
||||
----------------------
|
||||
Ray zero-copy deserializes NumPy arrays when possible. If you execute compiled graph with a NumPy array output multiple times,
|
||||
you could possibly run into issues if a NumPy array output from a previous Compiled Graph execution isn't deleted before attempting to get the result
|
||||
of a following execution of the same Compiled Graph. This is because the NumPy array stays in the buffer of the Compiled Graph until you or Python delete it.
|
||||
It's recommended to explicitly delete the NumPy array as Python may not always garbage collect the NumPy array immediately as you may expect.
|
||||
|
||||
For example, the following code sample could result in a hang or RayChannelTimeoutError if the NumPy array isn't deleted:
|
||||
|
||||
.. literalinclude:: ../doc_code/cgraph_troubleshooting.py
|
||||
:language: python
|
||||
:start-after: __numpy_troubleshooting_start__
|
||||
:end-before: __numpy_troubleshooting_end__
|
||||
|
||||
In the preceding code snippet, Python may not garbage collect the NumPy array in `result` on each iteration of the loop.
|
||||
Therefore, you should explicitly delete the NumPy array before you try to get the result of subsequent Compiled Graph executions.
|
||||
|
||||
.. _troubleshoot-teardown:
|
||||
|
||||
Explicitly teardown before reusing the same actors
|
||||
--------------------------------------------------
|
||||
If you want to reuse the actors of a Compiled Graph, it's important to explicitly teardown the Compiled Graph before reusing the actors.
|
||||
Without explicitly tearing down the Compiled Graph, the resources created for actors in a Compiled Graph may have conflicts with further usage of those actors.
|
||||
|
||||
For example, in the following code, Python could delay garbage collection, which triggers the implicit teardown of the first Compiled Graph. This could lead to a segfault due to the resource conflicts mentioned:
|
||||
|
||||
.. literalinclude:: ../doc_code/cgraph_troubleshooting.py
|
||||
:language: python
|
||||
:start-after: __teardown_troubleshooting_start__
|
||||
:end-before: __teardown_troubleshooting_end__
|
||||
Reference in New Issue
Block a user