chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,122 @@
|
||||
.. _observability-debug-failures:
|
||||
|
||||
Debugging Failures
|
||||
==================
|
||||
|
||||
What Kind of Failures Exist in Ray?
|
||||
-----------------------------------
|
||||
|
||||
Ray consists of two major APIs. ``.remote()`` to create a Task or Actor, and :func:`ray.get <ray.get>` to get the result.
|
||||
Debugging Ray means identifying and fixing failures from remote processes that run functions and classes (Tasks and Actors) created by the ``.remote`` API.
|
||||
|
||||
Ray APIs are future APIs (indeed, it is :ref:`possible to convert Ray object references to standard Python future APIs <async-ref-to-futures>`),
|
||||
and the error handling model is the same. When any remote Tasks or Actors fail, the returned object ref contains an exception.
|
||||
When you call ``get`` API to the object ref, it raises an exception.
|
||||
|
||||
.. testcode::
|
||||
|
||||
import ray
|
||||
@ray.remote
|
||||
def f():
|
||||
raise ValueError("it's an application error")
|
||||
|
||||
# Raises a ValueError.
|
||||
try:
|
||||
ray.get(f.remote())
|
||||
except ValueError as e:
|
||||
print(e)
|
||||
|
||||
.. testoutput::
|
||||
|
||||
...
|
||||
ValueError: it's an application error
|
||||
|
||||
In Ray, there are three types of failures. See exception APIs for more details.
|
||||
|
||||
- **Application failures**: This means the remote task/actor fails by the user code. In this case, ``get`` API will raise the :func:`RayTaskError <ray.exceptions.RayTaskError>` which includes the exception raised from the remote process.
|
||||
- **Intentional system failures**: This means Ray is failed, but the failure is intended. For example, when you call cancellation APIs like ``ray.cancel`` (for task) or ``ray.kill`` (for actors), the system fails remote tasks and actors, but it is intentional.
|
||||
- **Unintended system failures**: This means the remote tasks and actors failed due to unexpected system failures such as processes crashing (for example, by out-of-memory error) or nodes failing.
|
||||
|
||||
1. `Linux Out of Memory killer <https://www.kernel.org/doc/gorman/html/understand/understand016.html>`_ or :ref:`Ray Memory Monitor <ray-oom-monitor>` kills processes with high memory usages to avoid out-of-memory.
|
||||
2. The machine shuts down (e.g., spot instance termination) or a :term:`raylet <raylet>` crashed (e.g., by an unexpected failure).
|
||||
3. System is highly overloaded or stressed (either machine or system components like Raylet or :term:`GCS <GCS / Global Control Service>`), which makes the system unstable and fail.
|
||||
|
||||
Debugging Application Failures
|
||||
------------------------------
|
||||
|
||||
Ray distributes users' code to multiple processes across many machines. Application failures mean bugs in users' code.
|
||||
Ray provides a debugging experience that's similar to debugging a single-process Python program.
|
||||
|
||||
print
|
||||
~~~~~
|
||||
|
||||
``print`` debugging is one of the most common ways to debug Python programs.
|
||||
:ref:`Ray's Task and Actor logs are printed to the Ray Driver <ray-worker-logs>` by default,
|
||||
which allows you to simply use the ``print`` function to debug the application failures.
|
||||
|
||||
Debugger
|
||||
~~~~~~~~
|
||||
|
||||
Many Python developers use a debugger to debug Python programs, and `Python pdb <https://docs.python.org/3/library/pdb.html>`_) is one of the popular choices.
|
||||
Ray has native integration to ``pdb``. You can simply add ``breakpoint()`` to Actors and Tasks code to enable ``pdb``. View :ref:`Ray Debugger <ray-debugger>` for more details.
|
||||
|
||||
|
||||
Running out of file descriptors (``Too many open files``)
|
||||
---------------------------------------------------------
|
||||
|
||||
In a Ray cluster, arbitrary two system components can communicate with each other and make 1 or more connections.
|
||||
For example, some workers may need to communicate with GCS to schedule Actors (worker <-> GCS connection).
|
||||
Your Driver can invoke Actor methods (worker <-> worker connection).
|
||||
|
||||
Ray can support 1000s of raylets and 10000s of worker processes. When a Ray cluster gets larger,
|
||||
each component can have an increasing number of network connections, which requires file descriptors.
|
||||
|
||||
Linux typically limits the default file descriptors per process to 1024. When there are
|
||||
more than 1024 connections to the component, it can raise error messages below.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
Too many open files
|
||||
|
||||
It is especially common for the head node GCS process because it is a centralized
|
||||
component that many other components in Ray communicate with. When you see this error message,
|
||||
we recommend you adjust the max file descriptors limit per process via the ``ulimit`` command.
|
||||
|
||||
We recommend you apply ``ulimit -n 65536`` to your host configuration. However, you can also selectively apply it for
|
||||
Ray components (view below example). Normally, each worker has 2~3 connections to GCS. Each raylet has 1~2 connections to GCS.
|
||||
65536 file descriptors can handle 10000~15000 of workers and 1000~2000 of nodes.
|
||||
If you have more workers, you should consider using a higher number than 65536.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Start head node components with higher ulimit.
|
||||
ulimit -n 65536 ray start --head
|
||||
|
||||
# Start worker node components with higher ulimit.
|
||||
ulimit -n 65536 ray start --address <head_node>
|
||||
|
||||
# Start a Ray driver with higher ulimit.
|
||||
ulimit -n 65536 <python script>
|
||||
|
||||
If that fails, double-check that the hard limit is sufficiently large by running ``ulimit -Hn``.
|
||||
If it is too small, you can increase the hard limit as follows (these instructions work on EC2).
|
||||
|
||||
* Increase the hard ulimit for open file descriptors system-wide by running
|
||||
the following.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo bash -c "echo $USER hard nofile 65536 >> /etc/security/limits.conf"
|
||||
|
||||
* Logout and log back in.
|
||||
|
||||
|
||||
Failures due to memory issues
|
||||
--------------------------------
|
||||
View :ref:`debugging memory issues <ray-core-mem-profiling>` for more details.
|
||||
|
||||
|
||||
This document discusses some common problems that people run into when using Ray
|
||||
as well as some known problems. If you encounter other problems, `let us know`_.
|
||||
|
||||
.. _`let us know`: https://github.com/ray-project/ray/issues
|
||||
@@ -0,0 +1,45 @@
|
||||
.. _observability-debug-hangs:
|
||||
|
||||
Debugging Hangs
|
||||
===============
|
||||
View stack traces in Ray Dashboard
|
||||
-----------------------------------
|
||||
The :ref:`Ray dashboard <observability-getting-started>` lets you profile Ray Driver or Worker processes, by clicking on the "CPU profiling" or "Stack Trace" actions for active Worker processes, Tasks, Actors, and Job's driver process.
|
||||
|
||||
.. image:: /images/profile.png
|
||||
:align: center
|
||||
:width: 80%
|
||||
|
||||
Clicking "Stack Trace" will return the current stack trace sample using ``py-spy``. By default, only the Python stack
|
||||
trace is shown. To show native code frames, set the URL parameter ``native=1`` (only supported on Linux).
|
||||
|
||||
.. image:: /images/stack.png
|
||||
:align: center
|
||||
:width: 60%
|
||||
|
||||
.. note::
|
||||
You may run into permission errors when using py-spy in the docker containers. To fix the issue:
|
||||
|
||||
* if you start Ray manually in a Docker container, follow the `py-spy documentation`_ to resolve it.
|
||||
* if you are a KubeRay user, follow the :ref:`guide to configure KubeRay <kuberay-pyspy-integration>` and resolve it.
|
||||
|
||||
.. note::
|
||||
The following errors are conditional and not signals of failures for your Python programs:
|
||||
|
||||
* If you see "No such file or direction", check if your worker process has exited.
|
||||
* If you see "No stack counts found", check if your worker process was sleeping and not active in the last 5s.
|
||||
|
||||
.. _`py-spy documentation`: https://github.com/benfred/py-spy#how-do-i-run-py-spy-in-docker
|
||||
|
||||
|
||||
Use ``ray stack`` CLI command
|
||||
------------------------------
|
||||
|
||||
Once ``py-spy`` is installed (it is automatically installed if "Ray Dashboard" component is included when :ref:`installing Ray <installation>`), you can run ``ray stack`` to dump the stack traces of all Ray Worker processes on
|
||||
the current node.
|
||||
|
||||
This document discusses some common problems that people run into when using Ray
|
||||
as well as some known problems. If you encounter other problems, please
|
||||
`let us know`_.
|
||||
|
||||
.. _`let us know`: https://github.com/ray-project/ray/issues
|
||||
@@ -0,0 +1,323 @@
|
||||
.. _ray-core-mem-profiling:
|
||||
|
||||
Debugging Memory Issues
|
||||
=======================
|
||||
|
||||
|
||||
.. _troubleshooting-out-of-memory:
|
||||
|
||||
Debugging Out of Memory
|
||||
-----------------------
|
||||
|
||||
Before reading this section, familiarize yourself with the Ray :ref:`Memory Management <memory>` model.
|
||||
|
||||
- If your cluster has out-of-memory problems, view :ref:`How to Detect Out-of-Memory Errors <troubleshooting-out-of-memory-how-to-detect>`.
|
||||
- To locate the source of the memory problems, view :ref:`Find per Task and Actor Memory Usage <troubleshooting-out-of-memory-task-actor-mem-usage>`.
|
||||
- Once you've identified the source, address it at :ref:`Eliminating worker out-of-memory errors <troubleshooting-out-of-memory-eliminate-worker-oom>`.
|
||||
- If your head node has high memory usage, view :ref:`Head Node Out-of-Memory Error <troubleshooting-out-of-memory-head>`.
|
||||
- If your memory usage is high due to high parallelism, view :ref:`Reduce Parallelism <troubleshooting-out-of-memory-reduce-parallelism>`.
|
||||
- If you want to profile per Task and Actor memory usage, view :ref:`Profile Task and Actor Memory Usage <troubleshooting-out-of-memory-profile>`.
|
||||
|
||||
What's the Out-of-Memory Error?
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Memory is a limited resource. When a process requests memory and the OS fails to allocate it, the OS executes a routine to free up memory
|
||||
by killing a process that has high memory usage (via SIGKILL) to avoid the OS becoming unstable. This routine is called the `Linux Out of Memory killer <https://www.kernel.org/doc/gorman/html/understand/understand016.html>`_.
|
||||
|
||||
For Ray, the Linux out-of-memory (OOM) killer kills Ray processes without the control plane noticing. This causes the following problems:
|
||||
|
||||
1. The Linux OOM killer indiscriminately kills processes based on memory footprint.
|
||||
For Ray, this can result in significant loss of progress and, in some scenarios, in the death of critical
|
||||
Ray components, which leads to node deaths.
|
||||
2. The Linux OOM killer uses SIGKILL to kill processes. Because processes can't handle SIGKILL,
|
||||
Ray has difficulty raising a proper error message and taking proper actions for fault tolerance.
|
||||
|
||||
To solve this problem, Ray has (from Ray 2.2) an application-level :ref:`memory monitor <ray-oom-monitor>`,
|
||||
which continually monitors the memory usage of the host and kills the Ray Workers before the Linux out-of-memory killer executes.
|
||||
|
||||
.. _troubleshooting-out-of-memory-how-to-detect:
|
||||
|
||||
Detecting Out-of-Memory errors
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can monitor out-of-memory errors on the Ray Dashboard's :ref:`metrics page <dash-metrics-view>` via the Ray OOM Kills panel and the Unexpected System Level Worker Failures panel.
|
||||
The Ray OOM Kills panel shows the number of workers killed by the Ray OOM killer.
|
||||
The Unexpected System Level Worker Failures panel shows the number of workers that failed unexpectedly. These failures are typically
|
||||
caused by the Linux out-of-memory killer; correlate with memory usage metrics to confirm.
|
||||
|
||||
.. image:: ../../images/ray-oom-kills.png
|
||||
:align: center
|
||||
|
||||
.. image:: ../../images/unexpected-system-level-worker-failures.png
|
||||
:align: center
|
||||
|
||||
|
||||
If the Linux out-of-memory killer terminates Tasks or Actors, Ray Worker processes are unable to catch and display an exact root cause
|
||||
because SIGKILL cannot be handled by processes. If you call ``ray.get`` into the Tasks and Actors that were executed from the dead worker,
|
||||
it raises an exception with one of the following error messages (which indicates the worker is killed unexpectedly).
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
Worker exit type: UNEXPECTED_SYSTEM_EXIT Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
Worker exit type: SYSTEM_ERROR Worker exit detail: The leased worker has unrecoverable failure. Worker is requested to be destroyed when it is returned.
|
||||
|
||||
You can also use the `dmesg <https://phoenixnap.com/kb/dmesg-linux#:~:text=The%20dmesg%20command%20is%20a,take%20place%20during%20system%20startup.>`_ CLI command to verify the processes are killed by the Linux out-of-memory killer.
|
||||
|
||||
.. image:: ../../images/dmsg.png
|
||||
:align: center
|
||||
|
||||
|
||||
As mentioned above, having the Linux OOM killer trigger before the Ray OOM killer is undesirable.
|
||||
In Ray 2.56 and above, enable resource isolation mode by passing ``--enable-resource-isolation`` when starting Ray
|
||||
to ensure that the Ray OOM killer triggers before the Linux OOM killer.
|
||||
|
||||
It's still possible for Linux OOM kills to occur if the system overhead consumes more memory than
|
||||
what's reserved for it (the default is 10%, with a minimum of 500 MB and a maximum of 10 GB).
|
||||
Ray logs something similar to the following example if it detects this scenario.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
System slice memory usage 10869600256 bytes has exceeded the reserved system memory of 10737418240 bytes. This can prevent Ray from being able to provide the proper protection to critical system processes and can lead to node deaths and significant loss of progress. Please consider passing a system reserved memory value that is higher than the current system slice memory usage via the --system-reserved-memory flag when starting the raylet.
|
||||
|
||||
When you see a kernel OOM or this log message with resource isolation enabled, try increasing the memory reserved for system processes
|
||||
by setting a higher value than the reported system slice memory usage for the ``--system-reserved-memory`` flag when starting Ray.
|
||||
Try to allocate at least a GiB (depending on host size) of buffer space between the reported/expected system slice memory usage and the system reserved memory.
|
||||
|
||||
.. note::
|
||||
|
||||
To enable resource isolation, complete the prerequisite steps in :ref:`How to Enable Cgroup v2 for Resource Isolation <enable-cgroupv2>`.
|
||||
|
||||
If Ray's memory monitor kills the worker, Ray retries it automatically (see the :ref:`retry policy <ray-oom-retry-policy>` for details).
|
||||
Ray's memory monitor also logs the details of the out-of-memory kill to the ``raylet.out`` log file.
|
||||
An example log is shown below.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
Task hungry_hippo failed due to oom. There are infinite oom retries remaining, so the task will be retried. Error: 2 worker(s) were killed due to the node running low on memory. Memory on the node (IP: <ip address>, ID: 92edc4e97e4dac3cee61126133ee7ab6d0a2ee73803623d24a02979d) was 110.69GB / 124.35GB (0.890161)
|
||||
OOM kill reason: user cgroup memory upper bound was met or exceeded
|
||||
Object store memory usage: [- objects spillable: 0
|
||||
- bytes spillable: 0
|
||||
- objects unsealed: 0
|
||||
- bytes unsealed: 0
|
||||
- objects in use: 0
|
||||
- bytes in use: 0
|
||||
- objects evictable: 0
|
||||
- bytes evictable: 0
|
||||
|
||||
- objects created by worker: 0
|
||||
- bytes created by worker: 0
|
||||
- objects restored: 0
|
||||
- bytes restored: 0
|
||||
- objects received: 0
|
||||
- bytes received: 0
|
||||
- objects errored: 0
|
||||
- bytes errored: 0
|
||||
|
||||
Eviction Stats:
|
||||
(global lru) capacity: 35098657996
|
||||
(global lru) used: 0%
|
||||
(global lru) num objects: 0
|
||||
(global lru) num evictions: 0
|
||||
(global lru) bytes evicted: 0]
|
||||
Ray killed 2 worker(s) based on the killing policy
|
||||
Considered workers: [
|
||||
Selected to kill: (Task: job ID=01000000, lease ID=0600000001000000ffffffffffffffffffffffffffffffffffffffffffffffff, task name=hungry_hippo, required resources={CPU: 1}, pid=3310152, actual memory used=0.67GB, worker ID=3e3d8f80b70d48b643d79ed2292b5d4f779820a964e55ad65413687d)
|
||||
Selected to kill: (Task: job ID=01000000, lease ID=0400000001000000ffffffffffffffffffffffffffffffffffffffffffffffff, task name=hungry_hippo, required resources={CPU: 1}, pid=3310153, actual memory used=21.64GB, worker ID=0e5649d39c15609c0db6a5cf95de94befded2ee7da2facbf64b52e6f)
|
||||
(Task: job ID=01000000, lease ID=0500000001000000ffffffffffffffffffffffffffffffffffffffffffffffff, task name=hungry_hippo, required resources={CPU: 1}, pid=3310151, actual memory used=21.95GB, worker ID=34241048bfb59ac29bd5e32d706c9bd41eafc6972c9bcbada99464e7)
|
||||
(Task: job ID=01000000, lease ID=0000000001000000ffffffffffffffffffffffffffffffffffffffffffffffff, task name=hungry_hippo, required resources={CPU: 1}, pid=3310149, actual memory used=21.87GB, worker ID=2cc6dbeef4ebc06789de65fb43e04fbe1feebf1e699902ece89a8328)
|
||||
(Task: job ID=01000000, lease ID=0200000001000000ffffffffffffffffffffffffffffffffffffffffffffffff, task name=hungry_hippo, required resources={CPU: 1}, pid=3310155, actual memory used=21.85GB, worker ID=14d4cc84e2f21ba3edbc9948b780d67013afbffda99dd33e829d56d3)
|
||||
(Task: job ID=01000000, lease ID=0100000001000000ffffffffffffffffffffffffffffffffffffffffffffffff, task name=hungry_hippo, required resources={CPU: 1}, pid=3310147, actual memory used=21.53GB, worker ID=c90db9af23d78530d1f848c1301bc4b877925fe9122bc70948ad9489)]
|
||||
Total non-selected idle workers: 25
|
||||
Total non-selected idle workers USS bytes: 1.00GB
|
||||
To see more information about memory usage on this node, use `ray logs raylet.out -ip <ip address>`
|
||||
Top 10 memory users: PID MEM(GB) COMMAND
|
||||
3310151 21.95 ray::hungry_hippo
|
||||
3310149 21.87 ray::hungry_hippo
|
||||
3310155 21.85 ray::hungry_hippo
|
||||
3310153 21.64 ray::hungry_hippo
|
||||
3310147 21.53 ray::hungry_hippo
|
||||
3108574 1.95 bazel
|
||||
3180337 1.61 ray::foo_actor
|
||||
3310152 0.67 ray::hungry_hippo
|
||||
2924839 0.53 ray::idle_worker
|
||||
3149737 0.47 ray::idle_worker
|
||||
Refer to the documentation on how to address the out of memory issue: https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html. Consider provisioning more memory on this node or reducing task parallelism by requesting more CPUs per task. To adjust the kill threshold, set the environment variable `RAY_memory_usage_threshold` when starting Ray. To disable worker killing, set the environment variable `RAY_memory_monitor_refresh_ms` to zero. Since 2.56, Ray updated the oom killing policy to enabling killing multiple workers and selecting workers based on the time since the task start executing. To revert to the legacy policy of determining worker to oom kill based on owner group size or only selecting a single worker to kill at a time, set the environment variable `RAY_worker_killing_policy_by_group` to true before starting Ray. If the idle workers have a non-trivial memory footprint at the time of OOM (check OOM log for non-selected idle workers), consider setting the environment variable `RAY_idle_worker_killing_memory_threshold_bytes` to a lower value to consider idle workers with lower memory footprint for killing.
|
||||
|
||||
If Tasks or Actors can't be retried, they raise an exception with
|
||||
a similar error message when you call ``ray.get`` on them.
|
||||
|
||||
Ray memory monitor also periodically prints the aggregated out-of-memory killer summary to Ray drivers.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
(raylet) [2023-04-09 07:23:59,445 E 395 395] (raylet) node_manager.cc:3049: 10 Workers (tasks / actors) killed due to memory pressure (OOM), 0 Workers crashed due to other reasons at node (ID: e5d953ef03e55e26f13973ea1b5a0fd0ecc729cd820bc89e4aa50451, IP: 10.0.62.231) over the last time period. To see more information about the Workers killed on this node, use `ray logs raylet.out -ip 10.0.62.231`
|
||||
(raylet)
|
||||
(raylet) Refer to the documentation on how to address the out of memory issue: https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html. Consider provisioning more memory on this node or reducing task parallelism by requesting more CPUs per task. To adjust the kill threshold, set the environment variable `RAY_memory_usage_threshold` when starting Ray. To disable worker killing, set the environment variable `RAY_memory_monitor_refresh_ms` to zero.
|
||||
|
||||
Ray Dashboard's :ref:`event page <dash-event>` also provides the out-of-memory killer-specific events and metrics.
|
||||
|
||||
.. image:: ../../images/oom-events.png
|
||||
:align: center
|
||||
|
||||
.. _troubleshooting-out-of-memory-task-actor-mem-usage:
|
||||
|
||||
Find per Task and Actor Memory Usage
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
If Tasks or Actors fail because of out-of-memory errors, they are retried based on :ref:`retry policies <ray-oom-retry-policy>`.
|
||||
However, it is often preferred to find the root causes of memory issues and fix them instead of relying on fault tolerance mechanisms.
|
||||
This section explains how to debug out-of-memory errors in Ray.
|
||||
|
||||
To view the memory usage of tasks and actors at the time of OOM, see the considered workers in the OOM log above. This information
|
||||
helps identify the tasks and actors responsible for the OOM. The OOM log also includes the memory footprint of all idle workers.
|
||||
See :ref:`ray-oom-worker-killing-policy` for how Ray considers idle workers for killing.
|
||||
|
||||
To view the memory usage of Tasks and Actors based on type over time, view the :ref:`per Task and Actor memory usage graph <dash-workflow-cpu-memory-analysis>` for more details.
|
||||
The memory usage from the per component graph uses RSS - SHR. See below for reasoning.
|
||||
|
||||
Alternatively, you can also use the CLI command `htop <https://htop.dev/>`_.
|
||||
|
||||
.. image:: ../../images/htop.png
|
||||
:align: center
|
||||
|
||||
See the ``allocate_memory`` row. See two columns, RSS and SHR.
|
||||
|
||||
SHR usage is typically the memory usage from the Ray object store. The Ray object store allocates 30% of host memory to the shared memory (``/dev/shm``, unless you specify ``--object-store-memory``).
|
||||
If Ray workers access the object inside the object store using ``ray.get``, SHR usage increases. Since the Ray object store supports the :ref:`zero-copy <serialization-guide>`
|
||||
deserialization, several workers can access the same object without copying them to in-process memory. For example, if
|
||||
8 workers access the same object inside the Ray object store, each process' ``SHR`` usage increases. However, they are not using 8 * SHR memory (there's only 1 copy in the shared memory).
|
||||
Also note that Ray object store triggers :ref:`object spilling <object-spilling>` when the object usage goes beyond the limit, which means the memory usage from the shared memory won't exceed 30%
|
||||
of the host memory.
|
||||
|
||||
Out-of-memory issues from a host, are due to RSS usage from each worker. Calculate per
|
||||
process memory usage by RSS - SHR because SHR is for Ray object store as explained above. The total memory usage is typically
|
||||
``SHR (object store memory usage, 30% of memory) + sum(RSS - SHR from each ray proc) + sum(RSS - SHR from system components. e.g., raylet, GCS. Usually small)``.
|
||||
|
||||
.. _troubleshooting-out-of-memory-eliminate-worker-oom:
|
||||
|
||||
Eliminating worker Out-Of-Memory errors
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Most out-of-memory errors come from oversubscribing memory on a node.
|
||||
By default, tasks and actors have no memory requirements, so the scheduler is unaware
|
||||
of their memory footprint and may schedule too many memory-hungry tasks or actors onto a single node.
|
||||
|
||||
To prevent this oversubscription and eliminate OOM issues, pass a ``memory`` resource request to tasks or actors
|
||||
to reserve memory for them. See :ref:`resource requirements <resource-requirements>` for more details.
|
||||
This request doesn't impose any limit on memory usage; it's used for scheduling only.
|
||||
As shown in the example OOM log above, the log includes the resource request for each active worker. If the worker memory usage
|
||||
exceeds the requested memory at the time of OOM, adjust the resource request accordingly.
|
||||
|
||||
.. _troubleshooting-out-of-memory-head:
|
||||
|
||||
Head node out-of-Memory error
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
First, check the head node memory usage from the metrics page. Find the head node address from the cluster page.
|
||||
|
||||
.. image:: ../../images/head-node-addr.png
|
||||
:align: center
|
||||
|
||||
Then check the memory usage from the head node from the node memory usage view inside the Dashboard :ref:`metrics view <dash-metrics-view>`.
|
||||
|
||||
.. image:: ../../images/metrics-node-view.png
|
||||
:align: center
|
||||
|
||||
The Ray head node has more memory-demanding system components such as GCS or the dashboard.
|
||||
Also, the driver runs from a head node by default. If the head node has the same memory capacity as worker nodes
|
||||
and if you execute the same number of Tasks and Actors from a head node, it can easily have out-of-memory problems.
|
||||
In this case, do not run any Tasks and Actors on the head node by specifying ``--num-cpus=0`` when starting a head node by ``ray start --head``.
|
||||
If you use KubeRay, view :ref:`here <kuberay-num-cpus>`.
|
||||
|
||||
.. _troubleshooting-out-of-memory-reduce-parallelism:
|
||||
|
||||
Reduce Parallelism
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
High parallelism can trigger out-of-memory errors. For example, if
|
||||
you have 8 training workers that perform the data preprocessing -> training.
|
||||
If you load too much data into each worker, the total memory usage (``training worker mem usage * 8``) can exceed the
|
||||
memory capacity.
|
||||
|
||||
Verify the memory usage by looking at the :ref:`per Task and Actor memory usage graph <dash-workflow-cpu-memory-analysis>` and the Task metrics.
|
||||
|
||||
First, see the memory usage of an ``allocate_memory`` task. The total is 18GB.
|
||||
At the same time, verify the 15 concurrent tasks that are running.
|
||||
|
||||
.. image:: ../../images/component-memory.png
|
||||
:align: center
|
||||
|
||||
.. image:: ../../images/tasks-graph.png
|
||||
:align: center
|
||||
|
||||
Each task uses about 18GB / 15 == 1.2 GB. To reduce the parallelism:
|
||||
|
||||
- `Limit the max number of running tasks <https://docs.ray.io/en/latest/ray-core/patterns/limit-running-tasks.html>`_.
|
||||
- Increase the ``num_cpus`` options for :func:`ray.remote`. Modern hardware typically has 4GB of memory per CPU, so you can choose the CPU requirements accordingly. This example specifies 1 CPU per ``allocate_memory`` Task. Doubling the CPU requirements, runs only half(7) of the Tasks at the same time, and memory usage doesn't exceed 9GB.
|
||||
|
||||
.. _troubleshooting-out-of-memory-profile:
|
||||
|
||||
Profiling Task and Actor memory usage
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
It is also possible tasks and actors use more memory than you expect. For example, actors or tasks can have a memory leak or have unnecessary copies.
|
||||
|
||||
View the instructions below to learn how to memory profile individual actors and tasks.
|
||||
|
||||
.. _memray-profiling:
|
||||
|
||||
Memory Profiling Ray tasks and actors
|
||||
--------------------------------------
|
||||
|
||||
To memory profile Ray tasks or actors, use `memray <https://bloomberg.github.io/memray/>`_.
|
||||
Note that you can also use other memory profiling tools if it supports a similar API.
|
||||
|
||||
First, install ``memray``.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pip install memray
|
||||
|
||||
``memray`` supports a Python context manager to enable memory profiling. You can write the ``memray`` profiling file wherever you want.
|
||||
But in this example, we will write them to `/tmp/ray/session_latest/logs` because Ray dashboard allows you to download files inside the log folder.
|
||||
This will allow you to download profiling files from other nodes.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Actors
|
||||
|
||||
.. literalinclude:: ../../doc_code/memray_profiling.py
|
||||
:language: python
|
||||
:start-after: __memray_profiling_start__
|
||||
:end-before: __memray_profiling_end__
|
||||
|
||||
.. tab-item:: Tasks
|
||||
|
||||
Note that tasks have a shorter lifetime, so there could be lots of memory profiling files.
|
||||
|
||||
.. literalinclude:: ../../doc_code/memray_profiling.py
|
||||
:language: python
|
||||
:start-after: __memray_profiling_task_start__
|
||||
:end-before: __memray_profiling_task_end__
|
||||
|
||||
Once the task or actor runs, go to the :ref:`Logs view <dash-logs-view>` of the dashboard. Find and click the log file name.
|
||||
|
||||
.. image:: ../../images/memory-profiling-files.png
|
||||
:align: center
|
||||
|
||||
Click the download button.
|
||||
|
||||
.. image:: ../../images/download-memory-profiling-files.png
|
||||
:align: center
|
||||
|
||||
Now, you have the memory profiling file. Running
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
memray flamegraph <memory profiling bin file>
|
||||
|
||||
And you can see the result of the memory profiling!
|
||||
|
||||
@@ -0,0 +1,210 @@
|
||||
.. _observability-general-debugging:
|
||||
|
||||
Common Issues
|
||||
=======================
|
||||
|
||||
Distributed applications offer great power but also increased complexity.
|
||||
Some of Ray's behaviors may initially surprise users, but these design choices serve important purposes in distributed computing environments.
|
||||
|
||||
This document outlines common issues encountered when running Ray in a cluster, highlighting key differences compared to running Ray locally.
|
||||
|
||||
Environment variables aren't passed from the Driver process to Worker processes
|
||||
---------------------------------------------------------------------------------
|
||||
|
||||
**Issue:** When you set an environment variable on your Driver, it isn't propagated to the Worker processes.
|
||||
|
||||
**Example:** Suppose you have a file ``baz.py`` in the directory where you run Ray, and you execute the following command:
|
||||
|
||||
.. literalinclude:: /ray-observability/doc_code/gotchas.py
|
||||
:language: python
|
||||
:start-after: __env_var_start__
|
||||
:end-before: __env_var_end__
|
||||
|
||||
|
||||
**Expected behavior:** Users may expect that setting environment variables on the Driver sends them to all Worker processes as if running on a single machine, but it doesn't.
|
||||
|
||||
**Fix:** Enable Runtime Environments to explicitly pass environment variables. When you call ``ray.init(runtime_env=...)``, it sends the specified environment variables to the Workers.
|
||||
Alternatively, you can set the environment variables as part of your cluster setup configuration.
|
||||
|
||||
.. literalinclude:: /ray-observability/doc_code/gotchas.py
|
||||
:language: python
|
||||
:start-after: __env_var_fix_start__
|
||||
:end-before: __env_var_fix_end__
|
||||
|
||||
|
||||
Filenames work sometimes and not at other times
|
||||
-----------------------------------------------
|
||||
|
||||
**Issue:** Referencing a file by its name in a Task or Actor may sometimes succeed and sometimes fail.
|
||||
This inconsistency arises because the Task or Actor finds the file when running on the Head Node, but the file might not exist on other machines.
|
||||
|
||||
**Example:** Consider the following scenario:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
% touch /tmp/foo.txt
|
||||
|
||||
And this code:
|
||||
|
||||
.. testcode::
|
||||
|
||||
import os
|
||||
import ray
|
||||
|
||||
@ray.remote
|
||||
def check_file():
|
||||
foo_exists = os.path.exists("/tmp/foo.txt")
|
||||
return foo_exists
|
||||
|
||||
futures = []
|
||||
for _ in range(1000):
|
||||
futures.append(check_file.remote())
|
||||
|
||||
print(ray.get(futures))
|
||||
|
||||
In this case, you might receive a mixture of True and False. If ``check_file()`` runs on the Head Node or locally, it finds the file; however, on a Worker Node, it doesn't.
|
||||
|
||||
**Expected behavior:** Users generally expect file references to either work consistently or to reliably fail, rather than behaving inconsistently.
|
||||
|
||||
**Fix:**
|
||||
|
||||
— Use only shared file paths for such applications. For example, a network file system or S3 storage can provide the required consistency.
|
||||
— Avoid relying on local files to be consistent across machines.
|
||||
|
||||
|
||||
Placement Groups aren't composable
|
||||
-----------------------------------
|
||||
|
||||
**Issue:** If you schedule a new task from the tasks or actors running within a Placement Group, the system might fail to allocate resources properly, causing the operation to hang.
|
||||
|
||||
**Example:** Imagine you are using Ray Tune (which creates Placement Groups) and want to apply it to an objective function that in turn uses Ray Tasks. For example:
|
||||
|
||||
.. testcode::
|
||||
|
||||
import ray
|
||||
from ray import tune
|
||||
from ray.util.scheduling_strategies import PlacementGroupSchedulingStrategy
|
||||
|
||||
def create_task_that_uses_resources():
|
||||
@ray.remote(num_cpus=10)
|
||||
def sample_task():
|
||||
print("Hello")
|
||||
return
|
||||
|
||||
return ray.get([sample_task.remote() for i in range(10)])
|
||||
|
||||
def objective(config):
|
||||
create_task_that_uses_resources()
|
||||
|
||||
tuner = tune.Tuner(objective, param_space={"a": 1})
|
||||
tuner.fit()
|
||||
|
||||
This code errors with the message:
|
||||
|
||||
.. code-block::
|
||||
|
||||
ValueError: Cannot schedule create_task_that_uses_resources.<locals>.sample_task with the placement group
|
||||
because the resource request {'CPU': 10} cannot fit into any bundles for the placement group, [{'CPU': 1.0}].
|
||||
|
||||
**Expected behavior:** The code executes successfully without resource allocation issues.
|
||||
|
||||
**Fix:** Ensure that in the ``@ray.remote`` declaration of tasks called within ``create_task_that_uses_resources()``, you include the parameter
|
||||
``scheduling_strategy=PlacementGroupSchedulingStrategy(placement_group=None)``.
|
||||
|
||||
.. code-block:: diff
|
||||
|
||||
def create_task_that_uses_resources():
|
||||
+ @ray.remote(num_cpus=10, scheduling_strategy=PlacementGroupSchedulingStrategy(placement_group=None))
|
||||
- @ray.remote(num_cpus=10)
|
||||
|
||||
|
||||
Outdated Function Definitions
|
||||
-----------------------------
|
||||
|
||||
Because of Python's subtleties, redefining a remote function may not always update Ray to use the latest version.
|
||||
For example, suppose you define a remote function ``f`` and then redefine it; Ray should use the new definition:
|
||||
|
||||
.. testcode::
|
||||
|
||||
import ray
|
||||
|
||||
@ray.remote
|
||||
def f():
|
||||
return 1
|
||||
|
||||
@ray.remote
|
||||
def f():
|
||||
return 2
|
||||
|
||||
print(ray.get(f.remote())) # This should print 2.
|
||||
|
||||
.. testoutput::
|
||||
|
||||
2
|
||||
|
||||
However, there are cases where modifying a remote function doesn't take effect without restarting the cluster:
|
||||
|
||||
— **Imported function issue:** If ``f`` is defined in an external file (e.g., ``file.py``), and you modify its definition, re-importing the file may be ignored because Python treats the subsequent import as a no-op. A solution is to use ``from importlib import reload; reload(file)`` instead of a second import.
|
||||
|
||||
— **Helper function dependency:** If ``f`` depends on a helper function ``h`` defined in an external file, changes to ``h`` may not propagate. The easiest solution is to restart the Ray cluster. Alternatively, you can redefine ``f`` to reload ``file.py`` before invoking ``h``:
|
||||
|
||||
.. testcode::
|
||||
|
||||
@ray.remote
|
||||
def f():
|
||||
from importlib import reload
|
||||
reload(file)
|
||||
return file.h()
|
||||
|
||||
This forces the external module to reload on the Workers. Note that in Python 3, you must use ``from importlib import reload``.
|
||||
|
||||
|
||||
Capture task and actor call sites
|
||||
---------------------------------
|
||||
|
||||
Ray captures and displays a stack trace when you invoke a task, create an actor, or call an actor method.
|
||||
|
||||
To enable call site capture, set the environment variable ``RAY_record_task_actor_creation_sites=true``. When enabled:
|
||||
|
||||
— Ray captures a stack trace when creating tasks, actors, or invoking actor methods.
|
||||
— The captured stack trace is available in the Ray Dashboard (under task and actor details), output of the state CLI command ``ray list task --detail``, and state API responses.
|
||||
|
||||
Note that Ray turns off stack trace capture by default due to potential performance impacts. Enable it only when you need it for debugging.
|
||||
|
||||
Example:
|
||||
|
||||
.. NOTE(edoakes): test is skipped because it reinitializes Ray.
|
||||
.. testcode::
|
||||
:skipif: True
|
||||
|
||||
import ray
|
||||
|
||||
# Enable stack trace capture
|
||||
ray.init(runtime_env={"env_vars": {"RAY_record_task_actor_creation_sites": "true"}})
|
||||
|
||||
@ray.remote
|
||||
def my_task():
|
||||
return 42
|
||||
|
||||
# Capture the stack trace upon task invocation.
|
||||
future = my_task.remote()
|
||||
result = ray.get(future)
|
||||
|
||||
@ray.remote
|
||||
class Counter:
|
||||
def __init__(self):
|
||||
self.value = 0
|
||||
|
||||
def increment(self):
|
||||
self.value += 1
|
||||
return self.value
|
||||
|
||||
# Capture the stack trace upon actor creation.
|
||||
counter = Counter.remote()
|
||||
|
||||
# Capture the stack trace upon method invocation.
|
||||
counter.increment.remote()
|
||||
|
||||
This document outlines common problems encountered when using Ray along with potential solutions. If you encounter additional issues, please report them.
|
||||
|
||||
.. _`let us know`: https://github.com/ray-project/ray/issues
|
||||
@@ -0,0 +1,24 @@
|
||||
(observability-debug-apps)=
|
||||
|
||||
# Debugging Applications
|
||||
|
||||
```{toctree}
|
||||
:hidden:
|
||||
|
||||
general-debugging
|
||||
debug-memory
|
||||
debug-hangs
|
||||
debug-failures
|
||||
optimize-performance
|
||||
../../ray-distributed-debugger
|
||||
ray-debugging
|
||||
```
|
||||
|
||||
These guides help you perform common debugging or optimization tasks for your distributed application on Ray:
|
||||
* {ref}`observability-general-debugging`
|
||||
* {ref}`ray-core-mem-profiling`
|
||||
* {ref}`observability-debug-hangs`
|
||||
* {ref}`observability-debug-failures`
|
||||
* {ref}`observability-optimize-performance`
|
||||
* {ref}`ray-distributed-debugger`
|
||||
* {ref}`ray-debugger` (deprecated)
|
||||
@@ -0,0 +1,360 @@
|
||||
.. _observability-optimize-performance:
|
||||
|
||||
Optimizing Performance
|
||||
======================
|
||||
|
||||
No speedup
|
||||
----------
|
||||
|
||||
You just ran an application using Ray, but it wasn't as fast as you expected it
|
||||
to be. Or worse, perhaps it was slower than the serial version of the
|
||||
application! The most common reasons are the following.
|
||||
|
||||
- **Number of cores:** How many cores is Ray using? When you start Ray, it will
|
||||
determine the number of CPUs on each machine with ``psutil.cpu_count()``. Ray
|
||||
usually will not schedule more tasks in parallel than the number of CPUs. So
|
||||
if the number of CPUs is 4, the most you should expect is a 4x speedup.
|
||||
|
||||
- **Physical versus logical CPUs:** Do the machines you're running on have fewer
|
||||
**physical** cores than **logical** cores? You can check the number of logical
|
||||
cores with ``psutil.cpu_count()`` and the number of physical cores with
|
||||
``psutil.cpu_count(logical=False)``. This is common on a lot of machines and
|
||||
especially on EC2. For many workloads (especially numerical workloads), you
|
||||
often cannot expect a greater speedup than the number of physical CPUs.
|
||||
|
||||
- **Small tasks:** Are your tasks very small? Ray introduces some overhead for
|
||||
each task (the amount of overhead depends on the arguments that are passed
|
||||
in). You will be unlikely to see speedups if your tasks take less than ten
|
||||
milliseconds. For many workloads, you can easily increase the sizes of your
|
||||
tasks by batching them together.
|
||||
|
||||
- **Variable durations:** Do your tasks have variable duration? If you run 10
|
||||
tasks with variable duration in parallel, you shouldn't expect an N-fold
|
||||
speedup (because you'll end up waiting for the slowest task). In this case,
|
||||
consider using ``ray.wait`` to begin processing tasks that finish first.
|
||||
|
||||
- **Multi-threaded libraries:** Are all of your tasks attempting to use all of
|
||||
the cores on the machine? If so, they are likely to experience contention and
|
||||
prevent your application from achieving a speedup.
|
||||
This is common with some versions of ``numpy``. To avoid contention, set an
|
||||
environment variable like ``MKL_NUM_THREADS`` (or the equivalent depending on
|
||||
your installation) to ``1``.
|
||||
|
||||
For many - but not all - libraries, you can diagnose this by opening ``top``
|
||||
while your application is running. If one process is using most of the CPUs,
|
||||
and the others are using a small amount, this may be the problem. The most
|
||||
common exception is PyTorch, which will appear to be using all the cores
|
||||
despite needing ``torch.set_num_threads(1)`` to be called to avoid contention.
|
||||
|
||||
If you are still experiencing a slowdown, but none of the above problems apply,
|
||||
we'd really like to know! Create a `GitHub issue`_ and Submit a minimal code example that demonstrates the problem.
|
||||
|
||||
.. _`Github issue`: https://github.com/ray-project/ray/issues
|
||||
|
||||
This document discusses some common problems that people run into when using Ray
|
||||
as well as some known problems. If you encounter other problems, `let us know`_.
|
||||
|
||||
.. _`let us know`: https://github.com/ray-project/ray/issues
|
||||
|
||||
.. _ray-core-timeline:
|
||||
|
||||
Visualizing Tasks with Ray Timeline
|
||||
-------------------------------------
|
||||
View :ref:`how to use Ray Timeline in the Dashboard <dashboard-timeline>` for more details.
|
||||
|
||||
Instead of using Dashboard UI to download the tracing file, you can also export the tracing file as a JSON file by running ``ray timeline`` from the command line or ``ray.timeline`` from the Python API.
|
||||
|
||||
.. testcode::
|
||||
|
||||
import ray
|
||||
|
||||
ray.init()
|
||||
|
||||
ray.timeline(filename="timeline.json")
|
||||
|
||||
|
||||
.. _dashboard-profiling:
|
||||
|
||||
Python CPU profiling in the Dashboard
|
||||
-------------------------------------
|
||||
|
||||
The :ref:`Ray dashboard <observability-getting-started>` lets you profile Ray worker processes by clicking on the "Stack Trace" or "CPU Flame Graph"
|
||||
actions for active workers, actors, and jobs.
|
||||
|
||||
.. image:: /images/profile.png
|
||||
:align: center
|
||||
:width: 80%
|
||||
|
||||
Clicking "Stack Trace" returns the current stack trace sample using ``py-spy``. By default, only the Python stack
|
||||
trace is shown. To show native code frames, set the URL parameter ``native=1`` (only supported on Linux). To also
|
||||
dump stack traces for child processes of the target (for example, data loader or multiprocess inference workers),
|
||||
set the URL parameter ``subprocesses=1``.
|
||||
|
||||
.. image:: /images/stack.png
|
||||
:align: center
|
||||
:width: 60%
|
||||
|
||||
Clicking "CPU Flame Graph" takes a number of stack trace samples and combine them into a flame graph visualization.
|
||||
This flame graph can be useful for understanding the CPU activity of the particular process. To adjust the duration
|
||||
of the flame graph, you can change the ``duration`` parameter in the URL. Similarly, you can change the ``native``
|
||||
parameter to enable native profiling. To also include off-CPU (sleeping) threads, such as threads blocked on locks,
|
||||
I/O, or CUDA syncs, set the URL parameter ``idle=1``. To also profile child processes of the target (for example,
|
||||
data loader or multiprocess inference workers), set the URL parameter ``subprocesses=1``.
|
||||
|
||||
.. image:: /images/flamegraph.png
|
||||
:align: center
|
||||
:width: 80%
|
||||
|
||||
The profiling feature requires ``py-spy`` to be installed. If it is not installed, or if the ``py-spy`` binary does
|
||||
not have root permissions, the Dashboard prompts with instructions on how to setup ``py-spy`` correctly:
|
||||
|
||||
.. code-block::
|
||||
|
||||
This command requires `py-spy` to be installed with root permissions. You
|
||||
can install `py-spy` and give it root permissions as follows:
|
||||
$ pip install py-spy
|
||||
$ sudo chown root:root `which py-spy`
|
||||
$ sudo chmod u+s `which py-spy`
|
||||
|
||||
Alternatively, you can start Ray with passwordless sudo / root permissions.
|
||||
|
||||
.. note::
|
||||
You may run into permission errors when using py-spy in the docker containers. To fix the issue:
|
||||
|
||||
* If you start Ray manually in a Docker container, follow the `py-spy documentation`_ to resolve it.
|
||||
* if you are a KubeRay user, follow the :ref:`guide to configure KubeRay <kuberay-pyspy-integration>` and resolve it.
|
||||
|
||||
.. _`py-spy documentation`: https://github.com/benfred/py-spy#how-do-i-run-py-spy-in-docker
|
||||
|
||||
|
||||
|
||||
.. _dashboard-cprofile:
|
||||
|
||||
Profiling using Python's cProfile
|
||||
---------------------------------
|
||||
|
||||
You can use Python's native cProfile `profiling module`_ to profile the performance of your Ray application. Rather than tracking
|
||||
line-by-line of your application code, cProfile can give the total runtime
|
||||
of each loop function, as well as list the number of calls made and
|
||||
execution time of all function calls made within the profiled code.
|
||||
|
||||
.. _`profiling module`: https://docs.python.org/3/library/profile.html#module-cProfile
|
||||
|
||||
Unlike ``line_profiler`` above, this detailed list of profiled function calls
|
||||
**includes** internal function calls and function calls made within Ray.
|
||||
|
||||
However, similar to ``line_profiler``, cProfile can be enabled with minimal
|
||||
changes to your application code (given that each section of the code you want
|
||||
to profile is defined as its own function). To use cProfile, add an import
|
||||
statement, then replace calls to the loop functions as follows:
|
||||
|
||||
.. testcode::
|
||||
:skipif: True
|
||||
|
||||
import cProfile # Added import statement
|
||||
|
||||
def ex1():
|
||||
list1 = []
|
||||
for i in range(5):
|
||||
list1.append(ray.get(func.remote()))
|
||||
|
||||
def main():
|
||||
ray.init()
|
||||
cProfile.run('ex1()') # Modified call to ex1
|
||||
cProfile.run('ex2()')
|
||||
cProfile.run('ex3()')
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
Now, when you execute your Python script, a cProfile list of profiled function
|
||||
calls are printed on the terminal for each call made to ``cProfile.run()``.
|
||||
At the very top of cProfile's output gives the total execution time for
|
||||
``'ex1()'``:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
601 function calls (595 primitive calls) in 2.509 seconds
|
||||
|
||||
Following is a snippet of profiled function calls for ``'ex1()'``. Most of
|
||||
these calls are quick and take around 0.000 seconds, so the functions of
|
||||
interest are the ones with non-zero execution times:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ncalls tottime percall cumtime percall filename:lineno(function)
|
||||
...
|
||||
1 0.000 0.000 2.509 2.509 your_script_here.py:31(ex1)
|
||||
5 0.000 0.000 0.001 0.000 remote_function.py:103(remote)
|
||||
5 0.000 0.000 0.001 0.000 remote_function.py:107(_remote)
|
||||
...
|
||||
10 0.000 0.000 0.000 0.000 worker.py:2459(__init__)
|
||||
5 0.000 0.000 2.508 0.502 worker.py:2535(get)
|
||||
5 0.000 0.000 0.000 0.000 worker.py:2695(get_global_worker)
|
||||
10 0.000 0.000 2.507 0.251 worker.py:374(retrieve_and_deserialize)
|
||||
5 0.000 0.000 2.508 0.502 worker.py:424(get_object)
|
||||
5 0.000 0.000 0.000 0.000 worker.py:514(submit_task)
|
||||
...
|
||||
|
||||
The 5 separate calls to Ray's ``get``, taking the full 0.502 seconds each call,
|
||||
can be noticed at ``worker.py:2535(get)``. Meanwhile, the act of calling the
|
||||
remote function itself at ``remote_function.py:103(remote)`` only takes 0.001
|
||||
seconds over 5 calls, and thus is not the source of the slow performance of
|
||||
``ex1()``.
|
||||
|
||||
|
||||
Profiling Ray Actors with cProfile
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Considering that the detailed output of cProfile can be quite different depending
|
||||
on what Ray functionalities we use, let us see what cProfile's output might look
|
||||
like if our example involved Actors (for an introduction to Ray actors, see our
|
||||
:ref:`Actor documentation <actor-guide>`).
|
||||
|
||||
Now, instead of looping over five calls to a remote function like in ``ex1``,
|
||||
let's create a new example and loop over five calls to a remote function
|
||||
**inside an actor**. Our actor's remote function again just sleeps for 0.5
|
||||
seconds:
|
||||
|
||||
.. testcode::
|
||||
|
||||
# Our actor
|
||||
@ray.remote
|
||||
class Sleeper:
|
||||
def __init__(self):
|
||||
self.sleepValue = 0.5
|
||||
|
||||
# Equivalent to func(), but defined within an actor
|
||||
def actor_func(self):
|
||||
time.sleep(self.sleepValue)
|
||||
|
||||
Recalling the suboptimality of ``ex1``, let's first see what happens if we
|
||||
attempt to perform all five ``actor_func()`` calls within a single actor:
|
||||
|
||||
.. testcode::
|
||||
|
||||
def ex4():
|
||||
# This is suboptimal in Ray, and should only be used for the sake of this example
|
||||
actor_example = Sleeper.remote()
|
||||
|
||||
five_results = []
|
||||
for i in range(5):
|
||||
five_results.append(actor_example.actor_func.remote())
|
||||
|
||||
# Wait until the end to call ray.get()
|
||||
ray.get(five_results)
|
||||
|
||||
We enable cProfile on this example as follows:
|
||||
|
||||
.. testcode::
|
||||
:skipif: True
|
||||
|
||||
def main():
|
||||
ray.init()
|
||||
cProfile.run('ex4()')
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
Running our new Actor example, cProfile's abbreviated output is as follows:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
12519 function calls (11956 primitive calls) in 2.525 seconds
|
||||
|
||||
ncalls tottime percall cumtime percall filename:lineno(function)
|
||||
...
|
||||
1 0.000 0.000 0.015 0.015 actor.py:546(remote)
|
||||
1 0.000 0.000 0.015 0.015 actor.py:560(_remote)
|
||||
1 0.000 0.000 0.000 0.000 actor.py:697(__init__)
|
||||
...
|
||||
1 0.000 0.000 2.525 2.525 your_script_here.py:63(ex4)
|
||||
...
|
||||
9 0.000 0.000 0.000 0.000 worker.py:2459(__init__)
|
||||
1 0.000 0.000 2.509 2.509 worker.py:2535(get)
|
||||
9 0.000 0.000 0.000 0.000 worker.py:2695(get_global_worker)
|
||||
4 0.000 0.000 2.508 0.627 worker.py:374(retrieve_and_deserialize)
|
||||
1 0.000 0.000 2.509 2.509 worker.py:424(get_object)
|
||||
8 0.000 0.000 0.001 0.000 worker.py:514(submit_task)
|
||||
...
|
||||
|
||||
It turns out that the entire example still took 2.5 seconds to execute, or the
|
||||
time for five calls to ``actor_func()`` to run in serial. If you recall ``ex1``,
|
||||
this behavior was because we did not wait until after submitting all five
|
||||
remote function tasks to call ``ray.get()``, but we can verify on cProfile's
|
||||
output line ``worker.py:2535(get)`` that ``ray.get()`` was only called once at
|
||||
the end, for 2.509 seconds. What happened?
|
||||
|
||||
It turns out Ray cannot parallelize this example, because we have only
|
||||
initialized a single ``Sleeper`` actor. Because each actor is a single,
|
||||
stateful worker, our entire code is submitted and ran on a single worker the
|
||||
whole time.
|
||||
|
||||
To better parallelize the actors in ``ex4``, we can take advantage
|
||||
that each call to ``actor_func()`` is independent, and instead
|
||||
create five ``Sleeper`` actors. That way, we are creating five workers
|
||||
that can run in parallel, instead of creating a single worker that
|
||||
can only handle one call to ``actor_func()`` at a time.
|
||||
|
||||
.. testcode::
|
||||
|
||||
def ex4():
|
||||
# Modified to create five separate Sleepers
|
||||
five_actors = [Sleeper.remote() for i in range(5)]
|
||||
|
||||
# Each call to actor_func now goes to a different Sleeper
|
||||
five_results = []
|
||||
for actor_example in five_actors:
|
||||
five_results.append(actor_example.actor_func.remote())
|
||||
|
||||
ray.get(five_results)
|
||||
|
||||
Our example in total now takes only 1.5 seconds to run:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
1378 function calls (1363 primitive calls) in 1.567 seconds
|
||||
|
||||
ncalls tottime percall cumtime percall filename:lineno(function)
|
||||
...
|
||||
5 0.000 0.000 0.002 0.000 actor.py:546(remote)
|
||||
5 0.000 0.000 0.002 0.000 actor.py:560(_remote)
|
||||
5 0.000 0.000 0.000 0.000 actor.py:697(__init__)
|
||||
...
|
||||
1 0.000 0.000 1.566 1.566 your_script_here.py:71(ex4)
|
||||
...
|
||||
21 0.000 0.000 0.000 0.000 worker.py:2459(__init__)
|
||||
1 0.000 0.000 1.564 1.564 worker.py:2535(get)
|
||||
25 0.000 0.000 0.000 0.000 worker.py:2695(get_global_worker)
|
||||
3 0.000 0.000 1.564 0.521 worker.py:374(retrieve_and_deserialize)
|
||||
1 0.000 0.000 1.564 1.564 worker.py:424(get_object)
|
||||
20 0.001 0.000 0.001 0.000 worker.py:514(submit_task)
|
||||
...
|
||||
|
||||
|
||||
.. _performance-debugging-gpu-profiling:
|
||||
|
||||
GPU Profiling with PyTorch Profiler
|
||||
-----------------------------------
|
||||
Here are the steps to use PyTorch Profiler during training with Ray Train or batch inference with Ray Data:
|
||||
|
||||
* Follow the `PyTorch Profiler documentation <https://pytorch.org/tutorials/intermediate/tensorboard_profiler_tutorial.html>`_ to record events in your PyTorch code.
|
||||
|
||||
* Convert your PyTorch script to a :ref:`Ray Train training script <train-pytorch>` or a :ref:`Ray Data batch inference script <batch_inference_home>`. (no change to your profiler-related code)
|
||||
|
||||
* Run your training or batch inference script.
|
||||
|
||||
* Collect the profiling results from all the nodes (compared to 1 node in a non-distributed setting).
|
||||
|
||||
* You may want to upload results on each Node to NFS or object storage like S3 so that you don't have to fetch results from each Node respectively.
|
||||
|
||||
* Visualize the results with tools like Tensorboard.
|
||||
|
||||
GPU Profiling with Nsight System Profiler
|
||||
------------------------------------------
|
||||
GPU profiling is critical for ML training and inference. Ray allows users to run Nsight System Profiler with Ray actors and tasks. :ref:`See for details <profiling-nsight-profiler>`.
|
||||
|
||||
Profiling for developers
|
||||
------------------------
|
||||
If you are developing Ray Core or debugging some system level failures, profiling the Ray Core could help. In this case, see :ref:`Profiling for Ray developers <ray-core-internal-profiling>`.
|
||||
|
||||
@@ -0,0 +1,280 @@
|
||||
.. _ray-debugger:
|
||||
|
||||
Using the Ray Debugger
|
||||
======================
|
||||
|
||||
Ray has a built in debugger that allows you to debug your distributed applications. It allows
|
||||
to set breakpoints in your Ray tasks and actors and when hitting the breakpoint you can
|
||||
drop into a PDB session that you can then use to:
|
||||
|
||||
- Inspect variables in that context
|
||||
- Step within that task or actor
|
||||
- Move up or down the stack
|
||||
|
||||
.. warning::
|
||||
|
||||
The Ray Debugger is deprecated. Use the :doc:`Ray Distributed Debugger <../../ray-distributed-debugger>` instead.
|
||||
Starting with Ray 2.39, the new debugger is the default and you need to set the environment variable `RAY_DEBUG=legacy` to
|
||||
use the old debugger (e.g. by using a runtime environment).
|
||||
|
||||
Getting Started
|
||||
---------------
|
||||
|
||||
Take the following example:
|
||||
|
||||
.. testcode::
|
||||
:skipif: True
|
||||
|
||||
import ray
|
||||
|
||||
ray.init(runtime_env={"env_vars": {"RAY_DEBUG": "legacy"}})
|
||||
|
||||
@ray.remote
|
||||
def f(x):
|
||||
breakpoint()
|
||||
return x * x
|
||||
|
||||
futures = [f.remote(i) for i in range(2)]
|
||||
print(ray.get(futures))
|
||||
|
||||
Put the program into a file named ``debugging.py`` and execute it using:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python debugging.py
|
||||
|
||||
|
||||
Each of the 2 executed tasks will drop into a breakpoint when the line
|
||||
``breakpoint()`` is executed. You can attach to the debugger by running
|
||||
the following command on the head node of the cluster:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ray debug
|
||||
|
||||
The ``ray debug`` command will print an output like this:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
2021-07-13 16:30:40,112 INFO scripts.py:216 -- Connecting to Ray instance at 192.168.2.61:6379.
|
||||
2021-07-13 16:30:40,112 INFO worker.py:740 -- Connecting to existing Ray cluster at address: 192.168.2.61:6379
|
||||
Active breakpoints:
|
||||
index | timestamp | Ray task | filename:lineno
|
||||
0 | 2021-07-13 23:30:37 | ray::f() | debugging.py:6
|
||||
1 | 2021-07-13 23:30:37 | ray::f() | debugging.py:6
|
||||
Enter breakpoint index or press enter to refresh:
|
||||
|
||||
|
||||
You can now enter ``0`` and hit Enter to jump to the first breakpoint. You will be dropped into PDB
|
||||
at the break point and can use the ``help`` to see the available actions. Run ``bt`` to see a backtrace
|
||||
of the execution:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
(Pdb) bt
|
||||
/home/ubuntu/ray/python/ray/workers/default_worker.py(170)<module>()
|
||||
-> ray.worker.global_worker.main_loop()
|
||||
/home/ubuntu/ray/python/ray/worker.py(385)main_loop()
|
||||
-> self.core_worker.run_task_loop()
|
||||
> /home/ubuntu/tmp/debugging.py(7)f()
|
||||
-> return x * x
|
||||
|
||||
You can inspect the value of ``x`` with ``print(x)``. You can see the current source code with ``ll``
|
||||
and change stack frames with ``up`` and ``down``. For now let us continue the execution with ``c``.
|
||||
|
||||
After the execution is continued, hit ``Control + D`` to get back to the list of break points. Select
|
||||
the other break point and hit ``c`` again to continue the execution.
|
||||
|
||||
The Ray program ``debugging.py`` now finished and should have printed ``[0, 1]``. Congratulations, you
|
||||
have finished your first Ray debugging session!
|
||||
|
||||
Running on a Cluster
|
||||
--------------------
|
||||
|
||||
The Ray debugger supports setting breakpoints inside of tasks and actors that are running across your
|
||||
Ray cluster. In order to attach to these from the head node of the cluster using ``ray debug``, you'll
|
||||
need to make sure to pass in the ``--ray-debugger-external`` flag to ``ray start`` when starting the
|
||||
cluster (likely in your ``cluster.yaml`` file or k8s Ray cluster spec).
|
||||
|
||||
Note that this flag will cause the workers to listen for PDB commands on an external-facing IP address,
|
||||
so this should *only* be used if your cluster is behind a firewall.
|
||||
|
||||
Debugger Commands
|
||||
-----------------
|
||||
|
||||
The Ray debugger supports the
|
||||
`same commands as PDB
|
||||
<https://docs.python.org/3/library/pdb.html#debugger-commands>`_.
|
||||
|
||||
Stepping between Ray tasks
|
||||
--------------------------
|
||||
|
||||
You can use the debugger to step between Ray tasks. Let's take the
|
||||
following recursive function as an example:
|
||||
|
||||
.. testcode::
|
||||
:skipif: True
|
||||
|
||||
import ray
|
||||
|
||||
ray.init(runtime_env={"env_vars": {"RAY_DEBUG": "legacy"}})
|
||||
|
||||
@ray.remote
|
||||
def fact(n):
|
||||
if n == 1:
|
||||
return n
|
||||
else:
|
||||
n_ref = fact.remote(n - 1)
|
||||
return n * ray.get(n_ref)
|
||||
|
||||
@ray.remote
|
||||
def compute():
|
||||
breakpoint()
|
||||
result_ref = fact.remote(5)
|
||||
result = ray.get(result_ref)
|
||||
|
||||
ray.get(compute.remote())
|
||||
|
||||
|
||||
After running the program by executing the Python file and calling
|
||||
``ray debug``, you can select the breakpoint by pressing ``0`` and
|
||||
enter. This will result in the following output:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
Enter breakpoint index or press enter to refresh: 0
|
||||
> /home/ubuntu/tmp/stepping.py(16)<module>()
|
||||
-> result_ref = fact.remote(5)
|
||||
(Pdb)
|
||||
|
||||
You can jump into the call with the ``remote`` command in Ray's debugger.
|
||||
Inside the function, print the value of `n` with ``p(n)``, resulting in
|
||||
the following output:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
-> result_ref = fact.remote(5)
|
||||
(Pdb) remote
|
||||
*** Connection closed by remote host ***
|
||||
Continuing pdb session in different process...
|
||||
--Call--
|
||||
> /home/ubuntu/tmp/stepping.py(5)fact()
|
||||
-> @ray.remote
|
||||
(Pdb) ll
|
||||
5 -> @ray.remote
|
||||
6 def fact(n):
|
||||
7 if n == 1:
|
||||
8 return n
|
||||
9 else:
|
||||
10 n_ref = fact.remote(n - 1)
|
||||
11 return n * ray.get(n_ref)
|
||||
(Pdb) p(n)
|
||||
5
|
||||
(Pdb)
|
||||
|
||||
Now step into the next remote call again with
|
||||
``remote`` and print `n`. You an now either continue recursing into
|
||||
the function by calling ``remote`` a few more times, or you can jump
|
||||
to the location where ``ray.get`` is called on the result by using the
|
||||
``get`` debugger command. Use ``get`` again to jump back to the original
|
||||
call site and use ``p(result)`` to print the result:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
Enter breakpoint index or press enter to refresh: 0
|
||||
> /home/ubuntu/tmp/stepping.py(14)<module>()
|
||||
-> result_ref = fact.remote(5)
|
||||
(Pdb) remote
|
||||
*** Connection closed by remote host ***
|
||||
Continuing pdb session in different process...
|
||||
--Call--
|
||||
> /home/ubuntu/tmp/stepping.py(5)fact()
|
||||
-> @ray.remote
|
||||
(Pdb) p(n)
|
||||
5
|
||||
(Pdb) remote
|
||||
*** Connection closed by remote host ***
|
||||
Continuing pdb session in different process...
|
||||
--Call--
|
||||
> /home/ubuntu/tmp/stepping.py(5)fact()
|
||||
-> @ray.remote
|
||||
(Pdb) p(n)
|
||||
4
|
||||
(Pdb) get
|
||||
*** Connection closed by remote host ***
|
||||
Continuing pdb session in different process...
|
||||
--Return--
|
||||
> /home/ubuntu/tmp/stepping.py(5)fact()->120
|
||||
-> @ray.remote
|
||||
(Pdb) get
|
||||
*** Connection closed by remote host ***
|
||||
Continuing pdb session in different process...
|
||||
--Return--
|
||||
> /home/ubuntu/tmp/stepping.py(14)<module>()->None
|
||||
-> result_ref = fact.remote(5)
|
||||
(Pdb) p(result)
|
||||
120
|
||||
(Pdb)
|
||||
|
||||
|
||||
Post Mortem Debugging
|
||||
---------------------
|
||||
|
||||
Often we do not know in advance where an error happens, so we cannot set a breakpoint. In these cases,
|
||||
we can automatically drop into the debugger when an error occurs or an exception is thrown. This is called *post-mortem debugging*.
|
||||
|
||||
Copy the following code into a file called ``post_mortem_debugging.py``. The flag ``RAY_DEBUG_POST_MORTEM=1`` will have the effect
|
||||
that if an exception happens, Ray will drop into the debugger instead of propagating it further.
|
||||
|
||||
.. testcode::
|
||||
:skipif: True
|
||||
|
||||
import ray
|
||||
|
||||
ray.init(runtime_env={"env_vars": {"RAY_DEBUG": "legacy", "RAY_DEBUG_POST_MORTEM": "1"}})
|
||||
|
||||
@ray.remote
|
||||
def post_mortem(x):
|
||||
x += 1
|
||||
raise Exception("An exception is raised.")
|
||||
return x
|
||||
|
||||
ray.get(post_mortem.remote(10))
|
||||
|
||||
Let's start the program:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python post_mortem_debugging.py
|
||||
|
||||
Now run ``ray debug``. After we do that, we see an output like the following:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
Active breakpoints:
|
||||
index | timestamp | Ray task | filename:lineno
|
||||
0 | 2024-11-01 20:14:00 | /Users/pcmoritz/ray/python/ray/_private/workers/default_worker.py --node-ip-address=127.0.0.1 --node-manager-port=49606 --object-store-name=/tmp/ray/session_2024-11-01_13-13-51_279910_8596/sockets/plasma_store --raylet-name=/tmp/ray/session_2024-11-01_13-13-51_279910_8596/sockets/raylet --redis-address=None --metrics-agent-port=58655 --runtime-env-agent-port=56999 --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --runtime-env-agent-port=56999 --gcs-address=127.0.0.1:6379 --session-name=session_2024-11-01_13-13-51_279910_8596 --temp-dir=/tmp/ray --webui=127.0.0.1:8265 --cluster-id=6d341469ae0f85b6c3819168dde27cceda12e95c8efdfc256e0fd8ce --startup-token=12 --worker-launch-time-ms=1730492039955 --node-id=0d43573a606286125da39767a52ce45ad101324c8af02cc25a9fbac7 --runtime-env-hash=-1746935720 | /Users/pcmoritz/ray/python/ray/_private/worker.py:920
|
||||
Traceback (most recent call last):
|
||||
|
||||
File "python/ray/_raylet.pyx", line 1856, in ray._raylet.execute_task
|
||||
|
||||
File "python/ray/_raylet.pyx", line 1957, in ray._raylet.execute_task
|
||||
|
||||
File "python/ray/_raylet.pyx", line 1862, in ray._raylet.execute_task
|
||||
|
||||
File "/Users/pcmoritz/ray-debugger-test/post_mortem_debugging.py", line 8, in post_mortem
|
||||
raise Exception("An exception is raised.")
|
||||
|
||||
Exception: An exception is raised.
|
||||
|
||||
Enter breakpoint index or press enter to refresh:
|
||||
|
||||
We now press ``0`` and then Enter to enter the debugger. With ``ll`` we can see the context and with
|
||||
``print(x)`` we an print the value of ``x``.
|
||||
|
||||
In a similar manner as above, you can also debug Ray actors. Happy debugging!
|
||||
|
||||
Debugging APIs
|
||||
--------------
|
||||
|
||||
See :ref:`package-ref-debugging-apis`.
|
||||
Reference in New Issue
Block a user