wehub-resource-sync
111 lines
4.1 KiB
ReStructuredText
111 lines
4.1 KiB
ReStructuredText
.. _head-node-memory-management:
|
|
|
|
Head Node Memory Management
|
|
============================
|
|
|
|
When running Ray clusters for extended periods, the head node's memory usage can steadily increase over time, potentially leading to out-of-memory (OOM) errors that can make the entire cluster unusable. This guide explains the causes of head node memory growth and provides mitigation strategies.
|
|
|
|
.. contents::
|
|
:local:
|
|
|
|
Why Head Node Memory Grows
|
|
---------------------------
|
|
|
|
- The Ray Dashboard provides a web interface for cluster monitoring and debugging. For more details, see :ref:`observability-getting-started`.
|
|
- The Ray Dashboard caches cluster events in memory for display and debugging purposes. The ``RAY_DASHBOARD_MAX_EVENTS_TO_CACHE`` environment variable controls the cache size. For implementation details, see the `event caching code <https://github.com/ray-project/ray/blob/814768317813afca2f0af740f58d024b059ae7d7/python/ray/dashboard/modules/event/event_head.py#L35>`_.
|
|
- The dashboard processes and stores logs and metadata from jobs and workers, which accumulate over time in long-running clusters.
|
|
|
|
Mitigation Strategies
|
|
---------------------
|
|
|
|
Avoid Scheduling on the Head Node
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Running tasks or actors on the head node isn't recommended because it hosts critical system components. Preventing scheduling on the head node helps reduce contention and memory pressure.
|
|
|
|
See :ref:`vms-large-cluster-configure-head-node` for head-node best practices.
|
|
|
|
Disable the Dashboard
|
|
~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
If you don't need the dashboard, disabling it removes event caching and related memory overhead. This reduces observability into the system so it's not recommended for production clusters.
|
|
|
|
**Python API:**
|
|
|
|
.. code-block:: python
|
|
|
|
import ray
|
|
ray.init(include_dashboard=False)
|
|
|
|
**CLI:**
|
|
|
|
.. code-block:: bash
|
|
|
|
ray start --head --include-dashboard=False
|
|
|
|
**Kubernetes:**
|
|
|
|
Set ``spec.headGroupSpec.rayStartParams.include-dashboard`` to ``"false"`` in your RayCluster configuration.
|
|
|
|
.. warning::
|
|
Disabling the dashboard prevents KubeRay's ``RayJob`` and ``RayService`` features from working properly.
|
|
|
|
|
|
Kubernetes Configuration
|
|
------------------------
|
|
|
|
Head Pod Memory Settings
|
|
~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
When deploying on Kubernetes, configure appropriate memory requests and limits for the head pod.
|
|
|
|
**Important:** Set memory and CPU resource requests equal to their limits. KubeRay uses the container's resource **limits** to configure Ray's logical resource capacities and ignores memory and CPU **requests**.
|
|
|
|
Example configuration:
|
|
|
|
.. code-block:: yaml
|
|
|
|
headGroupSpec:
|
|
template:
|
|
spec:
|
|
containers:
|
|
- name: ray-head
|
|
resources:
|
|
requests:
|
|
memory: "8Gi"
|
|
cpu: "4"
|
|
limits:
|
|
memory: "8Gi"
|
|
cpu: "4"
|
|
|
|
Recommended Head Node Specifications
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
For large clusters, a good starting specification for the head node is:
|
|
|
|
- **CPU:** 16 cores
|
|
- **Memory:** 64 GB
|
|
|
|
The actual requirements depend on your workload and cluster size.
|
|
|
|
Additionally, consider preventing Ray from scheduling tasks on the head node by setting ``num-cpus: "0"`` in ``rayStartParams``.
|
|
|
|
Best Practices
|
|
--------------
|
|
|
|
1. **Avoid scheduling on the head node** to reduce contention and memory pressure.
|
|
2. **Scale vertically and use a larger head node** before adjusting internal settings.
|
|
3. **Set appropriate Kubernetes resource limits** (match requests for memory and GPU).
|
|
|
|
.. note::
|
|
You *can* disable the dashboard, but doing so severely limits observability and isn't **recommended for production**. If you choose to disable it, see the `Disable the Dashboard` section in the preceding text.
|
|
|
|
Troubleshooting
|
|
---------------
|
|
|
|
If your head node experiences OOM issues:
|
|
|
|
1. Check current memory usage: ``ray memory``. See :ref:`debug-with-ray-memory`
|
|
2. Consider increasing head node memory allocation
|
|
|
|
For more information on OOM prevention, see :ref:`ray-oom-prevention`. |