wehub/ray-project--ray
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:17:40 +08:00
commit f1825c8ceb
10096 changed files with 2364182 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/ray-core/internals/autoscaler-v2.rst
+211
View File
@@ -0,0 +1,211 @@
.. _autoscaler-v2:
Autoscaler v2
=============
This document explains how the open-source autoscaler v2 works in Ray 2.48 and outlines its high-level responsibilities and implementation details.
Overview
--------
The autoscaler is responsible for resizing the cluster based on resource demand from tasks, actors, and placement groups.
To achieve this, it follows a structured process: evaluating worker group configurations, periodically reconciling cluster state with user constraints, applying bin-packing strategies to pending workload demands, and interacting with cloud instance providers through the Instance Manager.
The following sections describe these components in detail.
Worker Group Configurations
---------------------------
Worker groups (also referred to as node types) define the sets of nodes that the Ray autoscaler scales.
Each worker group represents a logical category of nodes with the same resource configurations, such as CPU, memory, GPU, or custom resources.
The autoscaler dynamically adjusts the cluster size by adding or removing nodes within each group as workload demands change. In other words, it scales the cluster by modifying the number of nodes per worker group according to the specified scaling rules and resource requirements.
Worker groups can be configured in these ways:
- The `available_node_types <https://docs.ray.io/en/releases-2.48.0/cluster/vms/references/ray-cluster-configuration.html#node-types>`__ field in the Cluster YAML file, if you are using the ``ray up`` cluster launcher.
- The `workerGroupSpecs <https://docs.ray.io/en/releases-2.48.0/cluster/kubernetes/user-guides/config.html#pod-configuration-headgroupspec-and-workergroupspecs>`__ field in the RayCluster CRD, if you are using KubeRay.
The configuration specifies the logical resources each node has in a worker group, along with the minimum and maximum number of nodes that should exist in each group.
.. note::
Although the autoscaler fulfills pending resource demands and releases idle nodes, it doesn't perform the actual scheduling of Ray tasks, actors, or placement groups. Scheduling is handled internally by Ray.
The autoscaler does its own simulation of scheduling decisions on pending demands periodically to determine which nodes to launch or to stop. See the next sections for details.
Periodic Reconciliation
-----------------------
The entry point of the autoscaler is `monitor.py <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/python/ray/autoscaler/v2/monitor.py#L332>`__, which starts a GCS client and runs the reconciliation loop.
This process is launched on the head node by the `start_head_processes <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/python/ray/_private/node.py#L1439>`__ function when using the ``ray up`` cluster launcher.
When running under KubeRay, it instead runs as a `separate autoscaler container <https://github.com/ray-project/kuberay/blob/94fa7d3eb793aa1278142f8e585cbe568fec3ae3/ray-operator/controllers/ray/common/pod.go#L191-L194>`__ in the Head Pod.
.. warning::
In the case of the cluster launcher, if the autoscaler process crashes, then there is no autoscaling.
While in the case of KubeRay, Kubernetes restarts the autoscaler container if it crashes by the default container restart policy.
The process periodically `reconciles <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/python/ray/autoscaler/v2/autoscaler.py#L200-L213>`__ against a snapshot of the following information using the Reconciler:
1. **The latest pending demands** (queried from the `get_cluster_resource_state <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/src/ray/protobuf/autoscaler.proto#L392-L394>`__ GCS RPC): Pending Ray tasks, actors, and placement groups.
2. **The latest user cluster constraints** (queried from the `get_cluster_resource_state <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/src/ray/protobuf/autoscaler.proto#L392-L394>`__ GCS RPC): The minimum cluster size, if specified via the ``ray.autoscaler.sdk.request_resources`` invocation.
3. **The latest Ray nodes information** (queried from the `get_cluster_resource_state <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/src/ray/protobuf/autoscaler.proto#L392-L394>`__ GCS RPC): The total and currently available resources of each Ray node in the cluster. Also includes each Ray node's status (ALIVE or DEAD) and other information such as idle duration. See Appendix for more details.
4. **The latest cloud instances** (`queried from the cloud instance provider's implementation <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/python/ray/autoscaler/v2/autoscaler.py#L205-L207>`__): The list of instances managed by the cloud instance provider implementation.
5. **The latest worker group configurations** (queried from the cluster YAML file or the RayCluster CRD).
The preceding information is retrieved at the beginning of each reconciliation loop.
The Reconciler uses this information to construct its internal state and perform "`passive <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/python/ray/autoscaler/v2/instance_manager/reconciler.py#L159>`__" instance lifecycle transitions by observations. This is the `sync phase <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/python/ray/autoscaler/v2/instance_manager/reconciler.py#L112-L120>`__.
After the sync phase, the Reconciler performs the `following steps <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/python/ray/autoscaler/v2/scheduler.py#L840>`__ in order with the ``ResourceDemandScheduler``:
1. Enforce configuration constraints, including min/max nodes for each worker group.
2. Enforce user cluster constraints (if specified by `ray.autoscaler.sdk.request_resources <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/python/ray/autoscaler/_private/commands.py#L186>`__ invocation).
3. Fit pending demands into available resources on the cluster snapshot. This is the simulation mentioned earlier.
4. Fit any remaining demands (left over from the previous step) against worker group configurations to determine which nodes to launch.
5. Terminate idle instances (nodes that are needed by the previous 1-4 steps aren't considered idle) according to each node's ``idle_duration_ms`` (queried from GCS) and the configured idle timeout for each group.
6. Send accumulated scaling decisions (steps 1–5) to the Instance Manager with `Reconciler._update_instance_manager <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/python/ray/autoscaler/v2/instance_manager/reconciler.py#L1157-L1193>`__.
7. `Sleep briefly (5s by default) <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/python/ray/autoscaler/v2/monitor.py#L178>`__, then return to the sync phase.
.. warning::
If any error occurs, such as an error from the cloud instance provider or a timeout in the sync phase, the current reconciliation is aborted and the loop jumps to step 7 to wait for the next reconciliation.
.. note::
All scaling decisions from steps 1–5 are accumulated purely in memory.
No interaction with the cloud instance provider occurs until step 6.
Bin Packing and Worker Group Selection
--------------------------------------
The autoscaler applies the following scoring logic to evaluate each existing node. It selects the node with the highest score and assigns it a subset of feasible demands.
It also applies the same scoring logic to each worker group and selects the one with the highest score to launch new instances.
`Scoring <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/python/ray/autoscaler/v2/scheduler.py#L430>`__ is based on a tuple of four values:
1. Whether the node is a GPU node and whether feasible requests require GPUs:
- ``0`` if the node is a GPU node and requests do **not** require GPUs.
- ``1`` if the node isn't a GPU node or requests do require GPUs.
2. The number of resource types on the node used by feasible requests.
3. The minimum `utilization rate <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/python/ray/autoscaler/v2/scheduler.py#L481-L489>`__ across all resource types used by feasible requests.
4. The average `utilization rate <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/python/ray/autoscaler/v2/scheduler.py#L481-L489>`__ across all resource types used by feasible requests.
.. note::
Utilization rate used by feasible requests is calculated as the difference between the total and available resources divided by the total resources.
In other words:
- The autoscaler avoids launching GPU nodes unless necessary.
- It prefers nodes that maximize utilization and minimize unused resources.
Example:
- Task requires **2 GPUs**.
- Two node types are available:
- A: [GPU: 6]
- B: [GPU: 2, TPU: 1]
Node type **A** should be selected, since node B would leave an unused TPU (with a utilization rate of 0% on TPU), making it less favorable with respect to the third scoring criterion.
This process repeats until all feasible pending demands are packed or the maximum cluster size is reached.
Instance Manager and Cloud Instance Provider
--------------------------------------------
`Cloud Instance Provider <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/python/ray/autoscaler/v2/instance_manager/node_provider.py#L149>`__ is an abstract interface that defines the operations for managing instances in the cloud.
`Instance Manager <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/python/ray/autoscaler/v2/instance_manager/instance_manager.py#L29>`__ is the component that tracks instance lifecycle and drives event subscribers that call the cloud instance provider.
As described in the previous section, the autoscaler accumulates scaling decisions (steps 1–5) in memory and reconciles them with the cloud instance provider through the Instance Manager.
Scaling decisions are represented as a list of `InstanceUpdateEvent <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/src/ray/protobuf/instance_manager.proto#L135>`__ records. For example:
- **For launching new instances**:
- ``instance_id``: A randomly generated ID for Instance Manager tracking.
- ``instance_type``: The type of instance to launch.
- ``new_instance_status``: ``QUEUED``.
- **For terminating instances**:
- ``instance_id``: The ID of the instance to stop.
- ``new_instance_status``: ``TERMINATING`` or ``RAY_STOP_REQUESTED``.
These update events are passed to the Instance Manager, which transitions instance statuses.
A normal transition flow for an instance is:
- ``(non-existent) -> QUEUED``: The Reconciler creates an instance with the ``QUEUED`` ``InstanceUpdateEvent`` when it decides to launch a new instance.
- ``QUEUED -> REQUESTED``: The Reconciler considers ``max_concurrent_launches`` and ``upscaling_speed`` when selecting an instance from the queue to transition to ``REQUESTED`` during each reconciliation iteration.
- ``REQUESTED -> ALLOCATED``: Once the Reconciler detects the instance is allocated from the cloud instance provider, it will transition the instance to ``ALLOCATED``.
- ``ALLOCATED -> RAY_INSTALLING``: If the cloud instance provider is not ``KubeRayProvider``, the Reconciler will transition the instance to ``RAY_INSTALLING`` when the instance is allocated.
- ``RAY_INSTALLING -> RAY_RUNNING``: Once the Reconciler detects from GCS that Ray has started on the instance, it will transition the instance to ``RAY_RUNNING``.
- ``RAY_RUNNING -> RAY_STOP_REQUESTED``: If the instance is idle for longer than the configured timeout, the Reconciler will transition the instance to ``RAY_STOP_REQUESTED`` to start draining the Ray process.
- ``RAY_STOP_REQUESTED -> RAY_STOPPING``: Once the Reconciler detects from GCS that the Ray process is draining, it will transition the instance to ``RAY_STOPPING``.
- ``RAY_STOPPING -> RAY_STOPPED``: Once the Reconciler detects from GCS that the Ray process has stopped, it will transition the instance to ``RAY_STOPPED``.
- ``RAY_STOPPED -> TERMINATING``: The Reconciler will transition the instance from ``RAY_STOPPED`` to ``TERMINATING``.
- ``TERMINATING -> TERMINATED``: Once the Reconciler detects that the instance has been terminated by the cloud instance provider, it will transition the instance to ``TERMINATED``.
.. note::
The drain request sent by ``RAY_STOP_REQUESTED`` can be rejected if the node is no longer idle when the drain request arrives the node. Then the instance will be transitioned back to ``RAY_RUNNING`` instead.
You can find all valid transitions in the `get_valid_transitions <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/python/ray/autoscaler/v2/instance_manager/common.py#L193>`__ method.
Once transitions are triggered by the Reconciler, subscribers perform side effects, such as:
- ``QUEUED -> REQUESTED``: CloudInstanceUpdater launches the instance through the Cloud Instance Provider.
- ``ALLOCATED -> RAY_INSTALLING``: ThreadedRayInstaller installs the Ray process.
- ``RAY_RUNNING -> RAY_STOP_REQUESTED``: RayStopper stops the Ray process on the instance.
- ``RAY_STOPPED -> TERMINATING``: CloudInstanceUpdater terminates the instance through the Cloud Instance Provider.
.. note::
These transitions trigger side effects, but side effects don't trigger new transitions directly.
Instead, their results are observed from external state during the sync phase; subsequent transitions are triggered based on those observations.
.. note::
Cloud instance provider implementations in autoscaler v2 must implement:
- **Listing instances**: Return the set of instances currently managed by the provider.
- **Launching instances**: Create new instances given the requested instance type and tags.
- **Terminating instances**: Safely remove instances identified by their IDs.
``KubeRayProvider`` is one such cloud instance provider implementation.
``NodeProviderAdapter`` is an adapter that can wrap a v1 node provider (such as ``AWSNodeProvider``) to act as a cloud instance provider.
Appendix
--------
How ``get_cluster_resource_state`` Aggregates Cluster State
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The autoscaler retrieves a cluster snapshot through the ``get_cluster_resource_state`` RPC served by GCS (`HandleGetClusterResourceState <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/src/ray/gcs/gcs_server/gcs_autoscaler_state_manager.cc#L48>`__) which builds the reply in `MakeClusterResourceStateInternal <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/src/ray/gcs/gcs_server/gcs_autoscaler_state_manager.cc#L179>`__. Internally, GCS assembles the reply by combining per-node resource reports, pending workload demand, and any user-requested cluster constraints into a single ``ClusterResourceState`` message.
- Data sources and ownership:
- `GcsAutoscalerStateManager <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/src/ray/gcs/gcs_server/gcs_autoscaler_state_manager.cc>`__ maintains a per-node cache of ``ResourcesData`` that includes totals, availables, and load-by-shape. GCS periodically polls each alive raylet (``GetResourceLoad``) and updates this cache (`GcsServer::InitGcsResourceManager <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/src/ray/gcs/gcs_server/gcs_server.cc#L375-L418>`__, `UpdateResourceLoadAndUsage <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/src/ray/gcs/gcs_server/gcs_autoscaler_state_manager.cc#L267-L281>`__), then uses it to construct snapshots.
- `GcsNodeInfo <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/src/ray/protobuf/gcs.proto#L307>`__ provides static and slowly changing node metadata (node ID, instance ID, node type name, IP, labels, instance type) and dead/alive status.
- Placement group demand comes from the `placement group manager <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/src/ray/gcs/gcs_server/gcs_placement_group_mgr.cc#L934>`__.
- User cluster constraints come from autoscaler SDK requests that GCS records.
- Fields assembled in the reply:
- ``node_states``: For each node, GCS sets identity and metadata from `GcsNodeInfo <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/src/ray/protobuf/gcs.proto#L307>`__ and pulls resources and status from the cached ``ResourcesData`` (`GetNodeStates <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/src/ray/gcs/gcs_server/gcs_autoscaler_state_manager.cc#L319>`__). Dead nodes are marked ``DEAD`` and omit resource details. For alive nodes, GCS also includes ``idle_duration_ms`` and any node activity strings.
- ``pending_resource_requests``: Computed by aggregating per-node load-by-shape across the cluster (`GetPendingResourceRequests <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/src/ray/gcs/gcs_server/gcs_autoscaler_state_manager.cc#L303-L317>`__). For each resource shape, the count is the sum of infeasible, backlog, and ready requests that haven't been scheduled yet.
- ``pending_gang_resource_requests``: Pending or rescheduling placement groups represented as gang requests (`GetPendingGangResourceRequests <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/src/ray/gcs/gcs_server/gcs_autoscaler_state_manager.cc#L193>`__).
- ``cluster_resource_constraints``: The set of minimal cluster resource constraints previously requested via ``ray.autoscaler.sdk.request_resources`` (`GetClusterResourceConstraints <https://github.com/ray-project/ray/blob/03491225d59a1ffde99c3628969ccf456be13efd/src/ray/gcs/gcs_server/gcs_autoscaler_state_manager.cc#L245>`__).
doc/source/ray-core/internals/metric-exporter.rst
+188
View File
@@ -0,0 +1,188 @@
.. _metric-exporter:
Metric Exporter Infrastructure
================================
This document is based on `upstream/master` at commit `05e7efd5` (2025-12-17).
Ray's metric exporting infrastructure collects metrics from C++ components (raylet, GCS, workers) and Python components, aggregates them, and exports them to Prometheus. This document explains how metrics flow through the system from registration to final export.
Architecture Overview
---------------------
Ray's metric system uses a multi-stage pipeline:
1. **C++ Components**: Raylet, GCS, and worker processes record metrics using the OpenTelemetry SDK
2. **OTLP Export**: Metrics are exported via OpenTelemetry Protocol (OTLP) over gRPC to the metrics agent
3. **Metrics Agent**: The Python metrics agent (ReporterAgent) receives and processes metrics
4. **Aggregation**: High-cardinality labels are filtered and values are aggregated
5. **Prometheus Export**: Final metrics are exported in Prometheus format
The following diagram shows the high-level flow:
.. code-block:: text
C++ Components (raylet, GCS, workers)
↓ (Record metrics via Metric::Record)
OpenTelemetryMetricRecorder (C++)
↓ (OTLP gRPC export)
Metrics Agent (Python - ReporterAgent)
↓ (Aggregate & process)
OpenTelemetryMetricRecorder (Python)
↓ (Prometheus format)
Prometheus Server
Metric Registration and Recording (C++ Side)
---------------------------------------------
Ray's C++ components register and record metrics through the `OpenTelemetryMetricRecorder <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/src/ray/observability/open_telemetry_metric_recorder.h>`__ singleton. The recorder supports four metric types: Gauge, Counter, Sum, and Histogram.
Metric Types
~~~~~~~~~~~~
- **Gauge**: Represents a current value that can go up or down (e.g., number of running tasks)
- **Counter**: A cumulative metric that only increases (e.g., total tasks submitted)
- **Sum (UpDownCounter)**: A cumulative metric that can increase or decrease (e.g., number of objects in object store)
- **Histogram**: Tracks the distribution of values over time (e.g., task execution time)
Registration Process
~~~~~~~~~~~~~~~~~~~~
Metrics are registered lazily on first use. The `OpenTelemetryMetricRecorder` uses a singleton pattern accessible via `GetInstance() <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/src/ray/observability/open_telemetry_metric_recorder.cc#L78>`__. When a metric is first recorded, it's automatically registered if it hasn't been registered already.
Registration methods (defined in `open_telemetry_metric_recorder.cc <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/src/ray/observability/open_telemetry_metric_recorder.cc>`__):
- `RegisterGaugeMetric() <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/src/ray/observability/open_telemetry_metric_recorder.cc#L164>`__: Registers an observable gauge with a callback
- `RegisterCounterMetric() <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/src/ray/observability/open_telemetry_metric_recorder.cc#L203>`__: Registers a synchronous counter
- `RegisterSumMetric() <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/src/ray/observability/open_telemetry_metric_recorder.cc#L216>`__: Registers a synchronous up-down counter
- `RegisterHistogramMetric() <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/src/ray/observability/open_telemetry_metric_recorder.cc#L229>`__: Registers a histogram with explicit bucket boundaries
Recording Mechanisms
~~~~~~~~~~~~~~~~~~~~
Ray uses two different recording mechanisms depending on the metric type:
**Observable Metrics (Gauges)**
Observable gauges store values in an intermediate map (`observations_by_name_`) until collection time. When you call `SetMetricValue() <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/src/ray/observability/open_telemetry_metric_recorder.cc#L269>`__ for a gauge, the value is stored with its tags. During export, a callback function (`DoubleGaugeCallback <https://github.com/ray-project/ray/blob/52ed7e3/src/ray/observability/open_telemetry_metric_recorder.cc#L42>`__) is invoked by the OpenTelemetry SDK, which collects all stored values and clears the map to prevent stale data. The callback implementation is in `CollectGaugeMetricValues() <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/src/ray/observability/open_telemetry_metric_recorder.cc#L150>`__.
**Synchronous Metrics (Counters, Sums, Histograms)**
Synchronous metrics record values directly to their instruments without intermediate storage. When you call `SetMetricValue() <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/src/ray/observability/open_telemetry_metric_recorder.cc#L269>`__ for these types, the value is immediately added to the counter or recorded in the histogram via `SetSynchronousMetricValue() <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/src/ray/observability/open_telemetry_metric_recorder.cc#L292>`__.
Key Implementation Details
~~~~~~~~~~~~~~~~~~~~~~~~~~~
- **Thread Safety**: The recorder uses a mutex (`mutex_`) to protect the observations map and registered instruments
- **Lock Ordering**: Callbacks are registered after releasing the mutex to prevent deadlocks between the recorder's mutex and OpenTelemetry SDK's internal locks (see `RegisterGaugeMetric() <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/src/ray/observability/open_telemetry_metric_recorder.cc#L183-L195>`__ for details)
- **Lazy Registration**: Metrics can be registered multiple times safely; the recorder checks if a metric is already registered before creating a new instrument
C++ components record metrics through the `Metric::Record() <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/src/ray/stats/metric.cc#L111>`__ method, which forwards to `OpenTelemetryMetricRecorder::SetMetricValue() <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/src/ray/stats/metric.cc#L135>`__.
Metric Export from C++ (OTLP gRPC)
-----------------------------------
C++ components export metrics to the metrics agent using the OpenTelemetry Protocol (OTLP) over gRPC. The export process is configured when the recorder is started.
OpenTelemetry SDK Integration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The `OpenTelemetryMetricRecorder` initializes the OpenTelemetry SDK in its `constructor <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/src/ray/observability/open_telemetry_metric_recorder.cc#L129>`__ and `Start() <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/src/ray/observability/open_telemetry_metric_recorder.cc#L87>`__ method with:
- **MeterProvider**: Manages meter instances and metric readers
- **PeriodicExportingMetricReader**: Collects metrics at regular intervals and exports them
- **OTLP gRPC Exporter**: Sends metrics to the metrics agent endpoint
Export Configuration
~~~~~~~~~~~~~~~~~~~~~
When `Start() <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/src/ray/observability/open_telemetry_metric_recorder.cc#L87>`__ is called, the recorder configures:
- **Endpoint**: The metrics agent's gRPC address (typically `127.0.0.1:port`)
- **Export Interval**: How often metrics are collected and exported (configurable)
- **Export Timeout**: Maximum time to wait for export completion
- **Aggregation Temporality**: Set to delta mode to prevent double-counting (see `exporter_options.aggregation_temporality <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/src/ray/observability/open_telemetry_metric_recorder.cc#L97>`__)
Delta Aggregation Temporality
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Ray uses delta aggregation temporality, which means only the changes since the last export are sent. This is important because the metrics agent accumulates metrics, and re-accumulating them during export would lead to double-counting.
Export Process
~~~~~~~~~~~~~~
During each export interval:
1. **Observable Gauges**: The OpenTelemetry SDK invokes registered callbacks, which collect values from `observations_by_name_` and clear the map
2. **Synchronous Metrics**: Values are read directly from the instruments
3. **OTLP Format**: Metrics are converted to OTLP format
4. **gRPC Export**: Metrics are sent to the metrics agent via gRPC
Metric Reception and Processing (Python Side)
----------------------------------------------
The metrics agent (ReporterAgent) receives metrics from C++ components via a gRPC service that implements the OpenTelemetry Metrics Service interface.
gRPC Service Implementation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The `ReporterAgent <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/python/ray/dashboard/modules/reporter/reporter_agent.py>`__ class implements `MetricsServiceServicer`, which provides the `Export() <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/python/ray/dashboard/modules/reporter/reporter_agent.py#L662>`__ method. This method receives `ExportMetricsServiceRequest` messages containing OTLP-formatted metrics from C++ components.
Metric Processing
~~~~~~~~~~~~~~~~~
When metrics are received, the `Export()` method processes them in the following structure:
- **Resource Metrics**: Top-level container for metrics from a specific resource (e.g., a raylet process)
- **Scope Metrics**: Groups metrics by instrumentation scope
- **Metrics**: Individual metric data points
The method routes metrics to appropriate handlers based on their type:
- **Histogram Metrics**: Processed by `_export_histogram_data() <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/python/ray/dashboard/modules/reporter/reporter_agent.py#L577>`__
- **Number Metrics** (Gauge, Counter, Sum): Processed by `_export_number_data() <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/python/ray/dashboard/modules/reporter/reporter_agent.py#L628>`__
For histogram metrics, the metrics agent receives pre-aggregated OTLP bucket counts. The system reconstructs observations from bucket midpoints and records them with a single batch call to reduce lock contention.
Conversion to internal format
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The metrics agent converts OTLP format to Ray's internal metric representation and forwards them to the Python `OpenTelemetryMetricRecorder <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/python/ray/_private/telemetry/open_telemetry_metric_recorder.py>`__ for further processing and aggregation.
Metric Aggregation and Cardinality Reduction (Python)
------------------------------------------------------
The Python `OpenTelemetryMetricRecorder <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/python/ray/_private/telemetry/open_telemetry_metric_recorder.py>`__ handles final aggregation and cardinality reduction before exporting to Prometheus. This step is crucial for managing metric cardinality and preventing metric explosion.
OpenTelemetryMetricRecorder (Python)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The Python recorder (defined in `open_telemetry_metric_recorder.py <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/python/ray/_private/telemetry/open_telemetry_metric_recorder.py>`__) has a similar structure to the C++ version but uses the Prometheus exporter instead of OTLP. It maintains:
- **Registered Instruments**: Maps metric names to OpenTelemetry instruments
- **Observations Maps**: Stores gauge, counter, and sum observations (with their tag sets) until collection.
- **Histogram Bucket Midpoints**: Pre-calculated midpoints for histogram bucket conversion when reconstructing observations from OTLP bucket counts.
For gauges, counters, and sums, the recorder uses observable (asynchronous) instruments. Calls to `set_metric_value()` store values internally, and OpenTelemetry invokes callbacks at collection time to export aggregated observations. For histograms, OpenTelemetry doesn't support an observable histogram, so the recorder calls `record()` synchronously.
High-cardinality labels can cause metric explosion, making metrics systems unusable. Ray implements cardinality reduction through label filtering and value aggregation.
**Label Filtering**
The system identifies high-cardinality labels based on the `RAY_metric_cardinality_level` environment variable. The logic is implemented in `MetricCardinality.get_high_cardinality_labels_to_drop() <https://github.com/ray-project/ray/blob/05e7efd5ef71dca7a396e6b5f15c8ff16960c5db/python/ray/_private/telemetry/metric_cardinality.py#L80>`__:
- **`legacy`**: All labels are preserved (default behavior before Ray 2.53)
- **`recommended`**: The `WorkerId` label is dropped (default since Ray 2.53)
- **`low`**: Both `WorkerId` and `Name` labels are dropped for tasks and actors
**Aggregation process**
For observable gauges, counters, and sums, aggregation happens in the callback registered with the OpenTelemetry SDK in the Python recorder:
- **Collection**: The callback collects all observations for a metric from an internal observations map.
- **Label Filtering**: The callback drops high-cardinality labels from tag sets based on `MetricCardinality.get_high_cardinality_labels_to_drop()`.
- **Grouping**: The callback groups observations that share the same filtered tag set.
- **Aggregation**: The callback aggregates each group with `MetricCardinality.get_aggregation_function()`:
- For counters and sums, the system always aggregates by summing values.
- For gauges, the system aggregates task and actor metrics by summing values, and uses the first value for other metrics.
- **Export**: The callback returns aggregated observations to OpenTelemetry for Prometheus export.
This process ensures that metrics remain manageable even when there are thousands of workers or unique task names.
doc/source/ray-core/internals/object-spilling.rst
+677
View File
@@ -0,0 +1,677 @@
.. _object-spilling-internals:
Object Spilling
===============
This document explains how Ray's object spilling mechanism works and outlines its high-level architecture, components, and end-to-end data flow.
Overview
--------
Ray stores task outputs and ``ray.put()`` values as **objects** in the Plasma object store, a shared-memory region on each node. An object goes through the following lifecycle:
1. **Creation**: ``ray.put()`` or a task return triggers a ``Create`` RPC to the Plasma store, which allocates space in shared memory for the serialized object.
2. **Pinning**: After the object is created in Plasma, the CoreWorker sends a ``PinObjectIDs`` RPC to the local Raylet. This ensures the Raylet holds a reference to the object (preventing eviction) for as long as it may be needed (e.g., until it is consumed or spilled).
3. **Consumption**: Other tasks and ``ray.get()`` calls read the pinned object directly from shared memory via zero-copy access.
4. **Deletion**: When the object owner determines the object is no longer referenced, the Raylet unpins it and frees the shared memory.
This works well when the working set fits in memory. However, when the Plasma store is **full** and new objects need to be created, allocation fails — blocking ``ray.put()`` and task returns until space is freed.
**Object spilling** solves this by extending the object lifecycle with an external storage tier: when memory pressure is detected, pinned objects are automatically *spilled* from shared memory to external storage (local disk or S3). When a spilled object is needed again, it is transparently *restored* back into Plasma. This allows the effective object store capacity to exceed physical memory, at the cost of I/O latency for accessing spilled objects.
.. note::
Object spilling is fully abstracted away to user applications. No application-level code changes are needed to automatically spill pinned objects to disk when under plasma store memory pressure.
Architecture
------------
The object spilling architecture is designed to minimize interference with the critical path of task execution. It decouples **memory pressure detection** (which happens in the latency-sensitive Plasma store) from **spill orchestration** (managed by the Raylet) and **I/O execution** (offloaded to separate worker processes).
The system consists of three main interaction layers:
1. **Detection (Plasma Store Thread)**: The ``CreateRequestQueue`` within the Plasma store monitors memory usage. When an allocation fails (OOM), it triggers a callback to the Raylet. This ensures that the single-threaded object store is never blocked by I/O operations.
2. **Orchestration (Raylet Main Thread)**: The ``LocalObjectManager`` in the Raylet receives the spill request. It decides *what* to spill (based on LRU and pinning status) and *when* to spill (batching requests for efficiency). It manages the state of all local objects (Pinned, PendingSpill, Spilled).
3. **Execution (IO Worker Processes)**: Actual disk or network I/O is performed by a pool of Python ``IO Workers``. The Raylet communicates with these workers via gRPC. This separation ensures that even if I/O is slow (e.g., writing to S3), the Raylet's main loop remains responsive to other cluster events (heartbeats, scheduling).
The following diagram illustrates this layered architecture and the data flow:
.. image:: ../images/object_spilling_architecture.png
:alt: Object Spilling Architecture
..
Mermaid source (generate image from this):
flowchart TD
%% Node Definitions for Parallelism
A1["User Application 1"]
A2["User Application 2"]
B1["CoreWorker 1"]
B2["CoreWorker 2"]
%% Entry point connections
A1 -- "ray.put()" --> B1
A2 -- "ray.put()" --> B2
%% Main Logic paths (Parallel)
B1 -- "Step 1. Create" --> C["PlasmaStore"]
B2 -- "Step 1. Create" --> C
B1 -- "Step 2. Pin RPC" --> NM["NodeManager"]
B2 -- "Step 2. Pin RPC" --> NM
%% Alignment constraint
C ~~~ NM
%% Left: Memory Allocation Logic
subgraph PlasmaThread["Plasma Store Thread"]
C --> E["CreateRequestQueue<br/>ProcessRequests()"]
end
%% Right: Scheduling & Management
subgraph RayletThread["Raylet Main Thread"]
NM -- "PinObjectsAndWaitForFree()" --> F["LocalObjectManager"]
F -- "TryToSpillObjects()<br/>→ PopSpillWorker()" --> G["WorkerPool<br/>(IO Worker Pool)"]
end
%% Spilling Link (Cross-thread callback)
E -- "OOM: spill_objects_callback()<br/>→ main_service.post()" --> F
%% Parallel IO Workers
subgraph IOWorkerProcesses["Python IO Worker Processes"]
H1["Python IO Worker 1"]
H2["Python IO Worker 2"]
Hn["Python IO Worker N"]
end
G -- "gRPC" --> H1
G -- "gRPC" --> H2
G -- "gRPC" --> Hn
%% Storage Destination
H1 & H2 & Hn --> I[("External Storage<br/>(Filesystem / S3)")]
%% Styling
classDef memory fill:#e1f5fe,stroke:#01579b,stroke-width:2px;
classDef logic fill:#fff3e0,stroke:#e65100,stroke-width:2px;
classDef storage fill:#f1f8e9,stroke:#33691e,stroke-width:2px;
classDef core fill:#f3e5f5,stroke:#4a148c,stroke-width:2px;
class C,E memory;
class NM,F,G logic;
class H1,H2,Hn,I storage;
class A1,A2,B1,B2 core;
.. note::
The Plasma store and the Raylet main event loop run in **separate threads**. The spill callback bridges them by posting work from the store thread to the main thread. Only ``IsSpillingInProgress()`` is called cross-thread (using ``std::atomic``).
Primary vs. Secondary Copies
----------------------------
Ray distinguishes between two types of object copies in the cluster, which determines how they are handled under memory pressure:
- **Primary Copy**: The initial copy of an object, created by a task or ``ray.put``. The owner of the object (the CoreWorker that created it) manages its lifetime. The primary copy is the "source of truth" and cannot be evicted; it needs to be **spilled** to external storage if memory is needed.
- **Secondary Copy**: A copy of an object transferred to another node (e.g., as a dependency for a remote task or via ``ray.get``). These are treated as cached replicas.
In the context of spilling, "primary copy" and "pinned object" are closely related but distinct concepts:
* A **Primary Copy** is the initial object created by the owner. The owner explicitly registers it with the **Raylet's LocalObjectManager** (`source <https://github.com/ray-project/ray/blob/master/src/ray/raylet/local_object_manager.cc#L31>`__) (via ``PinObjectIDs``), making it a **pinned object** eligible for spilling.
* A **Secondary Copy** (cached replica) is pinned in the Plasma store only *while actively referenced* (`source <https://github.com/ray-project/ray/blob/master/src/ray/object_manager/plasma/eviction_policy.cc#L136>`__) (e.g., by a running task or a worker). It is **not managed by the LocalObjectManager** and is evicted by the Plasma Store's LRU policy (`source <https://github.com/ray-project/ray/blob/master/src/ray/object_manager/plasma/eviction_policy.cc#L82>`__) once the reference count drops.
Therefore, the Raylet's spilling mechanism **only** sees and operates on primary copies.
When memory pressure built up from objects being created or moved into the Plasma store, Ray prioritizes **evicting** secondary copies (which can be re-fetched from the primary) to free up space. If memory pressure persists, Ray then resorts to **spilling** primary copies to external storage.
Triggering Spilling
-------------------
Object spilling can be triggered by any operation that adds objects to the Plasma store. At the user API level, this includes:
- ``ray.put(obj)`` — explicitly places an object into the object store.
- **Task return values** — the return value of a remote task is serialized and stored in Plasma.
- **Object transfer** — when ``ray.get()`` fetches a remote object, the object is copied into the local Plasma store on the receiving node.
Internally, there are **three code paths** that trigger spilling. The first is *reactive* — spilling is triggered because allocation has already failed. The other two are *proactive* — they check a memory threshold and spill preemptively to avoid OOM in the first place.
.. list-table::
:widths: 15 30 55
:header-rows: 1
* - Trigger
- When it fires
- Condition
* - **OOM on Create**
- A ``Create`` RPC to Plasma fails with ``OutOfMemory`` (`source <https://github.com/ray-project/ray/blob/master/src/ray/object_manager/plasma/create_request_queue.cc#L117>`__)
- Reactive: allocation already failed, must spill to make room
* - **Periodic threshold**
- Every ``free_objects_period_milliseconds`` (default 1000 ms) (`source <https://github.com/ray-project/ray/blob/master/src/ray/raylet/node_manager.cc#L427>`__)
- Proactive: primary object bytes / capacity >= ``object_spilling_threshold`` (default 0.8)
* - **Object sealed**
- Whenever a new object is sealed in Plasma (`SealObjects <https://github.com/ray-project/ray/blob/master/src/ray/object_manager/plasma/store.cc#L278>`__ → `HandleObjectLocal <https://github.com/ray-project/ray/blob/master/src/ray/raylet/node_manager.cc#L2451>`__)
- Proactive: same threshold check as above, triggered immediately on the new object
All three paths converge on ``LocalObjectManager::SpillObjectUptoMaxThroughput()``.
Reactive: OOM on Object Creation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When the Plasma store cannot allocate space for a new object, the `CreateRequestQueue <https://github.com/ray-project/ray/blob/master/src/ray/object_manager/plasma/create_request_queue.h#L34>`__ manages the queued request and kicks off a recovery sequence. The key decision logic lives in `ProcessRequests <https://github.com/ray-project/ray/blob/master/src/ray/object_manager/plasma/create_request_queue.cc#L85>`__:
1. **Try to allocate** the object in shared memory.
2. If allocation fails with ``OutOfMemory`` and the disk is full (checked via ``FileSystemMonitor``), return ``OutOfDisk`` immediately — there is nowhere to spill to.
3. **Trigger global GC** if configured — this may free Python-side references, allowing Plasma objects to be unpinned.
4. **Call** ``spill_objects_callback_()``. This callback is registered in `main.cc <https://github.com/ray-project/ray/blob/master/src/ray/raylet/main.cc#L752>`__ and runs **on the Plasma store thread**. It does two things:
.. code-block:: cpp
/*spill_objects_callback=*/
[&]() {
// 1) Post spill task to Raylet main thread (non-blocking, enqueue only)
main_service.post(
[&]() { local_object_manager->SpillObjectUptoMaxThroughput(); },
"NodeManager.SpillObjects");
// 2) Return whether spilling is active (std::atomic, safe to read cross-thread)
return local_object_manager->IsSpillingInProgress();
}
Based on the return value, ``CreateRequestQueue`` decides what to do next:
- ``true`` (spilling is in progress): The **LocalObjectManager** has identified eligible **pinned primary copies** (objects with reference count == 1, meaning only the owner holds a reference and no task is actively using it, see `PlasmaStore::IsObjectSpillable <https://github.com/ray-project/ray/blob/master/src/ray/object_manager/plasma/store.cc#L560>`__) and spill workers are actively writing them to external storage.
- ``false`` (no active spills): The **LocalObjectManager** has no ongoing spills. This occurs if:
* No eligible objects were found (e.g., all pinned objects are currently **in use** by running tasks).
* The total size of spillable objects is too small (below ``min_spilling_size``) to justify an immediate spill, so Ray waits to batch more objects.
* Spilling is disabled in the configuration.
In this case, the queue enters the **grace period** (``oom_grace_period_s``). During the grace period, retries continue — this accounts for global GC latency and the delay between spilling completing and space actually being freed in the object store.
5. If the **grace period expires** without progress, try the **fallback allocator** as a last resort. The fallback allocator uses ``mmap`` to allocate the object directly on the local filesystem instead of shared memory — this is slower but avoids blocking the caller indefinitely. If that also fails (e.g. disk full), return ``OutOfDisk``.
The Plasma store retries ``ProcessCreateRequests()`` periodically (controlled by ``delay_on_oom_ms``) as long as the queue is non-empty and status is not OK. See `PlasmaStore::ProcessCreateRequests <https://github.com/ray-project/ray/blob/master/src/ray/object_manager/plasma/store.cc#L508>`__.
Proactive: Threshold-Based Spilling
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The reactive OOM path only fires *after* the store is already full. To avoid hitting that cliff, Ray also **proactively** spills objects before the store is full. `NodeManager::SpillIfOverPrimaryObjectsThreshold <https://github.com/ray-project/ray/blob/master/src/ray/raylet/node_manager.cc#L2400>`__ checks whether the fraction of primary object bytes in the store exceeds ``object_spilling_threshold`` (default 0.8), and if so, calls ``SpillObjectUptoMaxThroughput()``.
This check is invoked from two places:
1. **Periodic timer** (`node_manager.cc <https://github.com/ray-project/ray/blob/master/src/ray/raylet/node_manager.cc#L427>`__): runs every ``free_objects_period_milliseconds`` (default 1000 ms). This is the steady-state proactive spilling path — even without any new object creation, the system periodically checks and spills if needed.
2. **Object sealed event** (`HandleObjectLocal <https://github.com/ray-project/ray/blob/master/src/ray/raylet/node_manager.cc#L2451>`__): every time a new object is sealed in Plasma (via `SealObjects <https://github.com/ray-project/ray/blob/master/src/ray/object_manager/plasma/store.cc#L278>`__), the ``add_object_callback_`` posts ``HandleObjectLocal()`` to the Raylet main thread, which calls ``SpillIfOverPrimaryObjectsThreshold()`` at the end. This ensures that spilling reacts immediately when a large object pushes memory usage over the threshold, rather than waiting up to 1 second for the next periodic check.
Alternatives to Spilling
------------------------
Ray includes other mechanisms to handle memory pressure aside from spilling:
1. **Eviction (Secondary Copies)**: As mentioned above, Ray creates replicas of objects on other nodes when they are needed for tasks or ``ray.get``. These **secondary copies** are evictable. When the object store is full, Ray deletes these copies (LRU) to free space before attempting to spill primary objects.
2. **Fallback Allocation (Mmap)**: If spilling is too slow or the object store is fragmented, Ray may use **fallback allocation**. This occurs when a create request fails with OOM even after attempting to spill. The object is allocated directly on the filesystem (using ``mmap``) rather than in the shared memory pool. This avoids application deadlock but offers lower performance than shared memory.
Object Pinning
--------------
Before objects can be spilled, they must be *pinned* by the Raylet. Pinning ensures the Raylet holds a reference to the object so it is not prematurely evicted from the object store.
When the CoreWorker creates an object in Plasma, it sends a ``PinObjectIDs`` RPC to the Raylet. The Raylet's `HandlePinObjectIDs <https://github.com/ray-project/ray/blob/master/src/ray/raylet/node_manager.cc#L2588>`__ fetches the objects from Plasma and calls `PinObjectsAndWaitForFree <https://github.com/ray-project/ray/blob/master/src/ray/raylet/local_object_manager.cc#L31>`__, which:
1. Stores object metadata (owner address, generator ID, size) in ``local_objects_``.
2. Holds the ``std::unique_ptr<RayObject>`` in ``pinned_objects_``, preventing Plasma eviction.
3. Subscribes to eviction notifications via pub/sub: when the object owner says the object can be freed (or the owner process dies), ``ReleaseFreedObject()`` is called.
Every object tracked by ``LocalObjectManager`` is registered in ``local_objects_`` (metadata map) and simultaneously in exactly one of three sub-maps, corresponding to its current state:
.. list-table::
:widths: 20 30 50
:header-rows: 1
* - State
- Sub-map
- Meaning
* - **Pinned**
- ``pinned_objects_``
- Object is held in shared memory, eligible for spilling
* - **PendingSpill**
- ``objects_pending_spill_``
- Object has been handed to an IO worker, spill in progress
* - **Spilled**
- ``spilled_objects_url_``
- Object has been written to external storage, in-memory copy released
When an object is **deleted** (freed by owner), it is removed from ``local_objects_`` and its corresponding sub-map — it is no longer tracked by ``LocalObjectManager``.
.. image:: ../images/object_spilling_states.png
:alt: Object State Transitions
..
Mermaid source (generate image from this):
stateDiagram-v2
[*] --> Pinned : PinObjectsAndWaitForFree()
Pinned --> PendingSpill : SpillObjectsInternal()
PendingSpill --> Spilled : OnObjectSpilled()
PendingSpill --> Pinned : Spill failed (rollback)
Pinned --> [*] : ReleaseFreedObject()<br/>(unpin, remove from local_objects_)
PendingSpill --> [*] : ReleaseFreedObject()<br/>(deferred to spill completion)
Spilled --> [*] : ProcessSpilledObjectsDeleteQueue()<br/>(decrement url_ref_count)
Spill Scheduling
----------------
The `LocalObjectManager <https://github.com/ray-project/ray/blob/master/src/ray/raylet/local_object_manager.h#L46>`__ orchestrates all spill operations.
Strategy: Optimistic Batching
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The core tension in spill scheduling is between **latency** and **efficiency**:
- Spilling should start as soon as possible when memory is under pressure — delaying risks blocking object creation.
- But fusing multiple objects into a single spill file amortizes I/O overhead (fewer syscalls, sequential writes), so larger batches are more efficient.
Ray resolves this with an **optimistic batching** strategy: spill immediately with whatever objects are available, but defer tiny batches when other spills are already in flight. The reasoning is:
- In-flight spills will soon free memory, reducing the urgency.
- Waiting gives time for more objects to accumulate, improving the next batch's efficiency.
- When **no** spills are in progress, even a small batch is dispatched immediately — there is nothing to wait for.
The entry point is `SpillObjectUptoMaxThroughput <https://github.com/ray-project/ray/blob/master/src/ray/raylet/local_object_manager.cc#L169>`__, called when memory pressure is detected. It aggressively tries to saturate all available IO workers by calling ``TryToSpillObjects()`` in a loop until either no more objects can be spilled or all workers are busy (``num_active_workers_ >= max_active_workers_``).
Batch Construction
~~~~~~~~~~~~~~~~~~
`TryToSpillObjects <https://github.com/ray-project/ray/blob/master/src/ray/raylet/local_object_manager.cc#L186>`__ constructs a single spill batch. It iterates through ``pinned_objects_``, skipping objects that are not currently spillable (``is_plasma_object_spillable_`` checks that the object is not actively used by a worker process), and accumulates candidates until one of the following limits is reached:
- ``max_fused_object_count_`` objects have been collected, **or**
- ``max_spilling_file_size_bytes_`` would be exceeded by adding the next object (when enabled, i.e. > 0; the first object is always included even if it alone exceeds the limit), **or**
- all pinned objects have been checked.
Deferral Decision
~~~~~~~~~~~~~~~~~
After constructing the candidate batch, ``TryToSpillObjects`` decides whether to spill now or defer. Spilling is **deferred** (returns ``false``) when **all three** of the following conditions hold simultaneously (see `source <https://github.com/ray-project/ray/blob/master/src/ray/raylet/local_object_manager.cc#L220>`__):
1. The scan visited all pinned objects without hitting ``max_fused_object_count_`` — the candidate batch is small, not limited by fusion constraints.
2. The total bytes to spill (``bytes_to_spill``) is below ``min_spilling_size_``.
3. There are already objects being spilled (``objects_pending_spill_`` is non-empty).
In other words: the batch is small, smaller than the minimum threshold, and other spills are already making progress — so waiting is safe. If **any** condition is not met — the batch hit a fusion limit (indicating enough objects to justify a spill), or the batch is large enough, or no other spills are in progress — spilling proceeds immediately.
Once the decision is to spill, ``SpillObjectsInternal()`` is called with the selected batch.
SpillObjectsInternal
~~~~~~~~~~~~~~~~~~~~
`SpillObjectsInternal <https://github.com/ray-project/ray/blob/master/src/ray/raylet/local_object_manager.cc#L282>`__ performs the actual spill:
1. Filters out objects that have already been freed or are already pending spill.
2. Moves objects from ``pinned_objects_`` to ``objects_pending_spill_`` (updates size counters).
3. Increments ``num_active_workers_`` and pops a spill worker from the IO worker pool.
4. Constructs a ``SpillObjectsRequest`` RPC with object refs and owner addresses, then sends it to the IO worker.
5. On RPC response: moves failed objects back to ``pinned_objects_``, and calls ``OnObjectSpilled()`` for successful ones.
.. note::
Spilling is ordered: if object N succeeds, all objects before N in the request are guaranteed to have succeeded as well. Failed objects (from the first failure onward) are moved back to pinned state.
Python IO Workers and External Storage
---------------------------------------
IO workers are specialized Python processes that perform the actual I/O operations. They are spawned and managed by the `WorkerPool <https://github.com/ray-project/ray/blob/master/src/ray/raylet/worker_pool.h#L280>`__.
The ``WorkerPool`` manages all worker processes on a node, including regular task workers, driver processes, and IO workers. Regular workers and IO workers are tracked in separate data structures: regular task workers go into the ``State::idle`` set, while IO workers have their own dedicated ``IOWorkerState`` (`source <https://github.com/ray-project/ray/blob/master/src/ray/raylet/worker_pool.h#L619>`__), each of which maintains an ``idle_io_workers`` set, a ``pending_io_tasks`` queue, and a ``started_io_workers`` count. This separation ensures that IO workers are never assigned regular tasks and vice versa.
Worker Types and Pool Management
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
IO workers handle three types of operations:
- **Spill**: write objects from Plasma to external storage.
- **Restore**: read a previously spilled object from external storage back into Plasma.
- **Delete**: remove spill files from external storage when all objects within them have gone out of scope.
Each operation type maps to an ``IOWorkerState``:
- ``SPILL_WORKER`` (`spill_io_worker_state <https://github.com/ray-project/ray/blob/master/src/ray/raylet/worker_pool.h#L658>`__): a dedicated pool that handles ``SpillObjects`` RPCs.
- ``RESTORE_WORKER`` (`restore_io_worker_state <https://github.com/ray-project/ray/blob/master/src/ray/raylet/worker_pool.h#L660>`__): a dedicated pool that handles ``RestoreSpilledObjects`` RPCs.
- Delete operations do not have a dedicated pool. `PopDeleteWorker <https://github.com/ray-project/ray/blob/master/src/ray/raylet/worker_pool.cc#L1063>`__ compares the number of idle workers in the spill and restore pools, and borrows a worker from whichever pool has more idle workers available. After the delete completes, the worker is returned to its original pool.
When `PopSpillWorker <https://github.com/ray-project/ray/blob/master/src/ray/raylet/worker_pool.cc#L990>`__ (or ``PopRestoreWorker``) is called:
- If an idle IO worker of the matching type is available in its ``idle_io_workers`` set, it is returned immediately via the callback.
- If no idle IO workers exist, the callback is queued in ``pending_io_tasks`` and `TryStartIOWorkers <https://github.com/ray-project/ray/blob/master/src/ray/raylet/worker_pool.cc#L1750>`__ spawns new Python IO worker processes up to ``max_io_workers`` per type.
When the operation completes, the worker is returned to its pool via ``PushSpillWorker`` (or ``PushRestoreWorker``). If pending tasks are queued, the worker is immediately assigned to the next task instead of going idle.
Object Fusion Format
~~~~~~~~~~~~~~~~~~~~
Rather than writing each object to its own file, multiple objects from a single spill batch are **fused** into a single file. This reduces I/O overhead (fewer ``open``/``close`` syscalls) and filesystem fragmentation. The `_write_multiple_objects <https://github.com/ray-project/ray/blob/master/python/ray/_private/external_storage.py#L133>`__ method writes objects sequentially, each prefixed with a 24-byte header:
.. code-block:: text
┌──────────────────────────────────────────────────────┐
│ Spill File │
│ │
│ Object 1: │
│ ┌──────────┬──────────────┬──────────┐ │
│ │ addr_len │ metadata_len │ buf_len │ (24 bytes) │
│ │ (8 bytes)│ (8 bytes) │ (8 bytes)│ │
│ ├──────────┴──────────────┴──────────┤ │
│ │ owner_address │ metadata │ buffer │ │
│ └───────────────┴──────────┴─────────┘ │
│ │
│ Object 2: [same format...] │
│ ... │
└──────────────────────────────────────────────────────┘
The header's three 8-byte fields (``addr_len``, ``metadata_len``, ``buf_len``) encode the sizes of the three variable-length sections that follow: the serialized owner address (needed for ``ReportObjectSpilled``), the object metadata, and the object data buffer. During restore, the IO worker reads this header first to determine how many bytes to read for each section.
Since multiple objects share a single file, each individual object needs to be addressable independently — for example, object 3 in a fused file might be restored while objects 1 and 2 are still alive. Ray solves this with a **spill URL** that encodes the object's position within the file:
.. code-block:: text
/tmp/ray/spill/ray_spilled_objects_<node_id>/<uuid>-multi-<count>?offset=<N>&size=<M>
The URL has two parts:
- **Base URL** (the path before ``?``): identifies the spill file. This is the same for all objects fused into the same file. It is also the key used by ``url_ref_count_`` to track how many live objects reference the file — the file is only deleted when this count reaches zero.
- **Query parameters** (``offset`` and ``size``): the byte offset and total size (header + data) of this specific object within the file. The restore IO worker seeks to the offset and reads exactly ``size`` bytes.
This URL is stored in ``spilled_objects_url_`` on the spilling node and reported to the object directory via ``ReportObjectSpilled()``, making it discoverable by any node in the cluster that needs to restore the object.
Storage Backends
~~~~~~~~~~~~~~~~
The `ExternalStorage <https://github.com/ray-project/ray/blob/master/python/ray/_private/external_storage.py#L72>`__ abstract class defines the interface for all backends. Two production implementations are provided:
- `FileSystemStorage <https://github.com/ray-project/ray/blob/master/python/ray/_private/external_storage.py#L271>`__ (default): writes to local filesystem. Supports multiple directories with round-robin distribution for I/O parallelism across mount points. Files are named ``{directory}/ray_spilled_objects_{node_id}/{uuid}-multi-{count}``.
- `ExternalStorageSmartOpenImpl <https://github.com/ray-project/ray/blob/master/python/ray/_private/external_storage.py#L398>`__: uses the ``smart_open`` library for cloud storage (S3, GCS, etc.). Reuses boto3 sessions and uses deferred seek for performance.
The backend is selected by `setup_external_storage <https://github.com/ray-project/ray/blob/master/python/ray/_private/external_storage.py#L577>`__ based on the ``object_spilling_config`` JSON configuration.
Post-Spill Processing
---------------------
After the IO worker successfully writes objects to external storage, `OnObjectSpilled <https://github.com/ray-project/ray/blob/master/src/ray/raylet/local_object_manager.cc#L399>`__ is called for each spilled object:
1. Parses the returned URL to extract the ``base_url`` (the file path without offset/size query parameters).
2. Increments ``url_ref_count_[base_url]``. Since multiple objects can be fused into one file, this ref count tracks how many live objects reference each file.
3. Records the ``object_id → url_with_offset`` mapping in ``spilled_objects_url_``.
4. Removes the object from ``objects_pending_spill_`` (releases the in-memory copy).
5. Updates spill metrics (``spilled_bytes_total_``, ``spilled_objects_total_``, etc.).
6. If the object has not already been freed, reports the spilled URL to the object owner via ``object_directory_->ReportObjectSpilled()`` so that other nodes in the cluster can locate the spilled object.
Object Restore
--------------
When a spilled object is needed again, it must be restored back into the Plasma store (or streamed directly over the network) before it can be used. Restore is triggered whenever a node determines that a required object is not available in any node's in-memory Plasma store but has a known spilled URL.
Triggering Restore
~~~~~~~~~~~~~~~~~~
At the user API level, the following operations can trigger a restore if the referenced object has been spilled:
- ``ray.get(ref)`` — blocking get on a spilled object.
- ``ray.wait(refs)`` — waiting for spilled objects to become available.
- **Task scheduling** — a task's input arguments are spilled; the scheduler must restore them before the task can run.
- **Actor task arguments** — an actor receives a task whose arguments are spilled.
- ``await ref`` — async Python get on a spilled object.
All of these operations go through the same internal path: the Raylet's `LeaseDependencyManager <https://github.com/ray-project/ray/blob/master/src/ray/raylet/lease_dependency_manager.cc>`__ issues an ``ObjectManager::Pull`` request for the missing object. The `PullManager <https://github.com/ray-project/ray/blob/master/src/ray/object_manager/pull_manager.cc#L446>`__ then consults the object directory for the object's location. If the object has been spilled, the directory returns the spilled URL (reported earlier by ``OnObjectSpilled`` → ``ReportObjectSpilled``), and the ``PullManager`` decides how to restore it based on the storage backend.
Additionally, a **periodic retry timer** (`ObjectManager::Tick <https://github.com/ray-project/ray/blob/master/src/ray/object_manager/object_manager.cc#L828>`__) re-evaluates all active pull requests, retrying restores that previously failed.
Two Restore Paths
~~~~~~~~~~~~~~~~~
The restore path depends on the storage backend:
**Filesystem storage** (spilled to local disk): the spill file only exists on the node that spilled the object. If the requesting node is the same node, it restores locally via `AsyncRestoreSpilledObject <https://github.com/ray-project/ray/blob/master/src/ray/raylet/local_object_manager.cc#L464>`__. If the requesting node is a *different* node, it sends a pull request to the spilling node; the spilling node reads the object directly from disk and streams it over the network via `PushFromFilesystem <https://github.com/ray-project/ray/blob/master/src/ray/object_manager/object_manager.cc#L409>`__ — **without** restoring the object into its own Plasma store. This avoids unnecessary memory pressure on the spilling node.
**Cloud storage** (S3, GCS, etc.): the spill file is accessible from any node. The requesting node restores the object locally via ``AsyncRestoreSpilledObject`` using the cloud URL directly. No cross-node RPC is needed.
Restore Mechanics
~~~~~~~~~~~~~~~~~
`AsyncRestoreSpilledObject <https://github.com/ray-project/ray/blob/master/src/ray/raylet/local_object_manager.cc#L464>`__ performs the local restore:
1. **Deduplication**: if the same object is already being restored (``objects_pending_restore_`` contains the ID), the call is a no-op to avoid duplicate restores.
2. Pops a restore worker from the IO worker pool.
3. Sends a ``RestoreSpilledObjectsRequest`` RPC with the spilled URL and object ID.
4. The Python IO worker reads the file at the specified offset, parses the 24-byte header, and puts the object back into the Plasma store via ``core_worker.put_file_like_object()``.
5. On completion, updates restore metrics and invokes the callback.
Object Deletion and Cleanup
----------------------------
Object deletion is a two-phase process that handles the complexity of objects being freed while they are still being spilled, and multiple objects sharing a single spill file.
Phase 1: Marking Objects as Freed
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When the object owner frees the object (via pub/sub eviction notification or owner death), `ReleaseFreedObject <https://github.com/ray-project/ray/blob/master/src/ray/raylet/local_object_manager.cc#L111>`__ is called:
1. Marks ``local_objects_[id].is_freed_ = true``.
2. If the object is **pinned**: removes it from ``pinned_objects_`` and erases the ``local_objects_`` entry immediately.
3. If the object is **being spilled or already spilled**: pushes it onto ``spilled_object_pending_delete_`` for deferred cleanup (cannot delete while spilling is in progress).
4. Adds the object ID to ``objects_pending_deletion_`` for batch eviction from Plasma across the cluster.
Phase 2: Batch Cleanup of Spilled Files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
`ProcessSpilledObjectsDeleteQueue <https://github.com/ray-project/ray/blob/master/src/ray/raylet/local_object_manager.cc#L523>`__ drains the ``spilled_object_pending_delete_`` queue up to a batch size limit. For each object:
1. If the object is **still being spilled** (``objects_pending_spill_`` contains the ID): **break out of the loop entirely** — the queue is FIFO, so no subsequent entries are processed either. The object is not removed from the queue; it remains at the front and will be retried next time the function is called. This is a deliberate simplification: deletion is low priority compared to spilling, and the spill will eventually complete, at which point the next call will make progress. No data structures are modified for this object.
2. If the object **has a spilled URL** (found in ``spilled_objects_url_``): parse the URL to extract the ``base_url`` (the file path without offset/size query parameters) and decrement ``url_ref_count_[base_url]``. If the ref count reaches zero, add the URL to the list of files to delete and remove the ref count entry. Remove the object from ``spilled_objects_url_`` and ``local_objects_``.
3. If the object **does not have a spilled URL** (it was freed while still being spilled, and the spill has since completed without recording a URL for it — or was rolled back): remove it from ``pinned_objects_`` (if present) and ``local_objects_`` to prevent a memory leak.
**Note on Delete Workers**: Ray does not maintain a dedicated pool of workers for deleting spilled objects. Instead, deletion tasks **borrow** an idle worker from either the **spill** or **restore** worker pools. The ``WorkerPool`` dynamically selects a worker from the pool with more idle capacity (to minimize impact on critical path operations, see `WorkerPool::PopDeleteWorker <https://github.com/ray-project/ray/blob/master/src/ray/raylet/worker_pool.cc#L1071>`__). Once the delete operation completes, the worker is returned to its original pool.
After processing the queue, if there are any URLs to delete (i.e., one or more files have had their ref counts drop to zero), `DeleteSpilledObjects <https://github.com/ray-project/ray/blob/master/src/ray/raylet/local_object_manager.cc#L579>`__ is called. This function pops a delete worker from the IO worker pool and sends a ``DeleteSpilledObjectsRequest`` RPC containing the list of URLs to delete. The delete worker receives the full list of file URLs and deletes each one — for filesystem storage, this is a simple ``os.remove(path)`` call per file (`source <https://github.com/ray-project/ray/blob/master/python/ray/_private/external_storage.py#L364>`__). The decision of *which* files to delete has already been made by the C++ side (via ref counting); the Python IO worker unconditionally deletes every URL it receives. If the RPC fails (e.g., worker crash), the entire batch is retried up to 3 times.
.. note::
The URL ref counting mechanism is critical for correctness: since multiple objects can be fused into a single file, the file must not be deleted until **all** objects within it have gone out of scope.
Complete Lifecycle
------------------
The following sequence diagram shows the end-to-end interactions between components during spill, restore, and delete:
**Spill Path** (triggered by memory pressure):
.. image:: ../images/object_spilling_spill_sequence.png
:alt: Spill Path Sequence Diagram
..
Mermaid source (generate image from this):
sequenceDiagram
participant App as User Application
participant CW as CoreWorker
participant PS as PlasmaStore<br/>(store thread)
participant CRQ as CreateRequestQueue
participant LOM as LocalObjectManager<br/>(main thread)
participant WP as WorkerPool
participant IO as Python IO Worker
participant FS as External Storage
App->>CW: ray.put(obj)
CW->>PS: Create(object_id, size)
PS->>CRQ: ProcessRequests()
alt Space available
CRQ-->>PS: OK
PS-->>CW: PlasmaObject
else OutOfMemory
CRQ->>CRQ: trigger_global_gc_()
CRQ->>LOM: spill_objects_callback_()<br/>[post to main_service]
LOM->>LOM: SpillObjectUptoMaxThroughput()
LOM->>LOM: TryToSpillObjects()<br/>[batch by size/count]
LOM->>LOM: SpillObjectsInternal()<br/>[pinned → pending_spill]
LOM->>WP: PopSpillWorker()
WP-->>LOM: io_worker
LOM->>IO: SpillObjects RPC<br/>[object_refs + owner_addrs]
IO->>FS: Write fused objects to file
FS-->>IO: file path
IO-->>LOM: spilled_objects_urls
LOM->>LOM: OnObjectSpilled()<br/>[pending_spill → spilled_url]<br/>[update url_ref_count]
LOM->>CW: ReportObjectSpilled()<br/>[notify owner]
LOM-->>CRQ: IsSpillingInProgress() = true
Note over CRQ: Retry ProcessRequests()<br/>after delay_on_oom_ms
end
**Restore Path** (spilled object needed again):
.. image:: ../images/object_spilling_restore_sequence.png
:alt: Restore Path Sequence Diagram
..
Mermaid source (generate image from this):
sequenceDiagram
participant Task as Task / ray.get()
participant OM as ObjectManager
participant OD as ObjectDirectory
participant LOM as LocalObjectManager<br/>(main thread)
participant WP as WorkerPool
participant IO as Python IO Worker
participant FS as External Storage
Task->>OM: Request object
OM->>OD: Lookup object location
OD-->>OM: spilled_url
OM->>LOM: AsyncRestoreSpilledObject()<br/>[object_id, url]
LOM->>LOM: Dedup check<br/>[objects_pending_restore_]
LOM->>WP: PopRestoreWorker()
WP-->>LOM: io_worker
LOM->>IO: RestoreSpilledObjects RPC
IO->>FS: Read file at offset
IO->>IO: Parse header (24 bytes)<br/>[addr_len, metadata_len, buf_len]
IO->>IO: put_file_like_object()<br/>[back into Plasma]
IO-->>LOM: bytes_restored_total
LOM-->>Task: Object available in Plasma
**Delete Path** (object goes out of scope):
.. image:: ../images/object_spilling_delete_sequence.png
:alt: Delete Path Sequence Diagram
..
Mermaid source (generate image from this):
sequenceDiagram
participant Owner as Object Owner
participant LOM as LocalObjectManager<br/>(main thread)
participant WP as WorkerPool
participant IO as Python IO Worker
participant FS as External Storage
Owner->>LOM: PubSub: object eviction<br/>(or owner death)
LOM->>LOM: ReleaseFreedObject()<br/>[is_freed_ = true]
alt Object is PINNED
LOM->>LOM: Unpin immediately<br/>[remove from pinned_objects_]
else Object is SPILLED / PENDING_SPILL
LOM->>LOM: Push to<br/>spilled_object_pending_delete_
end
LOM->>LOM: Batch: objects_pending_deletion_
LOM->>LOM: FlushFreeObjects()
Note over LOM: ProcessSpilledObjectsDeleteQueue()
LOM->>LOM: url_ref_count_[base_url] -= 1
alt ref_count == 0
LOM->>WP: PopDeleteWorker()
WP-->>LOM: io_worker
LOM->>IO: DeleteSpilledObjects RPC
IO->>FS: os.remove(file)<br/>[retry up to 3x on failure]
else ref_count > 0
Note over LOM: File still has live objects,<br/>skip deletion
end
Configuration
-------------
Object spilling is controlled by the following configuration parameters:
- ``object_spilling_config``: JSON string specifying the storage backend. Empty string disables spilling.
- ``object_spilling_threshold``: fraction (0.0–1.0) of available object store memory at which spilling begins. Default: ``0.8``.
- ``min_spilling_size``: minimum bytes to accumulate before triggering a spill batch.
- ``max_spilling_file_size_bytes``: maximum bytes allowed in a single fused spill file. When enabled (> 0), ``TryToSpillObjects`` stops fusing objects once adding the next object would exceed this limit (the first object is always included). Must be >= ``min_spilling_size`` when enabled. Set to ``-1`` (default) to disable.
- ``max_fused_object_count``: maximum number of objects fused into a single spill file. Default: ``2000``.
- ``max_io_workers``: maximum number of concurrent spill/restore IO worker processes.
- ``oom_grace_period_s``: seconds to wait after OOM before using the fallback allocator.
- ``free_objects_batch_size``: number of freed objects to batch before flushing.
- ``free_objects_period_milliseconds``: interval for flushing freed objects.
- ``verbose_spill_logs``: byte threshold for error-level spill log messages (uses exponential backoff).
Example configuration:
.. code-block:: python
import json
import ray
ray.init(
_system_config={
"object_spilling_config": json.dumps({
"type": "filesystem",
"params": {
"directory_path": ["/mnt/ssd1/spill", "/mnt/ssd2/spill"],
"buffer_size": 1048576,
}
}),
"min_spilling_size": 100 * 1024 * 1024, # 100 MB
"max_spilling_file_size_bytes": 1024 * 1024 * 1024, # 1 GB cap per file
"max_io_workers": 4,
}
)
Key Source Files
----------------
.. list-table::
:widths: 40 60
:header-rows: 1
* - File
- Role
* - ``src/ray/object_manager/plasma/create_request_queue.cc``
- Decides when to trigger spilling on OOM
* - ``src/ray/object_manager/plasma/store.cc``
- Plasma store; retries create requests periodically
* - ``src/ray/raylet/main.cc``
- Wires up the spill callback between Plasma and LocalObjectManager
* - ``src/ray/raylet/node_manager.cc``
- Handles ``PinObjectIDs`` RPC; integrates LocalObjectManager
* - ``src/ray/raylet/local_object_manager.h``
- Class definition, state tracking, and member variables
* - ``src/ray/raylet/local_object_manager.cc``
- Spill/restore/delete orchestration logic
* - ``src/ray/raylet/worker_pool.cc``
- IO worker pool management (pop/push/start workers)
* - ``python/ray/_private/external_storage.py``
- Storage backends (FileSystemStorage, SmartOpenImpl)
doc/source/ray-core/internals/port-service-discovery.rst
+118
View File
@@ -0,0 +1,118 @@
.. _ray-port-service-discovery:
Port Service Discovery
======================
This document describes Ray's dynamic port assignment and discovery.
Design Principle
----------------
When a port is not explicitly specified by the user or start script, Ray uses a
**bind-then-report** pattern: components first bind to a random port, then report
the actual port.
This avoids the TOCTOU (Time-Of-Check-Time-Of-Use) race condition: if Ray preassigned
a port and passed it to a component, another process might bind to that port before
the component does.
Two-Layer Discovery
-------------------
Ray uses two discovery mechanisms based on process relationships:
.. list-table::
:header-rows: 1
* - Layer
- Scope
- Mechanism
* - **Raylet Internal**
- Raylet discovers child process ports
- File-based (Agents) or IPC (Workers)
* - **GCS**
- All other Ray components
- Node Table / Actor Table
Raylet Internal Port Discovery
------------------------------
Raylet spawns child processes and needs to discover their ports. The mechanism
depends on the language boundary.
File-Based: Raylet ↔ Agents
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Raylet (C++) spawns Agents (Python). Cross-language communication is needed, and
files are the simplest solution that works reliably across both platforms (Linux/Windows)
and languages (C++/Python). Raylet spawns two agents:
1. **Dashboard Agent** - exposes three ports:
- ``dashboard_agent_listen_port``: HTTP for Dashboard UI (default: 52365)
- ``metrics_agent_port``: gRPC for internal communication (default: random)
- ``metrics_export_port``: Prometheus metrics export (default: random)
2. **Runtime Env Agent** - manages runtime environments:
- ``runtime_env_agent_port``: gRPC (default: random)
After binding, agents write their ports to
``{session_dir}/{port_name}_{node_id_hex}`` (see `port_persistence.h <https://github.com/ray-project/ray/blob/master/src/ray/util/port_persistence.h>`_).
Raylet polls these files and waits for all agent ports before registering the node to GCS.
IPC-Based: Raylet ↔ Core Workers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Raylet (C++) spawns Core Workers (C++). Same language, so they use socket-based
IPC with Flatbuffers protocol. On Linux/macOS this is a Unix domain socket; on
Windows it's a TCP socket on localhost.
**Default (no port range):**
Worker binds to port 0 (OS picks a random port), then tells Raylet the actual port.
**With port range (** ``--min-worker-port`` **/** ``--max-worker-port`` **):**
OS only supports binding to a specific port or port 0 (random). There's no syscall
for "bind to any port within this range". So Raylet must manage the range itself.
Raylet maintains a ``free_ports_`` queue. When a worker registers, Raylet assigns
it an unused port from the queue. The worker binds, then confirms with ``AnnounceWorkerPort``.
If the worker fails to bind the port (e.g., port already in use by external process),
it crashes. Raylet detects the socket disconnect via
`NodeManager::HandleClientConnectionError <https://github.com/ray-project/ray/blob/10869d565047ae02b398802e1efaf04109f27249/src/ray/raylet/node_manager.h>`_, which returns the port to the queue and starts a new worker.
GCS Port Discovery
------------------
Raylet Internal Port Discovery only serves Raylet discovering its own children's ports.
For everything else they query GCS.
Two common examples:
Node Table
~~~~~~~~~~
Each Raylet registers a GcsNodeInfo to GCS, containing its own ports
(``node_manager_port``, ``object_manager_port``) and agent ports
(``runtime_env_agent_port``, ``metrics_agent_port``, ``dashboard_agent_listen_port``).
Other components query GCS for this information:
- Object Manager needs ``object_manager_port`` of remote nodes to pull objects across nodes
- Core Worker needs ``node_manager_port`` of remote nodes for task cancellation, and object recovery
- Dashboard needs ``runtime_env_agent_port`` of each node to collect runtime env info
- Ray Client Server needs ``runtime_env_agent_port`` to set up runtime environments for client jobs
- ...
Actor Table
~~~~~~~~~~~
Actors are stateful—callers must reach the same worker every time. Actor method
calls require direct RPC to a specific worker.
When an actor is created, its worker address (``address.ip_address`` and ``address.port``)
is registered to GCS via `GcsActorManager::HandleRegisterActor <https://github.com/ray-project/ray/blob/10869d565047ae02b398802e1efaf04109f27249/src/ray/gcs/gcs_actor_manager.h>`_.
Callers query GCS to get the address, then communicate directly with the worker.
doc/source/ray-core/internals/ray-event-exporter.rst
+335
View File
@@ -0,0 +1,335 @@
.. _ray-event-exporter:
Ray Event Exporter Infrastructure
==================================
This document is based on Ray version 2.52.1.
Ray's event exporting infrastructure collects events from C++ components (GCS, workers) and Python components, buffers and merges them, and exports them to external HTTP services. This document explains how events flow through the system from creation to final export.
Architecture Overview
---------------------
Ray's event system uses a multi-stage pipeline:
1. **C++ Components**: GCS and worker processes create events implementing `RayEventInterface <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/src/ray/observability/ray_event_interface.h#L24>`__. Raylet does not emit any Ray events, but there are no technical limitations preventing it from doing so.
2. **Event Buffering**: Events are buffered in a bounded circular buffer
3. **Event Merging**: Events with the same entity ID and type are merged before export
4. **gRPC Export**: Events are exported via gRPC to the aggregator agent
5. **Python Aggregation**: The `AggregatorAgent <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/python/ray/dashboard/modules/aggregator/aggregator_agent.py>`__ receives and buffers events
6. **HTTP Publishing**: Events are filtered, converted to JSON, and published to external HTTP services
The following diagram shows the high-level flow:
.. code-block:: text
C++ Components (GCS, workers)
↓ (Create events via RayEventInterface)
RayEventRecorder (C++)
↓ (Buffer & merge events)
↓ (gRPC export via EventAggregatorClient)
AggregatorAgent (Python)
↓ (Add to MultiConsumerEventBuffer)
RayEventPublisher
↓ (Filter & convert to JSON)
↓ (HTTP POST)
External HTTP Service
Event Types and Structure
-------------------------
Ray events are structured using protobuf messages with a base `RayEvent` message that contains event-specific nested messages.
Event Types
~~~~~~~~~~~
Events are categorized by type, defined in the `EventType` enum in `events_base_event.proto <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/src/ray/protobuf/public/events_base_event.proto#L49>`__:
- **TASK_DEFINITION_EVENT**: Task definition information
- **TASK_LIFECYCLE_EVENT**: Task state transitions (this covers both normal tasks and actor tasks)
- **ACTOR_TASK_DEFINITION_EVENT**: Actor task definition
- **ACTOR_DEFINITION_EVENT**: Actor definition
- **ACTOR_LIFECYCLE_EVENT**: Actor state transitions
- **DRIVER_JOB_DEFINITION_EVENT**: Driver job definition
- **DRIVER_JOB_LIFECYCLE_EVENT**: Driver job state transitions
- **NODE_DEFINITION_EVENT**: Node definition
- **NODE_LIFECYCLE_EVENT**: Node state transitions
- **TASK_PROFILE_EVENT**: Task profiling data
Event Structure
~~~~~~~~~~~~~~~
The base `RayEvent <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/src/ray/protobuf/public/events_base_event.proto#L32>`__ message contains:
- **event_id**: Unique identifier for the event
- **source_type**: Component that generated the event
- **event_type**: Type of event (from EventType enum)
- **timestamp**: When the event was created
- **severity**: Event severity level (TRACE, DEBUG, INFO, WARNING, ERROR, FATAL)
- **message**: Optional string message
- **session_name**: Ray session identifier
- **Nested event messages**: One of the event-specific messages (e.g., `task_definition_event`, `actor_lifecycle_event`)
Entity ID Concept
~~~~~~~~~~~~~~~~~
The entity ID is a unique identifier for the entity associated with an event. It's used for two purposes:
1. **Association**: Links execution events with definition events (e.g., task lifecycle events with task definition events)
2. **Merging**: Groups events with the same entity ID and type for merging before export
For example:
- Task events use `task_id + task_attempt` as the entity ID
- Actor events use `actor_id` as the entity ID
- Driver job events use `job_id` as the entity ID
Event Recording and Buffering (C++ Side)
-----------------------------------------
C++ components record events through the `RayEventRecorder <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/src/ray/observability/ray_event_recorder.h>`__ class, which provides thread-safe event buffering and export.
RayEventRecorder
~~~~~~~~~~~~~~~~
The `RayEventRecorder` is a thread-safe event recorder that:
- Maintains a bounded circular buffer for events
- Merges events with the same entity ID and type before export
- Periodically exports events via gRPC to the aggregator agent using `EventAggregatorClient <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/src/ray/rpc/event_aggregator_client.h>`__
- Tracks dropped events when the buffer is full
Adding Events
~~~~~~~~~~~~~
Events are added to the recorder via the `AddEvents() <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/src/ray/observability/ray_event_recorder.cc#L92>`__ method, which accepts a vector of `RayEventInterface` pointers. The method:
1. Checks if event recording is enabled (via `enable_ray_event` config)
2. Calculates if adding events would exceed the buffer size
3. Drops old events if necessary and records metrics for dropped events
4. Adds new events to the circular buffer
Buffer Management
~~~~~~~~~~~~~~~~~
The recorder uses a `boost::circular_buffer <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/src/ray/observability/ray_event_recorder.h#L66>`__ to store events. When the buffer is full:
- Oldest events are dropped to make room for new ones
- Dropped events are tracked via the `dropped_events_counter` metric
- The metric includes the source component name for tracking
- The default buffer size is 10,000 events, but it can be configured via the `RAY_ray_event_recorder_max_queued_events` environment variable
Event Export from C++ (gRPC)
------------------------------
Events are exported from C++ components to the aggregator agent using gRPC. The export process is initiated by calling `StartExportingEvents() <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/src/ray/observability/ray_event_recorder.cc#L37>`__.
StartExportingEvents
~~~~~~~~~~~~~~~~~~~~
This method:
1. Checks if event recording is enabled
2. Verifies it hasn't been called before (should only be called once)
3. Sets up a `PeriodicalRunner` to periodically call `ExportEvents()`
4. Uses the configured export interval (`ray_events_report_interval_ms`)
ExportEvents Process
~~~~~~~~~~~~~~~~~~~~
The `ExportEvents() <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/src/ray/observability/ray_event_recorder.cc#L52>`__ method performs the following steps:
1. **Check Buffer**: Returns early if the buffer is empty
2. **Group Events**: Groups events by entity ID and type using a hash map
3. **Merge Events**: Events with the same key are merged using the `Merge() <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/src/ray/observability/ray_event_interface.h#L55>`__ method
4. **Serialize**: Each merged event is serialized to a `RayEvent` protobuf via `Serialize() <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/src/ray/observability/ray_event_interface.h#L58>`__
5. **Send via gRPC**: Events are sent to the aggregator agent via `EventAggregatorClient::AddEvents() <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/src/ray/rpc/event_aggregator_client.h>`__
6. **Clear Buffer**: The buffer is cleared after successful export
Event Merging Logic
~~~~~~~~~~~~~~~~~~~
Event merging is an optimization that reduces data size by combining related events. Events with the same entity ID and type are merged:
- **Definition Events**: Typically don't change when merged (e.g., actor definition)
- **Lifecycle Events**: State transitions are appended to form a time series (e.g., task state transitions: started → running → completed)
The merging maintains the order of events while combining them into a single event with all state transitions.
Error Handling
~~~~~~~~~~~~~~
If the gRPC export fails:
- An error is logged
- The process continues (doesn't crash)
- The next export interval will attempt to send events again
- Events remain in the buffer until successfully exported (or the buffer is full and old events are dropped)
Event Reception and Buffering (Python Side)
---------------------------------------------
The `AggregatorAgent <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/python/ray/dashboard/modules/aggregator/aggregator_agent.py>`__ receives events from C++ components via a gRPC service and buffers them for publishing.
AggregatorAgent
~~~~~~~~~~~~~~~
The `AggregatorAgent` is a dashboard agent module that:
- Implements `EventAggregatorServiceServicer` for gRPC event reception
- Maintains a `MultiConsumerEventBuffer` for event storage
- Manages `RayEventPublisher` instances for publishing to external http endpoints
- Tracks metrics for events received, buffer and publisher operations
AddEvents gRPC Handler
~~~~~~~~~~~~~~~~~~~~~~~
The `AddEvents() <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/python/ray/dashboard/modules/aggregator/aggregator_agent.py#L165>`__ method is the gRPC handler that receives events:
1. Checks if event processing is enabled
2. Iterates through events in the request
3. Records metrics for each received event
4. Adds each event to the `MultiConsumerEventBuffer` via `add_event() <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/python/ray/dashboard/modules/aggregator/multi_consumer_event_buffer.py#L62>`__
5. Handles errors if adding events fails
MultiConsumerEventBuffer
~~~~~~~~~~~~~~~~~~~~~~~~~
The `MultiConsumerEventBuffer <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/python/ray/dashboard/modules/aggregator/multi_consumer_event_buffer.py>`__ is an asyncio-friendly buffer that:
- **Supports Multiple Consumers**: Each consumer has an independent cursor index. RayEventPublisher and other consumers share this same buffer.
- **Tracks Evictions**: When the buffer is full, oldest events are dropped and tracked per consumer
- **Bounded Buffer**: Uses `deque` with `maxlen` to limit buffer size
- **Asyncio-Safe**: Uses `asyncio.Lock` and `asyncio.Condition` for synchronization
Key operations:
- **add_event()**: Adds an event to the buffer, dropping oldest if full
- **wait_for_batch()**: Waits for a batch of events up to `max_batch_size`, with timeout. The timeout only applies when there is at least one event in the buffer. If the buffer is empty, `wait_for_batch()` can block indefinitely.
- **register_consumer()**: Registers a new consumer with a unique name
Event Filtering
~~~~~~~~~~~~~~~
The agent checks if events can be exposed to external services via `_can_expose_event() <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/python/ray/dashboard/modules/aggregator/aggregator_agent.py#L195>`__. Only events whose type is in the `EXPOSABLE_EVENT_TYPES` set are allowed to be published externally.
Event Publishing to HTTP
------------------------
Events are published to external HTTP services by the `RayEventPublisher`, which reads from the event buffer and sends HTTP POST requests.
RayEventPublisher
~~~~~~~~~~~~~~~~~
The `RayEventPublisher <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/python/ray/dashboard/modules/aggregator/publisher/ray_event_publisher.py>`__ runs a worker loop that:
1. Registers as a consumer of the `MultiConsumerEventBuffer`
2. Continuously waits for batches of events via `wait_for_batch()`
3. Publishes batches using the configured `PublisherClientInterface`
4. Handles retries with exponential backoff on failures
5. Records metrics for publish success, failures, and latency
The publisher runs in an async context and uses `asyncio` for non-blocking operations.
AsyncHttpPublisherClient
~~~~~~~~~~~~~~~~~~~~~~~~~
The `AsyncHttpPublisherClient <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/python/ray/dashboard/modules/aggregator/publisher/async_publisher_client.py#L60>`__ handles HTTP publishing:
1. **Event Filtering**: Filters events using `events_filter_fn` (typically `_can_expose_event`)
2. **JSON Conversion**: Converts protobuf events to JSON dictionaries
- Uses `message_to_json()` from protobuf
- Optionally preserves proto field names or converts to camelCase
- Runs in `ThreadPoolExecutor` to avoid blocking the event loop
3. **HTTP POST**: Sends filtered events as JSON to the configured endpoint
4. **Error Handling**: Catches exceptions and returns failure status
5. **Session Management**: Uses `aiohttp.ClientSession` for HTTP requests
Batch Publishing
~~~~~~~~~~~~~~~~
Events are published in batches:
- Batch size is limited by `max_batch_size` (default: 10,000 events)
- Batches are created by `wait_for_batch()` which waits up to a timeout for events
- Larger batches reduce HTTP request overhead but increase latency
Retry Logic
~~~~~~~~~~~
The publisher implements retry logic with exponential backoff:
- Retries failed publishes up to `max_retries` times (default: infinite)
- Uses exponential backoff with jitter between retries
- If max retries are exhausted, we drop the events and record a metric for dropped events
Configuration
~~~~~~~~~~~~~
HTTP publishing is configured via environment variables:
- **RAY_DASHBOARD_AGGREGATOR_AGENT_EVENTS_EXPORT_ADDR**: HTTP endpoint URL (e.g., `http://localhost:8080/events`)
- **RAY_DASHBOARD_AGGREGATOR_AGENT_EXPOSABLE_EVENT_TYPES**: Comma-separated list of event types to expose
- **RAY_DASHBOARD_AGGREGATOR_AGENT_PUBLISH_EVENTS_TO_EXTERNAL_HTTP_SERVICE**: Enable/disable flag (default: True)
Creating New Event Types
-------------------------
To create a new event type, follow these steps:
Step 1: Define Protobuf Message
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Create a new `.proto` file in `src/ray/protobuf/public/` following the naming convention `events_<name>_event.proto`. For example, see `events_task_definition_event.proto <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/src/ray/protobuf/public/events_task_definition_event.proto>`__.
Define your event-specific message with the fields you need:
.. code-block:: protobuf
syntax = "proto3";
package ray.rpc.events;
message MyNewEvent {
// Define your event-specific fields here
string entity_id = 1;
// ... other fields
}
Step 2: Add to Base Event
~~~~~~~~~~~~~~~~~~~~~~~~~~
Update `events_base_event.proto <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/src/ray/protobuf/public/events_base_event.proto>`__:
1. Add import for your new proto file
2. Add new `EventType` enum value (e.g., `MY_NEW_EVENT = 11`)
3. Add new field to `RayEvent` message (e.g., `MyNewEvent my_new_event = 18`)
Step 3: Implement RayEventInterface
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Create a C++ class that implements `RayEventInterface <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/src/ray/observability/ray_event_interface.h>`__. The easiest approach is to extend `RayEvent<T>` template class, as shown in `ray_actor_definition_event.h <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/src/ray/observability/ray_actor_definition_event.h>`__.
You need to implement:
- **GetEntityId()**: Return a unique identifier for the entity (e.g., task ID + attempt, actor ID)
- **MergeData()**: Implement merging logic for events with the same entity ID
- Definition events typically don't change when merged
- Lifecycle events append state transitions
- **SerializeData()**: Convert the event data to a `RayEvent` protobuf
- **GetEventType()**: Return the `EventType` enum value for this event
See `ray_actor_definition_event.cc <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/src/ray/observability/ray_actor_definition_event.cc>`__ for a complete example.
Step 4: Update Exposable Event Types (if needed)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If your event should be exposed to external HTTP services, add it to `DEFAULT_EXPOSABLE_EVENT_TYPES <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/python/ray/dashboard/modules/aggregator/aggregator_agent.py#L56>`__ in `aggregator_agent.py`. Alternatively, users can configure it via the `RAY_DASHBOARD_AGGREGATOR_AGENT_EXPOSABLE_EVENT_TYPES` environment variable.
Step 5: Update RayEventRecorder to publish your new event type
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use `RayEventRecorder::AddEvent() <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/src/ray/observability/ray_event_recorder.cc#L92>`__ to add your new event type to the buffer.
Step 6: Update AggregatorAgent to publish your new event type
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Update `AggregatorAgent <https://github.com/ray-project/ray/blob/4ebdc0abe5e5a551625fe7f87053c7e668a6ff74/python/ray/dashboard/modules/aggregator/aggregator_agent.py#L56>`__ to publish your new event type.
doc/source/ray-core/internals/rpc-fault-tolerance.rst
+223
View File
@@ -0,0 +1,223 @@
.. _rpc-fault-tolerance:
RPC Fault Tolerance
===================
All RPCs added to Ray Core should be fault tolerant and use the retryable gRPC client.
Ideally, they should be idempotent, or at the very least, the lack of idempotency should be
documented and the client must be able to take retries into account. If you aren't familiar
with what idempotency is, consider a function that writes "hello" to a file. On retry,
it writes "hello" again, resulting in "hellohello". This isn't idempotent. To make it
idempotent, you could check the file contents before writing "hello" again, ensuring that the
observable state after multiple identical function calls is the same as after a single call.
This guide walks you through a case study of a RPC that wasn't fault tolerant or idempotent,
and how it was fixed. By the end of this guide, you should understand what to look for
when adding new RPCs and which testing methods to use to verify fault tolerance.
Case study: RequestWorkerLease
-------------------------------
Problem
~~~~~~~
Prior to the fix described here, ``RequestWorkerLease`` could not be made retryable because
its handler in the Raylet was not idempotent.
This was because once leases were granted, they were considered occupied until ``ReturnWorker``
was called. Until this RPC was called, the worker and its resources were never returned to the pool
of available workers and resources. The raylet assumed that the original RPC and its retry were
both fresh lease requests and couldn't deduplicate them.
For example, consider the following sequence of operations:
1. Request a new worker lease (Owner → Raylet) through ``RequestWorkerLease``.
2. Response is lost (Raylet → Owner).
3. Retry ``RequestWorkerLease`` for lease (Owner → Raylet).
4. Two sets of resources and workers are now granted, one for the original AND retry.
On the retry, the raylet should detect that the lease request is a retry and forward the
already leased worker address to the owner so a second lease isn't granted.
Solution
~~~~~~~~
To implement idempotency, a unique identifier called ``LeaseID`` was added in
`PR #55469 <https://github.com/ray-project/ray/pull/55469>`_, which allowed for the deduplication
of incoming lease requests. Once leases are granted, they're tracked in a ``leased_workers`` map
which maps lease IDs to workers. If the new lease request is already present in the
``leased_workers`` map, the system knows this lease request is a retry and responds with the
already leased worker address.
Hidden problem: long-polling RPCs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Network transient errors can happen at any time. For most RPCs, they finish in one I/O context
execution, so guarding against whether the request or response failed is sufficient.
However, there are a few RPCs that are long-polling, meaning that once the ``HandleX`` function
executes, it won't immediately respond to the client but will rather depend on some state change
in the future to trigger the response back to the client.
This was the case for ``RequestWorkerLease``. Leases aren't granted until all the args are pulled,
so the system can't respond to the client until the pulling process is over. What happens if the
client disconnects while the server logic executes on the raylet side and sends a retry? The
``leased_workers`` map only tracks leases that are granted, not in the process of granting.
This caused a
`RAY_CHECK to be triggered <https://github.com/ray-project/ray/blob/66c08b47a195bcfac6878a234dc804142e488fc2/src/ray/raylet/lease_dependency_manager.cc#L222>`_
in the ``lease_dependency_manager`` because the system wasn't able to deduplicate lease
request retries while the server logic was executing.
Specifically, consider this sequence of operations:
1. Request a new worker lease (Owner → Raylet) through ``RequestWorkerLease``.
2. The raylet is pulling the lease args asynchronously for the lease.
3. Retry ``RequestWorkerLease`` for the lease (Owner → Raylet).
4. The lease hasn't been granted yet, so it passes the idempotency check and the raylet
fails to deduplicate the lease request.
5. ``RAY_CHECK`` hit since the raylet tries to pull args for the same lease again.
The final fix was to take into account that the server logic could still be executing and track
the lease as it goes through the phases of lease granting. At any phase, the system should be
able to deduplicate requests.
For any long-polling RPC, you should be **particularly careful** about idempotency because the
client's retry won't necessarily wait for the response to be sent.
Retryable gRPC client
---------------------
The retryable gRPC client was updated during the RPC fault tolerance project. This section
describes how it works and some gotchas to watch out for.
For a basic introduction, read the
`retryable_grpc_client.h comment <https://github.com/ray-project/ray/blob/885e34f4029f8956a0440f3cdfc89c9fe8f3d395/src/ray/rpc/retryable_grpc_client.h#L67>`_.
How it works
~~~~~~~~~~~~
The retryable gRPC client works as follows:
- RPCs are sent using the retryable gRPC client.
- If the client encounters a
`gRPC transient network error <https://github.com/ray-project/ray/blob/78082d65fa7081172d2848ced56d68cc612f8fd1/src/ray/common/grpc_util.h#L130>`_,
it pushes the callback into a queue.
- Several checks are done on a periodic basis:
- **Cheap gRPC channel state check**: This checks the state of the
`gRPC channel <https://github.com/ray-project/ray/blob/885e34f4029f8956a0440f3cdfc89c9fe8f3d395/src/ray/rpc/retryable_grpc_client.cc#L74>`_
to see whether the system can start sending messages again. This check happens every
second by default, but is configurable through
`check_channel_status_interval_milliseconds <https://github.com/ray-project/ray/blob/9b217e9ad01763e0b78c9161a4ebdd512289a748/src/ray/common/ray_config_def.h#L449>`_.
- **Potentially expensive GCS node status check**: If the exponential backoff period has
passed and the channel is still down, the system calls
`server_unavailable_timeout_callback_ <https://github.com/ray-project/ray/blob/885e34f4029f8956a0440f3cdfc89c9fe8f3d395/src/ray/rpc/retryable_grpc_client.cc#L84>`_.
This callback is set in the client pool classes
(`raylet_client_pool <https://github.com/ray-project/ray/blob/9b217e9ad01763e0b78c9161a4ebdd512289a748/src/ray/raylet_rpc_client/raylet_client_pool.cc#L24>`_,
`core_worker_client_pool <https://github.com/ray-project/ray/blob/9b217e9ad01763e0b78c9161a4ebdd512289a748/src/ray/core_worker_rpc_client/core_worker_client_pool.cc#L28>`_).
It checks if the client is subscribed for node status updates, and then checks the local
subscriber cache to see whether a node death notification from the GCS has been received.
If the client isn't subscribed or if there's no status for the node in the cache, it
makes a RPC to the GCS. Note that for the GCS client, the ``server_unavailable_timeout_callback_``
`kills the process once called <https://github.com/ray-project/ray/blob/888083bedf31458fb0fb33bf5613fb80f8fc0a6a/src/ray/gcs_rpc_client/rpc_client.h#L201>`_.
This happens after ``gcs_rpc_server_reconnect_timeout_s`` seconds (60 by default).
- **Per-RPC timeout check**: There's a
`timeout check <https://github.com/ray-project/ray/blob/9b217e9ad01763e0b78c9161a4ebdd512289a748/src/ray/rpc/retryable_grpc_client.cc#L57>`_
that's customizable per RPC, but it's functionally disabled because it's
`always set to -1 <https://github.com/ray-project/ray/blob/9b217e9ad01763e0b78c9161a4ebdd512289a748/src/ray/core_worker_rpc_client/core_worker_client.h#L72>`_
(infinity) for each RPC.
- With each additional failed RPC, the
`exponential backoff period is increased <https://github.com/ray-project/ray/blob/9b217e9ad01763e0b78c9161a4ebdd512289a748/src/ray/rpc/retryable_grpc_client.cc#L95>`_,
agnostic of the type of RPC that fails. The backoff period caps out to a max that you can
customize for the core worker and raylet clients using either the
``core_worker_rpc_server_reconnect_timeout_max_s`` or ``raylet_rpc_server_reconnect_timeout_max_s``
config options. The GCS client doesn't have a max backoff period as noted above.
- Once the channel check succeeds, the
`exponential backoff period is reset and all RPCs in the queue are retried <https://github.com/ray-project/ray/blob/9b217e9ad01763e0b78c9161a4ebdd512289a748/src/ray/rpc/retryable_grpc_client.cc#L117>`_.
- If the system successfully receives a node death notification (either through subscription or
querying the GCS directly), it destroys the RPC client, which posts each callback to the I/O
context with a
`gRPC Disconnected error <https://github.com/ray-project/ray/blob/75f8562759d4a5ef84163bb68ae9f7401b85728f/src/ray/rpc/retryable_grpc_client.cc#L32>`_.
Important considerations
~~~~~~~~~~~~~~~~~~~~~~~~
A few important points to keep in mind:
- **Per-client queuing**: Each retryable gRPC client is unique to the client (``WorkerID`` for
core worker clients, ``NodeID`` for raylet clients), not to the type of RPC. If you first
submit RPC A that fails due to a transient network error, then RPC B to the same client that
fails due to a transient network error, the queue will have two items: RPC A then RPC B.
There isn't a separate queue on an RPC basis, but on a client basis.
- **Client-level timeouts**: Each timeout needs to wait for the previous timeout to complete.
If both RPC A and RPC B are submitted in short succession, then RPC A will wait in total
for 1 second, and RPC B will wait in total for 1 + 2 = 3 seconds. Different RPCs don't
matter and are treated the same. The reasoning is that transient network errors aren't RPC
specific. If RPC A sees a network failure, you can assume that RPC B, if sent to the same
client, will experience the same failure. Hence, the time that an RPC waits is the sum of
the timeouts of all the previous RPCs in the queue and its own timeout.
- **Destructor behavior**: In the destructor for ``RetryableGrpcClient``, the system fails all
pending RPCs by posting their I/O contexts. These callbacks should ideally never modify state
held by the client classes such as ``RayletClient``. If absolutely necessary, they must check
if the client is still alive somehow, such as using a weak pointer. An example of this is in
`PR #58744 <https://github.com/ray-project/ray/pull/58744>`_. The application code should
also take into account the
`Disconnected error <https://github.com/ray-project/ray/blob/75f8562759d4a5ef84163bb68ae9f7401b85728f/src/ray/rpc/retryable_grpc_client.cc#L32>`_.
Testing RPC fault tolerance
----------------------------
Ray Core has three layers of testing for RPC fault tolerance and idempotency.
C++ unit tests
~~~~~~~~~~~~~~
For each RPC, there should be some form of C++ idempotency test that calls the ``HandleX`` server
function twice and checks that the same result is outputted each time. Different state changes
between the ``HandleX`` server function calls should be taken into account. For example, in
``RequestWorkerLease``, a C++ unit test was written to model the situation where the retry comes
while the initial lease request is stuck in the args pulling stage.
Python integration tests
~~~~~~~~~~~~~~~~~~~~~~~~
For each RPC, there should ideally be a Python integration test if it's straightforward. For some
RPCs, it's challenging to test them fully deterministically using Python APIs, so having
sufficient C++ unit testing can act as a good proxy. Hence, it's more of a nice-to-have, as
integration tests also act as examples of how a user could run into idempotency issues.
The main testing mechanism uses the ``RAY_testing_rpc_failure`` config option, which allows
you to:
- Trigger the RPC callback immediately with a gRPC error without sending the RPC
(simulating a request failure).
- Trigger the RPC callback with a gRPC error once the response arrives from the server
(simulating a response failure).
- Trigger the RPC callback immediately with a gRPC error but send the RPC to the server as well
(simulating an in-flight failure, where the retry should ideally hit the server while it's
executing the server code for long-polling RPCs).
For more details, see the comment in the Ray config file at
`ray_config_def.h <https://github.com/ray-project/ray/blob/a24e625f409a5c638414e5d104fd265547e4d1b4/src/ray/common/ray_config_def.h#L860>`_.
Chaos network release tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
IP table blackout
^^^^^^^^^^^^^^^^^
The IP table blackout approach involves SSH-ing into each node and blacking out the IP tables
for a small amount of time (5 seconds) to simulate transient network errors. The IP table script
runs in the background, periodically (60 seconds) causing network blackouts while the test script
executes.
For core release tests, we've added the IP table blackout approach to all existing chaos release
tests in `PR #58868 <https://github.com/ray-project/ray/pull/58868>`_.
.. note::
Initially, Amazon FIS was considered. However, it has a 60-second minimum which caused node
death due to configuration settings which was hard to debug, so the IP table approach was
simpler and more flexible to use.
doc/source/ray-core/internals/streaming-generator.md
+267
View File
@@ -0,0 +1,267 @@
(streaming-generator)=
# Streaming Generator
This document explains how streaming generator tasks work in Ray 2.55 and outlines the main implementation differences from normal Ray tasks. Please make sure you read the doc for the normal task lifecycle first. See {ref}`task-lifecycle`.
## Overview
A streaming generator task is a Python generator function executed as a Ray task. Unlike a normal Ray task, it doesn't return all results in the final `PushTask` reply. Instead, after each `yield`, the remote executor converts the yielded value into a Ray object and reports it to the caller before the overall generator task finishes.
The following example creates a streaming generator task:
```{eval-rst}
.. testcode::
import ray
@ray.remote
def numbers():
for i in range(3):
yield i
gen = numbers.remote()
for ref in gen:
print(ray.get(ref))
```
```{eval-rst}
.. testoutput::
0
1
2
```
The `gen` variable is an `ObjectRefGenerator`. Iterating over it returns one `ObjectRef` per yielded value. But note that `ObjectRefGenerator` is not serializable and can't be passed to other remote tasks.
## Defining a Streaming Generator Function
A streaming generator function is defined using the {func}`ray.remote` decorator, the same as a normal remote function.
But inside the decorator, Ray checks whether the decorated Python function is a generator function by calling [inspect.isgeneratorfunction(function)](https://github.com/ray-project/ray/blob/ray-2.55.0/python/ray/remote_function.py#L149).
Note that a user can explicitly set `num_returns` to an integer for a generator function. However, it is not the scope of this document because, in that case, Ray treats the invocation as a fixed-return task, iterates over the generator while buffering task outputs, and returns the fixed set of `ObjectRef` objects from `.remote()`.
## Task Submission and Return ObjectIDs
Invoking `generator.remote()` submits one streaming generator task and [returns one ObjectRefGenerator](https://github.com/ray-project/ray/blob/ray-2.55.0/python/ray/remote_function.py#L509-L514) to the Python caller. The submission path is the same as normal tasks until `CoreWorker` builds the `TaskSpecification`:
1. The caller exports the pickled function definition to GCS if needed.
2. The caller flattens and serializes the task arguments.
3. The caller calls into C++ `CoreWorker` to submit the task.
4. `CoreWorker` builds a `TaskSpecification`.
For streaming generator tasks, `CoreWorker` [converts the streaming return sentinel](https://github.com/ray-project/ray/blob/ray-2.55.0/src/ray/core_worker/core_worker.cc#L1896-L1902) and the `TaskSpecification` records:
1. `streaming_generator=true`: Marks the task as a streaming generator task.
2. `num_returns=1`: Creates one normal return ObjectID. This is the `generator ObjectRef`.
3. `returns_dynamic=true`: Indicates that the yielded values are dynamic returns, not fixed returns listed by `num_returns`.
4. `generator_backpressure_num_objects`: Stores the backpressure threshold. `-1` means backpressure is disabled.
The `generator ObjectRef` isn't a yielded value. It is only resolved when the generator task ends. Ray also uses its ObjectID as the `generator_id` in `ReportGeneratorItemReturns`. Yielded values get [deterministic object IDs derived from the task ID and the yield index](https://github.com/ray-project/ray/blob/ray-2.55.0/src/ray/common/task/task_spec.cc#L223-L229):
```
Generator task:
TaskID = T
Normal task return namespace:
ObjectID(T, index=1) generator ObjectRef, not a yielded value
used as generator_id and task completion/failure ref
Streaming generator namespace:
1st yield -> ObjectID(T, index=2)
2nd yield -> ObjectID(T, index=3)
3rd yield -> ObjectID(T, index=4)
...
end of stream -> ObjectID(T, index=2 + num_yields)
```
Task return object IDs are one-based, so index `0` isn't a valid task return index. Index `1` belongs to the `generator ObjectRef`. Streamed items start at index `2` to avoid sharing an object ID with the generator ObjectRef.
Before `.remote()` returns to Python, the caller-side core worker [creates an in-memory mapping](https://github.com/ray-project/ray/blob/ray-2.55.0/src/ray/core_worker/task_manager.cc#L321-L330) from the ObjectID of the generator ObjectRef to an `ObjectRefStream`. This mapping records reported yield indexes, exposes only the next in-order streamed ref to Python, and tracks where iteration should resume. The Python API wraps the generator ObjectRef in `ObjectRefGenerator` and uses it to read from the caller-side stream.
## Scheduling
Scheduling a streaming generator task follows the same worker lease path as a normal Ray task. See {ref}`task-lifecycle` for how `NormalTaskSubmitter` resolves dependencies, requests a worker lease, and sends `PushTask` to the leased worker.
The streaming generator specific difference is that the caller now has the `ObjectRefGenerator` before the task finishes. The caller can start iterating immediately, but iteration waits until the executor reports the next streamed return.
## Streaming Generator Task Execution
Execution starts the same as a normal task:
1. The leased worker receives the `PushTask` RPC.
2. The worker deserializes the arguments.
3. The worker fetches and unpickles the remote function.
4. The worker calls the user function.
For a normal task, calling the function produces the final return values. The executor worker serializes the return values. Small returns are sent back inline in the `PushTask` reply, while larger returns are put in the executor node's plasma object store and referenced from the reply.
For a streaming generator task, calling the function produces a Python generator or async generator object. The executor then drives the generator itself:
1. For a sync generator, the executor worker [repeatedly calls send(stats)](https://github.com/ray-project/ray/blob/ray-2.55.0/python/ray/_raylet.pyx#L1301-L1340).
2. For an async generator, the executor worker [runs the asend(stats) loop](https://github.com/ray-project/ray/blob/ray-2.55.0/python/ray/_raylet.pyx#L1355-L1444) with the worker event loop used for async task execution.
Each `send(stats)` or `asend(stats)` resumes the Python generator until the next `yield`. After a value is yielded, the generator is paused. Before the executor calls `send(stats)` or `asend(stats)` again, [it sends `ReportGeneratorItemReturns` to the caller](#reporting-yielded-values), and [waits for backpressure if needed](#backpressure). The loop ends when the generator raises `StopIteration` or `StopAsyncIteration`.
Note that Ray streaming generators are output-only streams. Callers can only iterate over the returned `ObjectRefGenerator` to receive yielded refs. Callers can't send values back into the remote generator, which is allowed for normal Python generators, because the remote executor owns the Python generator's `send` or `asend` channel. It passes `None` for the first `stats` value, and then passes a [StreamingGeneratorStats](https://github.com/ray-project/ray/blob/ray-2.55.0/python/ray/_raylet.pyx#L1166-L1168) object after each yielded value is reported. The stats object records executor-side metadata for the previous yielded value, such as object creation time.
## Reporting Yielded Values
For each yielded value, the executor runs [report_streaming_generator_output](https://github.com/ray-project/ray/blob/ray-2.55.0/python/ray/_raylet.pyx#L1170-L1233), which performs the following steps:
1. [Create the streamed return object](https://github.com/ray-project/ray/blob/ray-2.55.0/python/ray/_raylet.pyx#L1459-L1493) using the deterministic ObjectID for the yield index.
2. Store the yielded value using the same direct-return versus plasma-return rule as normal task returns. Direct return values are serialized into the report RPC. Plasma return values are stored in plasma and referenced from the report.
3. Send [ReportGeneratorItemReturns](https://github.com/ray-project/ray/blob/ray-2.55.0/src/ray/core_worker/core_worker.cc#L3155-L3217) from the executor to the caller.
When the caller [handles ReportGeneratorItemReturns](https://github.com/ray-project/ray/blob/ray-2.55.0/src/ray/core_worker/task_manager.cc#L779-L878), it performs the following steps:
1. Insert the returned object into the caller-side `ObjectRefStream` at the yield index.
2. Handle the reported return object as a direct return or a plasma return, using the same logic as normal task returns.
3. [Make the reported ObjectRef ready](https://github.com/ray-project/ray/blob/ray-2.55.0/src/ray/core_worker/task_manager.cc#L821-L847). If a caller is [waiting in next(gen)](https://github.com/ray-project/ray/blob/ray-2.55.0/python/ray/_private/object_ref_generator.py#L188-L237) for this in-order yield index, that wait can now finish.
The following diagram shows the reporting path for yielded values:
```
Caller / owner process Executor worker
---------------------- ---------------
gen = task.remote()
|
| generator_ref = ObjectRef(T, 1)
| (the generator ObjectRef, not a yielded value)
|
| Create ObjectRefGenerator
| and ObjectRefStream(generator_ref)
|
|------------------------- PushTask ---------------------->|
| task_spec.streaming_generator = true
| task_spec.num_returns = 1
|
| Run generator task
| output = 1st yield
| |
| Compute return ObjectID
| ObjectID(T, index=2)
| |
| Store yielded value
| inline or in plasma
| |
|<---------------- ReportGeneratorItemReturns -----------|
| returned_object.object_id = ObjectID(T, 2)
| # task return object index
| worker_addr = executor address
| item_index = 0
| # yield index
| generator_id = ObjectID(T, 1)
| # ObjectID inside generator_ref
| attempt_number = 0
|
| Insert ObjectRef(T, 2)
| into ObjectRefStream at yield index 0
|
| next(gen) returns ObjectRef(T, 2)
|
|---------------- ReportGeneratorItemReturnsReply ------>|
| total_num_object_consumed = 1
| (also releases backpressure if the
| caller delayed the reply)
|
| ...
|
|<---------------- ReportGeneratorItemReturns -----------|
| returned_object.object_id = ObjectID(T, 3)
| item_index = 1
| generator_id = ObjectID(T, 1)
| attempt_number = 0
|
| Insert ObjectRef(T, 3)
| into ObjectRefStream at yield index 1
|
| next(gen) returns ObjectRef(T, 3)
|
|---------------- ReportGeneratorItemReturnsReply ------>|
| total_num_object_consumed = 2
|
| ...
|
|<----------------------- PushTaskReply -------------------|
| return_objects[0] = generator_ref
| streaming_generator_return_ids = [...]
```
The executor repeats this protocol for every yielded value. Finally, the `PushTask` reply marks the generator task as complete. The caller handles this reply in [`TaskManager::CompletePendingTask`](https://github.com/ray-project/ray/blob/ray-2.55.0/src/ray/core_worker/task_manager.cc#L908-L1085). The reply contains [`streaming_generator_return_ids`](https://github.com/ray-project/ray/blob/ray-2.55.0/src/ray/protobuf/core_worker.proto#L172-L174), which are streamed return ObjectIDs and whether each return is stored in plasma. It doesn't contain the yielded values themselves because those values were already reported through `ReportGeneratorItemReturns`. The caller uses the number of entries in `streaming_generator_return_ids` to write the end-of-stream marker. For example, if the generator yielded three values, the marker is written at yield index `3`. This marker tells `ObjectRefGenerator` that the next index will never be reported, so iteration should stop instead of waiting forever.
One more detail is that, while the yield ordering is fixed, the arrival ordering of the [ReportGeneratorItemReturnsRequest](https://github.com/ray-project/ray/blob/ray-2.55.0/src/ray/protobuf/core_worker.proto#L434-L450) RPC on the caller isn't guaranteed. The executor sends report RPCs asynchronously. It [waits for a report response](https://github.com/ray-project/ray/blob/ray-2.55.0/src/ray/core_worker/generator_waiter.cc#L32-L57) only when generator backpressure is enabled and the number of generated but unconsumed objects reaches the configured threshold. When backpressure is disabled, or when the threshold still allows more unconsumed objects, multiple reports can be in flight. The caller uses `attempt_number` to reject stale reports from older task attempts after a retry has started.
## Backpressure
Streaming generators can produce objects faster than the caller consumes them. Ray supports a private task option named `_generator_backpressure_num_objects` to limit generated but unconsumed streamed objects.
The option has the following behavior:
1. `-1` disables generator backpressure.
2. A positive value sets the maximum number of generated but unconsumed objects.
3. `1` behaves most like a local Python generator: Ray produces the next object only after the caller consumes the previous `ObjectRef`.
4. `0` isn't valid.
Note that Ray doesn't support `_generator_backpressure_num_objects` for async generators.
Then after the executor sends `ReportGeneratorItemReturns`, it calls [WaitUntilObjectConsumed()](https://github.com/ray-project/ray/blob/ray-2.55.0/src/ray/core_worker/generator_waiter.cc#L32-L57) to block the executor thread while:
```text
generated_objects - consumed_objects >= _generator_backpressure_num_objects
```
If the stream is under the threshold, the caller replies to `ReportGeneratorItemReturns` when it handles the report. If the stream is at or above the threshold, the caller holds the report reply. The caller resumes the executor by consuming object refs from the `ObjectRefGenerator` by calling `next(gen)` or iterating over the generator. When the unconsumed count drops below the threshold, the caller replies to all held backpressured report RPCs with the cumulative number of consumed objects. Each report reply updates the executor-side consumed count and can wake an executor blocked in `WaitUntilObjectConsumed()`.
## Stopping Consumption
If the caller stops iterating but keeps the `ObjectRefGenerator` alive, Ray keeps the caller-side stream alive. With backpressure enabled, this can leave the executor paused in `WaitUntilObjectConsumed()`.
When the `ObjectRefGenerator` goes out of scope, its [destructor](https://github.com/ray-project/ray/blob/ray-2.55.0/python/ray/_private/object_ref_generator.py#L289-L295) asks the caller-side core worker to delete the `ObjectRefStream` for the ObjectID of the generator ObjectRef. [Deleting the stream](https://github.com/ray-project/ray/blob/ray-2.55.0/src/ray/core_worker/task_manager.cc#L690-L727) releases unconsumed streamed refs from the caller-side stream and replies to pending backpressured report RPCs with `NotFound`. This unblocks an executor that was waiting for the caller to consume more refs. If the executor reports more yielded values after the stream is deleted, the caller rejects those future `ReportGeneratorItemReturns` RPCs with `NotFound`. On the executor side, the failed report reply is treated as if [all generated refs were consumed](https://github.com/ray-project/ray/blob/ray-2.55.0/src/ray/core_worker/core_worker.cc#L3190-L3211) for backpressure accounting, so the executor doesn't stay blocked on the deleted stream. Note that deleting the `ObjectRefStream` doesn't cancel the remote generator task. The executor continues running user code until the generator finishes or the task is cancelled separately.
## Getting Streamed Return Values
The caller receives an `ObjectRefGenerator` from `.remote()`:
```python
gen = numbers.remote()
ref = next(gen)
value = ray.get(ref)
```
`next(gen)` doesn't return the yielded Python value. It returns an `ObjectRef` for the next yielded value.
Internally, `ObjectRefGenerator` [asks the caller-side core worker](https://github.com/ray-project/ray/blob/ray-2.55.0/python/ray/_private/object_ref_generator.py#L188-L282) to peek at the next expected object ID in the stream. If the item isn't ready, the caller waits until the executor reports that object reference through `ReportGeneratorItemReturns`. When the item is ready, the caller consumes the corresponding stream entry and returns the corresponding `ObjectRef` to Python. If the caller previously held report replies because of backpressure, this consume operation can release those held replies once the unconsumed count drops below the threshold.
The caller-side core worker can compute the ObjectID for the next yield index before the executor reports it. However, it doesn't return that ref until it is ready for the following reasons:
1. The generator may finish or fail before producing that index. In those cases, iteration should raise `StopIteration` or surface the task error through the generator ObjectRef instead of returning a normal-looking streamed ref that was never reported.
2. Returning refs without waiting would let the caller drain an unbounded number of refs from repeated calls to `next(gen)`. It would also break backpressure accounting because `next(gen)` is the signal that the caller consumed a stream item.
## Finishing the Generator
When the Python generator raises `StopIteration` or `StopAsyncIteration` on `send(stats)/asend(stats)`, the executor finishes the streaming task. The completion flow is:
1. The executor [waits for all in-flight ReportGeneratorItemReturns RPCs](https://github.com/ray-project/ray/blob/ray-2.55.0/python/ray/_raylet.pyx#L1345-L1352) to finish before sending the final `PushTask` reply. This prevents the task from appearing complete before the caller receives one of the yielded objects.
2. The executor sends the `PushTask` reply. This reply includes the generator ObjectRef and the [list of streamed return ObjectIDs](https://github.com/ray-project/ray/blob/ray-2.55.0/src/ray/core_worker/task_execution/task_receiver.cc#L54-L60) produced by the task.
3. The caller handles the `PushTask` reply, records how many streaming return objects the task produced, and marks the stream as ended.
4. The caller [writes an internal end-of-stream marker](https://github.com/ray-project/ray/blob/ray-2.55.0/src/ray/core_worker/task_manager.cc#L1078-L1085) into the caller-side `ObjectRefStream`.
5. Later, when `ObjectRefGenerator` reaches the marker, it checks the generator ObjectRef.
At that point:
1. If the generator task succeeded, iteration raises `StopIteration` or `StopAsyncIteration`.
2. If the generator task failed and the failure hasn't been surfaced yet, `ObjectRefGenerator` returns the generator ObjectRef once. `ray.get` on that ref raises the task error.
3. After surfacing that error ref, later iteration stops.
## Failures, Retries, and Reconstruction
If the generator raises an application exception, Ray reports an error object at the current stream index. The caller receives an `ObjectRef` for that item, and `ray.get` on that ref raises the task exception. Ray also uses the generator ObjectRef, available through `gen.completed()`, to represent completion or failure of the whole generator task.
Ray can retry a streaming generator task from its beginning through the same task retry machinery used for normal tasks. This assumes that the generator task is idempotent and deterministic. A retry can be triggered when the running task attempt fails, such as dependency resolution failure, worker or node death, eligible out-of-memory failure, or a retryable application exception when `retry_exceptions` is enabled. When retries remain, the task manager marks the current attempt failed, increments the attempt number, and resubmits the same task spec.
Object reconstruction is a separate resubmission path from immediate retry on task attempt failure. It starts when a streamed return object stored in plasma is lost while its lineage is still reconstructable. In that case, the object recovery manager calls [TaskManager::ResubmitTask](https://github.com/ray-project/ray/blob/ray-2.55.0/src/ray/core_worker/object_recovery_manager.cc#L140-L164) for the task that produced the lost object. If the streaming generator task is still running when reconstruction is requested, Ray [queues the resubmission](https://github.com/ray-project/ray/blob/ray-2.55.0/src/ray/core_worker/task_manager.cc#L353-L435) until the current attempt finishes or fails. If the task is already finished or failed, Ray sets up the task entry for resubmission immediately.
doc/source/ray-core/internals/task-lifecycle.rst
+83
View File
@@ -0,0 +1,83 @@
.. _task-lifecycle:
Task Lifecycle
==============
This doc talks about the lifecycle of a task in Ray Core, including how tasks are defined, scheduled and executed.
We will use the following code as an example and the internals are based on Ray 2.48.
.. testcode::
import ray
@ray.remote
def my_task(arg):
return f"Hello, {arg}!"
obj_ref = my_task.remote("Ray")
print(ray.get(obj_ref))
.. testoutput::
Hello, Ray!
Defining a remote function
--------------------------
The first step in the task lifecycle is defining a remote function using the :func:`ray.remote` decorator. :func:`ray.remote` wraps the Python function and returns an instance of `RemoteFunction <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/python/ray/remote_function.py#L41>`__.
``RemoteFunction`` stores the underlying function and all the user specified Ray task :meth:`options <ray.remote_function.RemoteFunction.options>` such as ``num_cpus``.
Invoking a remote function
--------------------------
Once a remote function is defined, it can be invoked using the `.remote()` method. Each invocation of a remote function creates a Ray task. This method submits the task for execution and returns an object reference (``ObjectRef``) that can be used to retrieve the result later.
Under the hood, `.remote()` does the following:
1. `Pickles the underlying function <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/python/ray/remote_function.py#L366>`__ into bytes and `stores the bytes in GCS key-value store <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/python/ray/remote_function.py#L372>`__ with a `key <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/python/ray/_private/function_manager.py#L223>`__ so that, later on, the remote executor (the core worker process that will execute the task) can get the bytes, unpickle, and execute the function. This is done once per remote function definition instead of once per invocation.
2. `Calls <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/python/ray/remote_function.py#L490>`__ Cython `submit_task <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/python/ray/_raylet.pyx#L3692>`__ which `prepares <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/python/ray/_raylet.pyx#L901>`__ the arguments (3 types) and calls the C++ `CoreWorker::SubmitTask <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/core_worker/core_worker.cc#L2514>`__.
1. Pass-by-reference argument: the argument is an ``ObjectRef``.
2. Pass-by-value inline argument: the argument is a `small <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/python/ray/_raylet.pyx#L967>`__ Python object and the total size of such arguments so far is below the `threshold <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/python/ray/_raylet.pyx#L968>`__. In this case, it will be pickled, sent to the remote executor (as part of the ``PushTask`` RPC), and unpickled there. This is called inlining and plasma store is not involved in this case.
3. Pass-by-value non-inline argument: the argument is a normal Python object but it doesn't meet the inline criteria (e.g. size is too big), it is `put <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/python/ray/_raylet.pyx#L987>`__ in the local plasma store and the argument is replaced by the generated ``ObjectRef``, so it's effectively equivalent to ``.remote(ray.put(arg))``.
3. ``CoreWorker`` `builds <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/core_worker/core_worker.cc#L2542>`__ a `TaskSpecification <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/common/task/task_spec.h#L258>`__ that contains all the information about the task including the `ID <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/python/ray/includes/function_descriptor.pxi#L265>`__ of the function, all the user specified options and the arguments. This spec will be sent to the executor for execution.
4. The TaskSpecification is `submitted <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/core_worker/core_worker.cc#L2587>`__ to `NormalTaskSubmitter <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/core_worker/transport/normal_task_submitter.cc#L28>`__ asynchronously. This means the ``.remote()`` call returns immediately and the task is scheduled and executed asynchronously.
Scheduling a task
-----------------
Once the task is submitted to ``NormalTaskSubmitter``, a worker process on some Ray node is selected to execute the task and this process is called scheduling.
1. ``NormalTaskSubmitter`` first `waits <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/core_worker/transport/normal_task_submitter.cc#L33>`__ for all the ``ObjectRef`` arguments to be available. Available means tasks that produce those ``ObjectRef``\s finished execution and the data is available somewhere in the cluster.
1. If the object pointed to by the ``ObjectRef`` is in the plasma store, the ``ObjectRef`` itself is sent to the executor and the executor will resolve the ``ObjectRef`` to the actual data (pull from remote plasma store if needed) before calling the user function.
2. If the object pointed to by the ``ObjectRef`` is in the caller memory store, the data is `inlined <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/core_worker/transport/dependency_resolver.cc#L26>`__ and sent to the executor as part of the ``PushTask`` RPC just like other pass-by-value inline arguments.
2. Once all the arguments are available, ``NormalTaskSubmitter`` will try to find an idle worker to execute the task. ``NormalTaskSubmitter`` gets workers for task execution from raylet via a process called worker lease and this is where scheduling happens.
Specifically, it will `send <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/core_worker/transport/normal_task_submitter.cc#L350>`__ a ``RequestWorkerLease`` RPC to a `selected <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/core_worker/transport/normal_task_submitter.cc#L339>`__ (it's either the local raylet or a data-locality-favored raylet) raylet for a worker lease.
3. Raylet `handles <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/raylet/node_manager.cc#L1754>`__ the ``RequestWorkerLease`` RPC.
4. When the ``RequestWorkerLease`` RPC returns with a leased worker address in the response, a worker lease is granted to the caller to execute the task. If the ``RequestWorkerLease`` response contains another raylet address instead, ``NormalTaskSubmitter`` will then request a worker lease from the specified raylet. This process continues until a worker lease is obtained.
Executing a task
----------------
Once a leased worker is obtained, the task execution starts.
1. ``NormalTaskSubmitter`` `sends <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/core_worker/transport/normal_task_submitter.cc#L568>`__ a ``PushTask`` RPC to the leased worker with the ``TaskSpecification`` to execute.
2. The executor `receives <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/core_worker/core_worker.cc#L3885>`__ the ``PushTask`` RPC and executes (`1 <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/core_worker/core_worker.cc#L3948>`__ -> `2 <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/core_worker/transport/task_receiver.cc#L62>`__ -> `3 <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/core_worker/core_worker.cc#L520>`__ -> `4 <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/core_worker/core_worker.cc#L3420>`__ -> `5 <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/python/ray/_raylet.pyx#L2318>`__) the task.
3. First step of executing the task is `getting <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/core_worker/core_worker.cc#L3789>`__ all the pass-by-reference arguments from the local plasma store (data is already pulled from remote plasma store to the local plasma store during scheduling).
4. Then the executor `gets <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/python/ray/_raylet.pyx#L2206>`__ the pickled function bytes from GCS key-value store and unpickles it.
5. The next step is `unpickling <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/python/ray/_raylet.pyx#L1871>`__ the arguments.
6. Finally, the user function is `called <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/python/ray/_raylet.pyx#L1925>`__.
Getting the return value
------------------------
After the user function is executed, the caller can get the return values.
1. After the user function returns, the executor `gets and stores <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/python/ray/_raylet.pyx#L4308>`__ all the return values. If the return value is a `small <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/core_worker/core_worker.cc#L3272>`__ object and the total size of such return values so far is below the `threshold <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/core_worker/core_worker.cc#L3274>`__, it is returned directly to the caller as part of the ``PushTask`` RPC response. `Otherwise <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/core_worker/core_worker.cc#L3279>`__, it is put in the local plasma store and the reference is returned to the caller.
2. When the caller `receives <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/core_worker/transport/normal_task_submitter.cc#L579>`__ the ``PushTask`` RPC response, it `stores <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/core_worker/task_manager.cc#L511>`__ the return values (actual data if the return value is small or a special value indicating the data is in plasma store if the return value is big) in the local memory store.
3. When the return value is `added <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/core_worker/task_manager.cc#L511>`__ to the local memory store, ``ray.get()`` is `unblocked <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/core_worker/store_provider/memory_store/memory_store.cc#L373>`__ and returns the value directly if the object is small, or it will `get <https://github.com/ray-project/ray/blob/e832bd843870cde7e66e7019ea82a366836f24d5/src/ray/core_worker/core_worker.cc#L1965>`__ from the local plasma store (pull from remote plasma store first if needed) if the object is big.
doc/source/ray-core/internals/token-authentication.rst
+225
View File
@@ -0,0 +1,225 @@
.. _token-authentication:
Token Authentication
====================
Ray v2.52.0 introduced support for token authentication, enabling Ray to enforce the use of a
single, statically generated token in the authorization header for all requests to the Ray
Dashboard, GCS server, and other control-plane services.
This document covers the design and architecture of token authentication in Ray, including
configuration, token loading, propagation, and verification across C++, Python, and the Ray dashboard.
Authentication Modes
--------------------
Ray's authentication behavior is controlled by the **RAY_AUTH_MODE** environment variable.
As of now, Ray supports two modes:
- ``disabled`` - Default; no authentication.
- ``token`` - Static bearer token authentication.
**RAY_AUTH_MODE** must be set via the environment and should be configured consistently on every
node in the Ray cluster. When ``RAY_AUTH_MODE=token``, token authentication is enabled and all
supported RPC and HTTP entry points enforce token based authentication.
Token Sources and Precedence
----------------------------
Once token auth is enabled, Ray looks for the token in the following order (highest to lowest
precedence):
1. **RAY_AUTH_TOKEN** (environment variable): If set and non-empty, this value is used directly
as the token string.
2. **RAY_AUTH_TOKEN_PATH** (environment variable pointing to file): If set, Ray reads the token
from that file. If the file cannot be read or is empty, Ray treats this as a fatal
misconfiguration and aborts rather than silently falling back.
3. **Default token file path**: If neither of the above are set, Ray falls back to a default path:
- ``~/.ray/auth_token`` on POSIX systems
- ``%USERPROFILE%\.ray\auth_token`` on Windows
For local clusters started with ``ray.init()`` and auth enabled, Ray automatically generates a
new token and persists it at the default path if no token exists.
.. note::
Whitespace is stripped when reading the token from files to avoid issues from trailing newlines.
Token Propagation and Verification
----------------------------------
Common Expectations
~~~~~~~~~~~~~~~~~~~
Across both C++ and Python, gRPC servers expect the token to be present in the authorization
metadata key as:
.. code-block:: text
Authorization: Bearer <token_value>
HTTP servers similarly expect one of:
1. ``Authorization: Bearer <token>`` - Used by Ray CLI and other internal HTTP clients.
2. Cookie ``ray-authentication-token=<token>`` - Used by the browser-based dashboard.
3. ``X-Ray-Authorization: Bearer <token>`` - Used by KubeRay and environments where the standard
``Authorization`` header may be stripped by a proxy.
C++ Clients and Servers
~~~~~~~~~~~~~~~~~~~~~~~
On the C++ side, token attachment to outgoing RPCs is automated using gRPC's interceptor API.
The client interceptor is defined in
`token_auth_client_interceptor.h <https://github.com/ray-project/ray/blob/master/src/ray/rpc/authentication/token_auth_client_interceptor.h>`_.
All production C++ gRPC channels must be created through the ``BuildChannel()`` helper, which
wires in the interceptor when token auth is enabled. Ray developers must not create channels
directly with ``grpc::CreateCustomChannel``; doing so would bypass token attachment.
``BuildChannel()`` is the central enforcement point that ensures all C++ clients automatically
add the correct ``Authorization: Bearer <token>`` metadata.
Server-side token validation compares the token presented by the client with the token the
cluster was started with. This check is performed in
`server_call.h <https://github.com/ray-project/ray/blob/master/src/ray/rpc/server_call.h>`_
inside the generic request handling path. Because all gRPC services inherit from the same base
call implementation, the validation applies uniformly to all C++ gRPC servers when token auth
is enabled.
Python Clients and Servers
~~~~~~~~~~~~~~~~~~~~~~~~~~
Most Python components use Cython bindings over the C++ clients, so they automatically inherit
the same token behavior without additional Python-level code.
For components that construct gRPC clients or servers directly in Python, explicit interceptors
(both sync and async) add and validate authentication metadata:
- `Client interceptors <https://github.com/ray-project/ray/blob/master/python/ray/_private/authentication/grpc_authentication_client_interceptor.py>`_
- `Server interceptors <https://github.com/ray-project/ray/blob/master/python/ray/_private/authentication/grpc_authentication_server_interceptor.py>`_
All Python gRPC clients and servers should be created using helper utilities from
`grpc_utils.py <https://github.com/ray-project/ray/blob/master/python/ray/_private/grpc_utils.py>`_.
These helpers automatically attach the correct client/server interceptors when token auth is
enabled. The convention is to always go through the shared utilities so that auth is consistently
enforced, never constructing raw gRPC channels or servers directly.
HTTP Clients and Servers
~~~~~~~~~~~~~~~~~~~~~~~~
For HTTP services, token authentication is implemented using aiohttp middleware in
`http_token_authentication.py <https://github.com/ray-project/ray/blob/master/python/ray/_private/authentication/http_token_authentication.py>`_.
The middleware must be explicitly added to each server's middleware list (e.g., ``dashboard_head``
service and ``runtime_env_agent`` service). Once configured, it:
- Extracts the token from ``Authorization`` header, ``X-Ray-Authorization`` header, or
``ray-authentication-token`` cookie.
- Validates the token and returns:
- **401 Unauthorized** for missing token.
- **403 Forbidden** for invalid token.
Client-side, HTTP callers can use the ``get_auth_headers_if_auth_enabled()`` helper to attach
headers. This helper computes ``Authorization: Bearer <token>`` if token auth is enabled and
merges it with any user-supplied headers.
.. note::
For HTTP, middleware and header injection are not automatically wired up for new services;
they must be added manually.
Ray Dashboard Flow
------------------
When a Ray cluster is started with ``RAY_AUTH_MODE=token``, accessing the dashboard triggers an
authentication flow in the UI:
1. The user sees a dialog prompting them to enter the authentication token.
2. Once the user submits the token, the frontend sends a ``POST`` request to the dashboard head's
``/api/authenticate`` endpoint with ``Authorization: Bearer <token>`` header.
3. The dashboard head validates the token.
4. If validation succeeds, the server responds with **200 OK** and instructs the browser to set
a cookie:
- Name: ``ray-authentication-token``
- Value: ``<token>``
- Attributes: ``HttpOnly``, ``SameSite=Strict`` (and ``Secure`` when running over HTTPS)
- max_age: 30 days (cookie is cleared after 30 days)
From this point on, subsequent dashboard UI API calls automatically include the cookie and
satisfy the middleware's authentication checks.
If a backend request returns **401 Unauthorized** (no token) or **403 Forbidden** (invalid token
or mode change), the dashboard UI interprets this as an authentication failure. It clears any
stale state and re-opens the authentication dialog, prompting the user to re-enter a valid token.
This approach keeps the token out of JavaScript-accessible storage and relies on standard browser
cookie mechanics to secure subsequent requests.
Ray CLI
-------
Ray CLI commands that talk to an authenticated cluster automatically load the token from the same
three mechanisms (in the same precedence order):
- **RAY_AUTH_TOKEN**, **RAY_AUTH_TOKEN_PATH**, or the default token file.
Once loaded, CLI commands pass the token along to their internal RPC calls. Depending on the
underlying implementation, they either:
- Use C++ clients (and thus C++ interceptors via ``BuildChannel()``), or
- Use Python gRPC clients/servers and the Python interceptors via ``grpc_utils.py``, or
- Use HTTP helpers that call ``get_auth_headers_if_auth_enabled()``.
From the user's perspective, as long as the token is configured via one of the supported
mechanisms, the CLI works against token-secured clusters.
ray get-auth-token Command
~~~~~~~~~~~~~~~~~~~~~~~~~~
To retrieve and share the token used by a local Ray cluster (for example, to paste into the
dashboard UI), Ray provides the ``ray get-auth-token`` command.
By default, ``ray get-auth-token`` attempts to load an existing token from:
- **RAY_AUTH_TOKEN**, **RAY_AUTH_TOKEN_PATH**, or the default token file.
If a token is found, it is printed to ``stdout`` (suitable for scripting and export). If no
token exists, the command fails with an error explaining that no token is configured.
Users can pass the ``--generate`` flag to generate a new token and store it in the default
token file path if no token is currently configured. This does not overwrite an existing token;
it only creates one when none is present.
Adding Token Authentication to New Services
-------------------------------------------
When adding new gRPC or HTTP services to Ray, follow these guidelines to ensure proper token
authentication support:
gRPC Services
~~~~~~~~~~~~~
**C++ Services:**
1. Always create gRPC channels through ``BuildChannel()`` - never use ``grpc::CreateCustomChannel``
directly.
2. Server-side validation is automatic if your service inherits from the standard base call
implementation.
**Python Services:**
1. Use helper utilities from ``grpc_utils.py`` to create clients and servers.
2. The interceptors are automatically attached when token auth is enabled.
HTTP Services
~~~~~~~~~~~~~
1. Add the authentication middleware from ``http_token_authentication.py`` to your server's
middleware list.
2. Use ``get_auth_headers_if_auth_enabled()`` for client-side header attachment.
.. note::
HTTP middleware and header injection are not automatically wired up - they must be added
manually to each new HTTP service.