wehub/ray-project--ray
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:17:40 +08:00
commit f1825c8ceb
10096 changed files with 2364182 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/serve/advanced-guides/advanced-autoscaling.md
+732
View File
@@ -0,0 +1,732 @@
(serve-advanced-autoscaling)=
# Advanced Ray Serve Autoscaling
This guide goes over more advanced autoscaling parameters in [autoscaling_config](../api/doc/ray.serve.config.AutoscalingConfig.rst) and an advanced model composition example.
(serve-autoscaling-config-parameters)=
## Autoscaling config parameters
In this section, we go into more detail about Serve autoscaling concepts as well as how to set your autoscaling config.
### [Required] Define the steady state of your system
To define what the steady state of your deployments should be, set values for `target_ongoing_requests` and `max_ongoing_requests`.
#### **[`target_ongoing_requests`](../api/doc/ray.serve.config.AutoscalingConfig.rst) [default=2]**
:::{note}
The default for `target_ongoing_requests` changed from 1.0 to 2.0 in Ray 2.32.0. You can continue to set it manually to override the default.
:::
Serve scales the number of replicas for a deployment up or down based on the average number of ongoing requests per replica. Specifically, Serve compares the *actual* number of ongoing requests per replica with the target value you set in the autoscaling config and makes upscale or downscale decisions from that. Set the target value with `target_ongoing_requests`, and Serve attempts to ensure that each replica has roughly that number of requests being processed and waiting in the queue.
Always load test your workloads. For example, if the use case is latency sensitive, you can lower the `target_ongoing_requests` number to maintain high performance. Benchmark your application code and set this number based on an end-to-end latency objective.
:::{note}
As an example, suppose you have two replicas of a synchronous deployment that has 100ms latency, serving a traffic load of 30 QPS. Then Serve assigns requests to replicas faster than the replicas can finish processing them; more and more requests queue up at the replica (these requests are "ongoing requests") as time progresses, and then the average number of ongoing requests at each replica steadily increases. Latency also increases because new requests have to wait for old requests to finish processing. If you set `target_ongoing_requests = 1`, Serve detects a higher than desired number of ongoing requests per replica, and adds more replicas. At 3 replicas, your system would be able to process 30 QPS with 1 ongoing request per replica on average.
:::
#### **`max_ongoing_requests` [default=5]**
:::{note}
The default for `max_ongoing_requests` changed from 100 to 5 in Ray 2.32.0. You can continue to set it manually to override the default.
:::
There is also a maximum queue limit that proxies respect when assigning requests to replicas. Define the limit with `max_ongoing_requests`. Set `max_ongoing_requests` to ~20 to 50% higher than `target_ongoing_requests`.
- Setting it too low can throttle throughput. Instead of being forwarded to replicas for concurrent execution, requests will tend to queue up at the proxy, waiting for replicas to finish processing existing requests.
:::{note}
`max_ongoing_requests` should be tuned higher especially for lightweight requests, else the overall throughput will be impacted.
:::
- Setting it too high can lead to imbalanced routing. Concretely, this can lead to very high tail latencies during upscale, because when the autoscaler is scaling a deployment up due to a traffic spike, most or all of the requests might be assigned to the existing replicas before the new replicas are started.
### [Required] Define upper and lower autoscaling limits
To use autoscaling, you need to define the minimum and maximum number of resources allowed for your system.
* **[`min_replicas`](../api/doc/ray.serve.config.AutoscalingConfig.rst) [default=1]**: This is the minimum number of replicas for the deployment. If you want to ensure your system can deal with a certain level of traffic at all times, set `min_replicas` to a positive number. On the other hand, if you anticipate periods of no traffic and want to scale to zero to save cost, set `min_replicas = 0`. Note that setting `min_replicas = 0` causes higher tail latencies; when you start sending traffic, the deployment scales up, and there will be a cold start time as Serve waits for replicas to be started to serve the request.
* **[`max_replicas`](../api/doc/ray.serve.config.AutoscalingConfig.rst) [default=1]**: This is the maximum number of replicas for the deployment. This should be greater than `min_replicas`. Ray Serve Autoscaling relies on the Ray Autoscaler to scale up more nodes when the currently available cluster resources (CPUs, GPUs, etc.) are not enough to support more replicas.
* **[`initial_replicas`](../api/doc/ray.serve.config.AutoscalingConfig.rst)**: This is the number of replicas that are started initially for the deployment. This defaults to the value for `min_replicas`.
### [Optional] Define how the system reacts to changing traffic
Given a steady stream of traffic and appropriately configured `min_replicas` and `max_replicas`, the steady state of your system is essentially fixed for a chosen configuration value for `target_ongoing_requests`. Before reaching steady state, however, your system is reacting to traffic shifts. How you want your system to react to changes in traffic determines how you want to set the remaining autoscaling configurations.
* **[`upscale_delay_s`](../api/doc/ray.serve.config.AutoscalingConfig.rst) [default=30s]**: This defines how long Serve waits before scaling up the number of replicas in your deployment. In other words, this parameter controls the frequency of upscale decisions. If the replicas are *consistently* serving more requests than desired for an `upscale_delay_s` number of seconds, then Serve scales up the number of replicas based on aggregated ongoing requests metrics. For example, if your service is likely to experience bursts of traffic, you can lower `upscale_delay_s` so that your application can react quickly to increases in traffic.
Ray Serve allows you to use different delays for different downscaling scenarios, providing more granular control over when replicas are removed. This is particularly useful when you want different behavior for scaling down to zero versus scaling down to a non-zero number of replicas.
* **[`downscale_delay_s`](../api/doc/ray.serve.config.AutoscalingConfig.rst) [default=600s]**: This defines how long Serve waits before scaling down the number of replicas in your deployment. If the replicas are *consistently* serving fewer requests than desired for a `downscale_delay_s` number of seconds, Serve scales down the number of replicas based on aggregated ongoing requests metrics. This delay applies to all downscaling decisions except for the optional 1→0 transition (see below). For example, if your application initializes slowly, you can increase `downscale_delay_s` to make downscaling happen more infrequently and avoid reinitialization costs when the application needs to upscale again.
* **[`downscale_to_zero_delay_s`](../api/doc/ray.serve.config.AutoscalingConfig.rst) [Optional]**: This defines how long Serve waits before scaling from one replica down to zero (only applies when `min_replicas = 0`). If not specified, the 1→0 transition uses the `downscale_delay_s` value. This is useful when you want more conservative scale-to-zero behavior. For example, you might set `downscale_delay_s = 300` for regular downscaling but `downscale_to_zero_delay_s = 1800` to wait 30 minutes before scaling to zero, avoiding cold starts for brief periods of inactivity.
* **[`upscale_smoothing_factor`](../api/doc/ray.serve.config.AutoscalingConfig.rst) [default_value=1.0] (DEPRECATED)**: This parameter is renamed to `upscaling_factor`. `upscale_smoothing_factor` will be removed in a future release.
* **[`downscale_smoothing_factor`](../api/doc/ray.serve.config.AutoscalingConfig.rst) [default_value=1.0] (DEPRECATED)**: This parameter is renamed to `downscaling_factor`. `downscale_smoothing_factor` will be removed in a future release.
* **[`upscaling_factor`](../api/doc/ray.serve.config.AutoscalingConfig.rst) [default_value=1.0]**: The multiplicative factor to amplify or moderate each upscaling decision. For example, when the application has high traffic volume in a short period of time, you can increase `upscaling_factor` to scale up the resource quickly. This parameter is like a "gain" factor to amplify the response of the autoscaling algorithm.
* **[`downscaling_factor`](../api/doc/ray.serve.config.AutoscalingConfig.rst) [default_value=1.0]**: The multiplicative factor to amplify or moderate each downscaling decision. For example, if you want your application to be less sensitive to drops in traffic and scale down more conservatively, you can decrease `downscaling_factor` to slow down the pace of downscaling.
* **[`metrics_interval_s`](../api/doc/ray.serve.config.AutoscalingConfig.rst) [default_value=10]**: In future this deployment level config will be removed in favor of cross-application level global config.
This controls how often each replica and handle sends reports on current ongoing requests to the autoscaler. ::{note} If metrics are reported infrequently, Ray Serve can take longer to notice a change in autoscaling metrics, so scaling can start later even if your delays are short. For example, if you set `upscale_delay_s = 3` but metrics are pushed every 10 seconds, Ray Serve might not see a change until the next push, so scaling up can be limited to about once every 10 seconds. ::
* **[`look_back_period_s`](../api/doc/ray.serve.config.AutoscalingConfig.rst) [default_value=30]**: This is the window over which the average number of ongoing requests per replica is calculated.
* **[`aggregation_function`](../api/doc/ray.serve.config.AutoscalingConfig.rst) [default_value="mean"]**: This controls how metrics are aggregated over the `look_back_period_s` time window. The aggregation function determines how Ray Serve combines multiple metric measurements into a single value for autoscaling decisions. Supported values:
- `"mean"` (default): Uses time-weighted average of metrics. This provides smooth scaling behavior that responds to sustained traffic patterns.
- `"max"`: Uses the maximum metric value observed. This makes autoscaling more sensitive to spikes, scaling up quickly when any replica experiences high load.
- `"min"`: Uses the minimum metric value observed. This results in more conservative scaling behavior.
For most workloads, the default `"mean"` aggregation provides the best balance. Use `"max"` if you need to react quickly to traffic spikes, or `"min"` if you prefer conservative scaling that avoids rapid fluctuations.
### How autoscaling metrics work
Understanding how metrics flow through the autoscaling system helps you configure the parameters effectively. The metrics pipeline involves several stages, each with its own timing parameters:
```
┌──────────────────────────────────────────────────────────────────────────┐
│ Metrics Pipeline Overview │
├──────────────────────────────────────────────────────────────────────────┤
│ │
│ Replicas/Handles Controller Autoscaling Policy │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Record │ Push │ Receive │ Decide │ Policy │ │
│ │ Metrics │────────────>│ Metrics │──────────>│ Runs │ │
│ │ (10s) │ (10s) │ │ (0.1s) │ │ │
│ └──────────┘ │ Aggregate│ └──────────┘ │
│ │ (30s) │ │
│ └──────────┘ │
│ │
└──────────────────────────────────────────────────────────────────────────┘
```
#### Stage 1: Metric recording
Replicas and deployment handles continuously record autoscaling metrics:
- **What**: Number of ongoing requests (queued + running)
- **Frequency**: Every 10s (configurable via [`metrics_interval_s`](../api/doc/ray.serve.config.AutoscalingConfig.rst))
- **Storage**: Metrics are stored locally as a timeseries
#### Stage 2: Metric pushing
Periodically, replicas and handles push their metrics to the controller:
- **Frequency**: Every 10s (configurable via `RAY_SERVE_REPLICA_AUTOSCALING_METRIC_PUSH_INTERVAL_S` and `RAY_SERVE_HANDLE_AUTOSCALING_METRIC_PUSH_INTERVAL_S`)
- **Data sent**: Both raw timeseries data and pre-aggregated metrics
- **Raw timeseries**: Data points are clipped to the [`look_back_period_s`](../api/doc/ray.serve.config.AutoscalingConfig.rst) window before sending (only recent measurements within the window are sent)
- **Pre-aggregated metrics**: A simple average computed over the [`look_back_period_s`](../api/doc/ray.serve.config.AutoscalingConfig.rst) window at the replica/handle
- **Controller usage**: The controller decides which data to use based on the `RAY_SERVE_AGGREGATE_METRICS_AT_CONTROLLER` setting (see Stage 3 below)
#### Stage 3: Metric aggregation
The controller aggregates metrics to compute total ongoing requests across all replicas. Ray Serve supports two aggregation modes (controlled by `RAY_SERVE_AGGREGATE_METRICS_AT_CONTROLLER`):
**Simple mode (default - `RAY_SERVE_AGGREGATE_METRICS_AT_CONTROLLER=0`):**
- **Input**: Pre-aggregated simple averages from replicas/handles (already clipped to [`look_back_period_s`](../api/doc/ray.serve.config.AutoscalingConfig.rst))
- **Method**: Sums the pre-aggregated values from all sources. Each component computes a simple average (arithmetic mean) before sending.
- **Output**: Single value representing total ongoing requests
- **Characteristics**: Lightweight and works well for most workloads. However, because it uses simple averages rather than time-weighted averages, it can be less accurate when replicas have different metric reporting intervals or when metrics arrive at different times.
**Aggregate mode (experimental - `RAY_SERVE_AGGREGATE_METRICS_AT_CONTROLLER=1`):**
- **Input**: Raw timeseries data from replicas/handles (already clipped to [`look_back_period_s`](../api/doc/ray.serve.config.AutoscalingConfig.rst))
- **Method**: Time-weighted aggregation using the [`aggregation_function`](../api/doc/ray.serve.config.AutoscalingConfig.rst) (mean, max, or min). Uses an instantaneous merge approach that treats metrics as right-continuous step functions.
- **Output**: Single value representing total ongoing requests
- **Characteristics**: Provides more mathematically accurate aggregation, especially when replicas report metrics at different intervals or you need precise time-weighted averages. The trade-off is increased controller overhead.
:::{note}
The [`aggregation_function`](../api/doc/ray.serve.config.AutoscalingConfig.rst) parameter only applies in aggregate mode. In simple mode, the aggregation is always a sum of the pre-computed simple averages.
:::
:::{note}
The long-term plan is to deprecate simple mode in favor of aggregate mode. Aggregate mode provides more accurate metrics aggregation and will become the default in a future release. Consider testing aggregate mode(`RAY_SERVE_AGGREGATE_METRICS_AT_CONTROLLER=1`) in your deployments to prepare for this transition.
:::
#### Stage 4: Policy execution
The autoscaling policy runs frequently to make scaling decisions, see [Custom policy for deployment](#custom-policy-for-deployment) for details on implementing custom scaling logic:
- **Frequency**: Every 0.1s (configurable via `RAY_SERVE_CONTROL_LOOP_INTERVAL_S`)
- **Input**: [`AutoscalingContext`](../api/doc/ray.serve.config.AutoscalingContext.rst)
- **Output**: Tuple of `(target_replicas, updated_policy_state)`
#### Timing parameter interactions
The timing parameters interact in important ways:
**Recording vs pushing intervals:**
- Push interval ≥ Recording interval
- Recording interval (10s) determines granularity of data
- Push interval (10s) determines how fresh the controller's data is
- With default values: Each push contains 1 data points (10s ÷ 10s)
**Push interval vs look-back period:**
- [`look_back_period_s`](../api/doc/ray.serve.config.AutoscalingConfig.rst) (30s) should be > push interval (10s)
- If look-back is too short, you won't have enough data for stable decisions
- If look-back is too long, autoscaling becomes less responsive
**Push interval vs control loop:**
- Control loop (0.1s) runs much faster than metrics arrive (10s)
- Most control loop iterations reuse existing metrics
- New scaling decisions only happen when fresh metrics arrive
**Push interval vs upscale/downscale delays:**
- Delays control when Ray Serve applies a scale up or scale down.
- The metrics push interval controls how quickly Ray Serve receives fresh metrics.
- If the push interval < delay, Ray Serve can use multiple metric updates before it scales.
- Example: push every 10s with `upscale_delay_s = 20` means up to 2 new metric updates before scaling
**Recommendation:** Keep default values unless you have specific needs. If you need faster autoscaling, decrease push intervals first, then adjust delays.
### Environment variables
Several environment variables control autoscaling behavior at a lower level. These variables affect metrics collection and the control loop timing:
#### Control loop and timeout settings
* **`RAY_SERVE_CONTROL_LOOP_INTERVAL_S`** (default: 0.1s): How often the Ray Serve controller runs the autoscaling control loop. Your autoscaling policy function executes at this frequency. The default value of 0.1s means policies run approximately 10 times per second.
* **`RAY_SERVE_RECORD_AUTOSCALING_STATS_TIMEOUT_S`** (default: 10.0s): Maximum time allowed for the `record_autoscaling_stats()` method to complete in custom metrics collection. If this timeout is exceeded, the metrics collection fails and a warning is logged.
* **`RAY_SERVE_MIN_HANDLE_METRICS_TIMEOUT_S`** (default: 10.0s): Minimum timeout for handle metrics collection. The system uses the maximum of this value and `2 * `[`metrics_interval_s`](../api/doc/ray.serve.config.AutoscalingConfig.rst) to determine when to drop stale handle metrics.
#### Advanced feature flags
* **`RAY_SERVE_AGGREGATE_METRICS_AT_CONTROLLER`** (default: false): Enables an experimental metrics aggregation mode where the controller aggregates raw timeseries data instead of using pre-aggregated metrics. This mode provides more accurate time-weighted averages but may increase controller overhead. See Stage 3 in "How autoscaling metrics work" for details.
## Model composition example
Determining the autoscaling configuration for a multi-model application requires understanding each deployment's scaling requirements. Every deployment has a different latency and differing levels of concurrency. As a result, finding the right autoscaling config for a model-composition application requires experimentation.
This example is a simple application with three deployments composed together to build some intuition about multi-model autoscaling. Assume these deployments:
* `HeavyLoad`: A mock 200ms workload with high CPU usage.
* `LightLoad`: A mock 100ms workload with high CPU usage.
* `Driver`: A driver deployment that fans out to the `HeavyLoad` and `LightLoad` deployments and aggregates the two outputs.
### Attempt 1: One `Driver` replica
First consider the following deployment configurations. Because the driver deployment has low CPU usage and is only asynchronously making calls to the downstream deployments, allocating one fixed `Driver` replica is reasonable.
::::{tab-set}
:::{tab-item} Driver
```yaml
- name: Driver
num_replicas: 1
max_ongoing_requests: 200
```
:::
:::{tab-item} HeavyLoad
```yaml
- name: HeavyLoad
max_ongoing_requests: 3
autoscaling_config:
target_ongoing_requests: 1
min_replicas: 0
initial_replicas: 0
max_replicas: 200
upscale_delay_s: 3
downscale_delay_s: 60
upscaling_factor: 0.3
downscaling_factor: 0.3
metrics_interval_s: 2
look_back_period_s: 10
```
:::
:::{tab-item} LightLoad
```yaml
- name: LightLoad
max_ongoing_requests: 3
autoscaling_config:
target_ongoing_requests: 1
min_replicas: 0
initial_replicas: 0
max_replicas: 200
upscale_delay_s: 3
downscale_delay_s: 60
upscaling_factor: 0.3
downscaling_factor: 0.3
metrics_interval_s: 2
look_back_period_s: 10
```
:::
:::{tab-item} Application Code
```{literalinclude} ../doc_code/autoscale_model_comp_example.py
:language: python
:start-after: __serve_example_begin__
:end-before: __serve_example_end__
```
:::
::::
Running the same Locust load test from the [Resnet workload](resnet-autoscaling-example) generates the following results:
| | |
| ----------------------- | ---------------------- |
| HeavyLoad and LightLoad Number Replicas | <img src="https://raw.githubusercontent.com/ray-project/images/master/docs/serve/autoscaling-guide/model_composition_replicas.png" alt="comp" width="600" /> |
As you might expect, the number of autoscaled `LightLoad` replicas is roughly half that of autoscaled `HeavyLoad` replicas. Although the same number of requests per second are sent to both deployments, `LightLoad` replicas can process twice as many requests per second as `HeavyLoad` replicas can, so the deployment should need half as many replicas to handle the same traffic load.
Unfortunately, the service latency rises to from 230 to 400 ms when the number of Locust users increases to 100.
| P50 Latency | QPS |
| ------- | --- |
| ![comp_latency](https://raw.githubusercontent.com/ray-project/images/master/docs/serve/autoscaling-guide/model_comp_latency.svg) | ![comp_rps](https://raw.githubusercontent.com/ray-project/images/master/docs/serve/autoscaling-guide/model_comp_rps.svg) |
Note that the number of `HeavyLoad` replicas should roughly match the number of Locust users to adequately serve the Locust traffic. However, when the number of Locust users increased to 100, the `HeavyLoad` deployment struggled to reach 100 replicas, and instead only reached 65 replicas. The per-deployment latencies reveal the root cause. While `HeavyLoad` and `LightLoad` latencies stayed steady at 200ms and 100ms, `Driver` latencies rose from 230 to 400 ms. This suggests that the high Locust workload may be overwhelming the `Driver` replica and impacting its asynchronous event loop's performance.
### Attempt 2: Autoscale `Driver`
For this attempt, set an autoscaling configuration for `Driver` as well, with the setting `target_ongoing_requests = 20`. Now the deployment configurations are as follows:
::::{tab-set}
:::{tab-item} Driver
```yaml
- name: Driver
max_ongoing_requests: 200
autoscaling_config:
target_ongoing_requests: 20
min_replicas: 1
initial_replicas: 1
max_replicas: 10
upscale_delay_s: 3
downscale_delay_s: 60
upscaling_factor: 0.3
downscaling_factor: 0.3
metrics_interval_s: 2
look_back_period_s: 10
```
:::
:::{tab-item} HeavyLoad
```yaml
- name: HeavyLoad
max_ongoing_requests: 3
autoscaling_config:
target_ongoing_requests: 1
min_replicas: 0
initial_replicas: 0
max_replicas: 200
upscale_delay_s: 3
downscale_delay_s: 60
upscaling_factor: 0.3
downscaling_factor: 0.3
metrics_interval_s: 2
look_back_period_s: 10
```
:::
:::{tab-item} LightLoad
```yaml
- name: LightLoad
max_ongoing_requests: 3
autoscaling_config:
target_ongoing_requests: 1
min_replicas: 0
initial_replicas: 0
max_replicas: 200
upscale_delay_s: 3
downscale_delay_s: 60
upscaling_factor: 0.3
downscaling_factor: 0.3
metrics_interval_s: 2
look_back_period_s: 10
```
:::
::::
Running the same Locust load test again generates the following results:
| | |
| ------------------------------------ | ------------------- |
| HeavyLoad and LightLoad Number Replicas | <img src="https://raw.githubusercontent.com/ray-project/images/master/docs/serve/autoscaling-guide/model_composition_improved_replicas.png" alt="heavy" width="600"/> |
| Driver Number Replicas | <img src="https://raw.githubusercontent.com/ray-project/images/master/docs/serve/autoscaling-guide/model_composition_improved_driver_replicas.png" alt="driver" width="600"/> |
With up to 6 `Driver` deployments to receive and distribute the incoming requests, the `HeavyLoad` deployment successfully scales up to 90+ replicas, and `LightLoad` up to 47 replicas. This configuration helps the application latency stay consistent as the traffic load increases.
| Improved P50 Latency | Improved RPS |
| ---------------- | ------------ |
| ![comp_latency](https://raw.githubusercontent.com/ray-project/images/master/docs/serve/autoscaling-guide/model_composition_improved_latency.svg) | ![comp_latency](https://raw.githubusercontent.com/ray-project/images/master/docs/serve/autoscaling-guide/model_comp_improved_rps.svg) |
## Troubleshooting guide
### Unstable number of autoscaled replicas
If the number of replicas in your deployment keeps oscillating even though the traffic is relatively stable, try the following:
* Set a smaller `upscaling_factor` and `downscaling_factor`. Setting both values smaller than one helps the autoscaler make more conservative upscale and downscale decisions. It effectively smooths out the replicas graph, and there will be less "sharp edges".
* Set a `look_back_period_s` value that matches the rest of the autoscaling config. For longer upscale and downscale delay values, a longer look back period can likely help stabilize the replica graph, but for shorter upscale and downscale delay values, a shorter look back period may be more appropriate. For instance, the following replica graphs show how a deployment with `upscale_delay_s = 3` works with a longer vs shorter look back period.
| `look_back_period_s = 30` | `look_back_period_s = 3` |
| ------------------------------------------------ | ----------------------------------------------- |
| ![look-back-before](https://raw.githubusercontent.com/ray-project/images/master/docs/serve/autoscaling-guide/look_back_period_before.png) | ![look-back-after](https://raw.githubusercontent.com/ray-project/images/master/docs/serve/autoscaling-guide/look_back_period_after.png) |
### High spikes in latency during bursts of traffic
If you expect your application to receive bursty traffic, and at the same time want the deployments to scale down in periods of inactivity, you are likely concerned about how quickly the deployment can scale up and respond to bursts of traffic. While an increase in latency initially during a burst in traffic may be unavoidable, you can try the following to improve latency during bursts of traffic.
* Set a lower `upscale_delay_s`. The autoscaler always waits `upscale_delay_s` seconds before making a decision to upscale, so lowering this delay allows the autoscaler to react more quickly to changes, especially bursts, of traffic.
* Set a larger `upscaling_factor`. If `upscaling_factor > 1`, then the autoscaler scales up more aggressively than normal. This setting can allow your deployment to be more sensitive to bursts of traffic.
* Lower the [`metrics_interval_s`](../api/doc/ray.serve.config.AutoscalingConfig.rst). Always set [`metrics_interval_s`](../api/doc/ray.serve.config.AutoscalingConfig.rst) to be less than or equal to `upscale_delay_s`, otherwise upscaling is delayed because the autoscaler doesn't receive fresh information often enough.
* Set a lower `max_ongoing_requests`. If `max_ongoing_requests` is too high relative to `target_ongoing_requests`, then when traffic increases, Serve might assign most or all of the requests to the existing replicas before the new replicas are started. This setting can lead to very high latencies during upscale.
### Deployments scaling down too quickly
You may observe that deployments are scaling down too quickly. Instead, you may want the downscaling to be much more conservative to maximize the availability of your service.
* Set a longer `downscale_delay_s`. The autoscaler always waits `downscale_delay_s` seconds before making a decision to downscale, so by increasing this number, your system has a longer "grace period" after traffic drops before the autoscaler starts to remove replicas.
* Set a smaller `downscaling_factor`. If `downscaling_factor < 1`, then the autoscaler removes *less replicas* than what it thinks it should remove to achieve the target number of ongoing requests. In other words, the autoscaler makes more conservative downscaling decisions.
| `downscaling_factor = 1` | `downscaling_factor = 0.5` |
| ------------------------------------------------ | ----------------------------------------------- |
| ![downscale-smooth-before](https://raw.githubusercontent.com/ray-project/images/master/docs/serve/autoscaling-guide/downscale_smoothing_factor_before.png) | ![downscale-smooth-after](https://raw.githubusercontent.com/ray-project/images/master/docs/serve/autoscaling-guide/downscale_smoothing_factor_after.png) |
(serve-custom-autoscaling-policies)=
## Custom autoscaling policies
:::{warning}
Custom autoscaling policies are experimental and may change in future releases.
:::
Ray Serve’s built-in, request-driven autoscaling works well for most apps. Use **custom autoscaling policies** when you need more control—e.g., scaling on external metrics (CloudWatch, Prometheus), anticipating predictable traffic (scheduled batch jobs), or applying business logic that goes beyond queue thresholds.
Custom policies let you implement scaling logic based on any metrics or rules you choose.
### Custom policy for deployment
A custom autoscaling policy is a user-provided Python function that takes an [`AutoscalingContext`](../api/doc/ray.serve.config.AutoscalingContext.rst) and returns a tuple `(target_replicas, policy_state)` for a single Deployment.
An `AutoscalingContext` object provides the following information to the custom autoscaling policy:
* **Current state:** Current replica count and deployment metadata.
* **Built-in metrics:** Total requests, queued requests, per-replica counts.
* **Custom metrics:** Values your deployment reports via `record_autoscaling_stats()`. (See below.)
* **Capacity bounds:** `min` / `max` replica limits adjusted for current cluster capacity.
* **Policy state:** A `dict` you can use to persist arbitrary state across control-loop iterations.
* **Timing:** Timestamps of the last scale actions and “now”.
The following example showcases a policy that scales up during business hours and evening batch processing, and scales down during off-peak hours:
`autoscaling_policy.py` file:
```{literalinclude} ../doc_code/autoscaling_policy.py
:language: python
:start-after: __begin_scheduled_batch_processing_policy__
:end-before: __end_scheduled_batch_processing_policy__
```
`main.py` file:
```{literalinclude} ../doc_code/scheduled_batch_processing.py
:language: python
:start-after: __serve_example_begin__
:end-before: __serve_example_end__
```
Policies are defined **per deployment**. If you don’t provide one, Ray Serve falls back to its built-in request-based policy.
The policy function is invoked by the Ray Serve controller every `RAY_SERVE_CONTROL_LOOP_INTERVAL_S` seconds (default **0.1s**), so your logic runs against near-real-time state.
Your policy can return an `int` or a `float` for `target_replicas`. If it returns a float, Ray Serve converts it to an integer replica count by rounding up to the next greatest integer.
:::{warning}
Keep policy functions **fast and lightweight**. Slow logic can block the Serve controller and degrade cluster responsiveness.
:::
### Applying standard autoscaling parameters to custom policies
Ray Serve automatically applies the following standard autoscaling parameters from your [`AutoscalingConfig`](../api/doc/ray.serve.config.AutoscalingConfig.rst) to custom policies:
- `upscale_delay_s`, `downscale_delay_s`, `downscale_to_zero_delay_s`
- `upscaling_factor`, `downscaling_factor`
- `min_replicas`, `max_replicas`
The following example shows a custom autoscaling policy with standard autoscaling parameters applied.
```{literalinclude} ../doc_code/autoscaling_policy.py
:language: python
:start-after: __begin_apply_autoscaling_config_example__
:end-before: __end_apply_autoscaling_config_example__
```
```{literalinclude} ../doc_code/autoscaling_policy.py
:language: python
:start-after: __begin_apply_autoscaling_config_usage__
:end-before: __end_apply_autoscaling_config_usage__
```
::::{note}
Your policy function should return the "raw" desired number of replicas. Ray Serve applies the `autoscaling_config` settings (delays, factors, and bounds) on top of your decision.
Your policy can return an `int` or a `float` "raw desired" replica count. Ray Serve returns an integer decision number.
::::
### Custom metrics
You can make richer decisions by emitting your own metrics from the deployment. Implement `record_autoscaling_stats()` to return a `dict[str, float]`. Ray Serve will surface these values in the [`AutoscalingContext`](../api/doc/ray.serve.config.AutoscalingContext.rst).
This example demonstrates how deployments can provide their own metrics (CPU usage, memory usage) and how autoscaling policies can use these metrics to make scaling decisions:
`autoscaling_policy.py` file:
```{literalinclude} ../doc_code/autoscaling_policy.py
:language: python
:start-after: __begin_custom_metrics_autoscaling_policy__
:end-before: __end_custom_metrics_autoscaling_policy__
```
`main.py` file:
```{literalinclude} ../doc_code/custom_metrics_autoscaling.py
:language: python
:start-after: __serve_example_begin__
:end-before: __serve_example_end__
```
:::{note}
The `record_autoscaling_stats()` method can be either synchronous or asynchronous. It must complete within the timeout specified by `RAY_SERVE_RECORD_AUTOSCALING_STATS_TIMEOUT_S` (default 10 seconds).
:::
In your policy, access custom metrics via:
* **`ctx.raw_metrics[metric_name]`** — A mapping of replica IDs to lists of raw metric values. The number of data points stored for each replica depends on the [`look_back_period_s`](../api/doc/ray.serve.config.AutoscalingConfig.rst) (the sliding window size) and [`metrics_interval_s`](../api/doc/ray.serve.config.AutoscalingConfig.rst) (the metric recording interval).
* **`ctx.aggregated_metrics[metric_name]`** — A time-weighted average computed from the raw metric values for each replica.
### Class-based policies
When your policy needs long-running setup — such as polling an external metrics service, maintaining a persistent connection, or running background computation — you can define it as a **class** instead of a plain function. Pass the class reference through `policy_function` and supply constructor arguments through `policy_kwargs`.
Ray Serve instantiates the class once on the controller when the deployment starts. `__init__` runs one-time setup, and `__call__` runs on every autoscaling tick with the current [`AutoscalingContext`](../api/doc/ray.serve.config.AutoscalingContext.rst).
The following example shows a policy that reads a target replica count from a JSON file in a background loop. In production you could replace the file read with an HTTP call, a message-queue consumer, or any other async IO operation:
`class_based_autoscaling_policy.py` file:
```{literalinclude} ../doc_code/class_based_autoscaling_policy.py
:language: python
:start-after: __begin_class_based_autoscaling_policy__
:end-before: __end_class_based_autoscaling_policy__
```
`main.py` file:
```{literalinclude} ../doc_code/class_based_autoscaling.py
:language: python
:start-after: __serve_example_begin__
:end-before: __serve_example_end__
```
:::{note}
The instance lives only on the Serve controller and is never serialized after creation, so it's safe to hold non-picklable state such as `asyncio.Task` objects, open connections, or thread pools. `policy_kwargs` values must be JSON-serializable because they travel through the deployment config.
:::
:::{tip}
If you're using `@task_consumer` deployments for asynchronous inference, Ray Serve provides a built-in `AsyncInferenceAutoscalingPolicy` that scales based on message queue length. See [Asynchronous Inference: Autoscaling](serve-async-inference-autoscaling) for setup and configuration.
:::
### Application level autoscaling
By default, each deployment in Ray Serve autoscales independently. When you have multiple deployments that need to scale in a coordinated way—such as deployments that share backend resources, have dependencies on each other, or need load-aware routing—you can define an **application-level autoscaling policy**. This policy makes scaling decisions for all deployments within an application simultaneously.
#### Define an application level policy
An application-level autoscaling policy is a function that takes a `dict[DeploymentID, AutoscalingContext]` objects (one per deployment) and returns a tuple of `(decisions, policy_state)`. Each context contains metrics and bounds for one deployment, and the policy returns target replica counts for all deployments.
The `policy_state` returned from an application-level policy must be a `Dict[DeploymentID, Dict]`— a dictionary mapping each deployment ID to its own state dictionary. Serve stores this per-deployment state and on the next control-loop iteration, injects each deployment's state back into that deployment's `AutoscalingContext.policy_state`. The per deployment number replicas returned from the policy can be an `int` or a `float`. If it returns a float, Ray Serve converts it to an integer replica count by rounding up to the next greatest integer.
Serve itself does not interpret the contents of `policy_state`. All the keys in each deployment's state dictionary are user-controlled except for internal keys that are used when default parameters are applied to custom autoscaling policies. The following example shows a policy that scales deployments based on their relative load, ensuring that downstream deployments have enough capacity for upstream traffic:
`autoscaling_policy.py` file:
```{literalinclude} ../doc_code/autoscaling_policy.py
:language: python
:start-after: __begin_application_level_autoscaling_policy__
:end-before: __end_application_level_autoscaling_policy__
```
The following example shows a stateful application-level policy that persists state between control-loop iterations:
`autoscaling_policy.py` file:
```{literalinclude} ../doc_code/autoscaling_policy.py
:language: python
:start-after: __begin_stateful_application_level_policy__
:end-before: __end_stateful_application_level_policy__
```
#### Configure application level autoscaling
To use an application-level policy, you need to define your deployments:
`main.py` file:
```{literalinclude} ../doc_code/application_level_autoscaling.py
:language: python
:start-after: __serve_example_begin__
:end-before: __serve_example_end__
```
Then specify the application-level policy in your application config:
`serve.yaml` file:
```{literalinclude} ../doc_code/application_level_autoscaling.yaml
:language: yaml
:emphasize-lines: 4-5
```
:::{note}
Programmatic configuration of application-level autoscaling policies through `serve.run()` will be supported in a future release.
:::
:::{note}
When you specify both a deployment-level policy and an application-level policy, the application-level policy takes precedence. Ray Serve logs a warning if you configure both.
:::
#### Applying standard autoscaling parameters to application-level policies
Ray Serve automatically applies standard autoscaling parameters (delays, factors, and min/max bounds) to application-level policies on a per-deployment basis. These parameters include:
- `upscale_delay_s`, `downscale_delay_s`, `downscale_to_zero_delay_s`
- `upscaling_factor`, `downscaling_factor`
- `min_replicas`, `max_replicas`
The YAML configuration file shows the default parameters applied to the application level policy.
```{literalinclude} ../doc_code/application_level_autoscaling_with_defaults.yaml
:language: yaml
```
Your application level policy can return per deployment desired replicas as `int` or `float` values. Ray Serve applies the autoscaling config parameters per deployment and returns integer decisions.
:::{warning}
### Gotchas and limitations
When you provide a custom policy, Ray Serve can fully support it as long as it's simple, self-contained Python code that relies only on the standard library. Once the policy becomes more complex, such as depending on other custom modules or packages, you need to bundle those modules into the Docker image or environment. This is because Ray Serve uses `cloudpickle` to serialize custom policies and it doesn't vendor transitive dependencies—if your policy inherits from a superclass in another module or imports custom packages, those must exist in the target environment. Additionally, environment parity matters: differences in Python version, `cloudpickle` version, or library versions can affect deserialization.
#### Alternatives for complex policies
When your custom autoscaling policy has complex dependencies or you want better control over versioning and deployment, you have several alternatives:
- **Contribute to Ray Serve**: If your policy is general-purpose and might benefit others, consider contributing it to Ray Serve as a built-in policy by opening a feature request or pull request on the [Ray GitHub repository](https://github.com/ray-project/ray/issues). The recommended location for the implementation is `python/ray/serve/autoscaling_policy.py`.
- **Ensure dependencies in your environment**: Make sure that the external dependencies are installed in your Docker image or environment.
:::
(serve-external-scale-api)=
### External scaling API
:::{warning}
This API is in alpha and may change before becoming stable.
:::
The external scaling API provides programmatic control over the number of replicas for any deployment in your Ray Serve application. Unlike Ray Serve's built-in autoscaling, which scales based on queue depth and ongoing requests, this API allows you to scale based on any external criteria you define.
#### Example: Predictive scaling
This example shows how to implement predictive scaling based on historical patterns or forecasts. You can preemptively scale up before anticipated traffic spikes by running an external script that adjusts replica counts based on time of day.
##### Define the deployment
The following example creates a simple text processing deployment that you can scale externally. Save this code to a file named `external_scaler_predictive.py`:
```{literalinclude} ../doc_code/external_scaler_predictive.py
:language: python
:start-after: __serve_example_begin__
:end-before: __serve_example_end__
```
##### Configure external scaling
Before using the external scaling API, enable it in your application configuration by setting `external_scaler_enabled: true`. Save this configuration to a file named `external_scaler_config.yaml`:
```{literalinclude} ../doc_code/external_scaler_config.yaml
:language: yaml
:start-after: __external_scaler_config_begin__
:end-before: __external_scaler_config_end__
```
:::{warning}
External scaling and built-in autoscaling are mutually exclusive. You can't use both for the same application. If you set `external_scaler_enabled: true`, you **must not** configure `autoscaling_config` on any deployment in that application. Attempting to use both results in an error.
:::
##### Implement the scaling logic
The following script implements predictive scaling based on time of day and historical traffic patterns. Save this script to a file named `external_scaler_predictive_client.py`:
```{literalinclude} ../doc_code/external_scaler_predictive_client.py
:language: python
:start-after: __client_script_begin__
:end-before: __client_script_end__
```
The script uses the external scaling API endpoint to scale deployments:
- **API endpoint**: `POST http://localhost:8265/api/v1/applications/{application_name}/deployments/{deployment_name}/scale`
- **Request body**: `{"target_num_replicas": <number>}` (must conform to the [`ScaleDeploymentRequest`](../api/doc/ray.serve.schema.ScaleDeploymentRequest.rst) schema)
The scaling client continuously adjusts the number of replicas based on the time of day:
- Business hours (9 AM - 5 PM): 10 replicas
- Off-peak hours: 3 replicas
##### Run the example
Follow these steps to run the complete example:
1. Start the Ray Serve application with the configuration:
```bash
serve run external_scaler_config.yaml
```
2. Run the predictive scaling client in a separate terminal:
```bash
python external_scaler_predictive_client.py
```
The client adjusts replica counts automatically based on the time of day. You can monitor the scaling behavior in the Ray dashboard or by checking the application logs.
#### Important considerations
Understanding how the external scaler interacts with your deployments helps you build reliable scaling logic:
- **Idempotent API calls**: The scaling API is idempotent. You can safely call it multiple times with the same `target_num_replicas` value without side effects. This makes it safe to run your scaling logic on a schedule or in response to repeated metric updates.
- **Interaction with serve deploy**: When you upgrade your service with `serve deploy`, the number of replicas you set through the external scaler API stays intact. This behavior matches what you'd expect from Ray Serve's built-in autoscaler—deployment updates don't reset replica counts.
- **Query current replica count**: You can get the current number of replicas for any deployment by querying the GET `/applications` API:
```bash
curl -X GET http://localhost:8265/api/serve/applications/ \
```
The response follows the [`ServeInstanceDetails`](../api/doc/ray.serve.schema.ServeInstanceDetails.rst) schema, which includes an `applications` field containing a dictionary with application names as keys. Each application includes detailed information about all its deployments, including current replica counts. Use this information to make informed scaling decisions. For example, you might scale up gradually by adding a percentage of existing replicas rather than jumping to a fixed number.
- **Initial replica count**: When you deploy an application for the first time, Ray Serve creates the number of replicas specified in the `num_replicas` field of your deployment configuration. The external scaler can then adjust this count dynamically based on your scaling logic.
doc/source/serve/advanced-guides/app-builder-guide.md
+128
View File
@@ -0,0 +1,128 @@
(serve-app-builder-guide)=
# Pass Arguments to Applications
This section describes how to pass arguments to your applications using an application builder function.
## Defining an application builder
When writing an application, there are often parameters that you want to be able to easily change in development or production. For example, you might have a path to trained model weights and want to test out a newly trained model. In Ray Serve, these parameters are typically passed to the constructor of your deployments using `.bind()`. This pattern allows you to configure deployments using ordinary Python code, but it requires modifying the code whenever one of the parameters needs to change.
To pass arguments without changing the code, define an "application builder" function that takes an arguments dictionary (or [Pydantic object](typed-app-builders)) and returns the built application to be run.
```{literalinclude} ../doc_code/app_builder.py
:start-after: __begin_untyped_builder__
:end-before: __end_untyped_builder__
:language: python
```
You can use this application builder function as the import path in the `serve run` CLI command or the config file (as shown below). To avoid writing code to handle type conversions and missing arguments, use a [Pydantic object](typed-app-builders) instead.
### Passing arguments via `serve run`
Pass arguments to the application builder from `serve run` using the following syntax:
```bash
$ serve run hello:app_builder key1=val1 key2=val2
```
The arguments are passed to the application builder as a dictionary, in this case `{"key1": "val1", "key2": "val2"}`. For example, to pass a new message to the `HelloWorld` app defined above (with the code saved in `hello.py`):
```bash
% serve run hello:app_builder message="Hello from CLI"
2023-05-16 10:47:31,641 INFO scripts.py:404 -- Running import path: 'hello:app_builder'.
2023-05-16 10:47:33,344 INFO worker.py:1615 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265
(ServeController pid=56826) INFO 2023-05-16 10:47:35,115 controller 56826 deployment_state.py:1244 - Deploying new version of deployment default_HelloWorld.
(ServeController pid=56826) INFO 2023-05-16 10:47:35,141 controller 56826 deployment_state.py:1483 - Adding 1 replica to deployment default_HelloWorld.
(ProxyActor pid=56828) INFO: Started server process [56828]
(ServeReplica:default_HelloWorld pid=56830) Message: Hello from CLI
2023-05-16 10:47:36,131 SUCC scripts.py:424 -- Deployed Serve app successfully.
```
Notice that the "Hello from CLI" message is printed from within the deployment constructor.
### Passing arguments via config file
Pass arguments to the application builder in the config file's `args` field:
```yaml
applications:
- name: MyApp
import_path: hello:app_builder
args:
message: "Hello from config"
```
For example, to pass a new message to the `HelloWorld` app defined above (with the code saved in `hello.py` and the config saved in `config.yaml`):
```bash
% serve run config.yaml
2023-05-16 10:49:25,247 INFO scripts.py:351 -- Running config file: 'config.yaml'.
2023-05-16 10:49:26,949 INFO worker.py:1615 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265
2023-05-16 10:49:28,678 SUCC scripts.py:419 -- Submitted deploy config successfully.
(ServeController pid=57109) INFO 2023-05-16 10:49:28,676 controller 57109 controller.py:559 - Building application 'MyApp'.
(ProxyActor pid=57111) INFO: Started server process [57111]
(ServeController pid=57109) INFO 2023-05-16 10:49:28,940 controller 57109 application_state.py:202 - Built application 'MyApp' successfully.
(ServeController pid=57109) INFO 2023-05-16 10:49:28,942 controller 57109 deployment_state.py:1244 - Deploying new version of deployment MyApp_HelloWorld.
(ServeController pid=57109) INFO 2023-05-16 10:49:29,016 controller 57109 deployment_state.py:1483 - Adding 1 replica to deployment MyApp_HelloWorld.
(ServeReplica:MyApp_HelloWorld pid=57113) Message: Hello from config
```
Notice that the "Hello from config" message is printed from within the deployment constructor.
(typed-app-builders)=
### Typing arguments with Pydantic
:::{warning}
**Pydantic v1 Deprecation Notice:** Pydantic v1 is deprecated and Ray will drop support for it in version 2.56. If you're using Pydantic v1, upgrade to Pydantic v2 by running `pip install -U pydantic`. See [GitHub issue #58876](https://github.com/ray-project/ray/issues/58876) for more details.
:::
To avoid writing logic to parse and validate the arguments by hand, define a [Pydantic model](https://pydantic-docs.helpmanual.io/usage/models/) as the single input parameter's type to your application builder function (the parameter must be type annotated). Arguments are passed the same way, but the resulting dictionary is used to construct the Pydantic model using `model.parse_obj(args_dict)`.
```{literalinclude} ../doc_code/app_builder.py
:start-after: __begin_typed_builder__
:end-before: __end_typed_builder__
:language: python
```
```bash
% serve run hello:typed_app_builder message="Hello from CLI"
2023-05-16 10:47:31,641 INFO scripts.py:404 -- Running import path: 'hello:typed_app_builder'.
2023-05-16 10:47:33,344 INFO worker.py:1615 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265
(ServeController pid=56826) INFO 2023-05-16 10:47:35,115 controller 56826 deployment_state.py:1244 - Deploying new version of deployment default_HelloWorld.
(ServeController pid=56826) INFO 2023-05-16 10:47:35,141 controller 56826 deployment_state.py:1483 - Adding 1 replica to deployment default_HelloWorld.
(ProxyActor pid=56828) INFO: Started server process [56828]
(ServeReplica:default_HelloWorld pid=56830) Message: Hello from CLI
2023-05-16 10:47:36,131 SUCC scripts.py:424 -- Deployed Serve app successfully.
```
## Common patterns
### Multiple parametrized applications using the same builder
You can use application builders to run multiple applications with the same code but different parameters. For example, multiple applications may share preprocessing and HTTP handling logic but use many different trained model weights. The same application builder `import_path` can take different arguments to define multiple applications as follows:
```yaml
applications:
- name: Model1
import_path: my_module:my_model_code
args:
model_uri: s3://my_bucket/model_1
- name: Model2
import_path: my_module:my_model_code
args:
model_uri: s3://my_bucket/model_2
- name: Model3
import_path: my_module:my_model_code
args:
model_uri: s3://my_bucket/model_3
```
### Configuring multiple composed deployments
You can use the arguments passed to an application builder to configure multiple deployments in a single application. For example a model composition application might take weights to two different models as follows:
```{literalinclude} ../doc_code/app_builder.py
:start-after: __begin_composed_builder__
:end-before: __end_composed_builder__
:language: python
```
doc/source/serve/advanced-guides/asyncio-best-practices.md
+456
View File
@@ -0,0 +1,456 @@
(serve-asyncio-best-practices)=
# Asyncio and concurrency best practices in Ray Serve
The code that runs inside of each replica in a Ray Serve deployment runs on an asyncio event loop. Asyncio enables efficient I/O bound concurrency but requires following a few best practices for optimal performance.
This guide explains:
- When to use `async def` versus `def` in Ray Serve.
- How Ray Serve executes your code (loops, threads, and the router).
- How `max_ongoing_requests` interacts with asyncio concurrency.
- How to think about Python's GIL, native code, and true parallelism.
The examples assume the following imports unless stated otherwise:
```{literalinclude} ../doc_code/asyncio_best_practices.py
:start-after: __imports_begin__
:end-before: __imports_end__
:language: python
```
## How to choose between `async def` and `def`
Use this decision table as a starting point:
| Workload type | Recommended handler | Reason |
| --- | --- | --- |
| I/O-bound (databases, HTTP calls, queues) | `async def` | Lets the event loop handle many requests while each waits on I/O. |
| CPU-bound (model inference, heavy numeric compute) | `def` or `async def` with offload | Async alone doesn't make CPU work faster. You need more replicas, threads, or native parallelism. |
| Streaming responses | `async def` generator | Integrates with backpressure and non-blocking iteration. |
| FastAPI ingress (`@serve.ingress`) | `def` or `async def` | FastAPI runs `def` endpoints in a threadpool, so they don't block the loop. |
## How Ray Serve executes your code
At a high level, requests go through a router to a replica actor that runs your code:
```text
Client
↓
Serve router (asyncio loop A)
↓
Replica actor
├─ System / control loop
└─ User code loop (your handlers)
└─ Optional threadpool for sync methods
```
The following are the key ideas to consider when deciding to use `async def` or `def`:
- Serve uses asyncio event loops for routing and for running replicas.
- By default, user code runs on a separate event loop from the replica's main/control loop, so blocking user code doesn't interfere with health checks and autoscaling.
- Depending on the value of `RAY_SERVE_RUN_SYNC_IN_THREADPOOL`, `def` handlers may run directly on the user event loop (blocking) or in a threadpool (non-blocking for the loop).
### Pure Serve deployments (no FastAPI ingress)
For a simple deployment:
```{literalinclude} ../doc_code/asyncio_best_practices.py
:start-after: __echo_async_begin__
:end-before: __echo_async_end__
:language: python
```
- `async def __call__` runs directly on the replica's user event loop.
- While this handler awaits `asyncio.sleep`, the loop is free to start handling other requests.
For a synchronous deployment:
```{literalinclude} ../doc_code/asyncio_best_practices.py
:start-after: __blocking_echo_begin__
:end-before: __blocking_echo_end__
:language: python
```
How this method executes depends on configuration:
- With `RAY_SERVE_RUN_SYNC_IN_THREADPOOL=0` (current default), `__call__` runs directly on the user event loop and blocks it for 1 second.
- With `RAY_SERVE_RUN_SYNC_IN_THREADPOOL=1`, Serve offloads `__call__` to a threadpool so the event loop stays responsive.
### FastAPI ingress (`@serve.ingress`)
When you use FastAPI ingress, FastAPI controls how endpoints run:
```{literalinclude} ../doc_code/asyncio_best_practices.py
:start-after: __fastapi_deployment_begin__
:end-before: __fastapi_deployment_end__
:language: python
```
Important differences:
- FastAPI always dispatches `def` endpoints to a threadpool.
- In pure Serve, `def` methods run on the event loop unless you opt into threadpool behavior.
## Threadpool sizing and overrides
Serve sets a default threadpool size for user code that mirrors Python's `ThreadPoolExecutor` defaults while respecting `ray_actor_options["num_cpus"]`.
In most cases, the default is fine. If you need to tune it, you can override the default executor inside your deployment:
```{literalinclude} ../doc_code/asyncio_best_practices.py
:start-after: __threadpool_override_begin__
:end-before: __threadpool_override_end__
:language: python
```
Guidance for choosing a size:
- Default is fine in most cases.
- For I/O-blocking code, consider a threadpool larger than `num_cpus`.
- For GIL-releasing compute (NumPy/Pandas/SciPy, etc.), keep the threadpool at or below `num_cpus`.
## Blocking versus non-blocking in practice
Blocking code keeps the event loop from processing other work. Non-blocking code yields control back to the loop when it's waiting on something.
### Blocking I/O versus asynchronous I/O
Blocking I/O example:
```{literalinclude} ../doc_code/asyncio_best_practices.py
:start-after: __blocking_http_begin__
:end-before: __blocking_http_end__
:language: python
```
Even though the method is `async def`, `requests.get` blocks the loop. No other requests can run on this replica during the request call. Blocking in `async def` is still blocking.
If the blocking call hangs and you've configured a request timeout, Ray Serve cancels the request when that timeout expires, but that cancellation is best-effort. Python only delivers `asyncio` cancellation when the task cooperates and yields control back to the event loop. A synchronous call such as `requests.get` doesn't do that, so one hung request can stall the replica's event loop and prevent later requests from running on that replica.
Non-blocking equivalent with async HTTP client:
```{literalinclude} ../doc_code/asyncio_best_practices.py
:start-after: __async_http_begin__
:end-before: __async_http_end__
:language: python
```
Non-blocking equivalent using a threadpool:
```{literalinclude} ../doc_code/asyncio_best_practices.py
:start-after: __threaded_http_begin__
:end-before: __threaded_http_end__
:language: python
```
## Concurrency doesn't equal parallelism in Python
It's common to expect `async` code to "use all the cores" or make CPU-heavy code faster. asyncio doesn't do that.
### Concurrency: Handling many waiting operations
Asyncio gives you **concurrency** for I/O-bound workloads:
- While one request waits on the database, another can wait on an HTTP call.
- Handlers yield back to the event loop at each `await`.
This is ideal for high-throughput APIs that mostly wait on external systems.
### Parallelism: Using multiple CPU cores
True CPU parallelism usually comes from:
- Multiple processes (for example, multiple Serve replicas).
- Native code that releases the GIL and runs across cores.
Python's GIL means that pure Python bytecode runs one thread at a time in a process, even if you use a threadpool.
### Using GIL-releasing native code
Many numeric and ML libraries release the GIL while doing heavy work in native code:
- NumPy, many linear algebra routines.
- PyTorch and some other deep learning frameworks.
- Some image-processing or compression libraries.
In these cases, you can still get useful parallelism from threads inside a single replica process:
```{literalinclude} ../doc_code/asyncio_best_practices.py
:start-after: __numpy_deployment_begin__
:end-before: __numpy_deployment_end__
:language: python
```
However:
- GIL-releasing behavior is library-specific and sometimes operation-specific.
- Some libraries use their own internal threadpools; combining them with your own threadpools can oversubscribe CPUs.
- You should verify that your model stack is thread-safe before relying on this form of parallelism.
For predictable CPU scaling, it's usually simpler to increase the number of replicas.
### Summary
- `async def` improves **concurrency** for I/O-bound code.
- CPU-bound code doesn't become faster merely because it's `async`.
- Parallel CPU scaling comes mostly from **more processes** (replicas or tasks) and, in some cases, native code that releases the GIL.
## How `max_ongoing_requests` and replica concurrency work
Each deployment has a `max_ongoing_requests` configuration that controls how many in-flight requests a replica handles at once.
```{literalinclude} ../doc_code/asyncio_best_practices.py
:start-after: __max_ongoing_requests_begin__
:end-before: __max_ongoing_requests_end__
:language: python
```
Key points:
- Ray Serve uses an internal semaphore to limit concurrent in-flight requests per replica to `max_ongoing_requests`.
- Requests beyond that limit queue in the router or handle until capacity becomes available, or they fail with backpressure depending on configuration.
How useful `max_ongoing_requests` is depends on how your handler behaves.
### `async` handlers and `max_ongoing_requests`
With an `async def` handler that spends most of its time awaiting I/O, `max_ongoing_requests` directly controls concurrency:
```{literalinclude} ../doc_code/asyncio_best_practices.py
:start-after: __async_io_bound_begin__
:end-before: __async_io_bound_end__
:language: python
```
- Up to 100 requests can be in-flight per replica.
- While one request is waiting, the event loop can work on others.
### Blocking `def` handlers and `max_ongoing_requests`
With a blocking `def` handler that runs on the event loop (threadpool disabled), `max_ongoing_requests` doesn't give you the concurrency you expect:
```{literalinclude} ../doc_code/asyncio_best_practices.py
:start-after: __blocking_cpu_begin__
:end-before: __blocking_cpu_end__
:language: python
```
In this case:
- The event loop can only run one handler at a time.
- Even though `max_ongoing_requests=100`, the replica effectively processes requests serially.
If you enable the sync-in-threadpool behavior (see the next section), each in-flight request can run in a thread:
```{literalinclude} ../doc_code/asyncio_best_practices.py
:start-after: __cpu_with_threadpool_begin__
:end-before: __cpu_with_threadpool_end__
:language: python
```
Now:
- Up to `max_ongoing_requests` calls can be running at once.
- Real throughput depends on:
- How many threads the threadpool uses.
- Whether your workload is CPU-bound or GIL-releasing.
- Underlying native libraries and system resources.
For heavily CPU-bound workloads, it's usually better to:
- Keep `max_ongoing_requests` modest (to avoid queueing too many heavy tasks), and
- Scale **replicas** (`num_replicas`) rather than pushing a single replica's concurrency too high.
## Environment flags and sync-in-threadpool warning
Ray Serve exposes several environment variables that control how user code interacts with event loops and threads.
### `RAY_SERVE_RUN_SYNC_IN_THREADPOOL`
By default (`RAY_SERVE_RUN_SYNC_IN_THREADPOOL=0`), which means synchronous methods in a deployment run directly on the user event loop. To help you migrate to a safer model, Serve emits a warning like:
> `RAY_SERVE_RUN_SYNC_IN_THREADPOOL_WARNING`: Calling sync method '...' directly on the asyncio loop. In a future version, sync methods will be run in a threadpool by default...
This warning means:
- You have a `def` method that is currently running on the event loop.
- In a future version, that method runs in a threadpool instead.
You can opt in to the future behavior now by setting:
```bash
export RAY_SERVE_RUN_SYNC_IN_THREADPOOL=1
```
When this flag is `1`:
- Serve runs synchronous methods in a threadpool.
- The event loop is free to keep serving other requests while sync methods run.
Before enabling this in production, make sure:
- Your handler code and any shared state are thread-safe.
- Your model objects can safely be used from multiple threads, or you protect them with locks.
### `RAY_SERVE_RUN_USER_CODE_IN_SEPARATE_THREAD`
By default, Serve runs user code in a separate event loop from the replica's main/control loop:
```bash
export RAY_SERVE_RUN_USER_CODE_IN_SEPARATE_THREAD=1 # default
```
This isolation:
- Protects system tasks (health checks, controller communication) from being blocked by user code.
- Adds some amount of overhead to cross-loop communication, resulting in higher latency in request. For throughput-optimized configurations, see [High throughput optimization](serve-high-throughput).
You can disable this behavior:
```bash
export RAY_SERVE_RUN_USER_CODE_IN_SEPARATE_THREAD=0
```
Only advanced users should change this. When user code and system tasks share a loop, any blocking operation in user code can interfere with replica health and control-plane operations.
### `RAY_SERVE_RUN_ROUTER_IN_SEPARATE_LOOP`
Serve's request router is also run on its own event loop by default:
```bash
export RAY_SERVE_RUN_ROUTER_IN_SEPARATE_LOOP=1 # default
```
This ensures:
- The router can continue routing and load balancing requests even if some replicas are running slow user code.
Disabling this:
```bash
export RAY_SERVE_RUN_ROUTER_IN_SEPARATE_LOOP=0
```
makes the router share an event loop with other work. This can reduce overhead in advanced, highly optimized scenarios, but makes the system more sensitive to blocking operations. See [High throughput optimization](serve-high-throughput).
For most production deployments, you should keep the defaults (`1`) for both separate-loop flags.
## Batching and streaming semantics
Batching and streaming both rely on the event loop to stay responsive. They don't change where your code runs: batched handlers and streaming handlers still run on the same user event loop as any other handler. This means that if you add batching or streaming on top of blocking code, you can make event loop blocking effects much worse.
### Batching
When you enable batching, Serve groups multiple incoming requests together and passes them to your handler as a list. The handler still runs on the user event loop, but each call now processes many requests at once instead of just one. If that batched work is blocking, it blocks the event loop for all of those requests at the same time.
The following example shows a batched deployment:
```{literalinclude} ../doc_code/asyncio_best_practices.py
:start-after: __batched_model_begin__
:end-before: __batched_model_end__
:language: python
```
The batch handler runs on the user event loop:
- If `_run_model` is CPU-heavy and runs inline, it blocks the loop for the duration of the batch.
- You can offload the batch computation:
```{literalinclude} ../doc_code/asyncio_best_practices.py
:start-after: __batched_model_offload_begin__
:end-before: __batched_model_offload_end__
:language: python
:emphasize-lines: 9-16
```
This keeps the event loop responsive while the model runs in a thread.
#### `max_concurrent_batches` and event loop yielding
The `@serve.batch` decorator accepts a `max_concurrent_batches` argument that controls how many batches can be processed concurrently. However, this argument only works effectively if your batch handler yields control back to the event loop during processing.
If your batch handler blocks the event loop (for example, by doing heavy CPU work without awaiting or offloading), `max_concurrent_batches` won't provide the concurrency you expect. The event loop can only start processing a new batch when the current batch yields control.
To get the benefit of `max_concurrent_batches`:
- Use `async def` for your batch handler and `await` I/O operations or offloaded CPU work.
- Offload CPU-heavy batch processing to a threadpool with `asyncio.to_thread()` or `loop.run_in_executor()`.
- Avoid blocking operations that prevent the event loop from scheduling other batches.
In the offloaded batch example above, the handler yields to the event loop when awaiting the threadpool executor, which allows multiple batches to be in flight simultaneously (up to the `max_concurrent_batches` limit).
### Streaming
Streaming is different from a regular response because the client starts receiving data while your handler is still running. Serve calls your handler once, gets back a generator or async generator, and then repeatedly asks it for the next chunk. That generator code still runs on the user event loop (or in a worker thread if you offload it).
Streaming is especially sensitive to blocking:
- If you block between chunks, you delay the next piece of data to the client.
- While the generator is blocked on the event loop, other requests on that loop can't make progress.
- The system also cannot react quickly to slow clients (backpressure) or cancellation.
Bad streaming example:
```{literalinclude} ../doc_code/asyncio_best_practices.py
:start-after: __blocking_stream_begin__
:end-before: __blocking_stream_end__
:language: python
```
Better streaming example:
```{literalinclude} ../doc_code/asyncio_best_practices.py
:start-after: __async_stream_begin__
:end-before: __async_stream_end__
:language: python
```
In streaming scenarios:
- Prefer `async def` generators that use `await` between yields.
- Avoid long CPU-bound loops between yields; offload them if needed.
## Offloading patterns: I/O, CPU
This section summarizes common offloading patterns you can use inside `async` handlers.
### Blocking I/O in `async def`
```{literalinclude} ../doc_code/asyncio_best_practices.py
:start-after: __offload_io_begin__
:end-before: __offload_io_end__
:language: python
```
### CPU-heavy code in `async def`
```{literalinclude} ../doc_code/asyncio_best_practices.py
:start-after: __offload_cpu_begin__
:end-before: __offload_cpu_end__
:language: python
```
### (Advance) Using Ray tasks or remote actors for true parallelism
:::{note}
While you can spawn Ray tasks from Ray Serve deployments, this approach isn't recommended because it lacks tooling for observability and debugging.
:::
```{literalinclude} ../doc_code/asyncio_best_practices.py
:start-after: __ray_parallel_begin__
:end-before: __ray_parallel_end__
:language: python
```
This pattern:
- Uses multiple Ray workers and processes.
- Bypasses the GIL limitation of a single Python process.
## Summary
- Use `async def` for I/O-bound and streaming work so the event loop can stay responsive.
- Use `max_ongoing_requests` to bound concurrency per replica, but remember that blocking `def` handlers can still serialize work if they run on the event loop.
- Consider enabling `RAY_SERVE_RUN_SYNC_IN_THREADPOOL` once your code is thread-safe, and be aware of the sync-in-threadpool warning.
- For CPU-heavy workloads, scale replicas or GIL-releasing native code for real parallelism.
doc/source/serve/advanced-guides/custom-request-router.md
+227
View File
@@ -0,0 +1,227 @@
(custom-request-router-guide)=
# Use Custom Algorithm for Request Routing
:::{warning}
This API is in alpha and may change before becoming stable.
:::
Different Ray serve applications demand different logics for load balancing. For example, in serving LLMs you might want to have a different policy than balancing number of requests across replicas: e.g. balancing ongoing input tokens, balancing kv-cache utilization, etc. [`RequestRouter`](../api/doc/ray.serve.request_router.RequestRouter.rst) is an abstraction in Ray Serve that allows extension and customization of load-balancing logic for each deployment.
This guide shows how to use [`RequestRouter`](../api/doc/ray.serve.request_router.RequestRouter.rst) API to achieve custom load balancing across replicas of a given deployment. It will cover the following:
- Define a simple uniform request router for load balancing
- Deploy an app with the uniform request router
- Utility mixins for request routing
- Define a complex throughput-aware request router
- Deploy an app with the throughput-aware request router
- Experimental: Use the round-robin request router
- Experimental: Use the consistent-hash request router for session stickiness
- Experimental: Define a centralized capacity queue request router
(simple-uniform-request-router)=
## Define simple uniform request router
Create a file `custom_request_router.py` with the following code:
```{literalinclude} ../doc_code/custom_request_router.py
:start-after: __begin_define_uniform_request_router__
:end-before: __end_define_uniform_request_router__
:language: python
```
This code defines a simple uniform request router that routes requests a random replica to distribute the load evenly regardless of the queue length of each replica or the body of the request. The router is defined as a class that inherits from [`RequestRouter`](../api/doc/ray.serve.request_router.RequestRouter.rst). It implements the [`choose_replicas`](../api/doc/ray.serve.request_router.RequestRouter.choose_replicas.rst) method, which returns the random replica for all incoming requests. The returned type is a list of lists of replicas, where each inner list represents a rank of replicas. The first rank is the most preferred and the last rank is the least preferred. The request will be attempted to be routed to the replica with the shortest request queue in each set of the rank in order until a replica is able to process the request. If none of the replicas are able to process the request, [`choose_replicas`](../api/doc/ray.serve.request_router.RequestRouter.choose_replicas.rst) will be called again with a backoff delay until a replica is able to process the request.
:::{note}
This request router also implements [`on_request_routed`](../api/doc/ray.serve.request_router.RequestRouter.on_request_routed.rst) which can help you update the state of the request router after a request is routed.
:::
(deploy-app-with-uniform-request-router)=
## Deploy an app with the uniform request router
To use a custom request router, you need to pass the `request_router_class` argument to the [`deployment`](../api/doc/ray.serve.deployment_decorator.rst) decorator. Also note that the `request_router_class` can be passed as the already imported class or as the string of import path to the class. Let's deploy a simple app that uses the uniform request router like this:
```{literalinclude} ../doc_code/custom_request_router_app.py
:start-after: __begin_deploy_app_with_uniform_request_router__
:end-before: __end_deploy_app_with_uniform_request_router__
:language: python
```
As the request is routed, both "UniformRequestRouter routing request" and "on_request_routed callback is called!!" messages will be printed to the console. The response will also be randomly routed to one of the replicas. You can test this by sending more requests and seeing the distribution of the replicas are roughly equal.
:::{note}
Currently, the only way to configure the request router is to pass it as an argument to the deployment decorator. This means that you cannot change the request router for an existing deployment handle with running router. If you have a particular usecase where you need to reconfigure a request router on the deployment handle, please open a feature request on the [Ray GitHub repository](https://github.com/ray-project/ray/issues)
:::
(utility-mixin)=
## Utility mixins for request routing
Ray Serve provides utility mixins that can be used to extend the functionality of the request router. These mixins can be used to implement common routing policies such as locality-aware routing, multiplexed model support, and FIFO request routing.
- [`FIFOMixin`](../api/doc/ray.serve.request_router.FIFOMixin.rst): This mixin implements first in first out (FIFO) request routing. The default behavior for the request router is OOO (out of order) which routes requests to the exact replica which got assigned by the request passed to [`choose_replicas`](../api/doc/ray.serve.request_router.RequestRouter.choose_replicas.rst). This mixin is useful for the routing algorithm that can work independently of the request content, so the requests can be routed as soon as possible in the order they were received. By including this mixin, in your custom request router, the request matching algorithm will be updated to route requests FIFO. There are no additional flags needs to be configured and no additional helper methods provided by this mixin.
- [`LocalityMixin`](../api/doc/ray.serve.request_router.LocalityMixin.rst): This mixin implements locality-aware request routing. It updates the internal states when between replica updates to track the location between replicas in the same node, same zone, and everything else. It offers helpers [`apply_locality_routing`](../api/doc/ray.serve.request_router.LocalityMixin.apply_locality_routing.rst) and [`rank_replicas_via_locality`](../api/doc/ray.serve.request_router.LocalityMixin.rank_replicas_via_locality.rst) to route and ranks replicas based on their locality to the request, which can be useful for reducing latency and improving performance.
- [`MultiplexMixin`](../api/doc/ray.serve.request_router.MultiplexMixin.rst): When you use model-multiplexing you need to route requests based on knowing which replica has already a hot version of the model. It updates the internal states when between replica updates to track the model loaded on each replica, and size of the model cache on each replica. It offers helpers [`apply_multiplex_routing`](../api/doc/ray.serve.request_router.MultiplexMixin.apply_multiplex_routing.rst) and [`rank_replicas_via_multiplex`](../api/doc/ray.serve.request_router.MultiplexMixin.rank_replicas_via_multiplex.rst) to route and ranks replicas based on their multiplexed model id of the request.
(throughput-aware-request-router)=
## Define a complex throughput-aware request router
A fully featured request router can be more complex and should take into account the multiplexed model, locality, the request queue length on each replica, and using custom statistics like throughput to decide which replica to route the request to. The following class defines a throughput-aware request router that routes requests to the replica with these factors in mind. Add the following code into the `custom_request_router.py` file:
```{literalinclude} ../doc_code/custom_request_router.py
:start-after: __begin_define_throughput_aware_request_router__
:end-before: __end_define_throughput_aware_request_router__
:language: python
```
This request router inherits from [`RequestRouter`](../api/doc/ray.serve.request_router.RequestRouter.rst), as well as [`FIFOMixin`](../api/doc/ray.serve.request_router.FIFOMixin.rst) for FIFO request routing, [`LocalityMixin`](../api/doc/ray.serve.request_router.LocalityMixin.rst) for locality-aware request routing, and [`MultiplexMixin`](../api/doc/ray.serve.request_router.MultiplexMixin.rst) for multiplexed model support. It implements [`choose_replicas`](../api/doc/ray.serve.request_router.RequestRouter.choose_replicas.rst) to take the highest ranked replicas from [`rank_replicas_via_multiplex`](../api/doc/ray.serve.request_router.MultiplexMixin.rank_replicas_via_multiplex.rst) and [`rank_replicas_via_locality`](../api/doc/ray.serve.request_router.LocalityMixin.rank_replicas_via_locality.rst) and uses the [`select_available_replicas`](../api/doc/ray.serve.request_router.RequestRouter.select_available_replicas.rst) helper to filter out replicas that have reached their maximum request queue length. Finally, it takes the replicas with the minimum throughput and returns the top one.
(deploy-app-with-throughput-aware-request-router)=
## Deploy an app with the throughput-aware request router
To use the throughput-aware request router, you can deploy an app like this:
```{literalinclude} ../doc_code/custom_request_router_app.py
:start-after: __begin_deploy_app_with_throughput_aware_request_router__
:end-before: __end_deploy_app_with_throughput_aware_request_router__
:language: python
```
Similar to the uniform request router, the custom request router can be defined in the `request_router_class` argument of the [`deployment`](../api/doc/ray.serve.deployment_decorator.rst) decorator. The Serve controller pulls statistics from the replica of each deployment by calling record_routing_stats. The `request_routing_stats_period_s` and `request_routing_stats_timeout_s` arguments control the frequency and timeout time of the serve controller pulling information from each replica in its background thread. You can customize the emission of these statistics by overriding `record_routing_stats` in the definition of the deployment class. The custom request router can then get the updated routing stats by looking up the `routing_stats` attribute of the running replicas and use it in the routing policy.
(round-robin-request-router)=
## Experimental: Use the round-robin request router
`RoundRobinRouter` cycles through replicas in round-robin fashion, starting at an arbitrary replica so routers spun up together don't synchronize on the same first replica. If the chosen replica is at capacity, the router falls back to the next replica in order and wraps around the candidate list. Each router instance keeps its own cursor, so with multiple routers (for example, one per Ray Serve proxy) the fan-out is round-robin per router and approximately uniform across replicas in aggregate. The router mixes in [`FIFOMixin`](../api/doc/ray.serve.request_router.FIFOMixin.rst) so queued requests are routed in arrival order.
### When to use
Use the round-robin router when you want a predictable, even distribution across replicas and don't need load-aware or locality-aware decisions. It fits stateless workloads with roughly uniform per-request latency. Unlike the default power-of-two-choices router, the round-robin router doesn't compare replicas by their number of ongoing requests. It only skips a replica once it reaches `max_ongoing_requests`. Therefore, requests can pile up behind a slow replica before it falls back to the next one.
### Example
Reference the router by import path through [`RequestRouterConfig`](../api/doc/ray.serve.config.RequestRouterConfig.rst):
```{literalinclude} ../doc_code/custom_request_router_app.py
:start-after: __begin_deploy_app_with_round_robin_router__
:end-before: __end_deploy_app_with_round_robin_router__
:language: python
```
(consistent-hash-request-router)=
## Experimental: Use the consistent-hash request router for session stickiness
`ConsistentHashRouter` pins each session to a specific replica using a consistent-hash ring with virtual nodes. The routing key is the session ID read from the HTTP header named by `RAY_SERVE_SESSION_ID_HEADER_KEY` (default `x-session-id`), so the same session ID consistently maps to the same replica. Requests without that header fall back to a per-request internal ID and spread uniformly across replicas. If the chosen replica rejects (for example, due to backpressure from the number of ongoing request), the router walks up to `num_fallback_replicas` clockwise successors on the ring. If those are also at capacity, the router sleeps with exponential backoff and retries the same primary and fallbacks until one accepts. When the replica set changes, the ring is rebuilt and only keys owned by added or removed replicas are reshuffled.
Two parameters are configurable through [`request_router_kwargs`](../api/doc/ray.serve.config.RequestRouterConfig.rst):
* `num_virtual_nodes` (default `100`): vnodes per replica on the ring. Higher values spread sessions more evenly across replicas.
* `num_fallback_replicas` (default `2`): clockwise successors tried after the primary rejects. Set to `0` for strict affinity with no fallback.
### When to use
Use the consistent-hash router when requests carry session-scoped state worth keeping warm on a single replica, for example in-memory caches keyed by user or KV-cache reuse for LLM chat sessions. Skip it for stateless workloads, since affinity sacrifices queue-aware balancing and a hot session can saturate its assigned replica while others are idle. The router does not combine with queue-depth, locality, or multiplexed-model signals, since mixing them in would break determinism and therefore break affinity.
### Example
Configure the router via [`RequestRouterConfig`](../api/doc/ray.serve.config.RequestRouterConfig.rst) and pass tuning parameters through `request_router_kwargs`:
```{literalinclude} ../doc_code/custom_request_router_app.py
:start-after: __begin_deploy_app_with_consistent_hash_router__
:end-before: __end_deploy_app_with_consistent_hash_router__
:language: python
```
If your clients send the session identifier under a different header (for example, `x-correlation-id`), point Ray Serve at it before the cluster starts:
```bash
export RAY_SERVE_SESSION_ID_HEADER_KEY=x-correlation-id
```
(capacity-queue-request-router)=
## Experimental: Define a centralized capacity queue request router
In the previous examples, the routing decisions are based on the locally visible state of the target replicas from the perspective of the router replica. This view is **eventually consistent** not strongly because the serve controller frequently broadcasts the replica information to the router. Under high concurrency with multiple routers, this information can drift from reality and can cause several routers to simultaneously pick the same replica, causing transient load imbalance or triggering rejections and retries. For some applications this can result in lower throughput. A **centralized** approach avoids this: a single actor tracks per-replica in-flight counts, and every router acquires a *capacity token* before forwarding a request. This way, each token guarantees the target replica has room, eliminating the rejection protocol entirely.
This example demonstrates how we can implement such routing policy. The example has three pieces:
1. An importable **`CapacityQueue`** actor that tracks per-replica capacity and hands out tokens using a least-loaded selection strategy.
2. An importable **`CapacityQueueRouter`** custom request router that acquires a token before routing and releases it when the request completes. In a real application, we can have multiple replicas of `CapacityQueueRouter` each one keeping tracking their own view of state of replicas. The centralized `CapacityQueue` actor is meant to keep their local information synchronized with reality.
3. A **deployment** that ties them together using a [deployment actor](../api/doc/ray.serve.config.DeploymentActorConfig.rst) for the queue and a [`RequestRouterConfig`](../api/doc/ray.serve.config.RequestRouterConfig.rst) for the router.
(deploy-app-with-capacity-queue-router)=
### Deploy an app with the capacity queue router
The deployment wires the pieces together: a `DeploymentActorConfig` for the capacity queue and a `RequestRouterConfig` pointing at the custom router:
```{literalinclude} ../doc_code/capacity_queue_request_router_app.py
:start-after: __begin_deploy_app_with_capacity_queue_router__
:end-before: __end_deploy_app_with_capacity_queue_router__
:language: python
```
When the app starts:
1. The Serve controller creates the `CapacityQueue` deployment actor **before** any replicas start. `CapacityQueue` subscribes to replica updates via long poll.
2. As the controller starts replicas, it sends deployment-target updates. The queue's long-poll callback automatically registers each replica with its `max_ongoing_requests` capacity and unregisters replicas that are removed during scale-down or crash recovery.
3. The `CapacityQueueRouter` running in each proxy discovers the singleton `CapacityQueue` deployment actor, acquires a token for every incoming request, and routes to the replica identified by the token.
4. When the request completes, `CapacityQueueRouter.on_request_completed` fires and the token is released back to the queue.
Because the queue is a deployment actor, the controller handles its lifecycle automatically — health checks, cleanup on app deletion, and versioning during rolling updates.
### Fault tolerance
The `CapacityQueueRouter` handles failures gracefully:
- **Queue unavailable** — if the queue actor is dead, not yet discovered, or errors, the router retries with exponential backoff and falls back to power-of-two-choices after `MAX_FAULT_RETRIES` consecutive failures. Requests never raise exceptions due to queue issues.
- **Capacity exhausted** — when all replicas are at capacity, the router backs off and retries until capacity frees up.
- **Queue restart** — a restarted queue has no knowledge of pre-crash in-flight counts and may temporarily over-provision. This self-heals: replicas reject excess requests, and the router does not release rejected tokens intentionally, ratcheting up `in_flight` on the queue until it matches reality. `token_ttl_s` (if configured) auto-reclaims any remaining leaked tokens.
- **Replica death** — the controller sends a long-poll update, the queue unregisters the dead replica, and tokens are only issued for live replicas.
### Usage
The centralized capacity queue request router could bring performance benefits particularly in a constrained supply deployment, i.e. `max_ongoing_request=1` or `2`.
### Benchmark
#### Benchmark Setup
- Deployment topology: Client -> `ParentDeployment` -> `ChildDeployment`. Request router selection is applied to both deployments, controlling how parent replicas are selected by the HTTP proxy and how child replicas are selected by parent's `DeploymentHandle`.
- Scale: small (8 replicas), medium (32 replicas), large (128 replicas), xlarge (512 replicas).
- Workload: Replica processing latency is drawn from an exponential distribution with mean 1s and capped at 10s.
- `max_ongoing_request` is set to `2`.
- Load generation: Applies closed-loop load generation where the load consistently keeps replicas saturated at `max_ongoing_request` concurrency.
- Warmup: 10s; metrics within the warmup window are discarded entirely.
#### Benchmark Metrics
- Throughput: Requests per second, i.e. `num_requests / duration`.
- Utilization: Measures what fraction of a replica's total processing capacity was consumed by actual work during the experiment. Concretely, `sum(replica_processing_latency_s) / (duration_s * max_ongoing_requests)`. For GPU deployments, utilization serves as an assessment proxy for GPU utilization.
- Latency: Measures the client-side end-to-end latency, covering the full round-trip -- client -> `ParentDeployment` -> `ChildDeployment` -> `ParentDeployment` -> client.
#### Normal Situation
Under normal (success) situations, `CapacityQueueRouter` yields higher throughput and utilization and lower latency.
```{image} ../images/capacity-queue-router-normal.png
:align: center
:width: 800px
```
#### Fault Situation
A fault is simulated by killing the `CapacityQueue` router, and upon recovery, `CapacityQueue` converges towards its pre-fault performance.
```{image} ../images/capacity-queue-router-fault.png
:align: center
:width: 800px
```
:::{note}
If you experience the following error when the `CapacityQueue` actor experiences faults and routing decisions fall back to the power-of-two-choices router, set `RAY_SERVE_QUEUE_LENGTH_RESPONSE_DEADLINE_S` to a higher value.
> Failed to get queue length from Replica(id='...', deployment='ParentDeployment', app='...') within 0.1s.
:::
:::{warning}
## Gotchas and limitations
When you provide a custom router, Ray Serve can fully support it as long as it's simple, self-contained Python code that relies only on the standard library. Once the router becomes more complex, such as depending on other custom modules or packages, you need to ensure those modules are bundled into the Docker image or environment. This is because Ray Serve uses `cloudpickle` to serialize custom routers and it doesn't vendor transitive dependencies—if your router inherits from a superclass in another module or imports custom packages, those must exist in the target environment. Additionally, environment parity matters: differences in Python version, `cloudpickle` version, or library versions can affect deserialization.
### Alternatives for complex routers
When your custom request router has complex dependencies or you want better control over versioning and deployment, you have several alternatives:
- **Use built-in routers**: Consider using the routers shipped with Ray Serve—these are well-tested, production-ready, and guaranteed to work across different environments.
- **Contribute to Ray Serve**: If your router is general-purpose and might benefit others, consider contributing it to Ray Serve as a built-in router by opening a feature request or pull request on the [Ray GitHub repository](https://github.com/ray-project/ray/issues). The recommended location for the implementation is `python/ray/serve/_private/request_router/`.
- **Ensure dependencies in your environment**: Make sure that the external dependencies are installed in your Docker image or environment.
:::
doc/source/serve/advanced-guides/deploy-vm.md
+89
View File
@@ -0,0 +1,89 @@
(serve-in-production-deploying)=
# Deploy on VM
You can deploy your Serve application to production on a Ray cluster using the Ray Serve CLI. `serve deploy` takes in a config file path and it deploys that file to a Ray cluster over HTTP. This could either be a local, single-node cluster as in this example or a remote, multi-node cluster started with the [Ray Cluster Launcher](cloud-vm-index).
This section should help you:
- understand how to deploy a Ray Serve config file using the CLI.
- understand how to update your application using the CLI.
- understand how to deploy to a remote cluster started with the [Ray Cluster Launcher](cloud-vm-index).
Start by deploying this [config](production-config-yaml) for the Text ML Application [example](serve-in-production-example):
```console
$ ls
text_ml.py
serve_config.yaml
$ ray start --head
...
$ serve deploy serve_config.yaml
2022-06-20 17:26:31,106 SUCC scripts.py:139 --
Sent deploy request successfully!
* Use `serve status` to check deployments' statuses.
* Use `serve config` to see the running app's config.
```
`ray start --head` starts a long-lived Ray cluster locally. `serve deploy serve_config.yaml` deploys the `serve_config.yaml` file to this local cluster. To stop Ray cluster, run the CLI command `ray stop`.
The message `Sent deploy request successfully!` means:
* The Ray cluster has received your config file successfully.
* It will start a new Serve application if one hasn't already started.
* The Serve application will deploy the deployments from your deployment graph, updated with the configurations from your config file.
It does **not** mean that your Serve application, including your deployments, has already started running successfully. This happens asynchronously as the Ray cluster attempts to update itself to match the settings from your config file. See [Inspect an application](serve-in-production-inspecting) for how to get the current status.
(serve-in-production-remote-cluster)=
## Using a remote cluster
By default, `serve deploy` deploys to a cluster running locally. However, you should also use `serve deploy` whenever you want to deploy your Serve application to a remote cluster. `serve deploy` takes in an optional `--address/-a` argument where you can specify your remote Ray cluster's dashboard address. This address should be of the form:
```
[RAY_CLUSTER_URI]:[DASHBOARD_PORT]
```
As an example, the address for the local cluster started by `ray start --head` is `http://127.0.0.1:8265`. We can explicitly deploy to this address using the command
```console
$ serve deploy config_file.yaml -a http://127.0.0.1:8265
```
The Ray Dashboard's default port is 8265. To set it to a different value, use the `--dashboard-port` argument when running `ray start`.
:::{note}
When running on a remote cluster, you need to ensure that the import path is accessible. See [Handle Dependencies](serve-handling-dependencies) for how to add a runtime environment.
:::
:::{tip}
By default, all the Serve CLI commands assume that you're working with a local cluster. All Serve CLI commands, except `serve start` and `serve run` use the Ray Dashboard address associated with a local cluster started by `ray start --head`. However, if the `RAY_DASHBOARD_ADDRESS` environment variable is set, these Serve CLI commands will default to that value instead.
Similarly, `serve start` and `serve run`, use the Ray head node address associated with a local cluster by default. If the `RAY_ADDRESS` environment variable is set, they will use that value instead.
You can check `RAY_DASHBOARD_ADDRESS`'s value by running:
```console
$ echo $RAY_DASHBOARD_ADDRESS
```
You can set this variable by running the CLI command:
```console
$ export RAY_DASHBOARD_ADDRESS=[YOUR VALUE]
```
You can unset this variable by running the CLI command:
```console
$ unset RAY_DASHBOARD_ADDRESS
```
Check for this variable in your environment to make sure you're using your desired Ray Dashboard address.
:::
To inspect the status of the Serve application in production, see [Inspect an application](serve-in-production-inspecting).
Make heavyweight code updates (like `runtime_env` changes) by starting a new Ray Cluster, updating your Serve config file, and deploying the file with `serve deploy` to the new cluster. Once the new deployment is finished, switch your traffic to the new cluster.
doc/source/serve/advanced-guides/deployment-scoped-actors.md
+102
View File
@@ -0,0 +1,102 @@
(serve-deployment-scoped-actors)=
# Use deployment-scoped actors
:::{warning}
Deployment-scoped actors are experimental and may change between Ray minor versions.
:::
A deployment-scoped actor is a Ray actor that the Serve controller starts and manages for one deployment. All replicas in that deployment share the actor. The actor outlives individual replica restarts, and Ray Serve cleans it up when you delete the application or call `serve.shutdown()`.
Ray Serve already uses this pattern in the [custom request router guide](custom-request-router). The experimental centralized capacity queue router uses a deployment-scoped actor to keep shared routing state in one place while the controller manages its lifecycle.
## Decide whether to use deployment-scoped actors
Deployment-scoped actors work best when you need controller-managed shared state or coordination for one deployment, such as:
- A shared counter, cache, prefix tree, or rate limiter that every replica should see.
- A coordinator that should survive replica restarts and rolling updates.
- A helper actor that should inherit the deployment's environment and be cleaned up with the deployment.
- Centralized control-plane logic, such as a queue or router helper, where one actor prevents each replica from keeping a conflicting local view.
Deployment-scoped actors aren't a good fit when:
- The state belongs to one replica. Keep it inside the replica instead.
- The state must stay durable across cluster loss. Use an external database or cache instead.
- The actor would sit on the hot path for high-volume data-plane work and become a serialization bottleneck.
- Multiple deployments or non-Serve jobs need to share the same service. Use a dedicated Ray actor or an external service instead.
## Define a deployment-scoped actor
The following example defines a shared counter actor and attaches it to a deployment with `DeploymentActorConfig`:
```{literalinclude} ../doc_code/deployment_scoped_actors.py
:start-after: __begin_define_deployment_scoped_actor__
:end-before: __end_define_deployment_scoped_actor__
:language: python
```
Every replica of `SharedCounterDeployment` reads and updates the same `SharedCounter` actor. `serve.get_deployment_actor()` only works inside a running Serve replica, so call it from the deployment's methods or constructor, not from driver code.
## Run the example
The example file is runnable as written:
```{literalinclude} ../doc_code/deployment_scoped_actors.py
:start-after: __begin_run_deployment_scoped_actor_example__
:end-before: __end_run_deployment_scoped_actor_example__
:language: python
```
When you run the script, the two requests return increasing values because they hit the same shared actor state.
## Handle actor recreation
Ray Serve health-checks deployment-scoped actors and may recreate them after failures. A handle cached before recreation can become stale. The safest pattern is to call `serve.get_deployment_actor()` when you need the actor. If you want to cache the handle, refresh it on `RayActorError` and retry the lookup until the new actor is registered:
```{literalinclude} ../doc_code/deployment_scoped_actors.py
:start-after: __begin_cached_handle_refresh__
:end-before: __end_cached_handle_refresh__
:language: python
```
This pattern comes from the existing Serve recovery tests. It matters most when the actor manages long-lived coordination state and you want to avoid a lookup on every request.
## Understand rollout behavior
Changes to `deployment_actors` are heavyweight deployment changes. If you change actor configuration, such as the actor class, actor name, `init_args`, `init_kwargs`, or actor options, Ray Serve rolls out a new deployment version and restarts replicas in that deployment.
That rollout has a few important properties:
- If you redeploy with the same deployment-actor config, Ray Serve keeps the existing actor.
- If you change the deployment-actor config, Ray Serve creates a new actor for the new version and the old actor state doesn't carry over.
- During an incremental rollout, Ray Serve can keep both old and new deployment-scoped actors alive at the same time.
- Ray Serve starts the new version's deployment-scoped actors before it starts any replicas for the new version.
- Ray Serve doesn't stop the old version's deployment-scoped actors until every old-version replica has drained and stopped.
This ordering matters because `serve.get_deployment_actor()` resolves the actor for the replica's own code version. In other words, old replicas keep talking to old actors while new replicas talk to new actors during the rollout window.
## Understand health and status behavior
Ray Serve health-checks deployment-scoped actors, and deployment health depends on them.
- If a deployment-scoped actor can't start after retries, the deployment moves to `DEPLOY_FAILED`.
- If one actor fails to start in a deployment with multiple deployment-scoped actors, Ray Serve doesn't partially succeed. The deployment still fails.
- If a deployment actor dies and Ray Serve is still managing it, the controller recreates it and the deployment can recover to healthy.
- If the replacement actor is blocked in its constructor or otherwise can't become ready, the deployment stays `UNHEALTHY` until the actor becomes healthy again.
- A replica health check can also depend on a deployment-scoped actor. If the replica's `check_health` method calls the actor and that call fails repeatedly, Ray Serve can restart the replica even if the actor itself still exists.
## Understand lifecycle and limits
- The actor is scoped to one deployment in one application. Different apps can reuse the same logical actor name without colliding.
- Ray Serve shares the actor across all replicas of that deployment, including autoscaled replicas.
- Ray Serve cleans up the actor when you delete the application or shut Serve down.
- When you delete an application, Ray Serve keeps deployment-scoped actors alive until in-flight requests finish and the old replicas stop draining.
- The actor can survive a Serve controller restart because the controller rediscovers it.
- If the actor constructor keeps failing, the deployment fails to deploy.
- `actor_options` on the deployment actor let you reserve different resources or override parts of the deployment `runtime_env`.
:::{note}
Deployment-scoped actors follow Ray actor concurrency semantics. If the actor uses threaded execution, Ray defaults `max_concurrency` to `1`. If the actor is an async actor, Ray defaults `max_concurrency` to `1000`. If those defaults don't fit your workload, set `max_concurrency` explicitly in `actor_options`.
:::
Even though the actor can survive replica restarts and controller restarts, it isn't a substitute for durable external storage.
doc/source/serve/advanced-guides/dev-workflow.md
+128
View File
@@ -0,0 +1,128 @@
(serve-dev-workflow)=
# Development Workflow
This page describes the recommended workflow for developing Ray Serve applications. If you're ready to go to production, jump to the [Production Guide](serve-in-production) section.
## Local Development using `serve.run`
You can use `serve.run` in a Python script to run and test your application locally, using a handle to send requests programmatically rather than over HTTP.
Benefits:
- Self-contained Python is convenient for writing local integration tests.
- No need to deploy to a cloud provider or manage infrastructure.
Drawbacks:
- Doesn't test HTTP endpoints.
- Can't use GPUs if your local machine doesn't have them.
Let's see a simple example.
```{literalinclude} ../doc_code/local_dev.py
:start-after: __local_dev_start__
:end-before: __local_dev_end__
:language: python
```
We can add the code below to deploy and test Serve locally.
```{literalinclude} ../doc_code/local_dev.py
:start-after: __local_dev_handle_start__
:end-before: __local_dev_handle_end__
:language: python
```
## Local Development with HTTP requests
You can use the `serve run` CLI command to run and test your application locally using HTTP to send requests (similar to how you might use the `uvicorn` command if you're familiar with [Uvicorn](https://www.uvicorn.org/)).
Recall our example above:
```{literalinclude} ../doc_code/local_dev.py
:start-after: __local_dev_start__
:end-before: __local_dev_end__
:language: python
```
Now run the following command in your terminal:
```bash
serve run local_dev:app
# 2022-08-11 11:31:47,692 INFO scripts.py:294 -- Deploying from import path: "local_dev:app".
# 2022-08-11 11:31:50,372 INFO worker.py:1481 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265.
# (ServeController pid=9865) INFO 2022-08-11 11:31:54,039 controller 9865 proxy_state.py:129 - Starting HTTP proxy with name 'SERVE_CONTROLLER_ACTOR:SERVE_PROXY_ACTOR-dff7dc5b97b4a11facaed746f02448224aa0c1fb651988ba7197e949' on node 'dff7dc5b97b4a11facaed746f02448224aa0c1fb651988ba7197e949' listening on '127.0.0.1:8000'
# (ServeController pid=9865) INFO 2022-08-11 11:31:55,373 controller 9865 deployment_state.py:1232 - Adding 1 replicas to deployment 'Doubler'.
# (ServeController pid=9865) INFO 2022-08-11 11:31:55,389 controller 9865 deployment_state.py:1232 - Adding 1 replicas to deployment 'HelloDeployment'.
# (HTTPProxyActor pid=9872) INFO: Started server process [9872]
# 2022-08-11 11:31:57,383 SUCC scripts.py:315 -- Deployed successfully.
```
The `serve run` command blocks the terminal and can be canceled with Ctrl-C. Typically, `serve run` should not be run simultaneously from multiple terminals, unless each `serve run` is targeting a separate running Ray cluster.
Now that Serve is running, we can send HTTP requests to the application. For simplicity, we'll just use the `curl` command to send requests from another terminal.
```bash
curl -X PUT "http://localhost:8000/?name=Ray"
# Hello, Ray! Hello, Ray!
```
After you're done testing, you can shut down Ray Serve by interrupting the `serve run` command (e.g., with Ctrl-C):
```console
^C2022-08-11 11:47:19,829 INFO scripts.py:323 -- Got KeyboardInterrupt, shutting down...
(ServeController pid=9865) INFO 2022-08-11 11:47:19,926 controller 9865 deployment_state.py:1257 - Removing 1 replicas from deployment 'Doubler'.
(ServeController pid=9865) INFO 2022-08-11 11:47:19,929 controller 9865 deployment_state.py:1257 - Removing 1 replicas from deployment 'HelloDeployment'.
```
Note that rerunning `serve run` redeploys all deployments. To prevent redeploying the deployments whose code hasn't changed, you can use `serve deploy`; see the [Production Guide](serve-in-production) for details.
### Local Testing Mode
:::{note}
This is an experimental feature.
:::
Ray Serve supports a local testing mode that allows you to run your deployments locally in a single process. This mode is useful for unit testing and debugging your application logic without the overhead of a full Ray cluster. To enable this mode, use the `_local_testing_mode` flag in the `serve.run` function:
```{literalinclude} ../doc_code/local_dev.py
:start-after: __local_dev_testing_start__
:end-before: __local_dev_testing_end__
:language: python
```
This mode runs each deployment in a background thread and supports most of the same features as running on a full Ray cluster. Note that some features, such as converting `DeploymentResponses` to `ObjectRefs`, are not supported in local testing mode. If you encounter limitations, consider filing a feature request on GitHub.
## Testing on a remote cluster
To test on a remote cluster, use `serve run` again, but this time, pass in an `--address` argument to specify the address of the Ray cluster to connect to. For remote clusters, this address has the form `ray://<head-node-ip-address>:10001`; see [Ray Client](ray-client-ref) for more information.
When making the transition from your local machine to a remote cluster, you'll need to make sure your cluster has a similar environment to your local machine--files, environment variables, and Python packages, for example.
Let's see a simple example that just packages the code. Run the following command on your local machine, with your remote cluster head node IP address substituted for `<head-node-ip-address>` in the command:
```bash
serve run --address=ray://<head-node-ip-address>:10001 --working-dir="./project/src" local_dev:app
```
This connects to the remote cluster with the Ray Client, uploads the `working_dir` directory, and runs your Serve application. Here, the local directory specified by `working_dir` must contain `local_dev.py` so that it can be uploaded to the cluster and imported by Ray Serve.
Once this is up and running, we can send requests to the application:
```bash
curl -X PUT http://<head-node-ip-address>:8000/?name=Ray
# Hello, Ray! Hello, Ray!
```
For more complex dependencies, including files outside the working directory, environment variables, and Python packages, you can use {ref}`Runtime Environments<runtime-environments>`. This example uses the --runtime-env-json argument:
```bash
serve run --address=ray://<head-node-ip-address>:10001 --runtime-env-json='{"env_vars": {"MY_ENV_VAR": "my-value"}, "working_dir": "./project/src", "pip": ["requests", "chess"]}' local_dev:app
```
You can also specify the `runtime_env` in a YAML file; see [serve run](#serve-cli) for details.
## What's Next?
View details about your Serve application in the [Ray dashboard](dash-serve-view). Once you are ready to deploy to production, see the [Production Guide](serve-in-production).
doc/source/serve/advanced-guides/dyn-req-batch.md
+124
View File
@@ -0,0 +1,124 @@
(serve-performance-batching-requests)=
# Dynamic Request Batching
Serve offers a request batching feature that can improve your service throughput without sacrificing latency. This improvement is possible because ML models can utilize efficient vectorized computation to process a batch of requests at a time. Batching is also necessary when your model is expensive to use and you want to maximize the utilization of hardware.
Machine Learning (ML) frameworks such as Tensorflow, PyTorch, and Scikit-Learn support evaluating multiple samples at the same time. Ray Serve allows you to take advantage of this feature with dynamic request batching. When a request arrives, Serve puts the request in a queue. This queue buffers the requests to form a batch. The deployment picks up the batch and evaluates it. After the evaluation, Ray Serve splits up the resulting batch, and returns each response individually.
## Enable batching for your deployment
You can enable batching by using the {mod}`ray.serve.batch` decorator. The following simple example modifies the `Model` class to accept a batch:
```{literalinclude} ../doc_code/batching_guide.py
---
start-after: __single_sample_begin__
end-before: __single_sample_end__
---
```
The batching decorators expect you to make the following changes in your method signature:
- Declare the method as an async method because the decorator batches in asyncio event loop.
- Modify the method to accept a list of its original input types as input. For example, `arg1: int, arg2: str` should be changed to `arg1: List[int], arg2: List[str]`.
- Modify the method to return a list. The length of the return list and the input list must be of equal lengths for the decorator to split the output evenly and return a corresponding response back to its respective request.
```{literalinclude} ../doc_code/batching_guide.py
---
start-after: __batch_begin__
end-before: __batch_end__
emphasize-lines: 11-12
---
```
You can supply 4 optional parameters to the decorators:
- `max_batch_size` controls the size of the batch. The default value is 10.
- `batch_wait_timeout_s` controls how long Serve should wait for a batch once the first request arrives. The default value is 0.01 (10 milliseconds).
- `max_concurrent_batches` maximum number of batches that can run concurrently. The default value is 1.
- `batch_size_fn` optional function to compute the effective batch size. If provided, this function takes a list of items and returns an integer representing the batch size. This is useful for batching based on custom metrics such as total nodes in graphs or total tokens in sequences. If `None` (the default), the batch size is computed as `len(batch)`.
Once the first request arrives, the batching decorator waits for a full batch (up to `max_batch_size`) until `batch_wait_timeout_s` is reached. If the timeout is reached, Serve sends the batch to the model regardless of the batch size.
:::{tip}
You can reconfigure your `batch_wait_timeout_s` and `max_batch_size` parameters using the `set_batch_wait_timeout_s` and `set_max_batch_size` methods:
```{literalinclude} ../doc_code/batching_guide.py
---
start-after: __batch_params_update_begin__
end-before: __batch_params_update_end__
---
```
Use these methods in the constructor or the `reconfigure` [method](serve-user-config) to control the `@serve.batch` parameters through your Serve configuration file.
:::
## Custom batch size functions
By default, Ray Serve measures batch size as the number of items in the batch (`len(batch)`). However, in many workloads, the computational cost depends on properties of the items themselves rather than just the count. For example:
- **Graph Neural Networks (GNNs)**: The cost depends on the total number of nodes across all graphs, not the number of graphs
- **Natural Language Processing (NLP)**: Transformer models batch by total token count, not the number of sequences
- **Variable-resolution images**: Memory usage depends on total pixels, not the number of images
Use the `batch_size_fn` parameter to define a custom metric for batch size:
### Graph Neural Network example
The following example shows how to batch graph data by total node count:
```{literalinclude} ../doc_code/batching_guide.py
---
start-after: __batch_size_fn_begin__
end-before: __batch_size_fn_end__
emphasize-lines: 20
---
```
In this example, `batch_size_fn=lambda graphs: sum(g.num_nodes for g in graphs)` ensures that the batch contains at most 10,000 total nodes, preventing GPU memory overflow regardless of how many individual graphs are in the batch.
### NLP token batching example
The following example shows how to batch text sequences by total token count:
```{literalinclude} ../doc_code/batching_guide.py
---
start-after: __batch_size_fn_nlp_begin__
end-before: __batch_size_fn_nlp_end__
emphasize-lines: 12
---
```
This pattern ensures that the total number of tokens doesn't exceed the model's context window or memory limits.
(serve-streaming-batched-requests-guide)=
## Streaming batched requests
Use an async generator to stream the outputs from your batched requests. The following example converts the `StreamingResponder` class to accept a batch.
```{literalinclude} ../doc_code/batching_guide.py
---
start-after: __single_stream_begin__
end-before: __single_stream_end__
---
```
Decorate async generator functions with the {mod}`ray.serve.batch` decorator. Similar to non-streaming methods, the function takes in a `List` of inputs and in each iteration it `yield`s an iterable of outputs with the same length as the input batch size.
```{literalinclude} ../doc_code/batching_guide.py
---
start-after: __batch_stream_begin__
end-before: __batch_stream_end__
---
```
Calling the `serve.batch`-decorated function returns an async generator that you can `await` to receive results.
Some inputs within a batch may generate fewer outputs than others. When a particular input has nothing left to yield, pass a `StopIteration` object into the output iterable. This action terminates the generator that Serve returns when it calls the `serve.batch` function with that input. When `serve.batch`-decorated functions return streaming generators over HTTP, this action allows the end client's connection to terminate once its call is done, instead of waiting until the entire batch is done.
## Tips for fine-tuning batching parameters
`max_batch_size` ideally should be a power of 2 (2, 4, 8, 16, ...) because CPUs and GPUs are both optimized for data of these shapes. Large batch sizes incur a high memory cost as well as latency penalty for the first few requests.
When using `batch_size_fn`, set `max_batch_size` based on your custom metric rather than item count. For example, if batching by total nodes in graphs, set `max_batch_size` to your GPU's maximum node capacity (such as 10,000 nodes) rather than a count of graphs.
Set `batch_wait_timeout_s` considering the end-to-end latency SLO (Service Level Objective). For example, if your latency target is 150ms, and the model takes 100ms to evaluate the batch, set the `batch_wait_timeout_s` to a value much lower than 150ms - 100ms = 50ms.
When using batching in a Serve Deployment Graph, the relationship between an upstream node and a downstream node might affect the performance as well. Consider a chain of two models where first model sets `max_batch_size=8` and second model sets `max_batch_size=6`. In this scenario, when the first model finishes a full batch of 8, the second model finishes one batch of 6 and then to fill the next batch, which Serve initially only partially fills with 8 - 6 = 2 requests, leads to incurring latency costs. The batch size of downstream models should ideally be multiples or divisors of the upstream models to ensure the batches work optimally together.
doc/source/serve/advanced-guides/gang-scheduling.md
+218
View File
@@ -0,0 +1,218 @@
(serve-gang-scheduling)=
# Gang scheduling
:::{note}
Gang scheduling is an **alpha** feature. The API may change in future releases.
:::
Gang scheduling enables you to co-schedule groups of deployment replicas atomically. A **gang** is a set of replicas that are reserved and started together using a single [Ray placement group](ray-placement-group-doc-ref). If the cluster doesn't have enough resources for the entire gang, none of the replicas in that gang are started.
This is useful for workloads where a partial set of replicas is useless, such as:
- **Data parallel attention deployment**: In WideEP deployments, data parallel attention - expert parallelism ranks are required coordinate with each other to perform dispatch-combine collective communication. Any rank failure leads to dispatch-combine collective hangs, and the entire data parallel attention - expert parallelism group needs to go through failover mechanism to re-establish collectives.
- **Any workload requiring coordinated startup**, where replicas need to discover each other's identities and establish communication before serving traffic.
## Getting started
Configure gang scheduling by passing a `GangSchedulingConfig` to the `@serve.deployment` decorator:
```{literalinclude} ../doc_code/gang_scheduling.py
:start-after: __basic_gang_start__
:end-before: __basic_gang_end__
:language: python
```
This creates 2 gangs of 4 replicas each resulting in a total of 8 replicas. Partial gang isn't allowed, and therefore `num_replicas` must be a multiple of `gang_size`.
### How resources are reserved within a gang
The `ray_actor_options` field defines the resource requirements for each replica actor (for example, CPU, GPU, memory). When gang scheduling is enabled **without** `placement_group_bundles`, Ray uses the resources from `ray_actor_options` as the bundle template for each replica's slot in the gang placement group. For example, with `ray_actor_options={"num_cpus": 0.25}` and `gang_size=4`, the gang placement group contains 4 bundles of `{"CPU": 0.25}` each.
When `placement_group_bundles` is also set, each replica occupies multiple consecutive bundles in the gang placement group instead of a single flat bundle. The replica actor runs in the **first** bundle, so the resources in `ray_actor_options` must fit within that first bundle. The remaining bundles are available for child actors or tasks spawned by the replica. See [Combining with placement group bundles](#combining-with-placement-group-bundles) for details.
You can also configure gang scheduling via `.options()`:
```{literalinclude} ../doc_code/gang_scheduling.py
:start-after: __options_start__
:end-before: __options_end__
:language: python
```
In production deployments, declarative YAML config files are often used:
```yaml
applications:
- name: my_app
route_prefix: /
import_path: my_module:app
deployments:
- name: MyModel
num_replicas: 8
ray_actor_options:
num_cpus: 0.25
gang_scheduling_config:
gang_size: 4
```
### Accessing gang context
Each replica in a gang has access to a `GangContext` through the replica context. This provides the information replicas need to discover each other and coordinate:
```{literalinclude} ../doc_code/gang_scheduling.py
:start-after: __gang_context_start__
:end-before: __gang_context_end__
:language: python
```
Here's the interface of `GangContext`:
```{eval-rst}
.. autoclass:: ray.serve.context.GangContext
:members:
:no-index:
```
Replicas can use `rank` and `world_size` to set up distributed communication, e.g. initializing NCCL process groups, and `member_replica_ids` to discover and connect to their peers.
## Placement group strategies
Gang scheduling supports two placement group strategies that control how replicas within a gang are distributed across nodes:
### PACK (default)
Packs all replicas in a gang onto as few nodes as possible. This is best for workloads that benefit from locality, such as data parallel ranks within data parallel attention - expert parallelism deployment for MoE LLMs.
Although PACK colocates as many replicas per node as possible, it does not guarantee that ranks are contiguous within a node. For example, a gang of 8 replicas split across 2 nodes may be assigned ranks `{0, 3, 5, 7}` on one node and `{1, 2, 4, 6}` on the other, rather than `{0, 1, 2, 3}` and `{4, 5, 6, 7}`. Applications should not assume that consecutive ranks share a node. This is particularly important when a single replica owns multiple bundles, such as a tensor-parallel group. In that case, the bundles belonging to one replica may land on different nodes, splitting collective communication across the network. To ensure that the bundles for a single replica land on the same node, sort the gang placement group's bundle indices by node IP and assign each replica a contiguous slice of the sorted order.
```{literalinclude} ../doc_code/gang_scheduling.py
:start-after: __pack_strategy_start__
:end-before: __pack_strategy_end__
:language: python
```
### SPREAD
Spreads replicas in a gang across as many distinct nodes as possible. This is useful for fault isolation, where you want to minimize the impact of a single node failure.
```{literalinclude} ../doc_code/gang_scheduling.py
:start-after: __spread_strategy_start__
:end-before: __spread_strategy_end__
:language: python
```
Both strategies are **best effort**. PACK tries to colocate but may spread across nodes if a single node lacks capacity. SPREAD tries to distribute but may colocate if there aren't enough nodes.
### Combining with placement group bundles
You can combine gang scheduling with `placement_group_bundles` to reserve additional resources per replica within the gang. When both are set, the gang placement group contains the flattened bundles for all replicas in the gang. Each replica occupies `len(placement_group_bundles)` consecutive bundles, with the replica actor running in the first bundle.
```{literalinclude} ../doc_code/gang_scheduling.py
:start-after: __placement_group_bundles_start__
:end-before: __placement_group_bundles_end__
:language: python
```
```{image} images/gang-single-pg.png
:alt: Gang scheduling with a single bundle per replica
:width: 600px
```
In this example, each gang of 2 replicas creates a single gang placement group with 2 bundles (one `{"CPU": 1, "GPU": 1}` bundle per replica) upon scheduling. Note that `ray_actor_options={"num_cpus": 0}` is set so the replica actor doesn't request resources outside the placement group — all resource reservation is handled through the bundles.
If each replica needed multiple bundles, for example, one for the replica actor and one for a worker, the gang PG would contain `gang_size * len(placement_group_bundles)` total bundles. Replica 0 would occupy bundle indices 0 and 1, while replica 1 would occupy indices 2 and 3.
```{literalinclude} ../doc_code/gang_scheduling.py
:start-after: __multi_placement_group_bundles_start__
:end-before: __multi_placement_group_bundles_end__
:language: python
```
```{image} images/gang-multi-pg.png
:alt: Gang scheduling with multiple bundles per replica
:width: 600px
```
You can also use `placement_group_bundle_label_selector` to control which nodes the gang's bundles are placed on. The per-replica label selector is replicated across all replicas in the gang, so every replica is steered to nodes matching the selector. For example, to schedule all gang members on nodes with A100 GPUs:
```{literalinclude} ../doc_code/gang_scheduling.py
:start-after: __label_selector_start__
:end-before: __label_selector_end__
:language: python
```
## Autoscaling
Gang scheduling works with Ray Serve autoscaling (`num_replicas="auto"`). When autoscaling is enabled, the replica count recommended by the base autoscaling policy is always rounded **up** to the next multiple of `gang_size`, so the deployment never operates below the capacity the base policy requested.
```{literalinclude} ../doc_code/gang_scheduling.py
:start-after: __autoscaling_start__
:end-before: __autoscaling_end__
:language: python
```
When using autoscaling with gang scheduling, `min_replicas`, `max_replicas`, and `initial_replicas` must all be multiples of `gang_size`.
:::{note}
Scale-to-zero (`min_replicas=0`) is not supported with gang scheduling.
:::
In Ray Serve autoscaler, gang quantization is handled automatically by a `GangSchedulingAutoscalingPolicy` wrapper that is injected around the base autoscaling policy.
**Example**: With `gang_size=4`, if the base autoscaling policy recommends 5 replicas, the `GangSchedulingAutoscalingPolicy` rounds up to 8. If the policy recommends 10 replicas, the gang-aware policy rounds up to 12. Always rounding up makes the output deterministic: the same desired count produces the same replica target regardless of the current replica count, which prevents oscillation between two gang-aligned values.
## Fault tolerance
### RESTART_GANG policy
The `runtime_failure_policy` controls what happens when a replica in a running gang fails a health check. The default policy is `RESTART_GANG`:
```{literalinclude} ../doc_code/gang_scheduling.py
:start-after: __fault_tolerance_start__
:end-before: __fault_tolerance_end__
:language: python
```
When any replica in a gang fails its health check, **all replicas in that gang** are torn down and a fresh gang is created. This ensures gang members always start together and can re-establish coordinated state (for example, NCCL communicators).
:::{note}
`RESTART_REPLICA` (restarting only the failed replica while keeping healthy gang members running) is not yet supported. If you need this behavior, file a [GitHub issue](https://github.com/ray-project/ray/issues).
:::
### Incomplete gang detection
If a gang member dies while the Serve controller is down, the controller detects the incomplete gang on recovery. It checks each running replica's `gang_context.member_replica_ids` against the set of tracked replicas. If any member is missing, the entire gang is restarted.
### Controller recovery
Gang scheduling state survives controller restarts: `GangContext` is persisted in replica metadata and restored when the controller reconnects to existing replicas.
## How gang scheduling works
This section describes the internal mechanics for users who want to understand the system in depth.
### Placement group lifecycle
1. **Reservation**: During each reconciliation loop, the Serve controller's deployment scheduler creates named placement groups for each gang. The bundles in the placement group are constructed by repeating the per-replica resource requirements `gang_size` times, or flattening `placement_group_bundles` across all replicas in the gang.
2. **Atomic startup**: Once a gang placement group is ready, all replicas in the gang are scheduled together. Each replica is assigned a `rank` (0 to `gang_size - 1`) and receives a `GangContext`. Replicas are scheduled into specific bundle indices within the gang placement group.
3. **Cleanup**: When a deployment is deleted or a gang is replaced, its placement group is removed. The controller also runs periodic leak detection to clean up orphaned gang placement groups.
### Scaling
**Upscaling**: The controller reserves gang placement groups, then starts all replicas in each gang together once the placement group is ready. Gangs that can't be scheduled due to insufficient resources are retried on subsequent reconciliation loops while successfully scheduled gangs proceed. If any replica fails during startup, the entire gang is stopped and retried.
**Downscaling**: The controller selects **complete gangs** for removal rather than individual replicas, ensuring no gang is left partially running.
### Rolling updates
When a deployment config or code changes, the rolling update process replaces **complete gangs** atomically. The rollout size is aligned to `gang_size` multiples, so each update wave stops and starts whole gangs.
### Migration
When replicas need to migrate due to node draining, entire gangs are migrated atomically rather than individual replicas, preserving the atomic scheduling guarantees.
## Constraints and limitations
- `max_replicas_per_node` cannot be used together with gang scheduling since gang scheduling uses placement groups.
doc/source/serve/advanced-guides/grpc-guide.md
+331
View File
@@ -0,0 +1,331 @@
(serve-set-up-grpc-service)=
# Set Up a gRPC Service
This section helps you understand how to:
- Build a user defined gRPC service and protobuf
- Start Serve with gRPC enabled
- Deploy gRPC applications
- Send gRPC requests to Serve deployments
- Check proxy health
- Work with gRPC metadata
- Use streaming and model composition
- Handle errors
- Use gRPC context
(custom-serve-grpc-service)=
## Define a gRPC service
Running a gRPC server starts with defining gRPC services, RPC methods, and protobufs similar to the one below.
```{literalinclude} ../doc_code/grpc_proxy/user_defined_protos.proto
:start-after: __begin_proto__
:end-before: __end_proto__
:language: proto
```
This example creates a file named `user_defined_protos.proto` with two gRPC services: `UserDefinedService` and `ImageClassificationService`. `UserDefinedService` has three RPC methods: `__call__`, `Multiplexing`, and `Streaming`. `ImageClassificationService` has one RPC method: `Predict`. Their corresponding input and output types are also defined specifically for each RPC method.
Once you define the `.proto` services, use `grpcio-tools` to compile python code for those services. Example command looks like the following:
```bash
python -m grpc_tools.protoc -I=. --python_out=. --grpc_python_out=. ./user_defined_protos.proto
```
It generates two files: `user_defined_protos_pb2.py` and `user_defined_protos_pb2_grpc.py`.
For more details on `grpcio-tools` see [https://grpc.io/docs/languages/python/basics/#generating-client-and-server-code](https://grpc.io/docs/languages/python/basics/#generating-client-and-server-code).
:::{note}
Ensure that the generated files are in the same directory as where the Ray cluster is running so that Serve can import them when starting the proxies.
:::
(start-serve-with-grpc-proxy)=
## Start Serve with gRPC enabled
The [Serve start](https://docs.ray.io/en/releases-2.7.0/serve/api/index.html#serve-start) CLI, [`ray.serve.start`](https://docs.ray.io/en/releases-2.7.0/serve/api/doc/ray.serve.start.html#ray.serve.start) API, and [Serve config files](https://docs.ray.io/en/releases-2.7.0/serve/production-guide/config.html#serve-config-files-serve-build) all support starting Serve with a gRPC proxy. Two options are related to Serve's gRPC proxy: `grpc_port` and `grpc_servicer_functions`. `grpc_port` is the port for gRPC proxies to listen to. It defaults to 9000. `grpc_servicer_functions` is a list of import paths for gRPC `add_servicer_to_server` functions to add to a gRPC proxy. It also serves as the flag to determine whether to start gRPC server. The default is an empty list, meaning no gRPC server is started.
::::{tab-set}
:::{tab-item} CLI
```bash
ray start --head
serve start \
--grpc-port 9000 \
--grpc-servicer-functions user_defined_protos_pb2_grpc.add_UserDefinedServiceServicer_to_server \
--grpc-servicer-functions user_defined_protos_pb2_grpc.add_ImageClassificationServiceServicer_to_server
```
:::
:::{tab-item} Python API
```{literalinclude} ../doc_code/grpc_proxy/grpc_guide.py
:start-after: __begin_start_grpc_proxy__
:end-before: __end_start_grpc_proxy__
:language: python
```
:::
:::{tab-item} Serve config file
```yaml
# config.yaml
grpc_options:
port: 9000
grpc_servicer_functions:
- user_defined_protos_pb2_grpc.add_UserDefinedServiceServicer_to_server
- user_defined_protos_pb2_grpc.add_ImageClassificationServiceServicer_to_server
applications:
- name: app1
route_prefix: /app1
import_path: test_deployment_v2:g
runtime_env: {}
- name: app2
route_prefix: /app2
import_path: test_deployment_v2:g2
runtime_env: {}
```
```bash
# Start Serve with above config file.
serve run config.yaml
```
:::
::::
:::{note}
The default max gRPC message size is ~2GB. To adjust it, set `RAY_SERVE_GRPC_MAX_MESSAGE_SIZE` (in bytes) before starting Ray, e.g., `export RAY_SERVE_GRPC_MAX_MESSAGE_SIZE=104857600` for 100MB.
:::
(deploy-serve-grpc-applications)=
## Deploy gRPC applications
gRPC applications in Serve works similarly to HTTP applications. The only difference is that the input and output of the methods need to match with what's defined in the `.proto` file and that the method of the application needs to be an exact match (case sensitive) with the predefined RPC methods. For example, if we want to deploy `UserDefinedService` with `__call__` method, the method name needs to be `__call__`, the input type needs to be `UserDefinedMessage`, and the output type needs to be `UserDefinedResponse`. Serve passes the protobuf object into the method and expects the protobuf object back from the method.
Example deployment:
```{literalinclude} ../doc_code/grpc_proxy/grpc_guide.py
:start-after: __begin_grpc_deployment__
:end-before: __end_grpc_deployment__
:language: python
```
Deploy the application:
```{literalinclude} ../doc_code/grpc_proxy/grpc_guide.py
:start-after: __begin_deploy_grpc_app__
:end-before: __end_deploy_grpc_app__
:language: python
```
:::{note}
`route_prefix` is still a required field as of Ray 2.7.0 due to a shared code path with HTTP. Future releases will make it optional for gRPC.
:::
(send-serve-grpc-proxy-request)=
## Send gRPC requests to serve deployments
Sending a gRPC request to a Serve deployment is similar to sending a gRPC request to any other gRPC server. Create a gRPC channel and stub, then call the RPC method on the stub with the appropriate input. The output is the protobuf object that your Serve application returns.
Sending a gRPC request:
```{literalinclude} ../doc_code/grpc_proxy/grpc_guide.py
:start-after: __begin_send_grpc_requests__
:end-before: __end_send_grpc_requests__
:language: python
```
Read more about gRPC clients in Python: [https://grpc.io/docs/languages/python/basics/#client](https://grpc.io/docs/languages/python/basics/#client)
(serve-grpc-proxy-health-checks)=
## Check proxy health
Similar to HTTP `/-/routes` and `/-/healthz` endpoints, Serve also provides gRPC service method to be used in health check.
- `/ray.serve.RayServeAPIService/ListApplications` is used to list all applications deployed in Serve.
- `/ray.serve.RayServeAPIService/Healthz` is used to check the health of the proxy. It returns `OK` status and "success" message if the proxy is healthy.
The service method and protobuf are defined as below:
```proto
message ListApplicationsRequest {}
message ListApplicationsResponse {
repeated string application_names = 1;
}
message HealthzRequest {}
message HealthzResponse {
string message = 1;
}
service RayServeAPIService {
rpc ListApplications(ListApplicationsRequest) returns (ListApplicationsResponse);
rpc Healthz(HealthzRequest) returns (HealthzResponse);
}
```
You can call the service method with the following code:
```{literalinclude} ../doc_code/grpc_proxy/grpc_guide.py
:start-after: __begin_health_check__
:end-before: __end_health_check__
:language: python
```
:::{note}
Serve provides the `RayServeAPIServiceStub` stub, and `HealthzRequest` and `ListApplicationsRequest` protobufs for you to use. You don't need to generate them from the proto file. They are available for your reference.
:::
(serve-grpc-metadata)=
## Work with gRPC metadata
Just like HTTP headers, gRPC also supports metadata to pass request related information. You can pass metadata to Serve's gRPC proxy and Serve knows how to parse and use them. Serve also passes trailing metadata back to the client.
List of Serve accepted metadata keys:
- `application`: The name of the Serve application to route to. If not passed and only one application is deployed, serve routes to the only deployed app automatically.
- `request_id`: The request ID to track the request.
- `multiplexed_model_id`: The model ID to do model multiplexing.
List of Serve returned trailing metadata keys:
- `request_id`: The request ID to track the request.
Example of using metadata:
```{literalinclude} ../doc_code/grpc_proxy/grpc_guide.py
:start-after: __begin_metadata__
:end-before: __end_metadata__
:language: python
```
(serve-grpc-proxy-more-examples)=
## Use streaming and model composition
gRPC proxy supports all four gRPC streaming types:
- **Unary-Unary**: Single request, single response (default)
- **Server streaming (Unary-Stream)**: Single request, streaming response
- **Client streaming (Stream-Unary)**: Streaming request, single response
- **Bidirectional streaming (Stream-Stream)**: Streaming request, streaming response
### Server Streaming
The `Streaming` method is deployed with the app named "app1" above. The following code gets a streaming response.
```{literalinclude} ../doc_code/grpc_proxy/grpc_guide.py
:start-after: __begin_streaming__
:end-before: __end_streaming__
:language: python
```
### Client Streaming
Client streaming allows clients to send a stream of requests and receive a single response. Use the `gRPCInputStream` class to iterate over incoming request messages.
Define a proto file with a client streaming RPC:
```proto
service UserDefinedService {
rpc ClientStreaming(stream UserDefinedMessage) returns (UserDefinedResponse);
}
```
Deployment example:
```{literalinclude} ../doc_code/grpc_proxy/grpc_guide.py
:start-after: __begin_client_streaming_deployment__
:end-before: __end_client_streaming_deployment__
:language: python
```
Client code:
```{literalinclude} ../doc_code/grpc_proxy/grpc_guide.py
:start-after: __begin_client_streaming_client__
:end-before: __end_client_streaming_client__
:language: python
```
### Bidirectional Streaming
Bidirectional streaming allows clients to send and receive streams of messages simultaneously.
Define a proto file with a bidirectional streaming RPC:
```proto
service UserDefinedService {
rpc BidiStreaming(stream UserDefinedMessage) returns (stream UserDefinedResponse);
}
```
Deployment example:
```{literalinclude} ../doc_code/grpc_proxy/grpc_guide.py
:start-after: __begin_bidi_streaming_deployment__
:end-before: __end_bidi_streaming_deployment__
:language: python
```
Client code:
```{literalinclude} ../doc_code/grpc_proxy/grpc_guide.py
:start-after: __begin_bidi_streaming_client__
:end-before: __end_bidi_streaming_client__
:language: python
```
### Using gRPC Context with Streaming
You can access the gRPC context in streaming methods by adding a `grpc_context` parameter:
```{literalinclude} ../doc_code/grpc_proxy/grpc_guide.py
:start-after: __begin_streaming_with_context__
:end-before: __end_streaming_with_context__
:language: python
```
### Model composition
Assuming we have the below deployments. `ImageDownloader` and `DataPreprocessor` are two separate steps to download and process the image before PyTorch can run inference. The `ImageClassifier` deployment initializes the model, calls both `ImageDownloader` and `DataPreprocessor`, and feed into the resnet model to get the classes and probabilities of the given image.
```{literalinclude} ../doc_code/grpc_proxy/grpc_guide.py
:start-after: __begin_model_composition_deployment__
:end-before: __end_model_composition_deployment__
:language: python
```
We can deploy the application with the following code:
```{literalinclude} ../doc_code/grpc_proxy/grpc_guide.py
:start-after: __begin_model_composition_deploy__
:end-before: __end_model_composition_deploy__
:language: python
```
The client code to call the application looks like the following:
```{literalinclude} ../doc_code/grpc_proxy/grpc_guide.py
:start-after: __begin_model_composition_client__
:end-before: __end_model_composition_client__
:language: python
```
:::{note}
At this point, two applications are running on Serve, "app1" and "app2". If more than one application is running, you need to pass `application` to the metadata so Serve knows which application to route to.
:::
(serve-grpc-proxy-error-handling)=
## Handle errors
Similar to any other gRPC server, request throws a `grpc.RpcError` when the response code is not "OK". Put your request code in a try-except block and handle the error accordingly.
```{literalinclude} ../doc_code/grpc_proxy/grpc_guide.py
:start-after: __begin_error_handle__
:end-before: __end_error_handle__
:language: python
```
Serve uses the following gRPC error codes:
- `NOT_FOUND`: When multiple applications are deployed to Serve and the application is not passed in metadata or passed but no matching application.
- `UNAVAILABLE`: Only on the health check methods when the proxy is in draining state. When the health check is throwing `UNAVAILABLE`, it means the health check failed on this node and you should no longer route to this node.
- `DEADLINE_EXCEEDED`: The request took longer than the timeout setting and got cancelled.
- `INTERNAL`: Other unhandled errors during the request.
(serve-grpc-proxy-grpc-context)=
## Use gRPC context
Serve provides a [gRPC context object](https://grpc.github.io/grpc/python/grpc.html#grpc.ServicerContext) to the deployment replica to get information about the request as well as setting response metadata such as code and details. If the handler function is defined with a `grpc_context` argument, Serve will pass a [RayServegRPCContext](../api/doc/ray.serve.grpc_util.RayServegRPCContext.rst) object in for each request. Below is an example of how to set a custom status code, details, and trailing metadata. You can also set a status code before raising an exception, and Serve will preserve that status code in the error response. This is useful for returning meaningful status codes like `RESOURCE_EXHAUSTED` (retryable) or `INVALID_ARGUMENT` (not retryable) instead of the generic `INTERNAL` error.
```{literalinclude} ../doc_code/grpc_proxy/grpc_guide.py
:start-after: __begin_grpc_context_define_app__
:end-before: __end_grpc_context_define_app__
:language: python
```
The client code is defined like the following to get those attributes.
```{literalinclude} ../doc_code/grpc_proxy/grpc_guide.py
:start-after: __begin_grpc_context_client__
:end-before: __end_grpc_context_client__
:language: python
```
:::{note}
If the handler raises an unhandled exception without setting a status code on the `RayServegRPCContext` object, Serve returns an `INTERNAL` error code with the exception message in the details. However, if you set a status code on the context before raising the exception, Serve preserves that status code in the response. This allows you to return meaningful status codes like `INVALID_ARGUMENT` or `RESOURCE_EXHAUSTED` even when raising exceptions.
:::
doc/source/serve/advanced-guides/images/gang-multi-pg.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 138 KiB

doc/source/serve/advanced-guides/images/gang-single-pg.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 115 KiB

doc/source/serve/advanced-guides/index.md
+45
View File
@@ -0,0 +1,45 @@
(serve-advanced-guides)=
# Advanced Guides
```{toctree}
:hidden:
app-builder-guide
advanced-autoscaling
asyncio-best-practices
performance
dyn-req-batch
inplace-updates
dev-workflow
grpc-guide
replica-ranks
replica-scheduling
gang-scheduling
managing-java-deployments
deploy-vm
multi-app-container
custom-request-router
deployment-scoped-actors
multi-node-gpu-troubleshooting
```
If you’re new to Ray Serve, start with the [Ray Serve Quickstart](serve-getting-started).
Use these advanced guides for more options and configurations:
- [Pass Arguments to Applications](app-builder-guide)
- [Advanced Ray Serve Autoscaling](serve-advanced-autoscaling)
- [Asyncio and Concurrency best practices in Ray Serve](serve-asyncio-best-practices)
- [Performance Tuning](serve-perf-tuning)
- [Dynamic Request Batching](serve-performance-batching-requests)
- [In-Place Updates for Serve](serve-inplace-updates)
- [Development Workflow](serve-dev-workflow)
- [gRPC Support](serve-set-up-grpc-service)
- [Replica Ranks](serve-replica-ranks)
- [Replica Scheduling](serve-replica-scheduling)
- [Gang Scheduling](serve-gang-scheduling)
- [Ray Serve Dashboard](dash-serve-view)
- [Experimental Java API](serve-java-api)
- [Run Applications in Different Containers](serve-container-runtime-env-guide)
- [Use Custom Algorithm for Request Routing](custom-request-router)
- [Use deployment-scoped actors](serve-deployment-scoped-actors)
- [Troubleshoot multi-node GPU setups for serving LLMs](multi-node-gpu-troubleshooting)
doc/source/serve/advanced-guides/inplace-updates.md
+115
View File
@@ -0,0 +1,115 @@
(serve-inplace-updates)=
# Updating Applications In-Place
You can update your Serve applications once they're in production by updating the settings in your config file and redeploying it using the `serve deploy` command. In the redeployed config file, you can add new deployment settings or remove old deployment settings. This is because `serve deploy` is **idempotent**, meaning your Serve application's config always matches (or honors) the latest config you deployed successfully – regardless of what config files you deployed before that.
(serve-in-production-lightweight-update)=
## Lightweight Config Updates
Lightweight config updates modify running deployment replicas without tearing them down and restarting them, so there's less downtime as the deployments update. For each deployment, modifying the following values is considered a lightweight config update, and won't tear down the replicas for that deployment:
- `num_replicas`
- `autoscaling_config`
- `user_config`
- `max_ongoing_requests`
- `graceful_shutdown_timeout_s`
- `graceful_shutdown_wait_loop_s`
- `health_check_period_s`
- `health_check_timeout_s`
(serve-updating-user-config)=
## Updating the user config
This example uses the text summarization and translation application [from the production guide](production-config-yaml). Both of the individual deployments contain a `reconfigure()` method. This method allows you to issue lightweight updates to the deployments by updating the `user_config`.
First let's deploy the graph. Make sure to stop any previous Ray cluster using the CLI command `ray stop` for this example:
```console
$ ray start --head
$ serve deploy serve_config.yaml
```
Then send a request to the application:
```{literalinclude} ../doc_code/production_guide/text_ml.py
:language: python
:start-after: __start_client__
:end-before: __end_client__
```
Change the language that the text is translated into from French to German by changing the `language` attribute in the `Translator` user config:
```yaml
...
applications:
- name: default
route_prefix: /
import_path: text_ml:app
runtime_env:
pip:
- torch
- transformers
deployments:
- name: Translator
num_replicas: 1
user_config:
language: german
...
```
Without stopping the Ray cluster, redeploy the app using `serve deploy`:
```console
$ serve deploy serve_config.yaml
...
```
We can inspect our deployments with `serve status`. Once the application's `status` returns to `RUNNING`, we can try our request one more time:
```console
$ serve status
proxies:
cef533a072b0f03bf92a6b98cb4eb9153b7b7c7b7f15954feb2f38ec: HEALTHY
applications:
default:
status: RUNNING
message: ''
last_deployed_time_s: 1694041157.2211847
deployments:
Translator:
status: HEALTHY
replica_states:
RUNNING: 1
message: ''
Summarizer:
status: HEALTHY
replica_states:
RUNNING: 1
message: ''
```
The language has updated. Now the returned text is in German instead of French.
```{literalinclude} ../doc_code/production_guide/text_ml.py
:language: python
:start-after: __start_second_client__
:end-before: __end_second_client__
```
## Code Updates
Changing the following values in a deployment's config will trigger redeployment and restart all the deployment's replicas.
- `ray_actor_options`
- `placement_group_bundles`
- `placement_group_strategy`
Changing the following application-level config values is also considered a code update, and all deployments in the application will be restarted.
- `import_path`
- `runtime_env`
:::{warning}
Although you can update your Serve application by deploying an entirely new deployment graph using a different `import_path` and a different `runtime_env`, this is NOT recommended in production.
The best practice for large-scale code updates is to start a new Ray cluster, deploy the updated code to it using `serve deploy`, and then switch traffic from your old cluster to the new one.
:::
doc/source/serve/advanced-guides/managing-java-deployments.md
+142
View File
@@ -0,0 +1,142 @@
(serve-java-api)=
# Experimental Java API
:::{warning}
Java API support is an experimental feature and subject to change.
The Java API is not currently supported on KubeRay.
:::
Java is a mainstream programming language for production services. Ray Serve offers a native Java API for creating, updating, and managing deployments. You can create Ray Serve deployments using Java and call them via Python, or vice versa.
This section helps you to:
- create, query, and update Java deployments
- configure Java deployment resources
- manage Python deployments using the Java API
```{contents}
```
## Creating a Deployment
By specifying the full name of the class as an argument to the `Serve.deployment()` method, as shown in the code below, you can create and deploy a deployment of the class.
```{literalinclude} ../../../../java/serve/src/test/java/io/ray/serve/docdemo/ManageDeployment.java
:start-after: docs-create-start
:end-before: docs-create-end
:language: java
```
## Accessing a Deployment
Once a deployment is deployed, you can fetch its instance by name.
```{literalinclude} ../../../../java/serve/src/test/java/io/ray/serve/docdemo/ManageDeployment.java
:start-after: docs-query-start
:end-before: docs-query-end
:language: java
```
## Updating a Deployment
You can update a deployment's code and configuration and then redeploy it. The following example updates the `"counter"` deployment's initial value to 2.
```{literalinclude} ../../../../java/serve/src/test/java/io/ray/serve/docdemo/ManageDeployment.java
:start-after: docs-update-start
:end-before: docs-update-end
:language: java
```
## Configuring a Deployment
Ray Serve lets you configure your deployments to:
- scale out by increasing the number of [deployment replicas](serve-architecture-high-level-view)
- assign [replica resources](serve-cpus-gpus) such as CPUs and GPUs.
The next two sections describe how to configure your deployments.
### Scaling Out
By specifying the `numReplicas` parameter, you can change the number of deployment replicas:
```{literalinclude} ../../../../java/serve/src/test/java/io/ray/serve/docdemo/ManageDeployment.java
:start-after: docs-scale-start
:end-before: docs-scale-end
:language: java
```
### Resource Management (CPUs, GPUs)
Through the `rayActorOptions` parameter, you can reserve resources for each deployment replica, such as one GPU:
```{literalinclude} ../../../../java/serve/src/test/java/io/ray/serve/docdemo/ManageDeployment.java
:start-after: docs-resource-start
:end-before: docs-resource-end
:language: java
```
## Managing a Python Deployment
A Python deployment can also be managed and called by the Java API. Suppose you have a Python file `counter.py` in the `/path/to/code/` directory:
```python
from ray import serve
@serve.deployment
class Counter(object):
def __init__(self, value):
self.value = int(value)
def increase(self, delta):
self.value += int(delta)
return str(self.value)
```
You can deploy it through the Java API and call it through a `RayServeHandle`:
```java
import io.ray.api.Ray;
import io.ray.serve.api.Serve;
import io.ray.serve.deployment.Deployment;
import io.ray.serve.generated.DeploymentLanguage;
import java.io.File;
public class ManagePythonDeployment {
public static void main(String[] args) {
System.setProperty(
"ray.job.code-search-path",
System.getProperty("java.class.path") + File.pathSeparator + "/path/to/code/");
Serve.start(true, false, null);
Deployment deployment =
Serve.deployment()
.setDeploymentLanguage(DeploymentLanguage.PYTHON)
.setName("counter")
.setDeploymentDef("counter.Counter")
.setNumReplicas(1)
.setInitArgs(new Object[] {"1"})
.create();
deployment.deploy(true);
System.out.println(Ray.get(deployment.getHandle().method("increase").remote("2")));
}
}
```
:::{note}
Before `Ray.init` or `Serve.start`, you need to specify a directory to find the Python code. For details, please refer to [Cross-Language Programming](cross_language).
:::
## Future Roadmap
In the future, Ray Serve plans to provide more Java features, such as:
- an improved Java API that matches the Python version
- HTTP ingress support
- bring-your-own Java Spring project as a deployment
doc/source/serve/advanced-guides/multi-app-container.md
+169
View File
@@ -0,0 +1,169 @@
(serve-container-runtime-env-guide)=
# Run Multiple Applications in Different Containers
This section explains how to run multiple Serve applications on the same cluster in separate containers with different images.
This feature is experimental and the API is subject to change. If you have additional feature requests or run into issues, please submit them on [Github](https://github.com/ray-project/ray/issues).
## Install Podman
The `image_uri` runtime environment feature uses [Podman](https://podman.io/) to start and run containers. Follow the [Podman Installation Instructions](https://podman.io/docs/installation) to install Podman in the environment for all head and worker nodes.
:::{note}
For Ubuntu, the Podman package is available in the official repositories for Ubuntu 20.10 and newer.
```bash
sudo apt-get update
sudo apt-get install podman -y
```
:::
## Run a Serve application in a container
This example deploys two applications in separate containers: a Whisper model and a Resnet50 image classification model.
First, install the required dependencies in the images.
:::{warning}
The Ray version and Python version in the container *must* match those of the host environment exactly. Note that for Python, the versions must match down to the patch number.
:::
Save the following to files named `whisper.Dockerfile` and `resnet.Dockerfile`.
::::{tab-set}
:::{tab-item} whisper.Dockerfile
```dockerfile
# Use the latest Ray GPU image, `rayproject/ray:latest-py38-gpu`, so the Whisper model can run on GPUs.
FROM rayproject/ray:latest-py38-gpu
# Install the package `faster_whisper`, which is a dependency for the Whisper model.
RUN pip install faster_whisper==0.10.0
RUN sudo apt-get update && sudo apt-get install curl -y
# Download the source code for the Whisper application into `whisper_example.py`.
RUN curl -O https://raw.githubusercontent.com/ray-project/ray/master/doc/source/serve/doc_code/whisper_example.py
# Add /home/ray path to PYTHONPATH avoid import module error
ENV PYTHONPATH "${PYTHONPATH}:/home/ray"
```
:::
:::{tab-item} resnet.Dockerfile
```dockerfile
# Use the latest Ray CPU image, `rayproject/ray:latest-py38-cpu`.
FROM rayproject/ray:latest-py38-cpu
# Install the packages `torch` and `torchvision`, which are dependencies for the ResNet model.
RUN pip install torch==2.0.1 torchvision==0.15.2
RUN sudo apt-get update && sudo apt-get install curl -y
# Download the source code for the ResNet application into `resnet50_example.py`.
RUN curl -O https://raw.githubusercontent.com/ray-project/ray/master/doc/source/serve/doc_code/resnet50_example.py
# Add /home/ray path to PYTHONPATH avoid import module error
ENV PYTHONPATH "${PYTHONPATH}:/home/ray"
```
:::
::::
Then, build the corresponding images and push it to your choice of container registry. This tutorial uses `alice/whisper_image:latest` and `alice/resnet_image:latest` as placeholder names for the images, but make sure to swap out `alice` for a repo name of your choice.
::::{tab-set}
:::{tab-item} Whisper
```bash
# Build the image from the Dockerfile using Podman
export IMG1=alice/whisper_image:latest
podman build -t $IMG1 -f whisper.Dockerfile .
# Push to a registry. This step is unnecessary if you are deploying Serve locally.
podman push $IMG1
```
:::
:::{tab-item} Resnet
```bash
# Build the image from the Dockerfile using Podman
export IMG2=alice/resnet_image:latest
podman build -t $IMG2 -f resnet.Dockerfile .
# Push to a registry. This step is unnecessary if you are deploying Serve locally.
podman push $IMG2
```
:::
::::
Finally, you can specify the container image within which you want to run each application in the `image_uri` field of an application's runtime environment specification.
:::{note}
Previously you could access the feature through the `container` field of the runtime environment. That API is now deprecated in favor of `image_uri`.
:::
The following Serve config runs the `whisper` app with the image `IMG1`, and the `resnet` app with the image `IMG2`. `podman images` command can be used to list the names of the images. Concretely, all deployment replicas in the applications start and run in containers with the respective images.
```yaml
applications:
- name: whisper
import_path: whisper_example:entrypoint
route_prefix: /whisper
runtime_env:
image_uri: {IMG1}
- name: resnet
import_path: resnet50_example:app
route_prefix: /resnet
runtime_env:
image_uri: {IMG2}
```
### Send queries
```python
>>> import requests
>>> audio_file = "https://storage.googleapis.com/public-lyrebird-test/test_audio_22s.wav"
>>> resp = requests.post("http://localhost:8000/whisper", json={"filepath": audio_file}) # doctest: +SKIP
>>> resp.json() # doctest: +SKIP
{
"language": "en",
"language_probability": 1,
"duration": 21.775,
"transcript_text": " Well, think about the time of our ancestors. A ping, a ding, a rustling in the bushes is like, whoo, that means an immediate response. Oh my gosh, what's that thing? Oh my gosh, I have to do it right now. And dude, it's not a tiger, right? Like, but our, our body treats stress as if it's life-threatening because to quote Robert Sapolsky or butcher his quote, he's a Robert Sapolsky is like one of the most incredible stress physiologists of",
"whisper_alignments": [
[
0.0,
0.36,
" Well,",
0.3125
],
...
]
}
>>> link_to_image = "https://serve-resnet-benchmark-data.s3.us-west-1.amazonaws.com/000000000019.jpeg"
>>> resp = requests.post("http://localhost:8000/resnet", json={"uri": link_to_image}) # doctest: +SKIP
>>> resp.text # doctest: +SKIP
ox
```
## Advanced
### Compatibility with other runtime environment fields
Currently, use of the `image_uri` field is only supported with `config` and `env_vars`. If you have a use case for pairing `image_uri` with another runtime environment feature, submit a feature request on [Github](https://github.com/ray-project/ray/issues).
### Environment variables
The following environment variables will be set for the process in your container, in order of highest to lowest priority:
1. Environment variables specified in `runtime_env["env_vars"]`.
2. All environment variables that start with the prefix `RAY_` (including the two special variables `RAY_RAYLET_PID` and `RAY_JOB_ID`) are inherited by the container at runtime.
3. Any environment variables set in the docker image.
### Running the Ray cluster in a Docker container
If raylet is running inside a container, then that container needs the necessary permissions to start a new container. To setup correct permissions, you need to start the container that runs the raylet with the flag `--privileged`.
### Troubleshooting
* **Permission denied: '/tmp/ray/session_2023-11-28_15-27-22_167972_6026/ports_by_node.json.lock'**
* This error likely occurs because the user running inside the Podman container is different from the host user that started the Ray cluster. The folder `/tmp/ray`, which is volume mounted into the podman container, is owned by the host user that started Ray. The container, on the other hand, is started with the flag `--userns=keep-id`, meaning the host user is mapped into the container as itself. Therefore, permissions issues should only occur if the user inside the container is different from the host user. For instance, if the user on host is `root`, and you're using a container whose base image is a standard Ray image, then by default the container starts with user `ray(1000)`, who won't be able to access the mounted `/tmp/ray` volume.
* **ERRO[0000] 'overlay' is not supported over overlayfs: backing file system is unsupported for this graph driver**
* This error should only occur when you're running the Ray cluster inside a container. If you see this error when starting the replica actor, try volume mounting `/var/lib/containers` in the container that runs raylet. That is, add `-v /var/lib/containers:/var/lib/containers` to the command that starts the Docker container.
* **cannot clone: Operation not permitted; Error: cannot re-exec process**
* This error should only occur when you're running the Ray cluster inside a container. This error implies that you don't have the permissions to use Podman to start a container. You need to start the container that runs raylet, with privileged permissions by adding `--privileged`.
* **Very slow or hanging container startup**
* This is typically caused by using the default podman storage driver (`vfs`) with large container images. Podman runs in rootless mode, so its startup sequence involves modifying permissions of files in the container. The default storage driver is very slow to do this. Try configuring podman to use the `overlay` storage driver instead. You may need to also configure the `mount_program` to point to `/usr/bin/fuse-overlayfs` (or your appropriate local path).
doc/source/serve/advanced-guides/multi-node-gpu-troubleshooting.md
+252
View File
@@ -0,0 +1,252 @@
(serve-multi-node-gpu-troubleshooting)=
# Troubleshoot multi-node GPU serving on KubeRay
This guide helps you diagnose and resolve common issues when deploying multi-node GPU workloads on KubeRay, particularly for large language model (LLM) serving with vLLM.
## Debugging strategy
When encountering issues with multi-node GPU serving, use this systematic approach to isolate the problem:
1. **Test on different platforms** Compare behavior between:
- Single node without KubeRay
- Standalone vLLM server on KubeRay
- Ray Serve LLM deployment on KubeRay
2. **Vary hardware configurations** Test with different GPU types—for example, A100s vs H100s—to identify hardware-specific issues
3. **Use minimal reproducers** Create simplified test cases that isolate specific components (NCCL, model loading, etc.)
## Common issues and solutions
### 1. Head pod scheduled on GPU node
**Symptoms**
- `ray status` shows duplicate GPU resources, for example, 24 GPUs when cluster only has 16 GPUs
- Model serving hangs when using pipeline parallelism (PP > 1)
- Resource allocation conflicts
**Root Cause** The Ray head pod is incorrectly scheduled on a GPU worker node, causing resource accounting issues.
**Solution** Configure the head pod to use zero GPUs in your RayCluster specification:
```yaml
apiVersion: ray.io/v1
kind: RayCluster
metadata:
name: my-cluster
spec:
headGroupSpec:
rayStartParams:
num-cpus: "0"
num-gpus: "0" # Ensure head pod doesn't claim GPU resources.
# ... other head group configuration
```
### 2. AWS OFI plugin version issues (H100-specific)
**Symptoms**
- NCCL initialization failures on H100 instances
- Works fine on A100 but fails on H100 with identical configuration
- Malformed topology files
**Root Cause** Outdated `aws-ofi-plugin` in container images causes NCCL topology detection to fail on H100 instances.
**Related issues**
- [NVIDIA NCCL Issue #1726](https://github.com/NVIDIA/nccl/issues/1726)
- [vLLM Issue #18997](https://github.com/vllm-project/vllm/issues/18997)
- [AWS OFI NCCL Fix](https://github.com/aws/aws-ofi-nccl/pull/916)
**Solution**
- Update to a newer container image with an updated `aws-ofi-plugin`
- Use the NCCL debugging script below to verify NCCL functions as expected
- Consider hardware-specific configuration adjustments
## Further troubleshooting
If you continue to experience issues after following this guide:
1. **Collect diagnostic information**: Run the NCCL debugging script below and save the output
2. **Check compatibility**: Verify Ray, vLLM, PyTorch, and CUDA versions are compatible
3. **Review logs**: Examine Ray cluster logs and worker pod logs for additional error details
4. **Hardware verification**: Test with different GPU types if possible
5. **Community support**: Share your findings with the Ray and vLLM communities for additional help
## Additional resources
- [Ray Multi-Node GPU Guide](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/gpu.html)
- [vLLM Distributed Serving Documentation](https://docs.vllm.ai/en/latest/serving/distributed_serving.html)
- [NCCL Troubleshooting Guide](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html)
## NCCL debugging script
Use this diagnostic script to identify NCCL-related issues in your multi-node GPU setup:
```python
#!/usr/bin/env python3
"""
NCCL Diagnostic Script for Multi-Node GPU Serving
This script helps identify NCCL configuration issues that can cause
multi-node GPU serving failures. Run this script on each node to verify
NCCL function before deploying distributed workloads.
Usage: python3 multi-node-nccl-check.py
"""
import os
import sys
import socket
import torch
from datetime import datetime
def log(msg):
"""Log messages with timestamp for better debugging."""
timestamp = datetime.now().strftime("%H:%M:%S")
print(f"[{timestamp}] {msg}", flush=True)
def print_environment_info():
"""Print relevant environment information for debugging."""
log("=== Environment Information ===")
log(f"Hostname: {socket.gethostname()}")
log(f"CUDA_VISIBLE_DEVICES: {os.environ.get('CUDA_VISIBLE_DEVICES', 'not set')}")
# Print all NCCL-related environment variables.
nccl_vars = [var for var in os.environ.keys() if var.startswith('NCCL_')]
if nccl_vars:
log("NCCL Environment Variables:")
for var in sorted(nccl_vars):
log(f" {var}: {os.environ[var]}")
else:
log("No NCCL environment variables set")
def check_cuda_availability():
"""Verify CUDA is available and functional."""
log("\n=== CUDA Availability Check ===")
if not torch.cuda.is_available():
log("ERROR: CUDA not available")
return False
device_count = torch.cuda.device_count()
log(f"CUDA device count: {device_count}")
log(f"PyTorch version: {torch.__version__}")
# Check NCCL availability in PyTorch.
try:
import torch.distributed as dist
if hasattr(torch.distributed, 'nccl'):
log(f"PyTorch NCCL available: {torch.distributed.is_nccl_available()}")
except Exception as e:
log(f"Error checking NCCL availability: {e}")
return True
def test_individual_gpus():
"""Test that each GPU is working individually."""
log("\n=== Individual GPU Tests ===")
for gpu_id in range(torch.cuda.device_count()):
log(f"\n--- Testing GPU {gpu_id} ---")
try:
torch.cuda.set_device(gpu_id)
device = torch.cuda.current_device()
log(f"Device {device}: {torch.cuda.get_device_name(device)}")
# Print device properties.
props = torch.cuda.get_device_properties(device)
log(f" Compute capability: {props.major}.{props.minor}")
log(f" Total memory: {props.total_memory / 1024**3:.2f} GB")
# Test basic CUDA operations.
log(" Testing basic CUDA operations...")
tensor = torch.ones(1000, device=f'cuda:{gpu_id}')
result = tensor.sum()
log(f" Basic CUDA test passed: sum = {result.item()}")
# Test cross-GPU operations if multiple GPUs are available.
if torch.cuda.device_count() > 1:
log(" Testing cross-GPU operations...")
try:
other_gpu = (gpu_id + 1) % torch.cuda.device_count()
test_tensor = torch.randn(10, 10, device=f'cuda:{gpu_id}')
tensor_copy = test_tensor.to(f'cuda:{other_gpu}')
log(f" Cross-GPU copy successful: GPU {gpu_id} -> GPU {other_gpu}")
except Exception as e:
log(f" Cross-GPU copy failed: {e}")
# Test memory allocation.
log(" Testing large memory allocations...")
try:
large_tensor = torch.zeros(1000, 1000, device=f'cuda:{gpu_id}')
log(" Large memory allocation successful")
del large_tensor
except Exception as e:
log(f" Large memory allocation failed: {e}")
except Exception as e:
log(f"ERROR testing GPU {gpu_id}: {e}")
import traceback
log(f"Traceback:\n{traceback.format_exc()}")
def test_nccl_initialization():
"""Test NCCL initialization and basic operations."""
log("\n=== NCCL Initialization Test ===")
try:
import torch.distributed as dist
# Set up single-process NCCL environment.
os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '29500'
os.environ['RANK'] = '0'
os.environ['WORLD_SIZE'] = '1'
log("Attempting single-process NCCL initialization...")
dist.init_process_group(
backend='nccl',
rank=0,
world_size=1
)
log("Single-process NCCL initialization successful!")
# Test basic NCCL operation.
if torch.cuda.is_available():
device = torch.cuda.current_device()
tensor = torch.ones(10, device=device)
# This is a no-op with world_size=1 but exercises NCCL
dist.all_reduce(tensor)
log("NCCL all_reduce test successful!")
dist.destroy_process_group()
log("NCCL cleanup successful!")
except Exception as e:
log(f"NCCL initialization failed: {e}")
import traceback
log(f"Full traceback:\n{traceback.format_exc()}")
def main():
"""Main diagnostic routine."""
log("Starting NCCL Diagnostic Script")
log("=" * 50)
print_environment_info()
if not check_cuda_availability():
sys.exit(1)
test_individual_gpus()
test_nccl_initialization()
log("\n" + "=" * 50)
log("NCCL diagnostic script completed")
log("If you encountered errors, check the specific error messages above")
log("and refer to the troubleshooting guide for solutions.")
if __name__ == "__main__":
main()
doc/source/serve/advanced-guides/performance.md
+327
View File
@@ -0,0 +1,327 @@
(serve-perf-tuning)=
# Performance Tuning
This section should help you:
- understand Ray Serve's performance characteristics
- find ways to debug and tune your Serve application's performance
:::{note}
This section offers some tips and tricks to improve your Ray Serve application's performance. Check out the [architecture page](serve-architecture) for helpful context, including an overview of the HTTP proxy actor and deployment replica actors.
:::
```{contents}
```
## Performance and benchmarks
Ray Serve is built on top of Ray, so its scalability is bounded by Ray’s scalability. See Ray’s [scalability envelope](https://github.com/ray-project/ray/blob/master/release/benchmarks/README.md) to learn more about the maximum number of nodes and other limitations.
## Debugging performance issues in request path
The performance issue you're most likely to encounter is high latency or low throughput for requests.
Once you set up [monitoring](serve-monitoring) with Ray and Ray Serve, these issues may appear as:
* `serve_num_router_requests_total` staying constant while your load increases
* `serve_deployment_processing_latency_ms` spiking up as queries queue up in the background
The following are ways to address these issues:
1. Make sure you are using the right hardware and resources:
* Are you reserving GPUs for your deployment replicas using `ray_actor_options` (e.g., `ray_actor_options={“num_gpus”: 1}`)?
* Are you reserving one or more cores for your deployment replicas using `ray_actor_options` (e.g., `ray_actor_options={“num_cpus”: 2}`)?
* Are you setting [OMP_NUM_THREADS](serve-omp-num-threads) to increase the performance of your deep learning framework?
2. Try batching your requests. See [Dynamic Request Batching](serve-performance-batching-requests).
3. Consider using `async` methods in your callable. See [the section below](serve-performance-async-methods).
4. Set an end-to-end timeout for your HTTP requests. See [the section below](serve-performance-e2e-timeout).
(serve-performance-async-methods)=
### Using `async` methods
:::{note}
According to the [FastAPI documentation](https://fastapi.tiangolo.com/async/#very-technical-details), `def` endpoint functions are called in a separate threadpool, so you might observe many requests running at the same time inside one replica, and this scenario might cause OOM or resource starvation. In this case, you can try to use `async def` to control the workload performance.
:::
Are you using `async def` in your callable? If you are using `asyncio` and hitting the same queuing issue mentioned above, you might want to increase `max_ongoing_requests`. By default, Serve sets this to a low value (5) to ensure clients receive proper backpressure. You can increase the value in the deployment decorator; for example, `@serve.deployment(max_ongoing_requests=1000)`.
(serve-performance-e2e-timeout)=
### Set an end-to-end request timeout
By default, Serve lets client HTTP requests run to completion no matter how long they take. However, slow requests could bottleneck the replica processing, blocking other requests that are waiting. Set an end-to-end timeout, so slow requests can be terminated and retried.
You can set an end-to-end timeout for HTTP requests by setting the `request_timeout_s` parameter in the `http_options` field of the Serve config. HTTP Proxies wait for that many seconds before terminating an HTTP request. This config is global to your Ray cluster, and you can't update it during runtime. Use [client-side retries](serve-best-practices-http-requests) to retry requests that time out due to transient failures.
:::{note}
Serve returns a response with status code `408` when a request times out. Clients can retry when they receive this `408` response.
:::
(serve-performance-per-request-headers)=
### Override timeout and disconnect behavior per request
Two request headers let callers override the global `request_timeout_s` and client-disconnect policy on a per-request basis.
`x-request-timeout-seconds`
: Overrides `request_timeout_s` for this single request.
- A **positive float** sets the timeout in seconds (for example, `x-request-timeout-seconds: 30`).
- A **non-positive value** (for example, `0` or `-1`) disables the timeout for this request regardless of the global setting.
- An **absent or non-numeric value** falls back to the global `request_timeout_s`.
`x-request-disconnect-disabled`
: Controls whether Serve monitors for client disconnects on this request.
- `?1` — disables disconnect detection; Serve continues processing even if the client closes the connection.
- `?0` or absent — disconnect detection is enabled (default behavior).
**Performance impact of disabling both headers together**
Setting `x-request-timeout-seconds` to a non-positive value *and* `x-request-disconnect-disabled: ?1` on the same request enables a fast path in the direct-ingress handler that yields a meaningful throughput improvement:
- Serve skips wrapping the user handler in an `asyncio.create_task()` call.
- Serve skips the `asyncio.wait()` call that would otherwise coordinate the handler task, the disconnect-monitoring task, and the timeout watcher.
- Instead, Serve directly `await`s the user handler, eliminating all per-request event-loop scheduling overhead for monitoring.
In the proxy path the gain is more modest: `asyncio.wait()` is called per response chunk with one fewer task and no timeout handle, which reduces event-loop overhead at high request rates.
### Set backoff time when choosing replica
Ray Serve allows you to fine-tune the backoff behavior of the request router, which can help reduce latency when waiting for replicas to become ready. It uses exponential backoff strategy when retrying to route requests to replicas that are temporarily unavailable. You can optimize this behavior for your workload by configuring the following environment variables:
- `RAY_SERVE_ROUTER_RETRY_INITIAL_BACKOFF_S`: The initial backoff time (in seconds) before retrying a request. Default is `0.025`.
- `RAY_SERVE_ROUTER_RETRY_BACKOFF_MULTIPLIER`: The multiplier applied to the backoff time after each retry. Default is `2`.
- `RAY_SERVE_ROUTER_RETRY_MAX_BACKOFF_S`: The maximum backoff time (in seconds) between retries. Default is `0.5`.
### Set timeouts while probing replicas for queue length
Ray Serve's request router probes replicas for their queue lengths to make intelligent load balancing decisions. You can tune the following environment variables to optimize this behavior for your workload:
- `RAY_SERVE_QUEUE_LENGTH_RESPONSE_DEADLINE_S`: The initial timeout (in seconds) for waiting for replicas to respond with their queue length information. Default is `0.1`.
- `RAY_SERVE_MAX_QUEUE_LENGTH_RESPONSE_DEADLINE_S`: The maximum timeout (in seconds) for queue length responses. When retrying with exponential backoff, the deadline increases but is capped at this value. Default is `1.0`.
- `RAY_SERVE_QUEUE_LENGTH_CACHE_TIMEOUT_S`: How long (in seconds) cached queue length information from replicas is considered valid. After this timeout, the cache entry expires and the router must probe the replica again. Default is `10.0`.
### Configure locality-based routing
Ray Serve routes requests to replicas based on locality to reduce network latency. The system applies locality routing in two scenarios: proxy-to-replica communication (HTTP/gRPC requests) and inter-deployment communication (replica-to-replica calls through `DeploymentHandle`).
#### Routing priority
When locality routing is enabled, the system selects replicas in the following priority order:
1. **Same node**: Replicas running on the same node as the caller (lowest latency)
2. **Same availability zone**: Replicas in the same availability zone as the caller
3. **Any replica**: All available replicas (fallback when local replicas are busy)
If replicas at a higher priority level are busy or unavailable, the system automatically falls back to the next level.
#### Proxy-to-replica routing
You can configure proxy routing behavior through environment variables:
- `RAY_SERVE_PROXY_PREFER_LOCAL_NODE_ROUTING`: When enabled, the proxy prefers routing requests to replicas on the same node. Default is `1` (enabled).
- `RAY_SERVE_PROXY_PREFER_LOCAL_AZ_ROUTING`: When enabled, the proxy prefers routing requests to replicas in the same availability zone. Default is `1` (enabled).
#### Inter-deployment routing
When one deployment calls another through a `DeploymentHandle`, you can enable locality routing to reduce latency between deployments.
**Same-node routing**: By default, inter-deployment calls don't prefer same-node replicas. To enable same-node routing, initialize the handle with the `_prefer_local_routing` option:
```python
from ray import serve
from ray.serve.handle import DeploymentHandle
@serve.deployment
class Caller:
def __init__(self, target_handle: DeploymentHandle):
# Enable same-node routing for this handle
self._handle = target_handle.options(_prefer_local_routing=True)
async def call_target(self):
# Requests prefer replicas on the same node as this Caller replica
return await self._handle.remote()
```
**Same-AZ routing**: The `RAY_SERVE_PROXY_PREFER_LOCAL_AZ_ROUTING` environment variable controls availability zone routing for both proxy and inter-deployment communication. You can't configure AZ routing per-handle.
(serve-high-throughput)=
### Enable throughput-optimized serving
:::{note}
In Ray v2.54.0, the defaults for `RAY_SERVE_RUN_USER_CODE_IN_SEPARATE_THREAD` and `RAY_SERVE_RUN_ROUTER_IN_SEPARATE_LOOP` will change to `0` for improved performance.
:::
This section details how to enable Ray Serve options focused on improving throughput and reducing latency. These configurations focus on the following:
- Reducing overhead associated with frequent logging.
- Disabling behavior that allowed Serve applications to include blocking operations.
If your Ray Serve code includes thread blocking operations, you must refactor your code to achieve enhanced throughput. The following table shows examples of blocking and non-blocking code:
<table>
<tr>
<th>Blocking operation (❌)</th>
<th>Non-blocking operation (✅)</th>
</tr>
<tr>
<td>
```python
from ray import serve
from fastapi import FastAPI
import time
app = FastAPI()
@serve.deployment
@serve.ingress(app)
class BlockingDeployment:
@app.get("/process")
async def process(self):
# ❌ Blocking operation
time.sleep(2)
return {"message": "Processed (blocking)"}
serve.run(BlockingDeployment.bind())
```
</td>
<td>
```python
from ray import serve
from fastapi import FastAPI
import asyncio
app = FastAPI()
@serve.deployment
@serve.ingress(app)
class NonBlockingDeployment:
@app.get("/process")
async def process(self):
# ✅ Non-blocking operation
await asyncio.sleep(2)
return {"message": "Processed (non-blocking)"}
serve.run(NonBlockingDeployment.bind())
```
</td>
</tr>
</table>
To configure all options to the recommended settings, set the environment variable `RAY_SERVE_THROUGHPUT_OPTIMIZED=1`.
You can also configure each option individually. The following table details the recommended configurations and their impact:
| Configured value | Impact |
| --- | --- |
| `RAY_SERVE_RUN_USER_CODE_IN_SEPARATE_THREAD=0` | Your code runs in the same event loop as the replica's main event loop. You must avoid blocking operations in your request path. Set this configuration to `1` to run your code in a separate event loop, which protects the replica's ability to communicate with the Serve Controller if your code has blocking operations. |
| `RAY_SERVE_RUN_ROUTER_IN_SEPARATE_LOOP=0`| The request router runs in the same event loop as the your code's event loop. You must avoid blocking operations in your request path. Set this configuration to `1` to run the router in a separate event loop, which protect Ray Serve's request routing ability when your code has blocking operations |
| `RAY_SERVE_REQUEST_PATH_LOG_BUFFER_SIZE=1000` | Sets the log buffer to batch writes to every `1000` logs, flushing the buffer on write. The system always flushes the buffer and writes logs when it detects a line with level ERROR. Set the buffer size to `1` to disable buffering and write logs immediately. |
| `RAY_SERVE_LOG_TO_STDERR=0` | Only write logs to files under the `logs/serve/` directory. Proxy, Controller, and Replica logs no longer appear in the console, worker files, or the Actor Logs section of the Ray Dashboard. Set this property to `1` to enable additional logging. |
You may want to enable throughput-optimized serving while customizing the options above. You can do this by setting `RAY_SERVE_THROUGHPUT_OPTIMIZED=1` and overriding the specific options. For example, to enable throughput-optimized serving and continue logging to stderr, you should set `RAY_SERVE_THROUGHPUT_OPTIMIZED=1` and override with `RAY_SERVE_LOG_TO_STDERR=1`.
(serve-haproxy)=
### Use HAProxy load balancing
By default, Ray Serve uses a Python-based HTTP/gRPC proxy to route requests to replicas. You can replace this with [HAProxy](https://www.haproxy.org/), a high-performance C-based load balancer, for improved throughput and lower latency at high request rates.
When HAProxy mode is enabled:
- An `HAProxyManager` actor runs on each node (by default) and translates Serve's routing table into HAProxy configuration reloads.
- Each ingress replica opens a port, and HAProxy routes traffic directly to replicas, replacing the Python proxy entirely.
- Live traffic flows through the HAProxy subprocess, not through any Python actor.
#### Prerequisites
HAProxy must be available on every node that runs a Serve proxy. On Linux, `pip install "ray[serve]"` installs the [`ray-haproxy`](https://pypi.org/project/ray-haproxy/) package, which ships a prebuilt HAProxy binary that Serve uses automatically. To use a different binary, set `RAY_SERVE_HAPROXY_BINARY_PATH` to its absolute path.
#### Enabling HAProxy
Set the `RAY_SERVE_ENABLE_HA_PROXY` environment variable to `1` on all nodes before starting Ray:
```bash
export RAY_SERVE_ENABLE_HA_PROXY=1
```
This environment variable must be set on all nodes in the ray cluster.
::::{tab-set}
:::{tab-item} KubeRay
```yaml
# In the Ray container spec for head and worker groups:
env:
- name: RAY_SERVE_ENABLE_HA_PROXY
value: "1"
```
:::
:::{tab-item} VM cluster
```bash
# On every node (head and workers)
export RAY_SERVE_ENABLE_HA_PROXY=1
ray start --head # or ray start --address=<head-ip>:6379 on workers
```
:::
::::
#### Installing HAProxy manually (example)
Some platforms have no `ray-haproxy` wheel, such as non-glibc Linux, macOS, and Windows. On those platforms, install HAProxy 2.8 from source on every node. These steps are provided as an example only.
The following steps are for Ubuntu/Debian:
```bash
# Install build dependencies
apt-get update -y && apt-get install -y --no-install-recommends \
build-essential ca-certificates curl libc6-dev \
liblua5.3-dev libpcre3-dev libssl-dev zlib1g-dev
# Build HAProxy from source
export HAPROXY_VERSION="2.8.20"
curl -sSfL -o /tmp/haproxy.tar.gz \
"https://www.haproxy.org/download/2.8/src/haproxy-${HAPROXY_VERSION}.tar.gz"
mkdir -p /tmp/haproxy-build && tar -xzf /tmp/haproxy.tar.gz -C /tmp/haproxy-build --strip-components=1
make -C /tmp/haproxy-build TARGET=linux-glibc \
USE_OPENSSL=1 USE_ZLIB=1 USE_PCRE=1 USE_LUA=1 USE_PROMEX=1 -j$(nproc)
make -C /tmp/haproxy-build install SBINDIR=/usr/local/bin
rm -rf /tmp/haproxy-build /tmp/haproxy.tar.gz
# Install runtime dependencies
apt-get install -y --no-install-recommends liblua5.3-0
# Create required directories
mkdir -p /etc/haproxy /run/haproxy /var/log/haproxy
```
The required build flags are `USE_OPENSSL=1 USE_ZLIB=1 USE_PCRE=1 USE_LUA=1 USE_PROMEX=1`. The runtime dependency is `liblua5.3-0` (Lua runtime library).
(serve-interdeployment-grpc)=
### Use gRPC for interdeployment communication
By default, when one deployment calls another via a `DeploymentHandle`, requests are sent through Ray's actor RPC system. You can switch this internal transport to gRPC by setting `RAY_SERVE_USE_GRPC_BY_DEFAULT=1` on all nodes before starting Ray. This makes all `DeploymentHandle` calls use gRPC transport, which serializes requests and sends them directly to the target replica's gRPC server. gRPC transport is most beneficial for high-throughput workloads with small payloads (under ~1 MB), where bypassing Ray's object store reduces per-request overhead.
#### When not to use gRPC
1. Since gRPC is required to serialize every payload, it should not be used for large payloads (greater than ~1 MB). If gRPC was enabled by default, individual handles' transport mechanism can be manually set to actor RPC with `handle.options(_by_reference=True)`, and passes larger objects by reference.
2. When passing a `DeploymentResponse` from one deployment into another (i.e., without `await`-ing it first), gRPC resolves the value at the caller and serializes it over the wire. With actor RPC, the underlying `ObjectRef` is forwarded without materializing the data. If your pipeline chains `DeploymentResponse` objects through multiple deployments with payload sizes >100KB, avoid using gRPC for those handles.
```{literalinclude} ../doc_code/interdeployment_grpc.py
:start-after: __start_grpc_override__
:end-before: __end_grpc_override__
:language: python
```
## Debugging performance issues in controller
The Serve Controller runs on the Ray head node and is responsible for a variety of tasks, including receiving autoscaling metrics from other Ray Serve components. If the Serve Controller becomes overloaded (symptoms might include high CPU usage and a large number of pending `ServeController.record_autoscaling_metrics_from_handle` tasks), you can tune the following environment variables:
- `RAY_SERVE_CONTROL_LOOP_INTERVAL_S`: The interval between cycles of the control loop (defaults to `0.1` seconds). Increasing this value gives the Controller more time to process requests and may help alleviate overload.
- `RAY_SERVE_CONTROLLER_MAX_CONCURRENCY`: The maximum number of concurrent requests the Controller can handle (defaults to `15000`). The Controller accepts one long poll request per handle, so its concurrency needs scale with the number of handles. Increase this value if you have a large number of deployment handles.
- `RAY_SERVE_MAX_CACHED_HANDLES`: The maximum number of cached deployment handles (defaults to `100`). Each handle maintains a long poll connection to the controller, so limiting the cache size reduces controller overhead. Decrease this value if you're experiencing controller overload due to many handles.
To get a quantitative view of controller health, query the GET `/api/serve/applications/` endpoint and inspect the [`controller_health_metrics`](../api/doc/ray.serve.schema.ControllerHealthMetrics.rst) field on the response. It exposes rolling-window statistics ([`DurationStats`](../api/doc/ray.serve.schema.DurationStats.rst): mean, std, min, max) for control-loop duration and component update latencies (deployment, application, proxy, and node state managers), along with the event-loop delay (actual vs. expected sleep — positive values indicate the loop is falling behind), the count of pending asyncio tasks, and the delay between when autoscaling metrics are generated and when they reach the controller. High `event_loop_delay_s` or growing `loop_duration_s.mean` are the clearest signals that the Controller is overloaded and the tuning knobs above should be applied.
doc/source/serve/advanced-guides/replica-ranks.md
+186
View File
@@ -0,0 +1,186 @@
(serve-replica-ranks)=
# Replica ranks
:::{warning}
This API is experimental and may change between Ray minor versions.
:::
Replica ranks provide a unique identifier for **each replica within a deployment**. Each replica receives a **`ReplicaRank` object** containing rank information and **a world size (the total number of replicas)**. The rank object includes a global rank (an integer from 0 to N-1), a node rank, and a local rank on the node.
## Access replica ranks
You can access the rank and world size from within a deployment through the replica context using [`serve.get_replica_context()`](../api/doc/ray.serve.get_replica_context.rst).
The following example shows how to access replica rank information:
```{literalinclude} ../doc_code/replica_rank.py
:start-after: __replica_rank_start__
:end-before: __replica_rank_end__
:language: python
```
```{literalinclude} ../doc_code/replica_rank.py
:start-after: __replica_rank_start_run_main__
:end-before: __replica_rank_end_run_main__
:language: python
```
The [`ReplicaContext`](../api/doc/ray.serve.context.ReplicaContext.rst) provides two key fields:
- `rank`: A [`ReplicaRank`](../api/doc/ray.serve.schema.ReplicaRank.rst) object containing rank information for this replica. Access the integer rank value with `.rank`.
- `world_size`: The target number of replicas for the deployment.
The `ReplicaRank` object contains three fields:
- `rank`: The global rank (an integer from 0 to N-1) representing this replica's unique identifier across all nodes.
- `node_rank`: The rank of the node this replica runs on (an integer from 0 to M-1 where M is the number of nodes).
- `local_rank`: The rank of this replica on its node (an integer from 0 to K-1 where K is the number of replicas on this node).
:::{note}
**Accessing rank values:**
To use the rank in your code, access the `.rank` attribute to get the integer value:
```python
context = serve.get_replica_context()
my_rank = context.rank.rank # Get the integer rank value
my_node_rank = context.rank.node_rank # Get the node rank
my_local_rank = context.rank.local_rank # Get the local rank on this node
```
Most use cases only need the global `rank` value. The `node_rank` and `local_rank` are useful for advanced scenarios such as coordinating replicas on the same node.
:::
## Handle rank changes with reconfigure
When a replica's rank changes (such as during downscaling), Ray Serve can automatically call the `reconfigure` method on your deployment class to notify it of the new rank. This allows you to update replica-specific state when ranks change.
The following example shows how to implement `reconfigure` to handle rank changes:
```{literalinclude} ../doc_code/replica_rank.py
:start-after: __reconfigure_rank_start__
:end-before: __reconfigure_rank_end__
:language: python
```
```{literalinclude} ../doc_code/replica_rank.py
:start-after: __reconfigure_rank_start_run_main__
:end-before: __reconfigure_rank_end_run_main__
:language: python
```
### When reconfigure is called
Ray Serve automatically calls your `reconfigure` method in the following situations:
1. **At replica startup:** When a replica starts, if your deployment has both a `reconfigure` method and a `user_config`, Ray Serve calls `reconfigure` after running `__init__`. This lets you initialize rank-aware state without duplicating code between `__init__` and `reconfigure`.
2. **When you update user_config:** When you redeploy with a new `user_config`, Ray Serve calls `reconfigure` on all running replicas. If your `reconfigure` method includes `rank` as a parameter, Ray Serve passes both the new `user_config` and the current rank as a `ReplicaRank` object.
3. **When a replica's rank changes:** During downscaling, ranks may be reassigned to maintain contiguity (0 to N-1). If your `reconfigure` method includes `rank` as a parameter and your deployment has a `user_config`, Ray Serve calls `reconfigure` with the existing `user_config` and the new rank as a `ReplicaRank` object.
:::{note}
**Requirements to receive rank updates:**
To get rank changes through `reconfigure`, your deployment needs:
- A class-based deployment (function deployments don't support `reconfigure`)
- A `reconfigure` method with `rank` as a parameter: `def reconfigure(self, user_config, rank: ReplicaRank)`
- A `user_config` in your deployment (even if it's just an empty dict: `user_config={}`)
Without a `user_config`, Ray Serve won't call `reconfigure` for rank changes.
:::
:::{tip}
If you'd like different behavior for when `reconfigure` is called with rank changes, [open a GitHub issue](https://github.com/ray-project/ray/issues/new/choose) to discuss your use case with the Ray Serve team.
:::
## How replica ranks work
:::{note}
**Rank reassignment is eventually consistent**
When replicas are removed during downscaling, rank reassignment to maintain contiguity (0 to N-1) doesn't happen immediately. The controller performs rank consistency checks and reassignment only when the deployment reaches a `HEALTHY` state in its update loop. This means there can be a brief period after downscaling where ranks are non-contiguous before the controller reassigns them.
This design choice prevents rank reassignment from interfering with ongoing deployment updates and rollouts. If you need immediate rank reassignment or different behavior, [open a GitHub issue](https://github.com/ray-project/ray/issues/new/choose) to discuss your use case with the Ray Serve team.
:::
:::{note}
**Ranks don't influence scheduling or eviction decisions**
Replica ranks are independent of scheduling and eviction decisions. The deployment scheduler doesn't consider ranks when placing replicas on nodes, so there's no guarantee that replicas with contiguous ranks (such as rank 0 and rank 1) will be on the same node. Similarly, during downscaling, the autoscaler's eviction decisions don't take replica ranks into account—any replica can be chosen for removal regardless of its rank.
If you need rank-aware scheduling or eviction (for example, to colocate replicas with consecutive ranks), [open a GitHub issue](https://github.com/ray-project/ray/issues/new/choose) to discuss your requirements with the Ray Serve team.
:::
Ray Serve manages replica ranks automatically throughout the deployment lifecycle. The system maintains these invariants:
1. Ranks are contiguous integers from 0 to N-1.
2. Each running replica has exactly one rank.
3. No two replicas share the same rank.
### Rank assignment lifecycle
The following table shows how ranks and world size behave during different events:
| Event | Local Rank | World Size |
|-------|------------|------------|
| Upscaling | No change for existing replicas | Increases to target count |
| Downscaling | Can change to maintain contiguity | Decreases to target count |
| Other replica dies(will be restarted) | No change | No change |
| Self replica dies | No change | No change |
:::{note}
World size always reflects the target number of replicas configured for the deployment, not the current number of running replicas. During scaling operations, the world size updates immediately to the new target, even while replicas are still starting or stopping.
:::
### Rank lifecycle state machine
```
┌─────────────────────────────────────────────────────────────┐
│ DEPLOYMENT LIFECYCLE │
└─────────────────────────────────────────────────────────────┘
Initial Deployment / Upscaling:
┌──────────┐ assign ┌──────────┐
│ No Rank │ ───────────────> │ Rank: N-1│
└──────────┘ └──────────┘
(Contiguous: 0, 1, 2, ..., N-1)
Replica Crash:
┌──────────┐ release ┌──────────┐ assign ┌──────────┐
│ Rank: K │ ───────────────> │ Released │ ────────────> │ Rank: K │
│ (Dead) │ │ │ │ (New) │
└──────────┘ └──────────┘ └──────────┘
(K can be any rank from 0 to N-1)
:::{note}
When a replica crashes, Ray Serve automatically starts a replacement replica and assigns it the **same rank** as the crashed replica. This ensures rank contiguity is maintained without reassigning other replicas.
:::
Downscaling:
┌──────────┐ release ┌──────────┐
│ Rank: K │ ───────────────> │ Released │
│ (Stopped)│ │ │
└──────────┘ └──────────┘
│
└──> Remaining replicas may be reassigned to maintain
contiguity: [0, 1, 2, ..., M-1] where M < N
(K can be any rank from 0 to N-1)
Controller Recovery:
┌──────────┐ recover ┌──────────┐
│ Running │ ───────────────> │ Rank: N │
│ Replicas │ │(Restored)│
└──────────┘ └──────────┘
(Controller queries replicas to reconstruct rank state)
```
### Detailed lifecycle events
1. **Rank assignment on startup**: Ranks are assigned when replicas start, such as during initial deployment, cold starts, or upscaling. The controller assigns ranks and propagates them to replicas during initialization. New replicas receive the lowest available rank.
2. **Rank release on shutdown**: Ranks are released only after a replica fully stops, which occurs during graceful shutdown or downscaling. Ray Serve preserves existing rank assignments as much as possible to minimize disruption.
3. **Handling replica crashes**: If a replica crashes unexpectedly, the system releases its rank and assigns the **same rank** to the replacement replica. This means if replica with rank 3 crashes, the new replacement replica will also receive rank 3. The replacement receives its rank during initialization, and other replicas keep their existing ranks unchanged.
4. **Controller crash and recovery**: When the controller recovers from a crash, it reconstructs the rank state by querying all running replicas for their assigned ranks. Ranks aren't checkpointed; the system re-learns them directly from replicas during recovery.
5. **Maintaining rank contiguity**: After downscaling, the system may reassign ranks to remaining replicas to maintain contiguity (0 to N-1). Ray Serve minimizes reassignments by only changing ranks when necessary.
doc/source/serve/advanced-guides/replica-scheduling.md
+301
View File
@@ -0,0 +1,301 @@
(serve-replica-scheduling)=
# Replica scheduling
This guide explains how Ray Serve schedules deployment replicas across your cluster and the APIs and environment variables you can use to control placement behavior.
## Quick reference: Choosing the right approach
| Goal | Solution | Example |
|------|----------|---------|
| Multi-GPU inference with tensor parallelism | `placement_group_bundles` + `STRICT_PACK` | vLLM with `tensor_parallel_size=4` |
| Target specific GPU types or zones | `label_selector` in `ray_actor_options` | Schedule on A100 nodes only |
| Limit replicas per node for high availability | `max_replicas_per_node` | Max 2 replicas of each deployment per node |
| Reduce cloud costs by packing nodes | `RAY_SERVE_USE_PACK_SCHEDULING_STRATEGY=1` | Many small models sharing nodes |
| Reserve resources for worker actors | `placement_group_bundles` | Replica spawns Ray Data workers |
| Shard large embeddings across nodes | `placement_group_bundles` + `STRICT_SPREAD` | Recommendation model with distributed embedding table |
| Simple deployment, no special needs | Default (just `ray_actor_options`) | Single-GPU model |
## How replica scheduling works
When you deploy an application, Ray Serve's deployment scheduler determines where to place each replica actor across the available nodes in your Ray cluster. The scheduler runs on the Serve Controller and makes batch scheduling decisions during each update cycle. For information on configuring CPU, GPU, and other resource requirements for your replicas, see [Resource allocation](serve-resource-allocation).
```text
┌──────────────────────────────────┐
│ serve.run(app) │
└────────────────┬─────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────────┐
│ Serve Controller │
│ ┌───────────────────────────────────────────────────────────────────────────┐ │
│ │ Deployment Scheduler │ │
│ │ │ │
│ │ 1. Check placement_group_bundles ──▶ PlacementGroupSchedulingStrategy │ │
│ │ 2. Check target node affinity ──▶ NodeAffinitySchedulingStrategy │ │
│ │ 3. Use default strategy ──▶ SPREAD (default) or PACK │ │
│ └───────────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────┘
│
┌─────────────────────────────────┴─────────────────────────────────┐
│ │
▼ ▼
┌─────────────────────────────────────┐ ┌─────────────────────────────────────┐
│ SPREAD Strategy (default) │ │ PACK Strategy │
│ │ │ │
│ Distributes replicas across nodes │ │ Packs replicas onto fewer nodes │
│ for fault tolerance │ │ to minimize resource waste │
│ │ │ │
│ ┌─────────┐ ┌─────────┐ ┌───────┐ │ │ ┌─────────┐ ┌─────────┐ ┌───────┐ │
│ │ Node 1 │ │ Node 2 │ │Node 3 │ │ │ │ Node 1 │ │ Node 2 │ │Node 3 │ │
│ │ ┌─────┐ │ │ ┌─────┐ │ │┌─────┐│ │ │ │ ┌─────┐ │ │ │ │ │ │
│ │ │ R1 │ │ │ │ R2 │ │ ││ R3 ││ │ │ │ │ R1 │ │ │ idle │ │ idle │ │
│ │ └─────┘ │ │ └─────┘ │ │└─────┘│ │ │ │ │ R2 │ │ │ │ │ │ │
│ │ │ │ │ │ │ │ │ │ │ R3 │ │ │ │ │ │ │
│ └─────────┘ └─────────┘ └───────┘ │ │ └─────────┘ └─────────┘ └───────┘ │
│ │ │ ▲ ▲ │
│ ✓ High availability │ │ └───────────┘ │
│ ✓ Load balanced │ │ Can be released │
│ ✓ Reduced contention │ │ ✓ Fewer nodes = lower cloud costs │
└─────────────────────────────────────┘ └────────────────────────────────────┘
```
By default, Ray Serve uses a **spread scheduling strategy** that distributes replicas across nodes with best effort. This approach:
- Maximizes fault tolerance by avoiding concentration of replicas on a single node
- Balances load across the cluster
- Helps prevent resource contention between replicas
### Scheduling priority
When scheduling a replica, the scheduler evaluates strategies in the following priority order:
1. **Placement groups**: If you specify `placement_group_bundles`, the scheduler uses a `PlacementGroupSchedulingStrategy` to co-locate the replica with its required resources. If you specify `placement_group_bundle_label_selector`, the scheduler will only select nodes with the required labels for each bundle.
2. **Pack scheduling with node affinity**: If pack scheduling is enabled, the scheduler identifies the best available node by preferring non-idle nodes (nodes already running replicas) and using a best-fit algorithm to minimize resource fragmentation. It then uses a `NodeAffinitySchedulingStrategy` with soft constraints to schedule the replica on that node.
- **With labels**: If a `label_selector` is provided, the scheduler strictly filters candidate nodes to match the labels before selecting the best fit.
- **With fallback**: If a `fallback_strategy` is provided, the scheduler first attempts to pack on nodes matching the labels. If no matching nodes are available, it retries using the next fallback option.
3. **Default strategy**: Falls back to `SPREAD` when pack scheduling isn't enabled.
### Downscaling behavior
When Ray Serve scales down a deployment, it intelligently selects which replicas to stop:
1. **Non-running replicas first**: Pending, launching, or recovering replicas are stopped before running replicas.
2. **Minimize node count**: Running replicas are stopped from nodes with the fewest total replicas across all deployments, helping to free up nodes faster. Among replicas on the same node, newer replicas are stopped before older ones.
3. **Head node protection**: Replicas on the head node have the lowest priority for removal since the head node can't be released. Among replicas on the head node, newer replicas are stopped before older ones.
:::{note}
Running replicas on the head node isn't recommended for production deployments. The head node runs critical cluster processes such as the GCS and Serve controller, and replica workloads can compete for resources.
:::
## APIs for controlling replica placement
Ray Serve provides several options to control where replicas are scheduled. These parameters are configured through the [`@serve.deployment`](serve-configure-deployment) decorator. For the full API reference, see the [deployment decorator documentation](../api/doc/ray.serve.deployment_decorator.rst).
### Limit replicas per node with `max_replicas_per_node`
Use [`max_replicas_per_node`](../api/doc/ray.serve.deployment_decorator.rst) to cap the number of replicas of a deployment that can run on a single node. This is useful when:
- You want to ensure high availability by spreading replicas across nodes
- You want to avoid resource contention between replicas of the same deployment
```{literalinclude} ../doc_code/replica_scheduling.py
:start-after: __max_replicas_per_node_start__
:end-before: __max_replicas_per_node_end__
:language: python
```
In this example, if you have 6 replicas and `max_replicas_per_node=2`, Ray Serve requires at least 3 nodes to schedule all replicas.
:::{note}
Valid values for `max_replicas_per_node` are `None` (default, no limit) or an integer. You can't set `max_replicas_per_node` together with `placement_group_bundles`.
:::
You can also specify this in a config file:
```yaml
applications:
- name: my_app
import_path: my_module:app
deployments:
- name: MyDeployment
num_replicas: 6
max_replicas_per_node: 2
```
### Reserve resources with placement groups
For more details on placement group strategies, see the [Ray Core placement groups documentation](ray-placement-group-doc-ref).
A **placement group** is a Ray primitive that reserves a group of resources (called **bundles**) across one or more nodes in your cluster. When you configure [`placement_group_bundles`](../api/doc/ray.serve.deployment_decorator.rst) for a Ray Serve deployment, Ray creates a dedicated placement group for *each replica*, ensuring those resources are reserved and available for that replica's use.
A **bundle** is a dictionary specifying resource requirements, such as `{"CPU": 2, "GPU": 1}`. When you define multiple bundles, you're telling Ray to reserve multiple sets of resources that can be placed according to your chosen strategy.
#### Controlling placement group location with label selectors
You can further refine where placement groups are scheduled using a `placement_group_bundle_label_selector`. This field defines a list of label selectors to apply per-bundle when scheduling the Serve deployment. This allows you to restrict the nodes where your bundles (and therefore your replicas) are placed based on Ray node labels. For more information on Ray label selectors, see [Use labels to control scheduling](https://docs.ray.io/en/latest/ray-core/scheduling/labels.html).
```{literalinclude} ../doc_code/replica_scheduling.py
:start-after: __placement_group_labels_start__
:end-before: __placement_group_labels_end__
:language: python
```
The `placement_group_bundle_label_selector` accepts a list of dictionaries.
- Single selector: If you provide a list containing a single dictionary, that selector is applied to all bundles in `placement_group_bundles`.
- Per-bundle selector: If you provide a list of multiple dictionaries, the length must match `placement_group_bundles`. The *i*-th selector applies to the *i*-th bundle.
#### What placement groups and bundles mean
The following diagram illustrates how a deployment with `placement_group_bundles=[{"GPU": 1}, {"GPU": 1}, {"CPU": 4}]` and [`placement_group_strategy`](../api/doc/ray.serve.deployment_decorator.rst)` set to "STRICT_PACK"` is scheduled:
```text
┌─────────────────────────────────────────────────────────────────────────────┐
│ Node (8 CPUs, 4 GPUs) │
│ ┌───────────────────────────────────────────────────────────────────────┐ │
│ │ Placement Group (per replica) │ │
│ │ │ │
│ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────────┐ │ │
│ │ │ Bundle 0 │ │ Bundle 1 │ │ Bundle 2 │ │ │
│ │ │ {"GPU": 1} │ │ {"GPU": 1} │ │ {"CPU": 4} │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ ┌─────────────┐ │ │ ┌─────────────┐ │ │ ┌─────────────────┐ │ │ │
│ │ │ │ Replica │ │ │ │ Worker │ │ │ │ Worker Tasks │ │ │ │
│ │ │ │ Actor │ │ │ │ Actor │ │ │ │ (preprocessing)│ │ │ │
│ │ │ │ (main GPU) │ │ │ │ (2nd GPU) │ │ │ │ │ │ │ │
│ │ │ └─────────────┘ │ │ └─────────────┘ │ │ └─────────────────┘ │ │ │
│ │ └─────────────────┘ └─────────────────┘ └─────────────────────┘ │ │
│ │ ▲ │ │
│ │ │ │ │
│ │ Replica runs in │ │
│ │ first bundle │ │
│ └───────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
With STRICT_PACK: All bundles guaranteed on same node
```
Consider a deployment with `placement_group_bundles=[{"GPU": 1}, {"GPU": 1}, {"CPU": 4}]`:
- Ray reserves 3 bundles of resources for each replica
- The replica actor runs in the **first bundle** (so `ray_actor_options` must fit within it)
- The remaining bundles are available for worker actors/tasks spawned by the replica
- All child actors and tasks are automatically scheduled within the placement group
This is different from simply requesting resources in `ray_actor_options`. With `ray_actor_options={"num_gpus": 2}`, your replica actor gets 2 GPUs but you have no control over where additional worker processes run. With placement groups, you explicitly reserve resources for both the replica and its workers.
#### When to use placement groups
| Scenario | Why placement groups help |
|----------|---------------------------|
| **Model parallelism** | Tensor parallelism or pipeline parallelism requires multiple GPUs that must communicate efficiently. Use `STRICT_PACK` to guarantee all GPUs are on the same node. For example, vLLM with `tensor_parallel_size=4` and the Ray distributed executor backend spawns 4 Ray worker actors (one per GPU shard), all of which must be on the same node for efficient inter-GPU communication via NVLink/NVSwitch. |
| **Replica spawns workers** | Your deployment creates Ray actors or tasks for parallel processing. Placement groups reserve resources for these workers. For example, a video processing service that spawns Ray tasks to decode frames in parallel, or a batch inference service using Ray Data to preprocess inputs before model inference. |
| **Cross-node distribution** | You need bundles spread across different nodes. Use `SPREAD` or `STRICT_SPREAD`. For example, serving a model with a massive embedding table (such as a recommendation model with billions of item embeddings) that must be sharded across multiple nodes because it exceeds single-node memory. Each bundle holds one shard, and `STRICT_SPREAD` ensures each shard is on a separate node. |
Don't use placement groups when:
- Your replica is self-contained and doesn't spawn additional actors/tasks
- You only need simple resource requirements (use `ray_actor_options` instead)
- You want to use `max_replicas_per_node`. The combination of these two options is not supported today.
:::{note}
**How `max_replicas_per_node` works:** Ray Serve creates a synthetic custom resource for each deployment. Every node implicitly has 1.0 of this resource, and each replica requests `1.0 / max_replicas_per_node` of it. For example, with `max_replicas_per_node=3`, each replica requests ~0.33 of the resource, so only 3 replicas can fit on a node before the resource is exhausted. This mechanism relies on Ray's standard resource scheduling, which conflicts with placement group scheduling.
:::
#### Configuring placement groups
The following example reserves 2 GPUs for each replica using a strict pack strategy:
```{literalinclude} ../doc_code/replica_scheduling.py
:start-after: __placement_group_start__
:end-before: __placement_group_end__
:language: python
```
The replica actor is scheduled in the first bundle, so the resources specified in `ray_actor_options` must be a subset of the first bundle's resources. All actors and tasks created by the replica are scheduled in the placement group by default (`placement_group_capture_child_tasks=True`).
### Target nodes with labels
You can use label selectors in [`ray_actor_options`](../api/doc/ray.serve.deployment_decorator.rst) to target replicas to specific nodes. This is the recommended approach for controlling which nodes run your replicas.
Then configure your deployment to require the specific labels:
```{literalinclude} ../doc_code/replica_scheduling.py
:start-after: __label_selectors_start__
:end-before: __label_selectors_end__
:language: python
```
First, start your Ray nodes with labels that identify their capabilities:
```{literalinclude} ../doc_code/replica_scheduling.py
:start-after: __label_selector_main_start__
:end-before: __label_selector_main_end__
:language: python
```
#### Soft constraints with `fallback_strategy`
By default, a `label_selector` acts as a hard constraint. If no node matches the selector, the replica remains pending indefinitely. You can relax this requirement by providing a `fallback_strategy` in `ray_actor_options`.
```{literalinclude} ../doc_code/replica_scheduling.py
:start-after: __fallback_strategy_start__
:end-before: __fallback_strategy_end__
:language: python
```
This allows you to express preferences. For example, when using PACK scheduling, the scheduler will attempt to find a node that matches the `label_selector` first. If no available node is found, the scheduler will retry scheduling using the rules defined in your fallback strategy.
Label selectors and fallback strategies offer several advantages for Ray Serve deployments:
- **Expressive placement constraints**: Ray automatically detects and populates labels for node attributes like `ray.io/accelerator-type`, or you can add custom labels at startup using the `--labels` flag. You can target these labels utilizing familiar Kubernetes-like syntax with complex operators (equality, negation (`!`), inclusion (`in`), and exclusion (`!in`)) to precisely filter which nodes run your replicas.
- **Autoscaler-aware**: The Ray autoscaler understands label selectors and can provision nodes with the required labels automatically.
- **Soft constraints**: Unlike custom resources which are strict requirements, label selectors can also be specified in the `fallback_strategy` field. This allows you to define preferred scheduling options while permitting the scheduler to utilize alternative nodes if the primary targets are unavailable, preventing deployments from stalling.
## Environment variables
These environment variables modify Ray Serve's scheduling behavior. Set them before starting Ray.
### `RAY_SERVE_USE_PACK_SCHEDULING_STRATEGY`
**Default**: `0` (disabled)
When enabled, switches from spread scheduling to **pack scheduling**. Pack scheduling:
- Packs replicas onto fewer nodes to minimize resource fragmentation
- Sorts pending replicas by resource requirements (largest first)
- Prefers scheduling on nodes that already have replicas (non-idle nodes)
- Uses best-fit bin packing to find the optimal node for each replica
```bash
export RAY_SERVE_USE_PACK_SCHEDULING_STRATEGY=1
ray start --head
```
**When to use pack scheduling:** When you run many small deployments (such as 10 models each needing 0.5 CPUs), spread scheduling scatters them across nodes, wasting capacity. Pack scheduling fills nodes efficiently before using new ones. Cloud providers bill per node-hour. Packing replicas onto fewer nodes allows idle nodes to be released by the autoscaler, directly reducing your bill.
**When to avoid pack scheduling:** High availability is critical and you want replicas spread across nodes
:::{note}
Pack scheduling automatically falls back to spread scheduling when any deployment uses placement groups with `PACK`, `SPREAD`, or `STRICT_SPREAD` strategies. This happens because pack scheduling needs to predict where resources will be consumed to bin-pack effectively. With `STRICT_PACK`, all bundles are guaranteed to land on one node, making resource consumption predictable. With other strategies, bundles may spread across multiple nodes unpredictably, so the scheduler can't accurately track available resources per node.
:::
### `RAY_SERVE_HIGH_PRIORITY_CUSTOM_RESOURCES`
**Default**: empty
A comma-separated list of custom resource names that should be prioritized when sorting replicas for pack scheduling. Resources listed earlier have higher priority.
```bash
export RAY_SERVE_HIGH_PRIORITY_CUSTOM_RESOURCES="TPU,custom_accelerator"
ray start --head
```
When pack scheduling is enabled, the scheduler first filters the cluster to find nodes that match the `label_selector` (if specified). It then sorts the pending replicas by resource requirements to pack them efficiently. The priority order for sorting replicas is:
1. Custom resources in `RAY_SERVE_HIGH_PRIORITY_CUSTOM_RESOURCES` (in order)
2. GPU
3. CPU
4. Memory
5. Other custom resources
This ensures that replicas requiring high-priority resources are scheduled first, reducing the chance of resource fragmentation.
## See also
- [Resource allocation](serve-resource-allocation) for configuring CPU, GPU, and other resources
- [Autoscaling](serve-autoscaling) for automatically adjusting replica count
- [Ray placement groups](ray-placement-group-doc-ref) for advanced resource co-location
doc/source/serve/api/index.md
+528
View File
@@ -0,0 +1,528 @@
(serve-api)=
# Ray Serve API
## Python API
(core-apis)=
```{eval-rst}
.. currentmodule:: ray
```
### Writing Applications
<!---
NOTE: `serve.deployment` and `serve.Deployment` have an autosummary-generated filename collision due to case insensitivity. This is fixed by added custom filename mappings in `source/conf.py` (look for "autosummary_filename_map"). --->
```{eval-rst}
.. autosummary::
:nosignatures:
:toctree: doc/
:template: autosummary/class_without_init_args.rst
serve.Deployment
serve.Application
```
#### Deployment Decorators
```{eval-rst}
.. autosummary::
:nosignatures:
:toctree: doc/
serve.deployment
:noindex:
serve.ingress
serve.batch
serve.multiplexed
```
#### Deployment Handles
:::{note}
The deprecated `RayServeHandle` and `RayServeSyncHandle` APIs have been fully removed as of Ray 2.10. See the [model composition guide](serve-model-composition) for how to update code to use the {mod}`DeploymentHandle <ray.serve.handle.DeploymentHandle>` API instead.
:::
```{eval-rst}
.. autosummary::
:nosignatures:
:toctree: doc/
:template: autosummary/class_without_init_args.rst
serve.handle.DeploymentHandle
serve.handle.DeploymentResponse
serve.handle.DeploymentResponseGenerator
serve.handle.DeploymentBroadcastResponse
```
### Running Applications
```{eval-rst}
.. autosummary::
:nosignatures:
:toctree: doc/
serve.start
serve.run
serve.delete
serve.status
serve.shutdown
serve.shutdown_async
```
### Configurations
```{eval-rst}
.. autosummary::
:nosignatures:
:toctree: doc/
:template: autosummary/class_without_autosummary.rst
serve.config.ProxyLocation
serve.config.AutoscalingContext
serve.autoscaling_policy.replica_queue_length_autoscaling_policy
serve.config.AggregationFunction
serve.config.GangPlacementStrategy
serve.config.GangRuntimeFailurePolicy
.. autosummary::
:nosignatures:
:toctree: doc/
:template: autosummary/autopydantic.rst
serve.config.ControllerOptions
serve.config.gRPCOptions
serve.config.HTTPOptions
serve.config.AutoscalingConfig
serve.config.AutoscalingPolicy
serve.config.RequestRouterConfig
serve.config.GangSchedulingConfig
serve.config.DeploymentActorConfig
```
### Schemas
```{eval-rst}
.. autosummary::
:nosignatures:
:toctree: doc/
:template: autosummary/class_without_init_args.rst
serve.schema.ServeActorDetails
serve.schema.ProxyDetails
serve.schema.ApplicationStatusOverview
serve.schema.ServeStatus
serve.schema.DeploymentStatusOverview
serve.schema.EncodingType
serve.schema.AutoscalingMetricsHealth
serve.schema.AutoscalingStatus
serve.schema.ScalingDecision
serve.schema.DeploymentAutoscalingDetail
serve.schema.ReplicaRank
```
```{eval-rst}
.. autosummary::
:nosignatures:
:toctree: doc/
:template: autosummary/class_without_autosummary.rst
serve.schema.TaskProcessorAdapter
```
### Request Router
```{eval-rst}
.. autosummary::
:nosignatures:
:toctree: doc/
serve.request_router.ReplicaID
serve.request_router.PendingRequest
serve.request_router.RunningReplica
serve.request_router.FIFOMixin
serve.request_router.LocalityMixin
serve.request_router.MultiplexMixin
serve.request_router.RequestRouter
```
#### Advanced APIs
```{eval-rst}
.. autosummary::
:nosignatures:
:toctree: doc/
serve.get_replica_context
serve.get_trace_context
serve.get_deployment_actor
serve.context.ReplicaContext
serve.context.GangContext
serve.get_multiplexed_model_id
serve.get_app_handle
serve.get_deployment_handle
serve.grpc_util.RayServegRPCContext
serve.grpc_util.gRPCInputStream
serve.exceptions.BackPressureError
serve.exceptions.RayServeException
serve.exceptions.RequestCancelledError
serve.exceptions.gRPCStatusError
serve.exceptions.DeploymentUnavailableError
serve.exceptions.ReplicaUnavailableError
```
(serve-cli)=
## Command Line Interface (CLI)
```{eval-rst}
.. click:: ray.serve.scripts:cli
:prog: serve
:nested: full
```
(serve-rest-api)=
## Serve REST API
The Serve REST API is exposed at the same port as the Ray Dashboard. The Dashboard port is `8265` by default. This port can be changed using the `--dashboard-port` argument when running `ray start`. All example requests in this section use the default port.
### `PUT "/api/serve/applications/"`
Declaratively deploys a list of Serve applications. If Serve is already running on the Ray cluster, removes all applications not listed in the new config. If Serve is not running on the Ray cluster, starts Serve. See [multi-app config schema](serve-rest-api-config-schema) for the request's JSON schema.
**Example Request**:
```http
PUT /api/serve/applications/ HTTP/1.1
Host: http://localhost:8265/
Accept: application/json
Content-Type: application/json
{
"applications": [
{
"name": "text_app",
"route_prefix": "/",
"import_path": "text_ml:app",
"runtime_env": {
"working_dir": "https://github.com/ray-project/serve_config_examples/archive/HEAD.zip"
},
"deployments": [
{"name": "Translator", "user_config": {"language": "french"}},
{"name": "Summarizer"},
]
},
]
}
```
**Example Response**
```http
HTTP/1.1 200 OK
Content-Type: application/json
```
### `GET "/api/serve/applications/"`
Gets cluster-level info and comprehensive details on all Serve applications deployed on the Ray cluster. See [metadata schema](serve-rest-api-response-schema) for the response's JSON schema.
```http
GET /api/serve/applications/ HTTP/1.1
Host: http://localhost:8265/
Accept: application/json
```
**Example Response (abridged JSON)**:
```http
HTTP/1.1 200 OK
Content-Type: application/json
{
"controller_info": {
"node_id": "cef533a072b0f03bf92a6b98cb4eb9153b7b7c7b7f15954feb2f38ec",
"node_ip": "10.0.29.214",
"actor_id": "1d214b7bdf07446ea0ed9d7001000000",
"actor_name": "SERVE_CONTROLLER_ACTOR",
"worker_id": "adf416ae436a806ca302d4712e0df163245aba7ab835b0e0f4d85819",
"log_file_path": "/serve/controller_29778.log"
},
"proxy_location": "EveryNode",
"http_options": {
"host": "0.0.0.0",
"port": 8000,
"root_path": "",
"request_timeout_s": null,
"keep_alive_timeout_s": 5
},
"grpc_options": {
"port": 9000,
"grpc_servicer_functions": [],
"request_timeout_s": null
},
"proxies": {
"cef533a072b0f03bf92a6b98cb4eb9153b7b7c7b7f15954feb2f38ec": {
"node_id": "cef533a072b0f03bf92a6b98cb4eb9153b7b7c7b7f15954feb2f38ec",
"node_ip": "10.0.29.214",
"actor_id": "b7a16b8342e1ced620ae638901000000",
"actor_name": "SERVE_CONTROLLER_ACTOR:SERVE_PROXY_ACTOR-cef533a072b0f03bf92a6b98cb4eb9153b7b7c7b7f15954feb2f38ec",
"worker_id": "206b7fe05b65fac7fdceec3c9af1da5bee82b0e1dbb97f8bf732d530",
"log_file_path": "/serve/http_proxy_10.0.29.214.log",
"status": "HEALTHY"
}
},
"deploy_mode": "MULTI_APP",
"applications": {
"app1": {
"name": "app1",
"route_prefix": "/",
"docs_path": null,
"status": "RUNNING",
"message": "",
"last_deployed_time_s": 1694042836.1912267,
"deployed_app_config": {
"name": "app1",
"route_prefix": "/",
"import_path": "src.text-test:app",
"deployments": [
{
"name": "Translator",
"num_replicas": 1,
"user_config": {
"language": "german"
}
}
]
},
"deployments": {
"Translator": {
"name": "Translator",
"status": "HEALTHY",
"message": "",
"deployment_config": {
"name": "Translator",
"num_replicas": 1,
"max_ongoing_requests": 100,
"user_config": {
"language": "german"
},
"graceful_shutdown_wait_loop_s": 2.0,
"graceful_shutdown_timeout_s": 20.0,
"health_check_period_s": 10.0,
"health_check_timeout_s": 30.0,
"ray_actor_options": {
"runtime_env": {
"env_vars": {}
},
"num_cpus": 1.0
},
"is_driver_deployment": false
},
"replicas": [
{
"node_id": "cef533a072b0f03bf92a6b98cb4eb9153b7b7c7b7f15954feb2f38ec",
"node_ip": "10.0.29.214",
"actor_id": "4bb8479ad0c9e9087fee651901000000",
"actor_name": "SERVE_REPLICA::app1#Translator#oMhRlb",
"worker_id": "1624afa1822b62108ead72443ce72ef3c0f280f3075b89dd5c5d5e5f",
"log_file_path": "/serve/deployment_Translator_app1#Translator#oMhRlb.log",
"replica_id": "app1#Translator#oMhRlb",
"state": "RUNNING",
"pid": 29892,
"start_time_s": 1694042840.577496
}
]
},
"Summarizer": {
"name": "Summarizer",
"status": "HEALTHY",
"message": "",
"deployment_config": {
"name": "Summarizer",
"num_replicas": 1,
"max_ongoing_requests": 100,
"user_config": null,
"graceful_shutdown_wait_loop_s": 2.0,
"graceful_shutdown_timeout_s": 20.0,
"health_check_period_s": 10.0,
"health_check_timeout_s": 30.0,
"ray_actor_options": {
"runtime_env": {},
"num_cpus": 1.0
},
"is_driver_deployment": false
},
"replicas": [
{
"node_id": "cef533a072b0f03bf92a6b98cb4eb9153b7b7c7b7f15954feb2f38ec",
"node_ip": "10.0.29.214",
"actor_id": "7118ae807cffc1c99ad5ad2701000000",
"actor_name": "SERVE_REPLICA::app1#Summarizer#cwiPXg",
"worker_id": "12de2ac83c18ce4a61a443a1f3308294caf5a586f9aa320b29deed92",
"log_file_path": "/serve/deployment_Summarizer_app1#Summarizer#cwiPXg.log",
"replica_id": "app1#Summarizer#cwiPXg",
"state": "RUNNING",
"pid": 29893,
"start_time_s": 1694042840.5789504
}
]
}
}
}
}
}
```
### `DELETE "/api/serve/applications/"`
Shuts down Serve and all applications running on the Ray cluster. Has no effect if Serve is not running on the Ray cluster.
**Example Request**:
```http
DELETE /api/serve/applications/ HTTP/1.1
Host: http://localhost:8265/
Accept: application/json
```
**Example Response**
```http
HTTP/1.1 200 OK
Content-Type: application/json
```
(serve-rest-api-config-schema)=
## Config Schemas
```{eval-rst}
.. currentmodule:: ray.serve
```
```{eval-rst}
.. autosummary::
:nosignatures:
:toctree: doc/
:template: autosummary/autopydantic.rst
schema.ServeDeploySchema
schema.gRPCOptionsSchema
schema.HTTPOptionsSchema
schema.ServeApplicationSchema
schema.DeploymentSchema
schema.RayActorOptionsSchema
schema.CeleryAdapterConfig
schema.TaskProcessorConfig
schema.TaskResult
schema.ScaleDeploymentRequest
```
(serve-rest-api-response-schema)=
## Response Schemas
```{eval-rst}
.. autosummary::
:nosignatures:
:toctree: doc/
:template: autosummary/autopydantic.rst
schema.ServeInstanceDetails
schema.ApplicationDetails
schema.DeploymentDetails
schema.ReplicaDetails
schema.TargetGroup
schema.Target
schema.DeploymentNode
schema.DeploymentTopology
schema.ControllerHealthMetrics
schema.DurationStats
.. autosummary::
:nosignatures:
:toctree: doc/
:template: autosummary/class_without_autosummary.rst
schema.APIType
schema.ApplicationStatus
schema.ProxyStatus
```
## Observability
```{eval-rst}
.. autosummary::
:nosignatures:
:toctree: doc/
:template: autosummary/class_without_autosummary.rst
metrics.Counter
metrics.Histogram
metrics.Gauge
.. autosummary::
:nosignatures:
:toctree: doc/
:template: autosummary/autopydantic.rst
schema.LoggingConfig
```
(serve-llm-api)=
## LLM API
```{eval-rst}
.. currentmodule:: ray
```
### Builders
```{eval-rst}
.. autosummary::
:nosignatures:
:toctree: doc/
serve.llm.build_llm_deployment
serve.llm.build_openai_app
```
### Configs
```{eval-rst}
.. autosummary::
:nosignatures:
:toctree: doc/
:template: autosummary/autopydantic.rst
serve.llm.LLMConfig
serve.llm.LLMServingArgs
serve.llm.ModelLoadingConfig
serve.llm.CloudMirrorConfig
serve.llm.LoraConfig
```
### Deployments
```{eval-rst}
.. autosummary::
:nosignatures:
:toctree: doc/
serve.llm.LLMServer
serve.llm.LLMRouter
```
doc/source/serve/architecture-2.0.svg
+1
View File
File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 114 KiB

doc/source/serve/architecture.md
+91
View File
@@ -0,0 +1,91 @@
(serve-architecture)=
# Architecture
In this section, we explore Serve's key architectural concepts and components. It will offer insight and overview into:
- the role of each component in Serve and how they work
- the different types of actors that make up a Serve application
% Figure source: https://docs.google.com/drawings/d/1jSuBN5dkSj2s9-0eGzlU_ldsRa3TsswQUZM-cMQ29a0/edit?usp=sharing
```{image} architecture-2.0.svg
:align: center
:width: 600px
```
(serve-architecture-high-level-view)=
## High-Level View
Serve runs on Ray and utilizes [Ray actors](actor-guide).
There are three kinds of actors that are created to make up a Serve instance:
- **Controller**: A global actor unique to each Serve instance that manages the control plane. The Controller is responsible for creating, updating, and destroying other actors. Serve API calls like creating or getting a deployment make remote calls to the Controller.
- **HTTP Proxy**: By default there is one HTTP proxy actor on the head node. This actor runs a [Uvicorn](https://www.uvicorn.org/) HTTP server that accepts incoming requests, forwards them to replicas, and responds once they are completed. For scalability and high availability, you can also run a proxy on each node in the cluster via the `proxy_location` field inside [`serve.start()`](core-apis) or [the config file](serve-in-production-config-file).
- **gRPC Proxy**: If Serve is started with valid `port` and `grpc_servicer_functions`, then the gRPC proxy is started alongside the HTTP proxy. This Actor runs a [grpcio](https://grpc.github.io/grpc/python/) server. The gRPC server accepts incoming requests, forwards them to replicas, and responds once they are completed.
- **Replicas**: Actors that actually execute the code in response to a request. For example, they may contain an instantiation of an ML model. Each replica processes individual requests from the proxy. The replica may batch the requests using `@serve.batch`. See the [batching](serve-performance-batching-requests) docs.
## Lifetime of a request
When an HTTP or gRPC request is sent to the corresponding HTTP or gRPC proxy, the following happens:
1. The request is received and parsed.
2. Ray Serve looks up the correct deployment associated with the HTTP URL path or application name metadata. Serve places the request in a queue.
3. For each request in a deployment's queue, an available replica is looked up and the request is sent to it. If no replicas are available (that is, more than `max_ongoing_requests` requests are outstanding at each replica), the request is left in the queue until a replica becomes available.
Each replica maintains a queue of requests and executes requests one at a time, possibly using `asyncio` to process them concurrently. If the handler (the deployment function or the `__call__` method of the deployment class) is declared with `async def`, the replica will not wait for the handler to run. Otherwise, the replica blocks until the handler returns.
When making a request via a [DeploymentHandle](serve-key-concepts-deployment-handle) instead of HTTP or gRPC for [model composition](serve-model-composition), the request is placed on a queue in the `DeploymentHandle`, and we skip to step 3 above.
(serve-ft-detail)=
## Fault tolerance
Application errors like exceptions in your model evaluation code are caught and wrapped. A 500 status code will be returned with the traceback information. The replica will be able to continue to handle requests.
Machine errors and faults are handled by Ray Serve as follows:
- When replica Actors fail, the Controller Actor replaces them with new ones.
- When the proxy Actor fails, the Controller Actor restarts it.
- When the Controller Actor fails, Ray restarts it.
- When using the [KubeRay RayService](kuberay-rayservice-quickstart), KubeRay recovers crashed nodes or a crashed cluster. You can avoid cluster crashes by using the [GCS FT feature](kuberay-gcs-ft).
- If you aren't using KubeRay, when the Ray cluster fails, Ray Serve cannot recover.
When a machine hosting any of the actors crashes, those actors are automatically restarted on another available machine. All data in the Controller (routing policies, deployment configurations, etc) is checkpointed to the Ray Global Control Store (GCS) on the head node. Transient data in the router and the replica (like network connections and internal request queues) will be lost for this kind of failure. See [the end-to-end fault tolerance guide](serve-e2e-ft) for more details on how actor crashes are detected.
(serve-autoscaling-architecture)=
## Ray Serve Autoscaling
Ray Serve's autoscaling feature automatically increases or decreases a deployment's number of replicas based on its load.
![pic](https://raw.githubusercontent.com/ray-project/images/master/docs/serve/autoscaling.svg)
- The Serve Autoscaler runs in the Serve Controller actor.
- Each `DeploymentHandle` and each replica periodically pushes its metrics to the autoscaler.
- For each deployment, the autoscaler periodically checks `DeploymentHandle` queues and in-flight queries on replicas to decide whether or not to scale the number of replicas.
- Each `DeploymentHandle` continuously polls the controller to check for new deployment replicas. Whenever new replicas are discovered, it sends any buffered or new queries to the replica until `max_ongoing_requests` is reached. Queries are sent to replicas using a [power of two choices](https://ieeexplore.ieee.org/document/963420) scheduling strategy, subject to the constraint that no replica is handling more than `max_ongoing_requests` requests at a time.
:::{note}
When the controller dies, requests can still be sent via HTTP, gRPC and `DeploymentHandle`, but autoscaling is paused. When the controller recovers, the autoscaling resumes, but all previous metrics collected are lost.
:::
## Ray Serve API Server
Ray Serve provides a [CLI](serve-cli) for managing your Ray Serve instance, as well as a [REST API](serve-rest-api). Each node in your Ray cluster provides a Serve REST API server that can connect to Serve and respond to Serve REST requests.
## FAQ
### How does Serve ensure horizontal scalability and availability?
You can configure Serve to start one proxy Actor per node with the `proxy_location` field inside [`serve.start()`](core-apis) or [the config file](serve-in-production-config-file). Each proxy binds to the same port. You should be able to reach Serve and send requests to any models with any of the servers. You can use your own load balancer on top of Ray Serve.
This architecture ensures horizontal scalability for Serve. You can scale your HTTP and gRPC ingress by adding more nodes. You can also scale your model inference by increasing the number of replicas via the `num_replicas` option of your deployment.
### How do DeploymentHandles work?
{mod}`DeploymentHandles <ray.serve.handle.DeploymentHandle>` wrap a handle to a "router" on the same node which routes requests to replicas for a deployment. When a request is sent from one replica to another via the handle, the requests go through the same data path as incoming HTTP or gRPC requests. This enables the same deployment selection and batching procedures to happen. DeploymentHandles are often used to implement [model composition](serve-model-composition).
### What happens to large requests?
Serve utilizes Ray’s [shared memory object store](plasma-store) and in process memory store. Small request objects are directly sent between actors via network call. Larger request objects (100KiB+) are written to the object store and the replica can read them via zero-copy read.
doc/source/serve/asynchronous-inference.md
+298
View File
@@ -0,0 +1,298 @@
(serve-asynchronous-inference)=
:::{warning}
This API is in alpha and may change before becoming stable.
:::
# Asynchronous Inference
This guide shows how to run long-running inference asynchronously in Ray Serve using background task processing. With asynchronous tasks, your HTTP APIs stay responsive while the system performs work in the background.
## Why asynchronous inference?
Ray Serve customers need a way to handle long-running API requests asynchronously. Some inference workloads (such as video processing or large document indexing) take longer than typical HTTP timeouts, so when a user submits one of these requests the system should enqueue the work in a background queue for later processing and immediately return a quick response. This decouples request lifetime from compute time while the task executes asynchronously, while still leveraging Serve's scalability.
## Use cases
Common use cases include video inference (such as transcoding, detection, and transcription over long videos) and document indexing pipelines that ingest, parse, and vectorize large files or batches. More broadly, any long-running AI/ML workload where immediate results aren't required benefits from running asynchronously.
## Key concepts
- **@task_consumer**: A Serve deployment that consumes and executes tasks from a queue. Requires a `TaskProcessorConfig` parameter to configure the task processor; by default it uses the Celery task processor, but you can provide your own implementation.
- **@task_handler**: A decorator applied to a method inside a `@task_consumer` class. Each handler declares the task it handles via `name=...`; if `name` is omitted, the method's function name is used as the task name. All tasks with that name in the consumer's configured queue (set via the `TaskProcessorConfig` above) are routed to this method for execution.
## Components and APIs
The following sections describe the core APIs for asynchronous inference, with minimal examples to get you started.
### `TaskProcessorConfig`
Configures the task processor, including queue name, adapter (default is Celery), adapter config, retry limits, and dead-letter queues. The following example shows how to configure the task processor:
```python
from ray.serve.schema import TaskProcessorConfig, CeleryAdapterConfig
processor_config = TaskProcessorConfig(
queue_name="my_queue",
# Optional: Override default adapter string (default is Celery)
# adapter="ray.serve.task_processor.CeleryTaskProcessorAdapter",
adapter_config=CeleryAdapterConfig(
broker_url="redis://localhost:6379/0", # Or "filesystem://" for local testing
backend_url="redis://localhost:6379/1", # Result backend (optional for fire-and-forget)
),
max_retries=5,
failed_task_queue_name="failed_tasks", # Application errors after retries
)
```
:::{note}
The filesystem broker is intended for local testing only and has limited functionality. For example, it doesn't support `cancel_tasks`. For production deployments, use a production-ready broker such as Redis or RabbitMQ. See the [Celery broker documentation](https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/) for the full list of supported brokers.
:::
### `@task_consumer`
Decorator that turns a Serve deployment into a task consumer using the provided `TaskProcessorConfig`. The following code creates a task consumer:
```python
from ray import serve
from ray.serve.task_consumer import task_consumer
@serve.deployment
@task_consumer(task_processor_config=processor_config)
class SimpleConsumer:
pass
```
### `@task_handler`
Decorator that registers a method on the consumer as a named task handler. The following example shows how to define a task handler:
```python
from ray.serve.task_consumer import task_handler, task_consumer
@serve.deployment
@task_consumer(task_processor_config=processor_config)
class SimpleConsumer:
@task_handler(name="process_request")
def process_request(self, data):
return f"processed: {data}"
```
:::{note}
Ray Serve currently supports only synchronous handlers. Declaring an `async def` handler raises `NotImplementedError`.
:::
### `instantiate_adapter_from_config`
Factory function that returns a task processor adapter instance for the given `TaskProcessorConfig`. You can use the returned object to enqueue tasks, fetch status, retrieve metrics, and more. The following example demonstrates creating an adapter and enqueuing tasks:
```python
from ray.serve.task_consumer import instantiate_adapter_from_config
adapter = instantiate_adapter_from_config(task_processor_config=processor_config)
# Enqueue synchronously (returns TaskResult)
result = adapter.enqueue_task_sync(task_name="process_request", args=["hello"])
# Later, fetch status synchronously
status = adapter.get_task_status_sync(result.id)
```
:::{note}
All Ray actor options specified in the `@serve.deployment` decorator (such as `num_gpus`, `num_cpus`, `resources`, etc.) are applied to the task consumer replicas. This allows you to allocate specific hardware resources for your task processing workloads.
:::
## End-to-end example: Document indexing
This example shows how to configure the processor, build a consumer with a handler, enqueue tasks from an ingress deployment, and check task status.
```python
import io
import logging
import requests
from fastapi import FastAPI
from pydantic import BaseModel, HttpUrl
from PyPDF2 import PdfReader
from ray import serve
from ray.serve.schema import CeleryAdapterConfig, TaskProcessorConfig
from ray.serve.task_consumer import (
instantiate_adapter_from_config,
task_consumer,
task_handler,
)
logger = logging.getLogger("ray.serve")
fastapi_app = FastAPI(title="Async PDF Processing API")
TASK_PROCESSOR_CONFIG = TaskProcessorConfig(
queue_name="pdf_processing_queue",
adapter_config=CeleryAdapterConfig(
broker_url="redis://127.0.0.1:6379/0",
backend_url="redis://127.0.0.1:6379/0",
),
max_retries=3,
failed_task_queue_name="failed_pdfs",
)
class ProcessPDFRequest(BaseModel):
pdf_url: HttpUrl
max_summary_paragraphs: int = 3
@serve.deployment(num_replicas=2, max_ongoing_requests=5)
@task_consumer(task_processor_config=TASK_PROCESSOR_CONFIG)
class PDFProcessor:
"""Background worker that processes PDF documents asynchronously."""
@task_handler(name="process_pdf")
def process_pdf(self, pdf_url: str, max_summary_paragraphs: int = 3):
"""Download PDF, extract text, and generate summary."""
try:
response = requests.get(pdf_url, timeout=30)
response.raise_for_status()
pdf_reader = PdfReader(io.BytesIO(response.content))
if not pdf_reader.pages:
raise ValueError("PDF contains no pages")
full_text = "\n".join(
page.extract_text() for page in pdf_reader.pages if page.extract_text()
)
if not full_text.strip():
raise ValueError("PDF contains no extractable text")
paragraphs = [p.strip() for p in full_text.split("\n\n") if p.strip()]
summary = "\n\n".join(paragraphs[:max_summary_paragraphs])
return {
"status": "success",
"pdf_url": pdf_url,
"page_count": len(pdf_reader.pages),
"word_count": len(full_text.split()),
"summary": summary,
}
except requests.exceptions.RequestException as e:
raise ValueError(f"Failed to download PDF: {str(e)}")
except Exception as e:
raise ValueError(f"Failed to process PDF: {str(e)}")
@serve.deployment()
@serve.ingress(fastapi_app)
class AsyncPDFAPI:
"""HTTP API for submitting and checking PDF processing tasks."""
def __init__(self, task_processor_config: TaskProcessorConfig, handler):
self.adapter = instantiate_adapter_from_config(task_processor_config)
@fastapi_app.post("/process")
def process_pdf(self, request: ProcessPDFRequest):
"""Submit a PDF processing task and return task_id immediately."""
task_result = self.adapter.enqueue_task_sync(
task_name="process_pdf",
kwargs={
"pdf_url": str(request.pdf_url),
"max_summary_paragraphs": request.max_summary_paragraphs,
},
)
return {
"task_id": task_result.id,
"status": task_result.status,
"message": "PDF processing task submitted successfully",
}
@fastapi_app.get("/status/{task_id}")
def get_status(self, task_id: str):
"""Get task status and results."""
status = self.adapter.get_task_status_sync(task_id)
return {
"task_id": task_id,
"status": status.status,
"result": status.result if status.status == "SUCCESS" else None,
"error": str(status.result) if status.status == "FAILURE" else None,
}
app = AsyncPDFAPI.bind(TASK_PROCESSOR_CONFIG, PDFProcessor.bind())
```
In this example:
- `DocumentIndexingConsumer` reads tasks from `document_indexing_queue` queue and processes them.
- `API` enqueues tasks through `enqueue_task_sync` and fetches status through `get_task_status_sync`.
- Passing `consumer` into `API.__init__` ensures both deployments are part of the Serve application graph.
## Concurrency and reliability
Manage concurrency by setting `max_ongoing_requests` on the consumer deployment; this caps how many tasks each replica can process simultaneously. For at-least-once delivery, adapters should acknowledge a task only after the handler completes successfully. Failed tasks are retried up to `max_retries`; once exhausted, they are routed to the failed-task DLQ when configured. The default Celery adapter acknowledges on success, providing at-least-once processing.
(serve-async-inference-autoscaling)=
## Autoscaling
For workloads with variable traffic you can enable autoscaling so that replicas scale up when messages pile up in the queue and scale back down (optionally to zero) when the queue drains.
Ray Serve provides a built-in `AsyncInferenceAutoscalingPolicy` — a [class-based autoscaling policy](serve-custom-autoscaling-policies) that polls your message broker for queue length and scales replicas to match demand from both pending queue messages and in-flight requests.
### Basic example
::::{tab-set}
:::{tab-item} Python (imperative)
```{literalinclude} doc_code/async_inference_autoscaling.py
:language: python
:start-after: __basic_example_begin__
:end-before: __basic_example_end__
```
:::
:::{tab-item} YAML (declarative)
```{literalinclude} doc_code/async_inference_autoscaling.yaml
:language: yaml
:start-after: __basic_example_begin__
:end-before: __basic_example_end__
```
:::
::::
:::{note}
The `broker_url` and `queue_name` in `policy_kwargs` must match the values in your `TaskProcessorConfig`. The policy reads queue length from the same broker that your task consumer reads tasks from.
:::
### Policy parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `broker_url` | `str` | *(required)* | URL of the message broker (e.g. `redis://localhost:6379/0` or `amqp://guest:guest@localhost:5672//`). |
| `queue_name` | `str` | *(required)* | Name of the queue to monitor. Must match `TaskProcessorConfig.queue_name`. |
| `rabbitmq_management_url` | `str` | `None` | RabbitMQ HTTP management API URL (e.g. `http://guest:guest@localhost:15672/api/`). Required only for RabbitMQ brokers. |
| `poll_interval_s` | `float` | `10.0` | How often (seconds) to poll the broker for queue length. Lower values increase responsiveness but add broker load. |
All standard `AutoscalingConfig` parameters (`upscale_delay_s`, `downscale_delay_s`, `upscaling_factor`, `downscaling_factor`, etc.) apply on top of this policy. See [Advanced Ray Serve Autoscaling](serve-advanced-autoscaling) for details.
## Dead letter queues (DLQs)
Dead letter queues handle two types of problematic tasks:
- **Unprocessable tasks**: The system routes tasks with no matching handler to `unprocessable_task_queue_name` if set.
- **Failed tasks**: The system routes tasks that raise application exceptions after exhausting retries, have mismatched arguments, and other errors to `failed_task_queue_name` if set.
## Rollouts and compatibility
During deployment upgrades, both old and new consumer replicas may run concurrently and pull from the same queue. If task schemas or names change, either version may see incompatible tasks.
Recommendations:
- **Version task names and payloads** to allow coexistence across versions.
- **Don't remove handlers** until you drain old tasks.
- **Monitor DLQs** for deserialization or handler resolution failures and re-enqueue or transform as needed.
## Limitations
- Ray Serve supports only synchronous `@task_handler` methods.
- External (non-Serve) workers are out of scope; all consumers run as Serve deployments.
- Delivery guarantees ultimately depend on the configured broker. Results are optional when you don't configure a result backend.
:::{note}
The APIs in this guide reflect the alpha interfaces in `ray.serve.schema` and `ray.serve.task_consumer`.
:::
doc/source/serve/autoscaling-guide.md
+103
View File
@@ -0,0 +1,103 @@
(serve-autoscaling)=
# Ray Serve Autoscaling
Each [Ray Serve deployment](serve-key-concepts-deployment) has one [replica](serve-architecture-high-level-view) by default. This means there is one worker process running the model and serving requests. When traffic to your deployment increases, the single replica can become overloaded. To maintain high performance of your service, you need to scale out your deployment.
## Manual Scaling
Before jumping into autoscaling, which is more complex, the other option to consider is manual scaling. You can increase the number of replicas by setting a higher value for [num_replicas](serve-configure-deployment) in the deployment options through [in place updates](serve-inplace-updates). By default, `num_replicas` is 1. Increasing the number of replicas will horizontally scale out your deployment and improve latency and throughput for increased levels of traffic.
```yaml
# Deploy with a single replica
deployments:
- name: Model
num_replicas: 1
# Scale up to 10 replicas
deployments:
- name: Model
num_replicas: 10
```
## Autoscaling Basic Configuration
Instead of setting a fixed number of replicas for a deployment and manually updating it, you can configure a deployment to autoscale based on incoming traffic. The Serve autoscaler reacts to traffic spikes by monitoring queue sizes and making scaling decisions to add or remove replicas. Turn on autoscaling for a deployment by setting `num_replicas="auto"`. You can further configure it by tuning the [autoscaling_config](../serve/api/doc/ray.serve.config.AutoscalingConfig.rst) in deployment options.
The following config is what we will use in the example in the following section.
```yaml
- name: Model
num_replicas: auto
```
Setting `num_replicas="auto"` is equivalent to the following deployment configuration.
```yaml
- name: Model
max_ongoing_requests: 5
autoscaling_config:
target_ongoing_requests: 2
min_replicas: 1
max_replicas: 100
```
:::{note}
When you set `num_replicas="auto"`, Ray Serve applies the defaults shown above, including `max_replicas: 100`. However, if you configure autoscaling manually without using `num_replicas="auto"`, the base default for `max_replicas` is 1, which means autoscaling won't occur unless you explicitly set a higher value. You can override any of these defaults by specifying `autoscaling_config` even when using `num_replicas="auto"`.
:::
Let's dive into what each of these parameters do.
* **target_ongoing_requests** is the average number of ongoing requests per replica that the Serve autoscaler tries to ensure. You can adjust it based on your request processing length (the longer the requests, the smaller this number should be) as well as your latency objective (the shorter you want your latency to be, the smaller this number should be).
* **max_ongoing_requests** is the maximum number of ongoing requests allowed for a replica. Note this parameter is not part of the autoscaling config because it's relevant to all deployments, but it's important to set it relative to the target value if you turn on autoscaling for your deployment.
* **min_replicas** is the minimum number of replicas for the deployment. Set this to 0 if there are long periods of no traffic and some extra tail latency during upscale is acceptable. Otherwise, set this to what you think you need for low traffic.
* **max_replicas** is the maximum number of replicas for the deployment. Set this to ~20% higher than what you think you need for peak traffic.
These guidelines are a great starting point. If you decide to further tune your autoscaling config for your application, see [Advanced Ray Serve Autoscaling](serve-advanced-autoscaling).
(resnet-autoscaling-example)=
## Basic example
This example is a synchronous workload that runs ResNet50. The application code and its autoscaling configuration are below. Alternatively, see the second tab for specifying the autoscaling config through a YAML file.
::::{tab-set}
:::{tab-item} Application Code
```{literalinclude} doc_code/resnet50_example.py
:language: python
:start-after: __serve_example_begin__
:end-before: __serve_example_end__
```
:::
:::{tab-item} (Alternative) YAML config
```yaml
applications:
- name: default
import_path: resnet:app
deployments:
- name: Model
num_replicas: auto
```
:::
::::
This example uses [Locust](https://locust.io/) to run a load test against this application. The Locust load test runs a certain number of "users" that ping the ResNet50 service, where each user has a [constant wait time](https://docs.locust.io/en/stable/writing-a-locustfile.html#wait-time-attribute) of 0. Each user (repeatedly) sends a request, waits for a response, then immediately sends the next request. The number of users running over time is shown in the following graph:
![users](https://raw.githubusercontent.com/ray-project/images/master/docs/serve/autoscaling-guide/resnet50_users.png)
The results of the load test are as follows:
| | | |
| -------- | --- | ------- |
| Replicas | <img src="https://raw.githubusercontent.com/ray-project/images/master/docs/serve/autoscaling-guide/resnet50_replicas.png" alt="replicas" width="600"/> |
| QPS | <img src="https://raw.githubusercontent.com/ray-project/images/master/docs/serve/autoscaling-guide/resnet50_rps.png" alt="qps"/> |
| P50 Latency | <img src="https://raw.githubusercontent.com/ray-project/images/master/docs/serve/autoscaling-guide/resnet50_latency.png" alt="latency"/> |
Notice the following:
- Each Locust user constantly sends a single request and waits for a response. As a result, the number of autoscaled replicas is roughly half the number of Locust users over time as Serve attempts to satisfy the `target_ongoing_requests=2` setting.
- The throughput of the system increases with the number of users and replicas.
- The latency briefly spikes when traffic increases, but otherwise stays relatively steady.
## Ray Serve Autoscaler vs Ray Autoscaler
The Ray Serve Autoscaler is an application-level autoscaler that sits on top of the [Ray Autoscaler](cluster-index). Concretely, this means that the Ray Serve autoscaler asks Ray to start a number of replica actors based on the request demand. If the Ray Autoscaler determines there aren't enough available resources (e.g. CPUs, GPUs, etc.) to place these actors, it responds by requesting more Ray nodes. The underlying cloud provider then responds by adding more nodes. Similarly, when Ray Serve scales down and terminates replica Actors, it attempts to make as many nodes idle as possible so the Ray Autoscaler can remove them. To learn more about the architecture underlying Ray Serve Autoscaling, see [Ray Serve Autoscaling Architecture](serve-autoscaling-architecture).
doc/source/serve/configure-serve-deployment.md
+113
View File
@@ -0,0 +1,113 @@
(serve-configure-deployment)=
# Configure Ray Serve deployments
Ray Serve default values for deployments are a good starting point for exploration. To further tailor scaling behavior, resource management, or performance tuning, you can configure parameters to alter the default behavior of Ray Serve deployments.
Use this guide to learn the essentials of configuring deployments:
- What parameters you can configure for a Ray Serve deployment
- The different locations where you can specify the parameters.
## Configurable parameters
You can also refer to the [API reference](../serve/api/doc/ray.serve.deployment_decorator.rst) for the `@serve.deployment` decorator.
- `name` - Name uniquely identifying this deployment within the application. If not provided, the name of the class or function is used.
- `num_replicas` - Controls the number of replicas to run that handle requests to this deployment. This can be a positive integer, in which case the number of replicas stays constant, or `auto`, in which case the number of replicas will autoscale with a default configuration (see [Ray Serve Autoscaling](serve-autoscaling) for more). Defaults to 1.
- `ray_actor_options` - Options to pass to the Ray Actor decorator, such as resource requirements. Valid options are: `accelerator_type`, `memory`, `num_cpus`, `num_gpus`, `object_store_memory`, `resources`, and `runtime_env` For more details - [Resource management in Serve](serve-cpus-gpus)
- `max_ongoing_requests` - Maximum number of queries that are sent to a replica of this deployment without receiving a response. Defaults to 5 (note the default changed from 100 to 5 in Ray 2.32.0). This may be an important parameter to configure for [performance tuning](serve-perf-tuning).
- `autoscaling_config` - Parameters to configure autoscaling behavior. If this is set, you can't set `num_replicas` to a number. For more details on configurable parameters for autoscaling, see [Ray Serve Autoscaling](serve-autoscaling).
- `max_queued_requests` - Maximum number of requests to this deployment that will be queued at each caller (proxy or DeploymentHandle). Once this limit is reached, subsequent requests will raise a BackPressureError (for handles) or return an HTTP 503 status code (for HTTP requests). Defaults to -1 (no limit).
- `user_config` - Config to pass to the reconfigure method of the deployment. This can be updated dynamically without restarting the replicas of the deployment. The user_config must be fully JSON-serializable. For more details, see [Serve User Config](serve-user-config).
- `health_check_period_s` - Duration between health check calls for the replica. Defaults to 10s. The health check is by default a no-op Actor call to the replica, but you can define your own health check using the "check_health" method in your deployment that raises an exception when unhealthy.
- `health_check_timeout_s` - Duration in seconds, that replicas wait for a health check method to return before considering it as failed. Defaults to 30s.
- `graceful_shutdown_wait_loop_s` - Duration that replicas wait until there is no more work to be done before shutting down. Defaults to 2s.
- `graceful_shutdown_timeout_s` - Duration to wait for a replica to gracefully shut down before being forcefully killed. Defaults to 20s.
- `logging_config` - Logging Config for the deployment (e.g. log level, log directory, JSON log format and so on). See [LoggingConfig](../serve/api/doc/ray.serve.schema.LoggingConfig.rst) for details.
## How to specify parameters
You can specify the above mentioned parameters in two locations:
1. In your application code.
2. In the Serve Config file, which is the recommended method for production.
### Specify parameters through the application code
You can specify parameters in the application code in two ways:
- In the `@serve.deployment` decorator when you first define a deployment
- With the `options()` method when you want to modify a deployment
Use the `@serve.deployment` decorator to specify deployment parameters when you are defining a deployment for the first time:
```{literalinclude} ../serve/doc_code/configure_serve_deployment/model_deployment.py
:start-after: __deployment_start__
:end-before: __deployment_end__
:language: python
```
Use the [`.options()`](../serve/api/doc/ray.serve.Deployment.rst) method to modify deployment parameters on an already-defined deployment. Modifying an existing deployment lets you reuse deployment definitions and dynamically set parameters at runtime.
```{literalinclude} ../serve/doc_code/configure_serve_deployment/model_deployment.py
:start-after: __deployment_end__
:end-before: __options_end__
:language: python
```
### Specify parameters through the Serve config file
In production, we recommend configuring individual deployments through the Serve config file. You can change parameter values without modifying your application code. Learn more about how to use the Serve Config in the [production guide](serve-in-production-config-file).
```yaml
applications:
- name: app1
import_path: configure_serve:translator_app
deployments:
- name: Translator
num_replicas: 2
max_ongoing_requests: 100
graceful_shutdown_wait_loop_s: 2.0
graceful_shutdown_timeout_s: 20.0
health_check_period_s: 10.0
health_check_timeout_s: 30.0
ray_actor_options:
num_cpus: 0.2
num_gpus: 0.0
```
### Order of Priority
You can set parameters to different values in various locations. For each individual parameter, the order of priority is (from highest to lowest):
1. Serve Config file
2. Application code (either through the `@serve.deployment` decorator or through `.options()`)
3. Serve defaults
In other words, if you specify a parameter for a deployment in the config file and the application code, Serve uses the config file's value. If it's only specified in the code, Serve uses the value you specified in the code. If you don't specify the parameter anywhere, Serve uses the default for that parameter.
For example, the following application code contains a single deployment `ExampleDeployment`:
```python
@serve.deployment(num_replicas=2, graceful_shutdown_timeout_s=6)
class ExampleDeployment:
...
example_app = ExampleDeployment.bind()
```
Then you deploy the application with the following config file:
```yaml
applications:
- name: default
import_path: models:example_app
deployments:
- name: ExampleDeployment
num_replicas: 5
```
Serve uses `num_replicas=5` from the value set in the config file and `graceful_shutdown_timeout_s=6` from the value set in the application code. All other deployment settings use Serve defaults because you didn't specify them in the code or the config. For instance, `health_check_period_s=10` because by default Serve health checks deployments once every 10 seconds.
:::{tip}
Remember that `ray_actor_options` counts as a single setting. The entire `ray_actor_options` dictionary in the config file overrides the entire `ray_actor_options` dictionary from the graph code. If you set individual options within `ray_actor_options` (e.g. `runtime_env`, `num_gpus`, `memory`) in the code but not in the config, Serve still won't use the code settings if the config has a `ray_actor_options` dictionary. It treats these missing options as though the user never set them and uses defaults instead. This dictionary overriding behavior also applies to `user_config` and `autoscaling_config`.
:::
doc/source/serve/develop-and-deploy.md
+161
View File
@@ -0,0 +1,161 @@
(serve-develop-and-deploy)=
# Develop and Deploy an ML Application
The flow for developing a Ray Serve application locally and deploying it in production covers the following steps:
* Converting a Machine Learning model into a Ray Serve application
* Testing the application locally
* Building Serve config files for production deployment
* Deploying applications using a config file
## Convert a model into a Ray Serve application
This example uses a text-translation model:
```{literalinclude} ../serve/doc_code/getting_started/models.py
:start-after: __start_translation_model__
:end-before: __end_translation_model__
:language: python
```
The Python file, called `model.py`, uses the `Translator` class to translate English text to French.
- The `Translator`'s `__init__` method loads the [t5-small](https://huggingface.co/t5-small) tokenizer and model. `t5-small` is a text-to-text model that performs a task when its input is prefixed with a description of that task.
- The `translate` method prefixes the input with `"translate English to French: "`, calls `model.generate()` to produce the output tokens, and decodes them back into a translated string.
Copy and paste the script and run it locally. It translates `"Hello world!"` into `"Bonjour Monde!"`.
```console
$ python model.py
Bonjour Monde!
```
Converting this model into a Ray Serve application with FastAPI requires three changes:
1. Import Ray Serve and FastAPI dependencies
2. Add decorators for Serve deployment with FastAPI: `@serve.deployment` and `@serve.ingress(app)`
3. `bind` the `Translator` deployment to the arguments that are passed into its constructor
For other HTTP options, see [Set Up FastAPI and HTTP](serve-set-up-fastapi-http).
```{literalinclude} ../serve/doc_code/develop_and_deploy.py
:start-after: __deployment_start__
:end-before: __deployment_end__
:language: python
```
Note that the code configures parameters for the deployment, such as `num_replicas` and `ray_actor_options`. These parameters help configure the number of copies of the deployment and the resource requirements for each copy. In this case, we set up 2 replicas of the model that take 0.2 CPUs and 0 GPUs each. For a complete guide on the configurable parameters on a deployment, see [Configure a Serve deployment](serve-configure-deployment).
## Test a Ray Serve application locally
To test locally, run the script with the `serve run` CLI command. This command takes in an import path formatted as `module:application`. Run the command from a directory containing a local copy of the script saved as `model.py`, so it can import the application:
```console
$ serve run model:translator_app
```
This command runs the `translator_app` application and then blocks, streaming logs to the console. You can kill it with `Ctrl-C`, which tears down the application.
Now test the model over HTTP. Reach it at the following default URL:
```
http://127.0.0.1:8000/
```
Send a POST request with JSON data containing the English text. This client script requests a translation for "Hello world!":
```{literalinclude} ../serve/doc_code/develop_and_deploy.py
:start-after: __client_function_start__
:end-before: __client_function_end__
:language: python
```
While a Ray Serve application is deployed, use the `serve status` CLI command to check the status of the application and deployment. For more details on the output format of `serve status`, see [Inspect Serve in production](serve-in-production-inspecting).
```console
$ serve status
proxies:
a85af35da5fcea04e13375bdc7d2c83c7d3915e290f1b25643c55f3a: HEALTHY
applications:
default:
status: RUNNING
message: ''
last_deployed_time_s: 1693428451.894696
deployments:
Translator:
status: HEALTHY
replica_states:
RUNNING: 2
message: ''
```
## Build Serve config files for production deployment
To deploy Serve applications in production, you need to generate a Serve config YAML file. A Serve config file is the single source of truth for the cluster, allowing you to specify system-level configuration and your applications in one place. It also allows you to declaratively update your applications. The `serve build` CLI command takes as input the import path and saves to an output file using the `-o` flag. You can specify all deployment parameters in the Serve config files.
```console
$ serve build model:translator_app -o config.yaml
```
The `serve build` command adds a default application name that can be modified. The resulting Serve config file is:
```
proxy_location: EveryNode
http_options:
host: 0.0.0.0
port: 8000
grpc_options:
port: 9000
grpc_servicer_functions: []
applications:
- name: app1
route_prefix: /
import_path: model:translator_app
runtime_env: {}
deployments:
- name: Translator
num_replicas: 2
ray_actor_options:
num_cpus: 0.2
num_gpus: 0.0
```
You can also use the Serve config file with `serve run` for local testing. For example:
```console
$ serve run config.yaml
```
```console
$ serve status
proxies:
1894261b372d34854163ac5ec88405328302eb4e46ac3a2bdcaf8d18: HEALTHY
applications:
app1:
status: RUNNING
message: ''
last_deployed_time_s: 1693430474.873806
deployments:
Translator:
status: HEALTHY
replica_states:
RUNNING: 2
message: ''
```
For more details, see [Serve Config Files](serve-in-production-config-file).
## Deploy Ray Serve in production
Deploy the Ray Serve application in production on Kubernetes using the [KubeRay] operator. Copy the YAML file generated in the previous step directly into the Kubernetes configuration. KubeRay supports zero-downtime upgrades, status reporting, and fault tolerance for your production application. See [Deploying on Kubernetes](serve-in-production-kubernetes) for more information. For production usage, consider implementing the recommended practice of setting up [head node fault tolerance](serve-e2e-ft-guide-gcs).
## Monitor Ray Serve
Use the Ray Dashboard to get a high-level overview of your Ray Cluster and Ray Serve application's states. The Ray Dashboard is available both during local testing and on a remote cluster in production. Ray Serve provides some in-built metrics and logging as well as utilities for adding custom metrics and logs in your application. For production deployments, exporting logs and metrics to your observability platforms is recommended. See [Monitoring](serve-monitoring) for more details.
[KubeRay]: kuberay-index
doc/source/serve/doc_code/app_builder.py
+85
View File
@@ -0,0 +1,85 @@
# flake8: noqa
# __begin_untyped_builder__
# hello.py
from typing import Dict
from ray import serve
from ray.serve import Application
@serve.deployment
class HelloWorld:
def __init__(self, message: str):
self._message = message
print("Message:", self._message)
def __call__(self, request):
return self._message
def app_builder(args: Dict[str, str]) -> Application:
return HelloWorld.bind(args["message"])
# __end_untyped_builder__
import requests
serve.run(app_builder({"message": "Hello bar"}))
resp = requests.get("http://localhost:8000")
assert resp.text == "Hello bar"
# __begin_typed_builder__
# hello.py
from pydantic import BaseModel
from ray import serve
from ray.serve import Application
class HelloWorldArgs(BaseModel):
message: str
@serve.deployment
class HelloWorld:
def __init__(self, message: str):
self._message = message
print("Message:", self._message)
def __call__(self, request):
return self._message
def typed_app_builder(args: HelloWorldArgs) -> Application:
return HelloWorld.bind(args.message)
# __end_typed_builder__
serve.run(typed_app_builder(HelloWorldArgs(message="Hello baz")))
resp = requests.get("http://localhost:8000")
assert resp.text == "Hello baz"
# __begin_composed_builder__
from pydantic import BaseModel
from ray.serve import Application
class ComposedArgs(BaseModel):
model1_uri: str
model2_uri: str
def composed_app_builder(args: ComposedArgs) -> Application:
return IngressDeployment.bind(
Model1.bind(args.model1_uri),
Model2.bind(args.model2_uri),
)
# __end_composed_builder__
doc/source/serve/doc_code/application_level_autoscaling.py
+36
View File
@@ -0,0 +1,36 @@
# __serve_example_begin__
import time
from ray import serve
@serve.deployment
class Preprocessor:
def __call__(self, input_data: str) -> str:
# Simulate preprocessing work
time.sleep(0.05)
return f"preprocessed_{input_data}"
@serve.deployment
class Model:
def __call__(self, preprocessed_data: str) -> str:
# Simulate model inference (takes longer than preprocessing)
time.sleep(0.1)
return f"result_{preprocessed_data}"
@serve.deployment
class Driver:
def __init__(self, preprocessor, model):
self._preprocessor = preprocessor
self._model = model
async def __call__(self, input_data: str) -> str:
# Coordinate preprocessing and model inference
preprocessed = await self._preprocessor.remote(input_data)
result = await self._model.remote(preprocessed)
return result
app = Driver.bind(Preprocessor.bind(), Model.bind())
# __serve_example_end__
doc/source/serve/doc_code/application_level_autoscaling.yaml
+14
View File
@@ -0,0 +1,14 @@
applications:
- name: MyApp
import_path: application_level_autoscaling:app
autoscaling_policy:
policy_function: autoscaling_policy:coordinated_scaling_policy
deployments:
- name: Preprocessor
autoscaling_config:
min_replicas: 1
max_replicas: 10
- name: Model
autoscaling_config:
min_replicas: 2
max_replicas: 20
doc/source/serve/doc_code/application_level_autoscaling_with_defaults.yaml
+20
View File
@@ -0,0 +1,20 @@
applications:
- name: MyApp
import_path: application_level_autoscaling:app
autoscaling_policy:
policy_function: autoscaling_policy:coordinated_scaling_policy_with_defaults
deployments:
- name: Preprocessor
autoscaling_config:
min_replicas: 1
max_replicas: 10
target_ongoing_requests: 1
upscale_delay_s: 2
downscale_delay_s: 5
- name: Model
autoscaling_config:
min_replicas: 2
max_replicas: 20
target_ongoing_requests: 1
upscale_delay_s: 3
downscale_delay_s: 5
doc/source/serve/doc_code/async_inference_autoscaling.py
+40
View File
@@ -0,0 +1,40 @@
# __basic_example_begin__
from ray import serve
from ray.serve.config import AutoscalingConfig, AutoscalingPolicy
from ray.serve.schema import CeleryAdapterConfig, TaskProcessorConfig
from ray.serve.task_consumer import task_consumer, task_handler
processor_config = TaskProcessorConfig(
queue_name="my_queue",
adapter_config=CeleryAdapterConfig(
broker_url="redis://localhost:6379/0",
backend_url="redis://localhost:6379/1",
),
)
@serve.deployment(
max_ongoing_requests=5,
autoscaling_config=AutoscalingConfig(
min_replicas=1,
max_replicas=10,
target_ongoing_requests=2,
policy=AutoscalingPolicy(
policy_function="ray.serve.async_inference_autoscaling_policy:AsyncInferenceAutoscalingPolicy",
policy_kwargs={
"broker_url": "redis://localhost:6379/0",
"queue_name": "my_queue",
},
),
),
)
@task_consumer(task_processor_config=processor_config)
class MyConsumer:
@task_handler(name="process")
def process(self, data):
return f"processed: {data}"
app = MyConsumer.bind()
serve.run(app)
# __basic_example_end__
doc/source/serve/doc_code/async_inference_autoscaling.yaml
+17
View File
@@ -0,0 +1,17 @@
# __basic_example_begin__
applications:
- name: default
import_path: my_module:app
deployments:
- name: MyConsumer
max_ongoing_requests: 5
autoscaling_config:
min_replicas: 1
max_replicas: 10
target_ongoing_requests: 2
policy:
policy_function: "ray.serve.async_inference_autoscaling_policy:AsyncInferenceAutoscalingPolicy"
policy_kwargs:
broker_url: "redis://localhost:6379/0"
queue_name: "my_queue"
# __basic_example_end__
doc/source/serve/doc_code/asyncio_best_practices.py
+435
View File
@@ -0,0 +1,435 @@
# flake8: noqa
"""
Code examples for the asyncio best practices guide.
All examples are structured to be runnable and demonstrate key concepts.
"""
# __imports_begin__
from ray import serve
import asyncio
# __imports_end__
# __echo_async_begin__
@serve.deployment
class Echo:
async def __call__(self, request):
await asyncio.sleep(0.1)
return "ok"
# __echo_async_end__
# __blocking_echo_begin__
@serve.deployment
class BlockingEcho:
def __call__(self, request):
# Blocking.
import time
time.sleep(1)
return "ok"
# __blocking_echo_end__
# __fastapi_deployment_begin__
from fastapi import FastAPI
app = FastAPI()
@serve.deployment
@serve.ingress(app)
class FastAPIDeployment:
@app.get("/sync")
def sync_endpoint(self):
# FastAPI runs this in a threadpool.
import time
time.sleep(1)
return "ok"
@app.get("/async")
async def async_endpoint(self):
# Runs directly on FastAPI's asyncio loop.
await asyncio.sleep(1)
return "ok"
# __fastapi_deployment_end__
# __blocking_http_begin__
@serve.deployment
class BlockingHTTP:
async def __call__(self, request):
# ❌ This blocks the event loop until the HTTP call finishes.
import requests
resp = requests.get("https://example.com/")
return resp.text
# __blocking_http_end__
# __async_http_begin__
@serve.deployment
class AsyncHTTP:
async def __call__(self, request):
import httpx
async with httpx.AsyncClient() as client:
resp = await client.get("https://example.com/")
return resp.text
# __async_http_end__
# __threaded_http_begin__
@serve.deployment
class ThreadedHTTP:
async def __call__(self, request):
import requests
def fetch():
return requests.get("https://example.com/").text
# ✅ Offload blocking I/O to a worker thread.
return await asyncio.to_thread(fetch)
# __threaded_http_end__
# __threadpool_override_begin__
from concurrent.futures import ThreadPoolExecutor
@serve.deployment
class CustomThreadPool:
def __init__(self):
loop = asyncio.get_running_loop()
loop.set_default_executor(ThreadPoolExecutor(max_workers=16))
async def __call__(self, request):
return await asyncio.to_thread(lambda: "ok")
# __threadpool_override_end__
# __numpy_deployment_begin__
@serve.deployment
class NumpyDeployment:
def _heavy_numpy(self, array):
import numpy as np
# Many NumPy ops release the GIL while executing C/Fortran code.
return np.linalg.svd(array)[0]
async def __call__(self, request):
import numpy as np
# Create a sample array from request data
array = np.random.rand(100, 100)
# ✅ Multiple threads can run _heavy_numpy in parallel if
# the underlying implementation releases the GIL.
return await asyncio.to_thread(self._heavy_numpy, array)
# __numpy_deployment_end__
# __max_ongoing_requests_begin__
@serve.deployment(max_ongoing_requests=32)
class MyService:
async def __call__(self, request):
await asyncio.sleep(1)
return "ok"
# __max_ongoing_requests_end__
# __async_io_bound_begin__
@serve.deployment(max_ongoing_requests=100)
class AsyncIOBound:
async def __call__(self, request):
# Mostly waiting on an external system.
await asyncio.sleep(0.1)
return "ok"
# __async_io_bound_end__
# __blocking_cpu_begin__
@serve.deployment(max_ongoing_requests=100)
class BlockingCPU:
def __call__(self, request):
# ❌ Blocks the user event loop.
import time
time.sleep(1)
return "ok"
# __blocking_cpu_end__
# __cpu_with_threadpool_begin__
@serve.deployment(max_ongoing_requests=100)
class CPUWithThreadpool:
def __call__(self, request):
# With RAY_SERVE_RUN_SYNC_IN_THREADPOOL=1, each call runs in a thread.
import time
time.sleep(1)
return "ok"
# __cpu_with_threadpool_end__
# __batched_model_begin__
@serve.deployment(max_ongoing_requests=64)
class BatchedModel:
@serve.batch(max_batch_size=32)
async def __call__(self, requests):
# requests is a list of request objects.
inputs = [r for r in requests]
outputs = await self._run_model(inputs)
return outputs
async def _run_model(self, inputs):
# Placeholder model function
return [f"result_{i}" for i in inputs]
# __batched_model_end__
# __batched_model_offload_begin__
@serve.deployment(max_ongoing_requests=64)
class BatchedModelOffload:
@serve.batch(max_batch_size=32)
async def __call__(self, requests):
# requests is a list of request objects.
inputs = [r for r in requests]
outputs = await self._run_model(inputs)
return outputs
async def _run_model(self, inputs):
def run_sync():
# Heavy CPU or GIL-releasing native code here.
# Placeholder model function
return [f"result_{i}" for i in inputs]
loop = asyncio.get_running_loop()
return await loop.run_in_executor(None, run_sync)
# __batched_model_offload_end__
# __blocking_stream_begin__
@serve.deployment
class BlockingStream:
def __call__(self, request):
# ❌ Blocks the event loop between yields.
import time
for i in range(10):
time.sleep(1)
yield f"{i}\n"
# __blocking_stream_end__
# __async_stream_begin__
@serve.deployment
class AsyncStream:
async def __call__(self, request):
# ✅ Yields items without blocking the loop.
async def generator():
for i in range(10):
await asyncio.sleep(1)
yield f"{i}\n"
return generator()
# __async_stream_end__
# __offload_io_begin__
@serve.deployment
class OffloadIO:
async def __call__(self, request):
import requests
def fetch():
return requests.get("https://example.com/").text
# Offload to a thread, free the event loop.
body = await asyncio.to_thread(fetch)
return body
# __offload_io_end__
# __offload_cpu_begin__
@serve.deployment
class OffloadCPU:
def _compute(self, x):
# CPU-intensive work.
total = 0
for i in range(10_000_000):
total += (i * x) % 7
return total
async def __call__(self, request):
x = 123
loop = asyncio.get_running_loop()
result = await loop.run_in_executor(None, self._compute, x)
return str(result)
# __offload_cpu_end__
# __ray_parallel_begin__
import ray
@ray.remote
def heavy_task(x):
# Heavy compute runs in its own worker process.
return x * x
@serve.deployment
class RayParallel:
async def __call__(self, request):
values = [1, 2, 3, 4]
refs = [heavy_task.remote(v) for v in values]
results = await asyncio.gather(*[r for r in refs])
return {"results": results}
# __ray_parallel_end__
if __name__ == "__main__":
import ray
# Initialize Ray if not already running
if not ray.is_initialized():
ray.init()
print("Testing Echo deployment...")
# Test Echo
echo_handle = serve.run(Echo.bind())
result = echo_handle.remote(None).result()
print(f"Echo result: {result}")
assert result == "ok"
print("\nTesting BlockingEcho deployment...")
# Test BlockingEcho
blocking_handle = serve.run(BlockingEcho.bind())
result = blocking_handle.remote(None).result()
print(f"BlockingEcho result: {result}")
assert result == "ok"
print("\nTesting MyService deployment...")
# Test MyService
service_handle = serve.run(MyService.bind())
result = service_handle.remote(None).result()
print(f"MyService result: {result}")
assert result == "ok"
print("\nTesting AsyncIOBound deployment...")
# Test AsyncIOBound
io_bound_handle = serve.run(AsyncIOBound.bind())
result = io_bound_handle.remote(None).result()
print(f"AsyncIOBound result: {result}")
assert result == "ok"
print("\nTesting AsyncStream deployment...")
# Test AsyncStream (just create it, don't fully consume)
stream_handle = serve.run(AsyncStream.bind())
print("AsyncStream deployment created successfully")
print("\nTesting OffloadCPU deployment...")
# Test OffloadCPU
cpu_handle = serve.run(OffloadCPU.bind())
result = cpu_handle.remote(None).result()
print(f"OffloadCPU result: {result}")
print("\nTesting NumpyDeployment...")
# Test NumpyDeployment
numpy_handle = serve.run(NumpyDeployment.bind())
result = numpy_handle.remote(None).result()
print(f"NumpyDeployment result shape: {result.shape}")
assert result.shape == (100, 100)
print("\nTesting BlockingCPU deployment...")
# Test BlockingCPU
blocking_cpu_handle = serve.run(BlockingCPU.bind())
result = blocking_cpu_handle.remote(None).result()
print(f"BlockingCPU result: {result}")
assert result == "ok"
print("\nTesting CPUWithThreadpool deployment...")
# Test CPUWithThreadpool
cpu_threadpool_handle = serve.run(CPUWithThreadpool.bind())
result = cpu_threadpool_handle.remote(None).result()
print(f"CPUWithThreadpool result: {result}")
assert result == "ok"
print("\nTesting CustomThreadPool deployment...")
custom_threadpool_handle = serve.run(CustomThreadPool.bind())
result = custom_threadpool_handle.remote(None).result()
print(f"CustomThreadPool result: {result}")
assert result == "ok"
print("\nTesting BlockingStream deployment...")
# Test BlockingStream - just verify it can be created and called
blocking_stream_handle = serve.run(BlockingStream.bind())
# For generator responses, we need to handle them differently
# Just verify deployment works
print("BlockingStream deployment created successfully")
print("\nTesting RayParallel deployment...")
# Test RayParallel
ray_parallel_handle = serve.run(RayParallel.bind())
result = ray_parallel_handle.remote(None).result()
print(f"RayParallel result: {result}")
assert result == {"results": [1, 4, 9, 16]}
print("\nTesting BatchedModel deployment...")
# Test BatchedModel
batched_model_handle = serve.run(BatchedModel.bind())
result = batched_model_handle.remote(1).result()
print(f"BatchedModel result: {result}")
assert result == "result_1"
print("\nTesting BatchedModelOffload deployment...")
# Test BatchedModelOffload
batched_model_offload_handle = serve.run(BatchedModelOffload.bind())
result = batched_model_offload_handle.remote(1).result()
print(f"BatchedModelOffload result: {result}")
assert result == "result_1"
# Test HTTP-related deployments with try-except
print("\n--- Testing HTTP-related deployments (may fail due to network) ---")
print("\nTesting BlockingHTTP deployment...")
try:
blocking_http_handle = serve.run(BlockingHTTP.bind())
result = blocking_http_handle.remote(None).result()
print(f"BlockingHTTP result (first 50 chars): {result[:50]}...")
print("✅ BlockingHTTP test passed")
except Exception as e:
print(f"⚠️ BlockingHTTP test failed (expected): {type(e).__name__}: {e}")
print("\nTesting AsyncHTTP deployment...")
try:
async_http_handle = serve.run(AsyncHTTP.bind())
result = async_http_handle.remote(None).result()
print(f"AsyncHTTP result (first 50 chars): {result[:50]}...")
print("✅ AsyncHTTP test passed")
except Exception as e:
print(f"⚠️ AsyncHTTP test failed (expected): {type(e).__name__}: {e}")
print("\nTesting ThreadedHTTP deployment...")
try:
threaded_http_handle = serve.run(ThreadedHTTP.bind())
result = threaded_http_handle.remote(None).result()
print(f"ThreadedHTTP result (first 50 chars): {result[:50]}...")
print("✅ ThreadedHTTP test passed")
except Exception as e:
print(f"⚠️ ThreadedHTTP test failed (expected): {type(e).__name__}: {e}")
print("\nTesting OffloadIO deployment...")
try:
offload_io_handle = serve.run(OffloadIO.bind())
result = offload_io_handle.remote(None).result()
print(f"OffloadIO result (first 50 chars): {result[:50]}...")
print("✅ OffloadIO test passed")
except Exception as e:
print(f"⚠️ OffloadIO test failed (expected): {type(e).__name__}: {e}")
print("\nTesting FastAPIDeployment...")
fastapi_handle = serve.run(FastAPIDeployment.bind())
# Give it a moment to start
import time
import requests
time.sleep(2)
# Test the sync endpoint
response = requests.get("http://127.0.0.1:8000/sync", timeout=5)
print(f"FastAPIDeployment /sync result: {response.json()}")
# Test the async endpoint
response = requests.get("http://127.0.0.1:8000/async", timeout=5)
print(f"FastAPIDeployment /async result: {response.json()}")
print("✅ FastAPIDeployment test passed")
print("\n✅ All core tests passed!")
doc/source/serve/doc_code/autoscale_model_comp_example.py
+48
View File
@@ -0,0 +1,48 @@
# __serve_example_begin__
import time
from ray import serve
from ray.serve.handle import DeploymentHandle
@serve.deployment
class LightLoad:
async def __call__(self) -> str:
start = time.time()
while time.time() - start < 0.1:
pass
return "light"
@serve.deployment
class HeavyLoad:
async def __call__(self) -> str:
start = time.time()
while time.time() - start < 0.2:
pass
return "heavy"
@serve.deployment
class Driver:
def __init__(self, a_handle, b_handle):
self.a_handle: DeploymentHandle = a_handle
self.b_handle: DeploymentHandle = b_handle
async def __call__(self) -> str:
a_future = self.a_handle.remote()
b_future = self.b_handle.remote()
return (await a_future), (await b_future)
app = Driver.bind(HeavyLoad.bind(), LightLoad.bind())
# __serve_example_end__
import requests # noqa
serve.run(app)
resp = requests.post("http://localhost:8000")
assert resp.json() == ["heavy", "light"]
doc/source/serve/doc_code/autoscaling_policy.py
+178
View File
@@ -0,0 +1,178 @@
# __begin_scheduled_batch_processing_policy__
from datetime import datetime
from typing import Any, Dict
from ray.serve.config import AutoscalingContext
def scheduled_batch_processing_policy(
ctx: AutoscalingContext,
) -> tuple[int, Dict[str, Any]]:
current_time = datetime.now()
current_hour = current_time.hour
# Scale up during business hours (9 AM - 5 PM)
if 9 <= current_hour < 17:
return 2, {"reason": "Business hours"}
# Scale up for evening batch processing (6 PM - 8 PM)
elif 18 <= current_hour < 20:
return 4, {"reason": "Evening batch processing"}
# Minimal scaling during off-peak hours
else:
return 1, {"reason": "Off-peak hours"}
# __end_scheduled_batch_processing_policy__
# __begin_custom_metrics_autoscaling_policy__
from typing import Any, Dict
from ray.serve.config import AutoscalingContext
def custom_metrics_autoscaling_policy(
ctx: AutoscalingContext,
) -> tuple[int, Dict[str, Any]]:
cpu_usage_metric = ctx.aggregated_metrics.get("cpu_usage", {})
memory_usage_metric = ctx.aggregated_metrics.get("memory_usage", {})
max_cpu_usage = list(cpu_usage_metric.values())[-1] if cpu_usage_metric else 0
max_memory_usage = (
list(memory_usage_metric.values())[-1] if memory_usage_metric else 0
)
if max_cpu_usage > 80 or max_memory_usage > 85:
return min(ctx.capacity_adjusted_max_replicas, ctx.current_num_replicas + 1), {}
elif max_cpu_usage < 30 and max_memory_usage < 40:
return max(ctx.capacity_adjusted_min_replicas, ctx.current_num_replicas - 1), {}
else:
return ctx.current_num_replicas, {}
# __end_custom_metrics_autoscaling_policy__
# __begin_application_level_autoscaling_policy__
from typing import Dict, Tuple
from ray.serve.config import AutoscalingContext
from ray.serve._private.common import DeploymentID
from ray.serve.config import AutoscalingContext
def coordinated_scaling_policy(
contexts: Dict[DeploymentID, AutoscalingContext]
) -> Tuple[Dict[DeploymentID, int], Dict]:
"""Scale deployments based on coordinated load balancing."""
decisions = {}
# Example: Scale a preprocessing deployment
preprocessing_id = [d for d in contexts if d.name == "Preprocessor"][0]
preprocessing_ctx = contexts[preprocessing_id]
# Scale based on queue depth
preprocessing_replicas = max(
preprocessing_ctx.capacity_adjusted_min_replicas,
min(
preprocessing_ctx.capacity_adjusted_max_replicas,
preprocessing_ctx.total_num_requests // 10,
),
)
decisions[preprocessing_id] = preprocessing_replicas
# Example: Scale a model deployment proportionally
model_id = [d for d in contexts if d.name == "Model"][0]
model_ctx = contexts[model_id]
# Scale model to handle preprocessing output
# Assuming model takes 2x longer than preprocessing
model_replicas = max(
model_ctx.capacity_adjusted_min_replicas,
min(model_ctx.capacity_adjusted_max_replicas, preprocessing_replicas * 2),
)
decisions[model_id] = model_replicas
return decisions, {}
# __end_application_level_autoscaling_policy__
# __begin_stateful_application_level_policy__
from typing import Dict, Tuple, Any
from ray.serve.config import AutoscalingContext
from ray.serve._private.common import DeploymentID
def stateful_application_level_policy(
contexts: Dict[DeploymentID, AutoscalingContext]
) -> Tuple[Dict[DeploymentID, int], Dict[DeploymentID, Dict[str, Any]]]:
"""Example policy demonstrating per-deployment state persistence."""
decisions = {}
policy_state = {}
for deployment_id, ctx in contexts.items():
# Read previous state for this deployment (persisted from last iteration)
prev_state = ctx.policy_state or {}
scale_count = prev_state.get("scale_count", 0)
last_replicas = prev_state.get("last_replicas", ctx.current_num_replicas)
# Simple scaling logic: scale based on queue depth
desired_replicas = max(
ctx.capacity_adjusted_min_replicas,
min(
ctx.capacity_adjusted_max_replicas,
ctx.total_num_requests // 10,
),
)
decisions[deployment_id] = desired_replicas
# Store per-deployment state that persists across iterations
policy_state[deployment_id] = {
"scale_count": scale_count + 1,
"last_replicas": desired_replicas,
}
return decisions, policy_state
# __end_stateful_application_level_policy__
# __begin_apply_autoscaling_config_example__
from typing import Any, Dict
from ray.serve.config import AutoscalingContext
def queue_length_based_autoscaling_policy(
ctx: AutoscalingContext,
) -> tuple[int, Dict[str, Any]]:
# This policy calculates the "raw" desired replicas based on queue length.
# Ray Serve automatically applies scaling factors, delays, and bounds from
# the deployment's autoscaling_config on top of this decision.
queue_length = ctx.total_num_requests
if queue_length > 50:
return 10, {}
elif queue_length > 10:
return 5, {}
else:
return 0, {}
# __end_apply_autoscaling_config_example__
# __begin_apply_autoscaling_config_usage__
from ray import serve
from ray.serve.config import AutoscalingConfig, AutoscalingPolicy
@serve.deployment(
autoscaling_config=AutoscalingConfig(
min_replicas=1,
max_replicas=10,
metrics_interval_s=0.1,
upscale_delay_s=1.0,
downscale_delay_s=1.0,
policy=AutoscalingPolicy(
policy_function=queue_length_based_autoscaling_policy
)
),
max_ongoing_requests=5,
)
class MyDeployment:
def __call__(self) -> str:
return "Hello, world!"
app = MyDeployment.bind()
# __end_apply_autoscaling_config_usage__
doc/source/serve/doc_code/aws_neuron_core_inference_serve.py
+102
View File
@@ -0,0 +1,102 @@
# flake8: noqa
# __compile_neuron_code_start__
from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch, torch_neuronx
hf_model = "j-hartmann/emotion-english-distilroberta-base"
neuron_model = "./sentiment_neuron.pt"
model = AutoModelForSequenceClassification.from_pretrained(hf_model)
tokenizer = AutoTokenizer.from_pretrained(hf_model)
sequence_0 = "The company HuggingFace is based in New York City"
sequence_1 = "HuggingFace's headquarters are situated in Manhattan"
example_inputs = tokenizer.encode_plus(
sequence_0,
sequence_1,
return_tensors="pt",
padding="max_length",
truncation=True,
max_length=128,
)
neuron_inputs = example_inputs["input_ids"], example_inputs["attention_mask"]
n_model = torch_neuronx.trace(model, neuron_inputs)
n_model.save(neuron_model)
print(f"Saved Neuron-compiled model {neuron_model}")
# __compile_neuron_code_end__
# __neuron_serve_code_start__
from fastapi import FastAPI
import torch
from ray import serve
from ray.serve.handle import DeploymentHandle
app = FastAPI()
hf_model = "j-hartmann/emotion-english-distilroberta-base"
neuron_model = "./sentiment_neuron.pt"
@serve.deployment(num_replicas=1)
@serve.ingress(app)
class APIIngress:
def __init__(self, bert_base_model_handle: DeploymentHandle) -> None:
self.handle = bert_base_model_handle
@app.get("/infer")
async def infer(self, sentence: str):
return await self.handle.infer.remote(sentence)
@serve.deployment(
ray_actor_options={"resources": {"neuron_cores": 1}},
autoscaling_config={"min_replicas": 1, "max_replicas": 2},
)
class BertBaseModel:
def __init__(self):
import torch, torch_neuronx # noqa
from transformers import AutoTokenizer
self.model = torch.jit.load(neuron_model)
self.tokenizer = AutoTokenizer.from_pretrained(hf_model)
self.classmap = {
0: "anger",
1: "disgust",
2: "fear",
3: "joy",
4: "neutral",
5: "sadness",
6: "surprise",
}
def infer(self, sentence: str):
inputs = self.tokenizer.encode_plus(
sentence,
return_tensors="pt",
padding="max_length",
truncation=True,
max_length=128,
)
output = self.model(*(inputs["input_ids"], inputs["attention_mask"]))
class_id = torch.argmax(output["logits"], dim=1).item()
return self.classmap[class_id]
entrypoint = APIIngress.bind(BertBaseModel.bind())
# __neuron_serve_code_end__
if __name__ == "__main__":
import requests
import ray
# On inf2.8xlarge instance, there will be 2 neuron cores.
ray.init(resources={"neuron_cores": 2})
serve.run(entrypoint)
prompt = "Ray is super cool."
resp = requests.get(f"http://127.0.0.1:8000/infer?sentence={prompt}")
print(resp.status_code, resp.json())
assert resp.status_code == 200
doc/source/serve/doc_code/aws_neuron_core_inference_serve_stable_diffusion.py
+72
View File
@@ -0,0 +1,72 @@
# __neuron_serve_code_start__
from io import BytesIO
from fastapi import FastAPI
from fastapi.responses import Response
from ray import serve
app = FastAPI()
neuron_cores = 2
@serve.deployment(num_replicas=1)
@serve.ingress(app)
class APIIngress:
def __init__(self, diffusion_model_handle) -> None:
self.handle = diffusion_model_handle
@app.get(
"/imagine",
responses={200: {"content": {"image/png": {}}}},
response_class=Response,
)
async def generate(self, prompt: str):
image_ref = await self.handle.generate.remote(prompt)
image = image_ref
file_stream = BytesIO()
image.save(file_stream, "PNG")
return Response(content=file_stream.getvalue(), media_type="image/png")
@serve.deployment(
ray_actor_options={"resources": {"neuron_cores": neuron_cores}},
autoscaling_config={"min_replicas": 1, "max_replicas": 2},
)
class StableDiffusionV2:
def __init__(self):
from optimum.neuron import NeuronStableDiffusionXLPipeline
compiled_model_id = "aws-neuron/stable-diffusion-xl-base-1-0-1024x1024"
self.pipe = NeuronStableDiffusionXLPipeline.from_pretrained(
compiled_model_id, device_ids=[0, 1]
)
async def generate(self, prompt: str):
assert len(prompt), "prompt parameter cannot be empty"
image = self.pipe(prompt).images[0]
return image
entrypoint = APIIngress.bind(StableDiffusionV2.bind())
# __neuron_serve_code_end__
if __name__ == "__main__":
import requests
import ray
# On inf2.8xlarge instance, there are 2 Neuron cores.
ray.init(resources={"neuron_cores": 2})
serve.run(entrypoint)
prompt = "a zebra is dancing in the grass, river, sunlit"
input = "%20".join(prompt.split(" "))
resp = requests.get(f"http://127.0.0.1:8000/imagine?prompt={input}")
print("Write the response to `output.png`.")
with open("output.png", "wb") as f:
f.write(resp.content)
assert resp.status_code == 200
doc/source/serve/doc_code/batching_guide.py
+245
View File
@@ -0,0 +1,245 @@
# flake8: noqa
# __single_sample_begin__
from ray import serve
from ray.serve.handle import DeploymentHandle
@serve.deployment
class Model:
def __call__(self, single_sample: int) -> int:
return single_sample * 2
handle: DeploymentHandle = serve.run(Model.bind())
assert handle.remote(1).result() == 2
# __single_sample_end__
# __batch_begin__
from typing import List
import numpy as np
from ray import serve
from ray.serve.handle import DeploymentHandle
@serve.deployment
class Model:
@serve.batch(max_batch_size=8, batch_wait_timeout_s=0.1)
async def __call__(self, multiple_samples: List[int]) -> List[int]:
# Use numpy's vectorized computation to efficiently process a batch.
return np.array(multiple_samples) * 2
handle: DeploymentHandle = serve.run(Model.bind())
responses = [handle.remote(i) for i in range(8)]
assert list(r.result() for r in responses) == [i * 2 for i in range(8)]
# __batch_end__
# __batch_params_update_begin__
from typing import Dict
@serve.deployment(
# These values can be overridden in the Serve config.
user_config={
"max_batch_size": 10,
"batch_wait_timeout_s": 0.5,
}
)
class Model:
@serve.batch(max_batch_size=8, batch_wait_timeout_s=0.1)
async def __call__(self, multiple_samples: List[int]) -> List[int]:
# Use numpy's vectorized computation to efficiently process a batch.
return np.array(multiple_samples) * 2
def reconfigure(self, user_config: Dict):
self.__call__.set_max_batch_size(user_config["max_batch_size"])
self.__call__.set_batch_wait_timeout_s(user_config["batch_wait_timeout_s"])
# __batch_params_update_end__
# __single_stream_begin__
import asyncio
from typing import AsyncGenerator
from starlette.requests import Request
from starlette.responses import StreamingResponse
from ray import serve
@serve.deployment
class StreamingResponder:
async def generate_numbers(self, max: str) -> AsyncGenerator[str, None]:
for i in range(max):
yield str(i)
await asyncio.sleep(0.1)
def __call__(self, request: Request) -> StreamingResponse:
max = int(request.query_params.get("max", "25"))
gen = self.generate_numbers(max)
return StreamingResponse(gen, status_code=200, media_type="text/plain")
# __single_stream_end__
import requests
serve.run(StreamingResponder.bind())
r = requests.get("http://localhost:8000/", stream=True)
chunks = []
for chunk in r.iter_content(chunk_size=None, decode_unicode=True):
chunks.append(chunk)
assert ",".join(list(map(str, range(25)))) == ",".join(chunks)
# __batch_stream_begin__
import asyncio
from typing import List, AsyncGenerator, Union
from starlette.requests import Request
from starlette.responses import StreamingResponse
from ray import serve
@serve.deployment
class StreamingResponder:
@serve.batch(max_batch_size=5, batch_wait_timeout_s=0.1)
async def generate_numbers(
self, max_list: List[str]
) -> AsyncGenerator[List[Union[int, StopIteration]], None]:
for i in range(max(max_list)):
next_numbers = []
for requested_max in max_list:
if requested_max > i:
next_numbers.append(str(i))
else:
next_numbers.append(StopIteration)
yield next_numbers
await asyncio.sleep(0.1)
async def __call__(self, request: Request) -> StreamingResponse:
max = int(request.query_params.get("max", "25"))
gen = self.generate_numbers(max)
return StreamingResponse(gen, status_code=200, media_type="text/plain")
# __batch_stream_end__
import requests
from functools import partial
from concurrent.futures.thread import ThreadPoolExecutor
serve.run(StreamingResponder.bind())
def issue_request(max) -> List[str]:
url = "http://localhost:8000/?max="
response = requests.get(url + str(max), stream=True)
chunks = []
for chunk in response.iter_content(chunk_size=None, decode_unicode=True):
chunks.append(chunk)
return chunks
requested_maxes = [1, 2, 5, 6, 9]
with ThreadPoolExecutor(max_workers=5) as pool:
futs = [pool.submit(partial(issue_request, max)) for max in requested_maxes]
chunks_list = [fut.result() for fut in futs]
for max, chunks in zip(requested_maxes, chunks_list):
assert chunks == [str(i) for i in range(max)]
# __batch_size_fn_begin__
from typing import List
from ray import serve
from ray.serve.handle import DeploymentHandle
class Graph:
"""Simple graph data structure for GNN workloads."""
def __init__(self, num_nodes: int, node_features: list):
self.num_nodes = num_nodes
self.node_features = node_features
@serve.deployment
class GraphNeuralNetwork:
@serve.batch(
max_batch_size=10000, # Maximum total nodes per batch
batch_wait_timeout_s=0.1,
batch_size_fn=lambda graphs: sum(g.num_nodes for g in graphs),
)
async def predict(self, graphs: List[Graph]) -> List[float]:
"""Process a batch of graphs, batching by total node count."""
# The batch_size_fn ensures that the total number of nodes
# across all graphs in the batch doesn't exceed max_batch_size.
# This prevents GPU memory overflow.
results = []
for graph in graphs:
# Your GNN model inference logic here
# For this example, just return a simple score
score = float(graph.num_nodes * 0.1)
results.append(score)
return results
async def __call__(self, graph: Graph) -> float:
return await self.predict(graph)
handle: DeploymentHandle = serve.run(GraphNeuralNetwork.bind())
# Create test graphs with varying node counts
graphs = [
Graph(num_nodes=100, node_features=[1.0] * 100),
Graph(num_nodes=5000, node_features=[2.0] * 5000),
Graph(num_nodes=3000, node_features=[3.0] * 3000),
]
# Send requests - they'll be batched by total node count
results = [handle.remote(g).result() for g in graphs]
print(f"Results: {results}")
# __batch_size_fn_end__
# __batch_size_fn_nlp_begin__
from typing import List
from ray import serve
from ray.serve.handle import DeploymentHandle
@serve.deployment
class TokenBatcher:
@serve.batch(
max_batch_size=512, # Maximum total tokens per batch
batch_wait_timeout_s=0.1,
batch_size_fn=lambda sequences: sum(len(s.split()) for s in sequences),
)
async def process(self, sequences: List[str]) -> List[int]:
"""Process text sequences, batching by total token count."""
# The batch_size_fn ensures total tokens don't exceed max_batch_size.
# This is useful for transformer models with fixed context windows.
return [len(seq.split()) for seq in sequences]
async def __call__(self, sequence: str) -> int:
return await self.process(sequence)
handle: DeploymentHandle = serve.run(TokenBatcher.bind())
# Create sequences with different lengths
sequences = [
"This is a short sentence",
"This is a much longer sentence with many more words to process",
"Short",
]
# Send requests - they'll be batched by total token count
results = [handle.remote(seq).result() for seq in sequences]
print(f"Token counts: {results}")
# __batch_size_fn_nlp_end__
doc/source/serve/doc_code/capacity_queue_request_router_app.py
+56
View File
@@ -0,0 +1,56 @@
# flake8: noqa
# __begin_deploy_app_with_capacity_queue_router__
import ray
from ray import serve
from ray.serve.config import DeploymentActorConfig, RequestRouterConfig
from ray.serve.context import _get_internal_replica_context
from ray.serve.experimental.capacity_queue import (
CapacityQueue,
)
@serve.deployment(
deployment_actors=[
DeploymentActorConfig(
name="capacity_queue",
actor_class=CapacityQueue,
init_kwargs={
"acquire_timeout_s": 0.5,
"token_ttl_s": 5,
},
actor_options={"num_cpus": 0},
),
],
request_router_config=RequestRouterConfig(
request_router_class=(
"ray.serve.experimental.capacity_queue_router:CapacityQueueRouter"
),
request_router_kwargs={
"capacity_queue_actor_name": "capacity_queue",
# Fall back to Pow2 after this many consecutive CapacityQueue faults.
"max_fault_retries": 3,
},
# Backoff between retries when the CapacityQueue is unavailable or capacity is exhausted.
initial_backoff_s=0.05,
backoff_multiplier=2.0,
max_backoff_s=1.0,
),
num_replicas=3,
max_ongoing_requests=5,
ray_actor_options={"num_cpus": 0},
)
class CapacityQueueApp:
def __init__(self):
context = _get_internal_replica_context()
self.replica_id = context.replica_id
async def __call__(self):
return self.replica_id
handle = serve.run(CapacityQueueApp.bind())
response = handle.remote().result()
print(f"Response from CapacityQueueApp: {response}")
# __end_deploy_app_with_capacity_queue_router__
doc/source/serve/doc_code/class_based_autoscaling.py
+46
View File
@@ -0,0 +1,46 @@
# __serve_example_begin__
import json
import tempfile
from ray import serve
from ray.serve.config import AutoscalingConfig, AutoscalingPolicy
# Create a JSON file with the initial target replica count.
# In production this file would be written by an external system.
scaling_file = tempfile.NamedTemporaryFile(
mode="w", suffix=".json", delete=False
)
json.dump({"replicas": 2}, scaling_file)
scaling_file.close()
@serve.deployment(
autoscaling_config=AutoscalingConfig(
min_replicas=1,
max_replicas=10,
upscale_delay_s=3,
downscale_delay_s=10,
policy=AutoscalingPolicy(
policy_function="class_based_autoscaling_policy:FileBasedAutoscalingPolicy",
policy_kwargs={
"file_path": scaling_file.name,
"poll_interval_s": 2.0,
},
),
),
max_ongoing_requests=100,
)
class MyDeployment:
async def __call__(self) -> str:
return "Hello, world!"
app = MyDeployment.bind()
# __serve_example_end__
if __name__ == "__main__":
import requests # noqa
serve.run(app)
resp = requests.get("http://localhost:8000/")
assert resp.text == "Hello, world!"
doc/source/serve/doc_code/class_based_autoscaling_policy.py
+57
View File
@@ -0,0 +1,57 @@
# __begin_class_based_autoscaling_policy__
import asyncio
import json
import logging
from pathlib import Path
from typing import Any, Dict, Tuple
from ray.serve.config import AutoscalingContext
logger = logging.getLogger("ray.serve")
class FileBasedAutoscalingPolicy:
"""Scale replicas based on a target written to a JSON file.
A background asyncio task re-reads the file every ``poll_interval_s``
seconds. ``__call__`` returns the latest value on every autoscaling
tick. In production you could replace the file read with an HTTP
call, a message-queue consumer, or any other async IO operation.
"""
def __init__(self, file_path: str, poll_interval_s: float = 5.0):
self._file_path = Path(file_path)
self._poll_interval_s = poll_interval_s
self._desired_replicas: int = 1
self._task: asyncio.Task = None
self._started: bool = False
def _ensure_started(self) -> None:
"""Lazily start the background poll on the controller event loop."""
if self._started:
return
self._started = True
loop = asyncio.get_running_loop()
self._task = loop.create_task(self._poll_file())
async def _poll_file(self) -> None:
"""Read the target replica count from the JSON file in a loop."""
while True:
try:
text = self._file_path.read_text()
data = json.loads(text)
self._desired_replicas = int(data["replicas"])
except Exception:
pass # Keep the last known value on failure.
await asyncio.sleep(self._poll_interval_s)
def __call__(
self, ctx: AutoscalingContext
) -> Tuple[int, Dict[str, Any]]:
self._ensure_started()
desired = self._desired_replicas
return desired, {"last_polled_value": self._desired_replicas}
# __end_class_based_autoscaling_policy__
doc/source/serve/doc_code/configure_serve_deployment/model_deployment.py
+36
View File
@@ -0,0 +1,36 @@
# flake8: noqa
import ray
# __deployment_start__
# File name: configure_serve.py
from ray import serve
@serve.deployment(
name="Translator",
num_replicas=2,
ray_actor_options={"num_cpus": 0.2, "num_gpus": 0},
max_ongoing_requests=100,
health_check_period_s=10,
health_check_timeout_s=30,
graceful_shutdown_timeout_s=20,
graceful_shutdown_wait_loop_s=2,
)
class Example:
...
example_app = Example.bind()
# __deployment_end__
example_app = Example.options(
ray_actor_options={"num_cpus": 0.2, "num_gpus": 0.0}
).bind()
# __options_end__
serve.run(example_app)
serve.shutdown()
ray.shutdown()
doc/source/serve/doc_code/cross_node_parallelism_example.py
+197
View File
@@ -0,0 +1,197 @@
# flake8: noqa
"""
Cross-node parallelism examples for Ray Serve LLM.
TP / PP / custom placement group strategies
for multi-node LLM deployments.
"""
# __cross_node_tp_example_start__
import vllm
from ray import serve
from ray.serve.llm import LLMConfig, build_openai_app
# Configure a model with tensor parallelism across 2 GPUs
# Tensor parallelism splits model weights across GPUs
llm_config = LLMConfig(
model_loading_config=dict(
model_id="llama-3.1-8b",
model_source="meta-llama/Llama-3.1-8B-Instruct",
),
deployment_config=dict(
autoscaling_config=dict(
min_replicas=1,
max_replicas=2,
)
),
accelerator_type="L4",
engine_kwargs=dict(
tensor_parallel_size=2,
max_model_len=8192,
),
)
# Deploy the application
app = build_openai_app({"llm_configs": [llm_config]})
serve.run(app, blocking=True)
# __cross_node_tp_example_end__
# __cross_node_pp_example_start__
from ray import serve
from ray.serve.llm import LLMConfig, build_openai_app
# Configure a model with pipeline parallelism across 2 GPUs
# Pipeline parallelism splits model layers across GPUs
llm_config = LLMConfig(
model_loading_config=dict(
model_id="llama-3.1-8b",
model_source="meta-llama/Llama-3.1-8B-Instruct",
),
deployment_config=dict(
autoscaling_config=dict(
min_replicas=1,
max_replicas=1,
)
),
accelerator_type="L4",
engine_kwargs=dict(
pipeline_parallel_size=2,
max_model_len=8192,
),
)
# Deploy the application
app = build_openai_app({"llm_configs": [llm_config]})
serve.run(app, blocking=True)
# __cross_node_pp_example_end__
# __cross_node_tp_pp_example_start__
from ray import serve
from ray.serve.llm import LLMConfig, build_openai_app
# Configure a model with both tensor and pipeline parallelism
# This example uses 4 GPUs total (2 TP * 2 PP)
llm_config = LLMConfig(
model_loading_config=dict(
model_id="llama-3.1-8b",
model_source="meta-llama/Llama-3.1-8B-Instruct",
),
deployment_config=dict(
autoscaling_config=dict(
min_replicas=1,
max_replicas=1,
)
),
accelerator_type="L4",
engine_kwargs=dict(
tensor_parallel_size=2,
pipeline_parallel_size=2,
max_model_len=8192,
enable_chunked_prefill=True,
max_num_batched_tokens=4096,
),
)
# Deploy the application
app = build_openai_app({"llm_configs": [llm_config]})
serve.run(app, blocking=True)
# __cross_node_tp_pp_example_end__
# __custom_placement_group_pack_example_start__
from ray import serve
from ray.serve.llm import LLMConfig, build_openai_app
# Configure a model with custom placement group using PACK strategy
# PACK tries to place workers on as few nodes as possible for locality
llm_config = LLMConfig(
model_loading_config=dict(
model_id="llama-3.1-8b",
model_source="meta-llama/Llama-3.1-8B-Instruct",
),
deployment_config=dict(
autoscaling_config=dict(
min_replicas=1,
max_replicas=1,
)
),
accelerator_type="L4",
engine_kwargs=dict(
tensor_parallel_size=2,
max_model_len=8192,
),
placement_group_config=dict(
bundles=[{"GPU": 1}] * 2,
strategy="PACK",
),
)
# Deploy the application
app = build_openai_app({"llm_configs": [llm_config]})
serve.run(app, blocking=True)
# __custom_placement_group_pack_example_end__
# __custom_placement_group_spread_example_start__
from ray import serve
from ray.serve.llm import LLMConfig, build_openai_app
# Configure a model with custom placement group using SPREAD strategy
# SPREAD distributes workers across nodes for fault tolerance
llm_config = LLMConfig(
model_loading_config=dict(
model_id="llama-3.1-8b",
model_source="meta-llama/Llama-3.1-8B-Instruct",
),
deployment_config=dict(
autoscaling_config=dict(
min_replicas=1,
max_replicas=1,
)
),
accelerator_type="L4",
engine_kwargs=dict(
tensor_parallel_size=4,
max_model_len=8192,
),
placement_group_config=dict(
bundles=[{"GPU": 1}] * 4,
strategy="SPREAD",
),
)
# Deploy the application
app = build_openai_app({"llm_configs": [llm_config]})
serve.run(app, blocking=True)
# __custom_placement_group_spread_example_end__
# __custom_placement_group_strict_pack_example_start__
from ray import serve
from ray.serve.llm import LLMConfig, build_openai_app
# Configure a model with custom placement group using STRICT_PACK strategy
# STRICT_PACK ensures all workers are placed on the same node
llm_config = LLMConfig(
model_loading_config=dict(
model_id="llama-3.1-8b",
model_source="meta-llama/Llama-3.1-8B-Instruct",
),
deployment_config=dict(
autoscaling_config=dict(
min_replicas=1,
max_replicas=2,
)
),
accelerator_type="A100",
engine_kwargs=dict(
tensor_parallel_size=2,
max_model_len=8192,
),
placement_group_config=dict(
bundles=[{"GPU": 1}] * 2,
strategy="STRICT_PACK",
),
)
# Deploy the application
app = build_openai_app({"llm_configs": [llm_config]})
serve.run(app, blocking=True)
# __custom_placement_group_strict_pack_example_end__
doc/source/serve/doc_code/custom_metrics_autoscaling.py
+54
View File
@@ -0,0 +1,54 @@
# __serve_example_begin__
import time
from typing import Dict
import psutil
from ray import serve
@serve.deployment(
autoscaling_config={
"min_replicas": 1,
"max_replicas": 5,
"metrics_interval_s": 0.1,
"policy": {
"policy_function": "autoscaling_policy:custom_metrics_autoscaling_policy"
},
},
max_ongoing_requests=5,
)
class CustomMetricsDeployment:
def __init__(self):
self.process = psutil.Process()
def __call__(self) -> str:
# Simulate some work
time.sleep(0.5)
return "Hello, world!"
def record_autoscaling_stats(self) -> Dict[str, float]:
# Get CPU usage as a percentage
cpu_usage = self.process.cpu_percent(interval=0.1)
# Get memory usage as a percentage of system memory
memory_info = self.process.memory_full_info()
system_memory = psutil.virtual_memory().total
memory_usage = (memory_info.uss / system_memory) * 100
return {
"cpu_usage": cpu_usage,
"memory_usage": memory_usage,
}
# Create the app
app = CustomMetricsDeployment.bind()
# __serve_example_end__
if __name__ == "__main__":
import requests # noqa
serve.run(app)
for _ in range(10):
resp = requests.get("http://localhost:8000/")
assert resp.text == "Hello, world!"
doc/source/serve/doc_code/custom_request_router.py
+123
View File
@@ -0,0 +1,123 @@
# flake8: noqa
# __begin_define_uniform_request_router__
import random
from ray.serve.request_router import (
PendingRequest,
RequestRouter,
ReplicaID,
ReplicaResult,
RunningReplica,
)
from typing import (
List,
Optional,
)
class UniformRequestRouter(RequestRouter):
async def choose_replicas(
self,
candidate_replicas: List[RunningReplica],
pending_request: Optional[PendingRequest] = None,
) -> List[List[RunningReplica]]:
print("UniformRequestRouter routing request")
index = random.randint(0, len(candidate_replicas) - 1)
return [[candidate_replicas[index]]]
def on_request_routed(
self,
pending_request: PendingRequest,
replica_id: ReplicaID,
result: ReplicaResult,
):
print("on_request_routed callback is called!!")
# __end_define_uniform_request_router__
# __begin_define_throughput_aware_request_router__
from ray.serve.request_router import (
FIFOMixin,
LocalityMixin,
MultiplexMixin,
PendingRequest,
RequestRouter,
ReplicaID,
ReplicaResult,
RunningReplica,
)
from typing import (
Dict,
List,
Optional,
)
class ThroughputAwareRequestRouter(
FIFOMixin, MultiplexMixin, LocalityMixin, RequestRouter
):
async def choose_replicas(
self,
candidate_replicas: List[RunningReplica],
pending_request: Optional[PendingRequest] = None,
) -> List[List[RunningReplica]]:
"""
This method chooses the best replica for the request based on
multiplexed, locality, and custom throughput stats. The algorithm
works as follows:
1. Populate top_ranked_replicas based on available replicas based on
multiplex_id
2. Populate and override top_ranked_replicas info based on locality
information of replicas (we want to prefer replicas that are in the
same vicinity to this deployment)
3. Select the replica with minimum throughput.
"""
# Dictionary to hold the top-ranked replicas
top_ranked_replicas: Dict[ReplicaID, RunningReplica] = {}
# Take the best set of replicas for the multiplexed model
if (
pending_request is not None
and pending_request.metadata.multiplexed_model_id
):
ranked_replicas_multiplex: List[RunningReplica] = (
self.rank_replicas_via_multiplex(
replicas=candidate_replicas,
multiplexed_model_id=pending_request.metadata.multiplexed_model_id,
)
)[0]
# Filter out replicas that are not available (queue length exceed max ongoing request)
ranked_replicas_multiplex = self.select_available_replicas(
candidates=ranked_replicas_multiplex
)
for replica in ranked_replicas_multiplex:
top_ranked_replicas[replica.replica_id] = replica
# Take the best set of replicas in terms of locality
ranked_replicas_locality: List[
RunningReplica
] = self.rank_replicas_via_locality(replicas=candidate_replicas)[0]
# Filter out replicas that are not available (queue length exceed max ongoing request)
ranked_replicas_locality = self.select_available_replicas(
candidates=ranked_replicas_locality
)
for replica in ranked_replicas_locality:
top_ranked_replicas[replica.replica_id] = replica
print("ThroughputAwareRequestRouter routing request")
# Take the replica with minimum throughput.
min_throughput_replicas = min(
[replica for replica in top_ranked_replicas.values()],
key=lambda r: r.routing_stats.get("throughput", 0),
)
return [[min_throughput_replicas]]
# __end_define_throughput_aware_request_router__
doc/source/serve/doc_code/custom_request_router_app.py
+164
View File
@@ -0,0 +1,164 @@
# flake8: noqa
# __begin_deploy_app_with_uniform_request_router__
from ray import serve
from ray.serve.request_router import ReplicaID
import time
from collections import defaultdict
from ray.serve.context import _get_internal_replica_context
from typing import Any, Dict
from ray.serve.config import RequestRouterConfig
@serve.deployment(
request_router_config=RequestRouterConfig(
request_router_class="custom_request_router:UniformRequestRouter",
),
num_replicas=10,
ray_actor_options={"num_cpus": 0},
)
class UniformRequestRouterApp:
def __init__(self):
context = _get_internal_replica_context()
self.replica_id: ReplicaID = context.replica_id
async def __call__(self):
return self.replica_id
handle = serve.run(UniformRequestRouterApp.bind())
response = handle.remote().result()
print(f"Response from UniformRequestRouterApp: {response}")
# Example output:
# Response from UniformRequestRouterApp:
# Replica(id='67vc4ts5', deployment='UniformRequestRouterApp', app='default')
# __end_deploy_app_with_uniform_request_router__
# __begin_deploy_app_with_throughput_aware_request_router__
def _time_ms() -> int:
return int(time.time() * 1000)
@serve.deployment(
request_router_config=RequestRouterConfig(
request_router_class="custom_request_router:ThroughputAwareRequestRouter",
request_routing_stats_period_s=1,
request_routing_stats_timeout_s=1,
),
num_replicas=3,
ray_actor_options={"num_cpus": 0},
)
class ThroughputAwareRequestRouterApp:
def __init__(self):
self.throughput_buckets: Dict[int, int] = defaultdict(int)
self.last_throughput_buckets = _time_ms()
context = _get_internal_replica_context()
self.replica_id: ReplicaID = context.replica_id
def __call__(self):
self.update_throughput()
return self.replica_id
def update_throughput(self):
current_timestamp_ms = _time_ms()
# Under high concurrency, requests can come in at different times. This
# early return helps to skip if the current_timestamp_ms is more than a
# second older than the last throughput bucket.
if current_timestamp_ms < self.last_throughput_buckets - 1000:
return
# Record the request to the bucket
self.throughput_buckets[current_timestamp_ms] += 1
self.last_throughput_buckets = current_timestamp_ms
def record_routing_stats(self) -> Dict[str, Any]:
current_timestamp_ms = _time_ms()
throughput = 0
for t, c in list(self.throughput_buckets.items()):
if t < current_timestamp_ms - 1000:
# Remove the bucket if it is older than 1 second
self.throughput_buckets.pop(t)
else:
throughput += c
return {
"throughput": throughput,
}
handle = serve.run(ThroughputAwareRequestRouterApp.bind())
response = handle.remote().result()
print(f"Response from ThroughputAwareRequestRouterApp: {response}")
# Example output:
# Response from ThroughputAwareRequestRouterApp:
# Replica(id='tkywafya', deployment='ThroughputAwareRequestRouterApp', app='default')
# __end_deploy_app_with_throughput_aware_request_router__
# __begin_deploy_app_with_round_robin_router__
@serve.deployment(
request_router_config=RequestRouterConfig(
request_router_class=(
"ray.serve.experimental.round_robin_router:RoundRobinRouter"
),
),
num_replicas=4,
ray_actor_options={"num_cpus": 0},
)
class RoundRobinRouterApp:
def __init__(self):
context = _get_internal_replica_context()
self.replica_id: ReplicaID = context.replica_id
async def __call__(self):
return self.replica_id
handle = serve.run(RoundRobinRouterApp.bind())
response = handle.remote().result()
print(f"Response from RoundRobinRouterApp: {response}")
# __end_deploy_app_with_round_robin_router__
# __begin_deploy_app_with_consistent_hash_router__
import requests
from starlette.requests import Request
@serve.deployment(
request_router_config=RequestRouterConfig(
request_router_class=(
"ray.serve.experimental.consistent_hash_router:ConsistentHashRouter"
),
request_router_kwargs={
"num_virtual_nodes": 100,
"num_fallback_replicas": 2,
},
),
num_replicas=4,
ray_actor_options={"num_cpus": 0},
)
class ConsistentHashRouterApp:
def __init__(self):
context = _get_internal_replica_context()
self.replica_id: ReplicaID = context.replica_id
async def __call__(self, request: Request) -> str:
return str(self.replica_id)
serve.run(ConsistentHashRouterApp.bind())
# Clients pin a session to a replica by sending the same `x-session-id`
# on every request.
response = requests.get(
"http://localhost:8000/",
headers={"x-session-id": "example-session-id"},
)
print(f"Response from ConsistentHashRouterApp: {response.text}")
# Example output:
# Response from ConsistentHashRouterApp:
# Replica(id='...', deployment='ConsistentHashRouterApp', app='default')
# __end_deploy_app_with_consistent_hash_router__
doc/source/serve/doc_code/delayed_import.py
+9
View File
@@ -0,0 +1,9 @@
from ray import serve
@serve.deployment
class MyDeployment:
def __call__(self, model_path):
from my_module import my_model
self.model = my_model.load(model_path)
doc/source/serve/doc_code/deployment_scoped_actors.py
+88
View File
@@ -0,0 +1,88 @@
import asyncio
import time
import ray
from ray import serve
from ray.exceptions import RayActorError
from ray.serve.config import DeploymentActorConfig
# __begin_define_deployment_scoped_actor__
@ray.remote
class SharedCounter:
def __init__(self, start: int = 0):
self._value = start
def increment(self) -> int:
self._value += 1
return self._value
@serve.deployment(
deployment_actors=[
DeploymentActorConfig(
name="counter",
actor_class=SharedCounter,
init_kwargs={"start": 10},
actor_options={"num_cpus": 0},
),
],
)
class SharedCounterDeployment:
async def __call__(self) -> int:
counter = serve.get_deployment_actor("counter")
return await counter.increment.remote()
# __end_define_deployment_scoped_actor__
# __begin_cached_handle_refresh__
@serve.deployment(
deployment_actors=[
DeploymentActorConfig(
name="counter",
actor_class=SharedCounter,
init_kwargs={"start": 0},
actor_options={"num_cpus": 0},
),
],
)
class CachedHandleDeployment:
def __init__(self):
self._counter = serve.get_deployment_actor("counter")
async def _refresh_counter(self) -> None:
deadline = time.monotonic() + 30
while time.monotonic() < deadline:
try:
self._counter = serve.get_deployment_actor("counter")
return
except ValueError:
# The replacement actor might not be registered yet.
await asyncio.sleep(0.05)
raise TimeoutError("Timed out waiting for the deployment-scoped actor.")
async def __call__(self) -> int:
try:
return await self._counter.increment.remote()
except RayActorError:
await self._refresh_counter()
return await self._counter.increment.remote()
# __end_cached_handle_refresh__
if __name__ == "__main__":
ray.init()
try:
# __begin_run_deployment_scoped_actor_example__
handle = serve.run(SharedCounterDeployment.bind())
print(handle.remote().result())
print(handle.remote().result())
# __end_run_deployment_scoped_actor_example__
finally:
serve.shutdown()
ray.shutdown()
doc/source/serve/doc_code/develop_and_deploy.py
+58
View File
@@ -0,0 +1,58 @@
# flake8: noqa
# __deployment_start__
import ray
from ray import serve
from fastapi import FastAPI
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
app = FastAPI()
@serve.deployment(num_replicas=2, ray_actor_options={"num_cpus": 0.2, "num_gpus": 0})
@serve.ingress(app)
class Translator:
def __init__(self):
# Load model
self.tokenizer = AutoTokenizer.from_pretrained("t5-small")
self.model = AutoModelForSeq2SeqLM.from_pretrained("t5-small")
@app.post("/")
def translate(self, text: str) -> str:
# Run inference
input_ids = self.tokenizer(
f"translate English to French: {text}", return_tensors="pt"
).input_ids
output_ids = self.model.generate(
input_ids, num_beams=4, early_stopping=True, max_length=300
)
# Post-process output to return only the translation text
translation = self.tokenizer.decode(
output_ids[0], skip_special_tokens=True, clean_up_tokenization_spaces=False
)
return translation
translator_app = Translator.bind()
# __deployment_end__
translator_app = Translator.options(ray_actor_options={}).bind()
serve.run(translator_app)
# __client_function_start__
# File name: model_client.py
import requests
response = requests.post("http://127.0.0.1:8000/", params={"text": "Hello world!"})
french_text = response.json()
print(french_text)
# __client_function_end__
assert french_text == "Bonjour monde!"
serve.shutdown()
ray.shutdown()
doc/source/serve/doc_code/distilbert.py
+61
View File
@@ -0,0 +1,61 @@
# __example_code_start__
from transformers import pipeline
from fastapi import FastAPI
import torch
from ray import serve
from ray.serve.handle import DeploymentHandle
app = FastAPI()
@serve.deployment(num_replicas=1)
@serve.ingress(app)
class APIIngress:
def __init__(self, distilbert_model_handle: DeploymentHandle) -> None:
self.handle = distilbert_model_handle
@app.get("/classify")
async def classify(self, sentence: str):
return await self.handle.classify.remote(sentence)
@serve.deployment(
ray_actor_options={"num_gpus": 1},
autoscaling_config={"min_replicas": 0, "max_replicas": 2},
)
class DistilBertModel:
def __init__(self):
self.classifier = pipeline(
"sentiment-analysis",
model="distilbert-base-uncased",
framework="pt",
# Transformers requires you to pass device with index
device=torch.device("cuda:0"),
)
def classify(self, sentence: str):
return self.classifier(sentence)
entrypoint = APIIngress.bind(DistilBertModel.bind())
# __example_code_end__
if __name__ == "__main__":
import requests
import ray
ray.init(runtime_env={"pip": ["transformers==4.36.2", "accelerate==0.28.0"]})
serve.run(entrypoint)
prompt = (
"This was a masterpiece. Not completely faithful to the books, but "
"enthralling from beginning to end. Might be my favorite of the three."
)
input = "%20".join(prompt.split(" "))
resp = requests.get(f"http://127.0.0.1:8000/classify?sentence={prompt}")
print(resp.status_code, resp.json())
assert resp.status_code == 200
doc/source/serve/doc_code/external_scaler_config.yaml
+10
View File
@@ -0,0 +1,10 @@
# __external_scaler_config_begin__
applications:
- name: my-app
import_path: external_scaler_predictive:app
external_scaler_enabled: true
deployments:
- name: TextProcessor
num_replicas: 1
# __external_scaler_config_end__
doc/source/serve/doc_code/external_scaler_predictive.py
+35
View File
@@ -0,0 +1,35 @@
# __serve_example_begin__
import time
from ray import serve
from typing import Any
@serve.deployment(num_replicas=3)
class TextProcessor:
"""A simple text processing deployment that can be scaled externally."""
def __init__(self):
self.request_count = 0
def __call__(self, text: Any) -> dict:
# Simulate text processing work
time.sleep(0.1)
self.request_count += 1
return {
"request_count": self.request_count,
}
app = TextProcessor.bind()
# __serve_example_end__
def main():
import requests
serve.run(app)
# Test the deployment
resp = requests.post(
"http://localhost:8000/",
json="hello world"
)
print(f"Response: {resp.json()}")
doc/source/serve/doc_code/external_scaler_predictive_client.py
+82
View File
@@ -0,0 +1,82 @@
# __client_script_begin__
import logging
import time
from datetime import datetime
import requests
APPLICATION_NAME = "my-app"
DEPLOYMENT_NAME = "TextProcessor"
SERVE_ENDPOINT = "http://localhost:8265"
SCALING_INTERVAL = 300 # Check every 5 minutes
logger = logging.getLogger(__name__)
def get_current_replicas(app_name: str, deployment_name: str) -> int:
"""Get current replica count. Returns -1 on error."""
try:
resp = requests.get(
f"{SERVE_ENDPOINT}/api/serve/applications/",
timeout=10
)
if resp.status_code != 200:
logger.error(f"Failed to get applications: {resp.status_code}")
return -1
apps = resp.json().get("applications", {})
if app_name not in apps:
logger.error(f"Application {app_name} not found")
return -1
deployments = apps[app_name].get("deployments", {})
if deployment_name in deployments:
return deployments[deployment_name]["target_num_replicas"]
logger.error(f"Deployment {deployment_name} not found")
return -1
except requests.exceptions.RequestException as e:
logger.error(f"Request failed: {e}")
return -1
def scale_deployment(app_name: str, deployment_name: str):
"""Scale deployment based on time of day."""
hour = datetime.now().hour
current = get_current_replicas(app_name, deployment_name)
# Check if we successfully retrieved the current replica count
if current == -1:
logger.error("Failed to get current replicas, skipping scaling decision")
return
target = 10 if 9 <= hour < 17 else 3 # Peak hours: 9am-5pm
delta = target - current
if delta == 0:
logger.info(f"Already at target ({current} replicas)")
return
action = "Adding" if delta > 0 else "Removing"
logger.info(f"{action} {abs(delta)} replicas ({current} -> {target})")
try:
resp = requests.post(
f"{SERVE_ENDPOINT}/api/v1/applications/{app_name}/deployments/{deployment_name}/scale",
headers={"Content-Type": "application/json"},
json={"target_num_replicas": target},
timeout=10
)
if resp.status_code == 200:
logger.info("Successfully scaled deployment")
else:
logger.error(f"Scale failed: {resp.status_code} - {resp.text}")
except requests.exceptions.RequestException as e:
logger.error(f"Request failed: {e}")
def main():
logger.info(f"Starting predictive scaling for {APPLICATION_NAME}/{DEPLOYMENT_NAME}")
while True:
scale_deployment(APPLICATION_NAME, DEPLOYMENT_NAME)
time.sleep(SCALING_INTERVAL)
# __client_script_end__
doc/source/serve/doc_code/fake_email_creator.py
+16
View File
@@ -0,0 +1,16 @@
# __fake_start__
from faker import Faker
from ray import serve
@serve.deployment
def create_fake_email():
return Faker().email()
app = create_fake_email.bind()
# __fake_end__
handle = serve.run(app)
assert handle.remote().result() == "fake@fake.com"
doc/source/serve/doc_code/fake_email_creator.yaml
+61
View File
@@ -0,0 +1,61 @@
# __fake_config_start__
apiVersion: ray.io/v1alpha1
kind: RayService
metadata:
name: rayservice-fake-emails
spec:
serviceUnhealthySecondThreshold: 300
deploymentUnhealthySecondThreshold: 300
serveConfigV2: |
applications:
- name: fake
import_path: fake:app
route_prefix: /
rayClusterConfig:
rayVersion: '2.5.0' # Should match Ray version in the containers
headGroupSpec:
rayStartParams:
dashboard-host: '0.0.0.0'
template:
spec:
containers:
- name: ray-head
image: shrekrisanyscale/serve-fake-email-example:example
resources:
limits:
cpu: 2
memory: 2Gi
requests:
cpu: 2
memory: 2Gi
ports:
- containerPort: 6379
name: gcs-server
- containerPort: 8265 # Ray dashboard
name: dashboard
- containerPort: 10001
name: client
- containerPort: 8000
name: serve
workerGroupSpecs:
- replicas: 1
minReplicas: 1
maxReplicas: 1
groupName: small-group
template:
spec:
containers:
- name: ray-worker
image: shrekrisanyscale/serve-fake-email-example:example
lifecycle:
preStop:
exec:
command: ["/bin/sh","-c","ray stop"]
resources:
limits:
cpu: "1"
memory: "2Gi"
requests:
cpu: "500m"
memory: "2Gi"
# __fake_config_end__
doc/source/serve/doc_code/faker.py
+8
View File
@@ -0,0 +1,8 @@
class Faker:
"""Mock Faker class to test fake_email_creator.py.
Meant to mock https://github.com/joke2k/faker package.
"""
def email(self) -> str:
return "fake@fake.com"
doc/source/serve/doc_code/fastapi_example.py
+23
View File
@@ -0,0 +1,23 @@
import requests
from fastapi import FastAPI
from ray import serve
# 1: Define a FastAPI app and wrap it in a deployment with a route handler.
app = FastAPI()
@serve.deployment
@serve.ingress(app)
class FastAPIDeployment:
# FastAPI will automatically parse the HTTP request for us.
@app.get("/hello")
def say_hello(self, name: str) -> str:
return f"Hello {name}!"
# 2: Deploy the deployment.
serve.run(FastAPIDeployment.bind(), route_prefix="/")
# 3: Query the deployment and print the result.
print(requests.get("http://localhost:8000/hello", params={"name": "Theodore"}).json())
# "Hello Theodore!"
doc/source/serve/doc_code/fault_tolerance/k8s_config.yaml
+178
View File
@@ -0,0 +1,178 @@
# File name: config.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: redis-config
labels:
app: redis
data:
redis.conf: |-
port 6379
bind 0.0.0.0
protected-mode no
requirepass 5241590000000000
---
apiVersion: v1
kind: Service
metadata:
name: redis
labels:
app: redis
spec:
type: ClusterIP
ports:
- name: redis
port: 6379
selector:
app: redis
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis
labels:
app: redis
spec:
replicas: 1
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
spec:
containers:
- name: redis
image: redis:5.0.8
command:
- "sh"
- "-c"
- "redis-server /usr/local/etc/redis/redis.conf"
ports:
- containerPort: 6379
volumeMounts:
- name: config
mountPath: /usr/local/etc/redis/redis.conf
subPath: redis.conf
volumes:
- name: config
configMap:
name: redis-config
---
apiVersion: ray.io/v1alpha1
kind: RayService
metadata:
name: rayservice-sample
annotations:
ray.io/ft-enabled: "true"
spec:
serviceUnhealthySecondThreshold: 300
deploymentUnhealthySecondThreshold: 300
serveConfig:
importPath: "sleepy_pid:app"
runtimeEnv: |
working_dir: "https://github.com/ray-project/serve_config_examples/archive/42d10bab77741b40d11304ad66d39a4ec2345247.zip"
deployments:
- name: SleepyPid
numReplicas: 6
rayActorOptions:
numCpus: 0
rayClusterConfig:
rayVersion: '2.3.0'
headGroupSpec:
replicas: 1
rayStartParams:
num-cpus: '2'
dashboard-host: '0.0.0.0'
redis-password: "5241590000000000"
template:
spec:
containers:
- name: ray-head
image: rayproject/ray:2.3.0
imagePullPolicy: Always
env:
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: RAY_REDIS_ADDRESS
value: redis:6379
resources:
limits:
cpu: 2
memory: 2Gi
requests:
cpu: 2
memory: 2Gi
ports:
- containerPort: 6379
name: redis
- containerPort: 8265
name: dashboard
- containerPort: 10001
name: client
- containerPort: 8000
name: serve
workerGroupSpecs:
- replicas: 2
minReplicas: 2
maxReplicas: 2
groupName: small-group
rayStartParams:
node-ip-address: $MY_POD_IP
template:
spec:
containers:
- name: machine-learning
image: rayproject/ray:2.3.0
imagePullPolicy: Always
env:
- name: RAY_DISABLE_DOCKER_CPU_WARNING
value: "1"
- name: TYPE
value: "worker"
- name: CPU_REQUEST
valueFrom:
resourceFieldRef:
containerName: machine-learning
resource: requests.cpu
- name: CPU_LIMITS
valueFrom:
resourceFieldRef:
containerName: machine-learning
resource: limits.cpu
- name: MEMORY_LIMITS
valueFrom:
resourceFieldRef:
containerName: machine-learning
resource: limits.memory
- name: MEMORY_REQUESTS
valueFrom:
resourceFieldRef:
containerName: machine-learning
resource: requests.memory
- name: MY_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
ports:
- containerPort: 80
name: client
lifecycle:
preStop:
exec:
command: ["/bin/sh","-c","ray stop"]
resources:
limits:
cpu: "1"
memory: "2Gi"
requests:
cpu: "500m"
memory: "2Gi"
doc/source/serve/doc_code/fault_tolerance/replica_health_check.py
+24
View File
@@ -0,0 +1,24 @@
# flake8: noqa
from ray import serve
# Stubs
def connect_to_db(*args, **kwargs):
pass
# __health_check_start__
@serve.deployment(health_check_period_s=10, health_check_timeout_s=30)
class MyDeployment:
def __init__(self, db_addr: str):
self._my_db_connection = connect_to_db(db_addr)
def __call__(self, request):
return self._do_something_cool()
# Called by Serve to check the replica's health.
def check_health(self):
if not self._my_db_connection.is_connected():
# The specific type of exception is not important.
raise RuntimeError("uh-oh, DB connection is broken.")
# __health_check_end__
doc/source/serve/doc_code/fault_tolerance/sleepy_pid.py
+23
View File
@@ -0,0 +1,23 @@
# flake8: noqa
# __start__
# File name: sleepy_pid.py
from ray import serve
@serve.deployment
class SleepyPid:
def __init__(self):
import time
time.sleep(10)
def __call__(self) -> int:
import os
return os.getpid()
app = SleepyPid.bind()
# __end__
doc/source/serve/doc_code/gang_scheduling.py
+183
View File
@@ -0,0 +1,183 @@
from ray import serve
# __basic_gang_start__
from ray import serve
from ray.serve.config import GangSchedulingConfig
@serve.deployment(
num_replicas=8,
ray_actor_options={"num_cpus": 0.25},
gang_scheduling_config=GangSchedulingConfig(gang_size=4),
)
class Gang:
def __call__(self, request):
return "Hello!"
app = Gang.bind()
# __basic_gang_end__
# __gang_context_start__
@serve.deployment(
num_replicas=4,
ray_actor_options={"num_cpus": 0.25},
gang_scheduling_config=GangSchedulingConfig(gang_size=2),
)
class GangWithContext:
def __init__(self):
ctx = serve.get_replica_context()
gc = ctx.gang_context
self.rank = gc.rank
self.world_size = gc.world_size
self.gang_id = gc.gang_id
self.member_ids = gc.member_replica_ids
def __call__(self, request):
return {
"gang_id": self.gang_id,
"rank": self.rank,
"world_size": self.world_size,
}
gang_context_app = GangWithContext.bind()
# __gang_context_end__
# __pack_strategy_start__
from ray import serve
from ray.serve.config import GangPlacementStrategy, GangSchedulingConfig
@serve.deployment(
num_replicas=4,
ray_actor_options={"num_cpus": 0.25},
gang_scheduling_config=GangSchedulingConfig(
gang_size=4,
gang_placement_strategy=GangPlacementStrategy.PACK,
),
)
class PackedGang:
def __call__(self, request):
return "Packed on same node"
packed_app = PackedGang.bind()
# __pack_strategy_end__
# __spread_strategy_start__
@serve.deployment(
num_replicas=4,
ray_actor_options={"num_cpus": 0.25},
gang_scheduling_config=GangSchedulingConfig(
gang_size=2,
gang_placement_strategy=GangPlacementStrategy.SPREAD,
),
)
class SpreadGang:
def __call__(self, request):
return "Spread across nodes"
spread_app = SpreadGang.bind()
# __spread_strategy_end__
# __options_start__
@serve.deployment
class BaseGang:
def __call__(self, request):
return "Hello!"
app_with_gang = BaseGang.options(
num_replicas=8,
ray_actor_options={"num_cpus": 0.25},
gang_scheduling_config=GangSchedulingConfig(gang_size=4),
).bind()
# __options_end__
# __autoscaling_start__
@serve.deployment(
autoscaling_config={
"min_replicas": 4,
"max_replicas": 16,
"initial_replicas": 8,
"target_ongoing_requests": 5,
},
ray_actor_options={"num_cpus": 0.25},
gang_scheduling_config=GangSchedulingConfig(gang_size=4),
)
class AutoscaledGang:
def __call__(self, request):
return "Hello!"
autoscaled_app = AutoscaledGang.bind()
# __autoscaling_end__
# __fault_tolerance_start__
from ray import serve
from ray.serve.config import GangRuntimeFailurePolicy, GangSchedulingConfig
@serve.deployment(
num_replicas=8,
ray_actor_options={"num_cpus": 0.25},
gang_scheduling_config=GangSchedulingConfig(
gang_size=4,
runtime_failure_policy=GangRuntimeFailurePolicy.RESTART_GANG,
),
)
class FaultTolerantGang:
def __call__(self, request):
return "Hello!"
fault_tolerant_app = FaultTolerantGang.bind()
# __fault_tolerance_end__
# __placement_group_bundles_start__
@serve.deployment(
num_replicas=4,
ray_actor_options={"num_cpus": 0},
placement_group_bundles=[{"CPU": 1, "GPU": 1}],
gang_scheduling_config=GangSchedulingConfig(gang_size=2),
)
class GangWithSingleBundleReplica:
def __call__(self, request):
return "Running on reserved GPUs"
gang_single_bundle_replica_app = GangWithSingleBundleReplica.bind()
# __placement_group_bundles_end__
# __multi_placement_group_bundles_start__
@serve.deployment(
num_replicas=4,
ray_actor_options={"num_cpus": 1},
placement_group_bundles=[{"CPU": 1, "GPU": 1}, {"GPU": 1}],
gang_scheduling_config=GangSchedulingConfig(gang_size=2),
)
class GangWithMultiBundlesReplica:
def __call__(self, request):
return "Running on reserved GPUs"
gang_multi_bundles_replica_app = GangWithMultiBundlesReplica.bind()
# __multi_placement_group_bundles_end__
# __label_selector_start__
@serve.deployment(
num_replicas=4,
ray_actor_options={"num_cpus": 0},
placement_group_bundles=[{"CPU": 1, "GPU": 1}],
placement_group_bundle_label_selector=[{"ray.io/accelerator-type": "A100"}],
gang_scheduling_config=GangSchedulingConfig(gang_size=2),
)
class GangOnA100:
def __call__(self, request):
return "Running on A100"
gang_a100_app = GangOnA100.bind()
# __label_selector_end__
doc/source/serve/doc_code/getting_started/model_deployment.py
+67
View File
@@ -0,0 +1,67 @@
# flake8: noqa
# __import_start__
from starlette.requests import Request
import ray
from ray import serve
# __import_end__
# __model_start__
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
@serve.deployment(num_replicas=2, ray_actor_options={"num_cpus": 0.2, "num_gpus": 0})
class Translator:
def __init__(self):
# Load model
self.tokenizer = AutoTokenizer.from_pretrained("t5-small")
self.model = AutoModelForSeq2SeqLM.from_pretrained("t5-small")
def translate(self, text: str) -> str:
# Run inference
input_ids = self.tokenizer(
f"translate English to French: {text}", return_tensors="pt"
).input_ids
output_ids = self.model.generate(
input_ids, num_beams=4, early_stopping=True, max_length=300
)
# Post-process output to return only the translation text
translation = self.tokenizer.decode(
output_ids[0], skip_special_tokens=True, clean_up_tokenization_spaces=False
)
return translation
async def __call__(self, http_request: Request) -> str:
english_text: str = await http_request.json()
return self.translate(english_text)
# __model_end__
# __model_deploy_start__
translator_app = Translator.bind()
# __model_deploy_end__
translator_app = Translator.options(ray_actor_options={}).bind()
serve.run(translator_app)
# __client_function_start__
# File name: model_client.py
import requests
english_text = "Hello world!"
response = requests.post("http://127.0.0.1:8000/", json=english_text)
french_text = response.text
print(french_text)
# __client_function_end__
assert french_text == "Bonjour monde!"
serve.shutdown()
ray.shutdown()
doc/source/serve/doc_code/getting_started/model_deployment_full.py
+53
View File
@@ -0,0 +1,53 @@
# flake8: noqa
# __deployment_full_start__
# File name: serve_quickstart.py
from starlette.requests import Request
import ray
from ray import serve
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
@serve.deployment(num_replicas=2, ray_actor_options={"num_cpus": 0.2, "num_gpus": 0})
class Translator:
def __init__(self):
# Load model
self.tokenizer = AutoTokenizer.from_pretrained("t5-small")
self.model = AutoModelForSeq2SeqLM.from_pretrained("t5-small")
def translate(self, text: str) -> str:
# Run inference
input_ids = self.tokenizer(
f"translate English to French: {text}", return_tensors="pt"
).input_ids
output_ids = self.model.generate(
input_ids, num_beams=4, early_stopping=True, max_length=300
)
# Post-process output to return only the translation text
translation = self.tokenizer.decode(
output_ids[0], skip_special_tokens=True, clean_up_tokenization_spaces=False
)
return translation
async def __call__(self, http_request: Request) -> str:
english_text: str = await http_request.json()
return self.translate(english_text)
translator_app = Translator.bind()
# __deployment_full_end__
translator_app = Translator.options(ray_actor_options={}).bind()
serve.run(translator_app)
import requests
response = requests.post("http://127.0.0.1:8000/", json="Hello world!").text
assert response == "Bonjour monde!"
serve.shutdown()
ray.shutdown()
doc/source/serve/doc_code/getting_started/models.py
+83
View File
@@ -0,0 +1,83 @@
# flake8: noqa
# __start_translation_model__
# File name: model.py
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
class Translator:
def __init__(self):
# Load model
self.tokenizer = AutoTokenizer.from_pretrained("t5-small")
self.model = AutoModelForSeq2SeqLM.from_pretrained("t5-small")
def translate(self, text: str) -> str:
# Run inference
input_ids = self.tokenizer(
f"translate English to French: {text}", return_tensors="pt"
).input_ids
output_ids = self.model.generate(
input_ids, num_beams=4, early_stopping=True, max_length=300
)
# Post-process output to return only the translation text
translation = self.tokenizer.decode(
output_ids[0], skip_special_tokens=True, clean_up_tokenization_spaces=False
)
return translation
translator = Translator()
translation = translator.translate("Hello world!")
print(translation)
# __end_translation_model__
# Test model behavior
assert translation == "Bonjour monde!"
# __start_summarization_model__
# File name: summary_model.py
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
class Summarizer:
def __init__(self):
# Load model
self.tokenizer = AutoTokenizer.from_pretrained("t5-small")
self.model = AutoModelForSeq2SeqLM.from_pretrained("t5-small")
def summarize(self, text: str) -> str:
# Run inference
input_ids = self.tokenizer(f"summarize: {text}", return_tensors="pt").input_ids
output_ids = self.model.generate(
input_ids,
num_beams=4,
early_stopping=True,
length_penalty=2.0,
no_repeat_ngram_size=3,
min_length=5,
max_length=15,
)
# Post-process output to return only the summary text
summary = self.tokenizer.decode(
output_ids[0], skip_special_tokens=True, clean_up_tokenization_spaces=False
)
return summary
summarizer = Summarizer()
summary = summarizer.summarize(
"It was the best of times, it was the worst of times, it was the age "
"of wisdom, it was the age of foolishness, it was the epoch of belief"
)
print(summary)
# __end_summarization_model__
# Test model behavior
assert summary == "it was the best of times, it was worst of times ."
doc/source/serve/doc_code/getting_started/translator.py
+91
View File
@@ -0,0 +1,91 @@
# flake8: noqa
# __start_graph__
# File name: serve_quickstart_composed.py
from starlette.requests import Request
import ray
from ray import serve
from ray.serve.handle import DeploymentHandle
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
@serve.deployment
class Translator:
def __init__(self):
# Load model
self.tokenizer = AutoTokenizer.from_pretrained("t5-small")
self.model = AutoModelForSeq2SeqLM.from_pretrained("t5-small")
def translate(self, text: str) -> str:
# Run inference
input_ids = self.tokenizer(
f"translate English to French: {text}", return_tensors="pt"
).input_ids
output_ids = self.model.generate(
input_ids, num_beams=4, early_stopping=True, max_length=300
)
# Post-process output to return only the translation text
translation = self.tokenizer.decode(
output_ids[0], skip_special_tokens=True, clean_up_tokenization_spaces=False
)
return translation
@serve.deployment
class Summarizer:
def __init__(self, translator: DeploymentHandle):
self.translator = translator
# Load model.
self.tokenizer = AutoTokenizer.from_pretrained("t5-small")
self.model = AutoModelForSeq2SeqLM.from_pretrained("t5-small")
def summarize(self, text: str) -> str:
# Run inference
input_ids = self.tokenizer(f"summarize: {text}", return_tensors="pt").input_ids
output_ids = self.model.generate(
input_ids, num_beams=4, early_stopping=True, max_length=15
)
# Post-process output to return only the summary text
summary = self.tokenizer.decode(
output_ids[0], skip_special_tokens=True, clean_up_tokenization_spaces=False
)
return summary
async def __call__(self, http_request: Request) -> str:
english_text: str = await http_request.json()
summary = self.summarize(english_text)
translation = await self.translator.translate.remote(summary)
return translation
app = Summarizer.bind(Translator.bind())
# __end_graph__
serve.run(app)
# __start_client__
# File name: composed_client.py
import requests
english_text = (
"It was the best of times, it was the worst of times, it was the age "
"of wisdom, it was the age of foolishness, it was the epoch of belief"
)
response = requests.post("http://127.0.0.1:8000/", json=english_text)
french_text = response.text
print(french_text)
# __end_client__
assert french_text == "C'était le meilleur des temps, c'était le pire des temps,"
serve.shutdown()
ray.shutdown()
doc/source/serve/doc_code/gradio-integration-parallel.py
+74
View File
@@ -0,0 +1,74 @@
import requests
# __doc_import_begin__
from ray import serve
from ray.serve.handle import DeploymentHandle
from ray.serve.gradio_integrations import GradioIngress
import gradio as gr
import asyncio
from transformers import pipeline
# __doc_import_end__
# __doc_models_begin__
@serve.deployment
class TextGenerationModel:
def __init__(self, model_name):
self.generator = pipeline("text-generation", model=model_name)
def __call__(self, text):
generated_list = self.generator(
text, do_sample=True, min_length=20, max_length=100
)
generated = generated_list[0]["generated_text"]
return generated
app1 = TextGenerationModel.bind("gpt2")
app2 = TextGenerationModel.bind("distilgpt2")
# __doc_models_end__
# __doc_gradio_server_begin__
@serve.deployment
class MyGradioServer(GradioIngress):
def __init__(
self, downstream_model_1: DeploymentHandle, downstream_model_2: DeploymentHandle
):
self._d1 = downstream_model_1
self._d2 = downstream_model_2
super().__init__(
lambda: gr.Interface(
self.fanout, "textbox", "textbox", api_name="predict"
)
)
async def fanout(self, text):
[result1, result2] = await asyncio.gather(
self._d1.remote(text), self._d2.remote(text)
)
return (
f"[Generated text version 1]\n{result1}\n\n"
f"[Generated text version 2]\n{result2}"
)
# __doc_gradio_server_end__
# __doc_app_begin__
app = MyGradioServer.bind(app1, app2)
# __doc_app_end__
# Test example code
serve.run(app)
response = requests.post(
"http://127.0.0.1:8000/gradio_api/run/predict/", json={"data": ["My name is Lewis"]}
)
assert response.status_code == 200
print(
"gradio-integration-parallel.py: Response from example code is",
response.json()["data"],
)
doc/source/serve/doc_code/gradio-integration.py
+70
View File
@@ -0,0 +1,70 @@
import requests
from ray import serve
# __doc_import_begin__
from ray.serve.gradio_integrations import GradioServer
import gradio as gr
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
# __doc_import_end__
# __doc_gradio_app_begin__
example_input = (
"HOUSTON -- Men have landed and walked on the moon. "
"Two Americans, astronauts of Apollo 11, steered their fragile "
"four-legged lunar module safely and smoothly to the historic landing "
"yesterday at 4:17:40 P.M., Eastern daylight time. Neil A. Armstrong, the "
"38-year-old commander, radioed to earth and the mission control room "
'here: "Houston, Tranquility Base here. The Eagle has landed." The '
"first men to reach the moon -- Armstrong and his co-pilot, Col. Edwin E. "
"Aldrin Jr. of the Air Force -- brought their ship to rest on a level, "
"rock-strewn plain near the southwestern shore of the arid Sea of "
"Tranquility. About six and a half hours later, Armstrong opened the "
"landing craft's hatch, stepped slowly down the ladder and declared as "
"he planted the first human footprint on the lunar crust: \"That's one "
'small step for man, one giant leap for mankind." His first step on the '
"moon came at 10:56:20 P.M., as a television camera outside the craft "
"transmitted his every move to an awed and excited audience of hundreds "
"of millions of people on earth."
)
def gradio_summarizer_builder():
tokenizer = AutoTokenizer.from_pretrained("t5-small")
summarizer_model = AutoModelForSeq2SeqLM.from_pretrained("t5-small")
def model(text):
input_ids = tokenizer(f"summarize: {text}", return_tensors="pt").input_ids
output_ids = summarizer_model.generate(
input_ids, num_beams=4, early_stopping=True, max_length=200
)
return tokenizer.decode(
output_ids[0], skip_special_tokens=True, clean_up_tokenization_spaces=False
)
return gr.Interface(
fn=model,
inputs=[gr.Textbox(value=example_input, label="Input prompt")],
outputs=[gr.Textbox(label="Model output")],
api_name="predict",
)
# __doc_gradio_app_end__
# __doc_app_begin__
app = GradioServer.options(ray_actor_options={"num_cpus": 4}).bind(
gradio_summarizer_builder
)
# __doc_app_end__
# Test example code
serve.run(app)
response = requests.post(
"http://127.0.0.1:8000/gradio_api/run/predict/", json={"data": [example_input]}
)
assert response.status_code == 200
print("gradio-integration.py: Response from example code is", response.json()["data"])
serve.shutdown()
doc/source/serve/doc_code/gradio-original.py
+36
View File
@@ -0,0 +1,36 @@
import gradio as gr
from transformers import pipeline
import requests
# __doc_code_begin__
generator1 = pipeline("text-generation", model="gpt2")
generator2 = pipeline("text-generation", model="distilgpt2")
def model1(text):
generated_list = generator1(text, do_sample=True, min_length=20, max_length=100)
generated = generated_list[0]["generated_text"]
return generated
def model2(text):
generated_list = generator2(text, do_sample=True, min_length=20, max_length=100)
generated = generated_list[0]["generated_text"]
return generated
demo = gr.Interface(
lambda text: f"{model1(text)}\n------------\n{model2(text)}",
"textbox",
"textbox",
api_name="predict",
)
# __doc_code_end__
# Test example code
demo.launch(prevent_thread_lock=True)
response = requests.post(
"http://127.0.0.1:7860/gradio_api/run/predict/", json={"data": ["My name is Lewis"]}
)
assert response.status_code == 200
print("gradio-original.py: Response from example code is", response.json()["data"])
doc/source/serve/doc_code/grpc_proxy/grpc_guide.py
+513
View File
@@ -0,0 +1,513 @@
# flake8: noqa
import ray
ray.init()
# __begin_start_grpc_proxy__
from ray import serve
from ray.serve.config import gRPCOptions
grpc_port = 9000
grpc_servicer_functions = [
"user_defined_protos_pb2_grpc.add_UserDefinedServiceServicer_to_server",
"user_defined_protos_pb2_grpc.add_ImageClassificationServiceServicer_to_server",
]
serve.start(
grpc_options=gRPCOptions(
port=grpc_port,
grpc_servicer_functions=grpc_servicer_functions,
),
)
# __end_start_grpc_proxy__
# __begin_grpc_deployment__
import time
from typing import Generator
from user_defined_protos_pb2 import (
UserDefinedMessage,
UserDefinedMessage2,
UserDefinedResponse,
UserDefinedResponse2,
)
import ray
from ray import serve
@serve.deployment
class GrpcDeployment:
def __call__(self, user_message: UserDefinedMessage) -> UserDefinedResponse:
greeting = f"Hello {user_message.name} from {user_message.origin}"
num = user_message.num * 2
user_response = UserDefinedResponse(
greeting=greeting,
num=num,
)
return user_response
@serve.multiplexed(max_num_models_per_replica=1)
async def get_model(self, model_id: str) -> str:
return f"loading model: {model_id}"
async def Multiplexing(
self, user_message: UserDefinedMessage2
) -> UserDefinedResponse2:
model_id = serve.get_multiplexed_model_id()
model = await self.get_model(model_id)
user_response = UserDefinedResponse2(
greeting=f"Method2 called model, {model}",
)
return user_response
def Streaming(
self, user_message: UserDefinedMessage
) -> Generator[UserDefinedResponse, None, None]:
for i in range(10):
greeting = f"{i}: Hello {user_message.name} from {user_message.origin}"
num = user_message.num * 2 + i
user_response = UserDefinedResponse(
greeting=greeting,
num=num,
)
yield user_response
time.sleep(0.1)
g = GrpcDeployment.bind()
# __end_grpc_deployment__
# __begin_deploy_grpc_app__
app1 = "app1"
serve.run(target=g, name=app1, route_prefix=f"/{app1}")
# __end_deploy_grpc_app__
# __begin_send_grpc_requests__
import grpc
from user_defined_protos_pb2_grpc import UserDefinedServiceStub
from user_defined_protos_pb2 import UserDefinedMessage
channel = grpc.insecure_channel("localhost:9000")
stub = UserDefinedServiceStub(channel)
request = UserDefinedMessage(name="foo", num=30, origin="bar")
response, call = stub.__call__.with_call(request=request)
print(f"status code: {call.code()}") # grpc.StatusCode.OK
print(f"greeting: {response.greeting}") # "Hello foo from bar"
print(f"num: {response.num}") # 60
# __end_send_grpc_requests__
# __begin_health_check__
import grpc
from ray.serve.generated.serve_pb2_grpc import RayServeAPIServiceStub
from ray.serve.generated.serve_pb2 import HealthzRequest, ListApplicationsRequest
channel = grpc.insecure_channel("localhost:9000")
stub = RayServeAPIServiceStub(channel)
request = ListApplicationsRequest()
response = stub.ListApplications(request=request)
print(f"Applications: {response.application_names}") # ["app1"]
request = HealthzRequest()
response = stub.Healthz(request=request)
print(f"Health: {response.message}") # "success"
# __end_health_check__
# __begin_metadata__
import grpc
from user_defined_protos_pb2_grpc import UserDefinedServiceStub
from user_defined_protos_pb2 import UserDefinedMessage2
channel = grpc.insecure_channel("localhost:9000")
stub = UserDefinedServiceStub(channel)
request = UserDefinedMessage2()
app_name = "app1"
request_id = "123"
multiplexed_model_id = "999"
metadata = (
("application", app_name),
("request_id", request_id),
("multiplexed_model_id", multiplexed_model_id),
)
response, call = stub.Multiplexing.with_call(request=request, metadata=metadata)
print(f"greeting: {response.greeting}") # "Method2 called model, loading model: 999"
for key, value in call.trailing_metadata():
print(f"trailing metadata key: {key}, value {value}") # "request_id: 123"
# __end_metadata__
# __begin_streaming__
import grpc
from user_defined_protos_pb2_grpc import UserDefinedServiceStub
from user_defined_protos_pb2 import UserDefinedMessage
channel = grpc.insecure_channel("localhost:9000")
stub = UserDefinedServiceStub(channel)
request = UserDefinedMessage(name="foo", num=30, origin="bar")
metadata = (("application", "app1"),)
responses = stub.Streaming(request=request, metadata=metadata)
for response in responses:
print(f"greeting: {response.greeting}") # greeting: n: Hello foo from bar
print(f"num: {response.num}") # num: 60 + n
# __end_streaming__
# __begin_model_composition_deployment__
import requests
import torch
from typing import List
from PIL import Image
from io import BytesIO
from torchvision import transforms
from torchvision.models import resnet18, ResNet18_Weights
from user_defined_protos_pb2 import (
ImageClass,
ImageData,
)
from ray import serve
from ray.serve.handle import DeploymentHandle
@serve.deployment
class ImageClassifier:
def __init__(
self,
_image_downloader: DeploymentHandle,
_data_preprocessor: DeploymentHandle,
):
self._image_downloader = _image_downloader
self._data_preprocessor = _data_preprocessor
self.model = resnet18(weights=ResNet18_Weights.DEFAULT)
self.model.eval()
self.categories = self._image_labels()
def _image_labels(self) -> List[str]:
categories = []
url = (
"https://raw.githubusercontent.com/pytorch/hub/master/imagenet_classes.txt"
)
labels = requests.get(url).text
for label in labels.split("\n"):
categories.append(label.strip())
return categories
async def Predict(self, image_data: ImageData) -> ImageClass:
# Download image
image = await self._image_downloader.remote(image_data.url)
# Preprocess image
input_batch = await self._data_preprocessor.remote(image)
# Predict image
with torch.no_grad():
output = self.model(input_batch)
probabilities = torch.nn.functional.softmax(output[0], dim=0)
return self.process_model_outputs(probabilities)
def process_model_outputs(self, probabilities: torch.Tensor) -> ImageClass:
image_classes = []
image_probabilities = []
# Show top categories per image
top5_prob, top5_catid = torch.topk(probabilities, 5)
for i in range(top5_prob.size(0)):
image_classes.append(self.categories[top5_catid[i]])
image_probabilities.append(top5_prob[i].item())
return ImageClass(
classes=image_classes,
probabilities=image_probabilities,
)
@serve.deployment
class ImageDownloader:
def __call__(self, image_url: str):
image_bytes = requests.get(image_url).content
return Image.open(BytesIO(image_bytes)).convert("RGB")
@serve.deployment
class DataPreprocessor:
def __init__(self):
self.preprocess = transforms.Compose(
[
transforms.Resize(256),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.Normalize(
mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]
),
]
)
def __call__(self, image: Image):
input_tensor = self.preprocess(image)
return input_tensor.unsqueeze(0) # create a mini-batch as expected by the model
image_downloader = ImageDownloader.bind()
data_preprocessor = DataPreprocessor.bind()
g2 = ImageClassifier.options(name="grpc-image-classifier").bind(
image_downloader, data_preprocessor
)
# __end_model_composition_deployment__
# __begin_model_composition_deploy__
app2 = "app2"
serve.run(target=g2, name=app2, route_prefix=f"/{app2}")
# __end_model_composition_deploy__
# __begin_model_composition_client__
import grpc
from user_defined_protos_pb2_grpc import ImageClassificationServiceStub
from user_defined_protos_pb2 import ImageData
channel = grpc.insecure_channel("localhost:9000")
stub = ImageClassificationServiceStub(channel)
request = ImageData(url="https://github.com/pytorch/hub/raw/master/images/dog.jpg")
metadata = (("application", "app2"),) # Make sure application metadata is passed.
response, call = stub.Predict.with_call(request=request, metadata=metadata)
print(f"status code: {call.code()}") # grpc.StatusCode.OK
print(f"Classes: {response.classes}") # ['Samoyed', ...]
print(f"Probabilities: {response.probabilities}") # [0.8846230506896973, ...]
# __end_model_composition_client__
# __begin_error_handle__
import grpc
from user_defined_protos_pb2_grpc import UserDefinedServiceStub
from user_defined_protos_pb2 import UserDefinedMessage
channel = grpc.insecure_channel("localhost:9000")
stub = UserDefinedServiceStub(channel)
request = UserDefinedMessage(name="foo", num=30, origin="bar")
try:
response = stub.__call__(request=request)
except grpc.RpcError as rpc_error:
print(f"status code: {rpc_error.code()}") # StatusCode.NOT_FOUND
print(f"details: {rpc_error.details()}") # Application metadata not set...
# __end_error_handle__
# __begin_grpc_context_define_app__
from user_defined_protos_pb2 import UserDefinedMessage, UserDefinedResponse
from ray import serve
from ray.serve.grpc_util import RayServegRPCContext
import grpc
from typing import Tuple
@serve.deployment
class GrpcDeployment:
def __init__(self):
self.nums = {}
def num_lookup(self, name: str) -> Tuple[int, grpc.StatusCode, str]:
if name not in self.nums:
self.nums[name] = len(self.nums)
code = grpc.StatusCode.INVALID_ARGUMENT
message = f"{name} not found, adding to nums."
else:
code = grpc.StatusCode.OK
message = f"{name} found."
return self.nums[name], code, message
def __call__(
self,
user_message: UserDefinedMessage,
grpc_context: RayServegRPCContext, # to use grpc context, add this kwarg
) -> UserDefinedResponse:
greeting = f"Hello {user_message.name} from {user_message.origin}"
num, code, message = self.num_lookup(user_message.name)
# Set custom code, details, and trailing metadata.
grpc_context.set_code(code)
grpc_context.set_details(message)
grpc_context.set_trailing_metadata([("num", str(num))])
# You can also set a status code before raising an exception.
# The status code will be preserved in the response.
if user_message.name == "error":
grpc_context.set_code(grpc.StatusCode.RESOURCE_EXHAUSTED)
grpc_context.set_details("Resource exhausted, please retry later.")
raise RuntimeError("Simulated error")
user_response = UserDefinedResponse(
greeting=greeting,
num=num,
)
return user_response
g = GrpcDeployment.bind()
app1 = "app1"
serve.run(target=g, name=app1, route_prefix=f"/{app1}")
# __end_grpc_context_define_app__
# __begin_grpc_context_client__
import grpc
from user_defined_protos_pb2_grpc import UserDefinedServiceStub
from user_defined_protos_pb2 import UserDefinedMessage
channel = grpc.insecure_channel("localhost:9000")
stub = UserDefinedServiceStub(channel)
request = UserDefinedMessage(name="foo", num=30, origin="bar")
metadata = (("application", "app1"),)
# First call is going to page miss and return INVALID_ARGUMENT status code.
try:
response, call = stub.__call__.with_call(request=request, metadata=metadata)
except grpc.RpcError as rpc_error:
assert rpc_error.code() == grpc.StatusCode.INVALID_ARGUMENT
assert rpc_error.details() == "foo not found, adding to nums."
assert any(
[key == "num" and value == "0" for key, value in rpc_error.trailing_metadata()]
)
assert any([key == "request_id" for key, _ in rpc_error.trailing_metadata()])
# Second call is going to page hit and return OK status code.
response, call = stub.__call__.with_call(request=request, metadata=metadata)
assert call.code() == grpc.StatusCode.OK
assert call.details() == "foo found."
assert any([key == "num" and value == "0" for key, value in call.trailing_metadata()])
assert any([key == "request_id" for key, _ in call.trailing_metadata()])
# __end_grpc_context_client__
# __begin_client_streaming_deployment__
from ray import serve
from ray.serve.grpc_util import gRPCInputStream
from user_defined_protos_pb2 import UserDefinedResponse
@serve.deployment
class ClientStreamingService:
async def ClientStreaming(self, request_stream: gRPCInputStream):
"""Receives stream of requests, returns a single response."""
total = 0
count = 0
async for request in request_stream:
total += request.num
count += 1
return UserDefinedResponse(
greeting=f"Received {count} messages",
num=total * 2,
)
serve.run(ClientStreamingService.bind())
# __end_client_streaming_deployment__
# __begin_client_streaming_client__
import grpc
from user_defined_protos_pb2_grpc import UserDefinedServiceStub
from user_defined_protos_pb2 import UserDefinedMessage
channel = grpc.insecure_channel("localhost:9000")
stub = UserDefinedServiceStub(channel)
metadata = (("application", "default"),)
def request_generator():
for i in range(5):
yield UserDefinedMessage(name=f"msg_{i}", num=i + 1, origin="client")
response = stub.ClientStreaming(request_generator(), metadata=metadata)
print(f"greeting: {response.greeting}") # greeting: Received 5 messages
print(f"num: {response.num}") # num: 30
# __end_client_streaming_client__
# __begin_bidi_streaming_deployment__
from ray import serve
from ray.serve.grpc_util import gRPCInputStream
from user_defined_protos_pb2 import UserDefinedResponse
@serve.deployment
class BidiStreamingService:
async def BidiStreaming(self, request_stream: gRPCInputStream):
"""Receives stream of requests, yields response for each."""
async for request in request_stream:
yield UserDefinedResponse(
greeting=f"Hello {request.name}",
num=request.num * 2,
)
serve.run(BidiStreamingService.bind())
# __end_bidi_streaming_deployment__
# __begin_bidi_streaming_client__
import grpc
from user_defined_protos_pb2_grpc import UserDefinedServiceStub
from user_defined_protos_pb2 import UserDefinedMessage
channel = grpc.insecure_channel("localhost:9000")
stub = UserDefinedServiceStub(channel)
metadata = (("application", "default"),)
def request_generator():
for i in range(3):
yield UserDefinedMessage(name=f"user_{i}", num=i * 10, origin="client")
responses = stub.BidiStreaming(request_generator(), metadata=metadata)
for response in responses:
print(f"greeting: {response.greeting}")
print(f"num: {response.num}")
# __end_bidi_streaming_client__
# __begin_streaming_with_context__
from ray import serve
from ray.serve.grpc_util import gRPCInputStream, RayServegRPCContext
from user_defined_protos_pb2 import UserDefinedResponse
@serve.deployment
class StreamingWithContext:
async def ClientStreaming(
self,
request_stream: gRPCInputStream,
grpc_context: RayServegRPCContext,
):
"""Receives stream and can modify gRPC context."""
count = 0
async for request in request_stream:
count += 1
grpc_context.set_trailing_metadata([("processed-count", str(count))])
return UserDefinedResponse(greeting=f"Processed {count} messages")
# __end_streaming_with_context__
doc/source/serve/doc_code/grpc_proxy/user_defined_protos.proto
+51
View File
@@ -0,0 +1,51 @@
// __begin_proto__
// user_defined_protos.proto
syntax = "proto3";
option java_multiple_files = true;
option java_package = "io.ray.examples.user_defined_protos";
option java_outer_classname = "UserDefinedProtos";
package userdefinedprotos;
message UserDefinedMessage {
string name = 1;
string origin = 2;
int64 num = 3;
}
message UserDefinedResponse {
string greeting = 1;
int64 num = 2;
}
message UserDefinedMessage2 {}
message UserDefinedResponse2 {
string greeting = 1;
}
message ImageData {
string url = 1;
string filename = 2;
}
message ImageClass {
repeated string classes = 1;
repeated float probabilities = 2;
}
service UserDefinedService {
rpc __call__(UserDefinedMessage) returns (UserDefinedResponse);
rpc Multiplexing(UserDefinedMessage2) returns (UserDefinedResponse2);
rpc Streaming(UserDefinedMessage) returns (stream UserDefinedResponse);
rpc ClientStreaming(stream UserDefinedMessage) returns (UserDefinedResponse);
rpc BidiStreaming(stream UserDefinedMessage) returns (stream UserDefinedResponse);
}
service ImageClassificationService {
rpc Predict(ImageData) returns (ImageClass);
}
// __end_proto__
doc/source/serve/doc_code/grpc_proxy/user_defined_protos_pb2_grpc.py
+259
View File
@@ -0,0 +1,259 @@
# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
"""Client and server classes corresponding to protobuf-defined services."""
import grpc
import user_defined_protos_pb2 as user__defined__protos__pb2
class UserDefinedServiceStub(object):
"""Missing associated documentation comment in .proto file."""
def __init__(self, channel):
"""Constructor.
Args:
channel: A grpc.Channel.
"""
self.__call__ = channel.unary_unary(
'/userdefinedprotos.UserDefinedService/__call__',
request_serializer=user__defined__protos__pb2.UserDefinedMessage.SerializeToString,
response_deserializer=user__defined__protos__pb2.UserDefinedResponse.FromString,
)
self.Multiplexing = channel.unary_unary(
'/userdefinedprotos.UserDefinedService/Multiplexing',
request_serializer=user__defined__protos__pb2.UserDefinedMessage2.SerializeToString,
response_deserializer=user__defined__protos__pb2.UserDefinedResponse2.FromString,
)
self.Streaming = channel.unary_stream(
'/userdefinedprotos.UserDefinedService/Streaming',
request_serializer=user__defined__protos__pb2.UserDefinedMessage.SerializeToString,
response_deserializer=user__defined__protos__pb2.UserDefinedResponse.FromString,
)
self.ClientStreaming = channel.stream_unary(
'/userdefinedprotos.UserDefinedService/ClientStreaming',
request_serializer=user__defined__protos__pb2.UserDefinedMessage.SerializeToString,
response_deserializer=user__defined__protos__pb2.UserDefinedResponse.FromString,
)
self.BidiStreaming = channel.stream_stream(
'/userdefinedprotos.UserDefinedService/BidiStreaming',
request_serializer=user__defined__protos__pb2.UserDefinedMessage.SerializeToString,
response_deserializer=user__defined__protos__pb2.UserDefinedResponse.FromString,
)
class UserDefinedServiceServicer(object):
"""Missing associated documentation comment in .proto file."""
def __call__(self, request, context):
"""Missing associated documentation comment in .proto file."""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def Multiplexing(self, request, context):
"""Missing associated documentation comment in .proto file."""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def Streaming(self, request, context):
"""Missing associated documentation comment in .proto file."""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def ClientStreaming(self, request_iterator, context):
"""Missing associated documentation comment in .proto file."""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def BidiStreaming(self, request_iterator, context):
"""Missing associated documentation comment in .proto file."""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def add_UserDefinedServiceServicer_to_server(servicer, server):
rpc_method_handlers = {
'__call__': grpc.unary_unary_rpc_method_handler(
servicer.__call__,
request_deserializer=user__defined__protos__pb2.UserDefinedMessage.FromString,
response_serializer=user__defined__protos__pb2.UserDefinedResponse.SerializeToString,
),
'Multiplexing': grpc.unary_unary_rpc_method_handler(
servicer.Multiplexing,
request_deserializer=user__defined__protos__pb2.UserDefinedMessage2.FromString,
response_serializer=user__defined__protos__pb2.UserDefinedResponse2.SerializeToString,
),
'Streaming': grpc.unary_stream_rpc_method_handler(
servicer.Streaming,
request_deserializer=user__defined__protos__pb2.UserDefinedMessage.FromString,
response_serializer=user__defined__protos__pb2.UserDefinedResponse.SerializeToString,
),
'ClientStreaming': grpc.stream_unary_rpc_method_handler(
servicer.ClientStreaming,
request_deserializer=user__defined__protos__pb2.UserDefinedMessage.FromString,
response_serializer=user__defined__protos__pb2.UserDefinedResponse.SerializeToString,
),
'BidiStreaming': grpc.stream_stream_rpc_method_handler(
servicer.BidiStreaming,
request_deserializer=user__defined__protos__pb2.UserDefinedMessage.FromString,
response_serializer=user__defined__protos__pb2.UserDefinedResponse.SerializeToString,
),
}
generic_handler = grpc.method_handlers_generic_handler(
'userdefinedprotos.UserDefinedService', rpc_method_handlers)
server.add_generic_rpc_handlers((generic_handler,))
# This class is part of an EXPERIMENTAL API.
class UserDefinedService(object):
"""Missing associated documentation comment in .proto file."""
@staticmethod
def __call__(request,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.unary_unary(request, target, '/userdefinedprotos.UserDefinedService/__call__',
user__defined__protos__pb2.UserDefinedMessage.SerializeToString,
user__defined__protos__pb2.UserDefinedResponse.FromString,
options, channel_credentials,
insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
@staticmethod
def Multiplexing(request,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.unary_unary(request, target, '/userdefinedprotos.UserDefinedService/Multiplexing',
user__defined__protos__pb2.UserDefinedMessage2.SerializeToString,
user__defined__protos__pb2.UserDefinedResponse2.FromString,
options, channel_credentials,
insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
@staticmethod
def Streaming(request,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.unary_stream(request, target, '/userdefinedprotos.UserDefinedService/Streaming',
user__defined__protos__pb2.UserDefinedMessage.SerializeToString,
user__defined__protos__pb2.UserDefinedResponse.FromString,
options, channel_credentials,
insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
@staticmethod
def ClientStreaming(request_iterator,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.stream_unary(request_iterator, target, '/userdefinedprotos.UserDefinedService/ClientStreaming',
user__defined__protos__pb2.UserDefinedMessage.SerializeToString,
user__defined__protos__pb2.UserDefinedResponse.FromString,
options, channel_credentials,
insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
@staticmethod
def BidiStreaming(request_iterator,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.stream_stream(request_iterator, target, '/userdefinedprotos.UserDefinedService/BidiStreaming',
user__defined__protos__pb2.UserDefinedMessage.SerializeToString,
user__defined__protos__pb2.UserDefinedResponse.FromString,
options, channel_credentials,
insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
class ImageClassificationServiceStub(object):
"""Missing associated documentation comment in .proto file."""
def __init__(self, channel):
"""Constructor.
Args:
channel: A grpc.Channel.
"""
self.Predict = channel.unary_unary(
'/userdefinedprotos.ImageClassificationService/Predict',
request_serializer=user__defined__protos__pb2.ImageData.SerializeToString,
response_deserializer=user__defined__protos__pb2.ImageClass.FromString,
)
class ImageClassificationServiceServicer(object):
"""Missing associated documentation comment in .proto file."""
def Predict(self, request, context):
"""Missing associated documentation comment in .proto file."""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def add_ImageClassificationServiceServicer_to_server(servicer, server):
rpc_method_handlers = {
'Predict': grpc.unary_unary_rpc_method_handler(
servicer.Predict,
request_deserializer=user__defined__protos__pb2.ImageData.FromString,
response_serializer=user__defined__protos__pb2.ImageClass.SerializeToString,
),
}
generic_handler = grpc.method_handlers_generic_handler(
'userdefinedprotos.ImageClassificationService', rpc_method_handlers)
server.add_generic_rpc_handlers((generic_handler,))
# This class is part of an EXPERIMENTAL API.
class ImageClassificationService(object):
"""Missing associated documentation comment in .proto file."""
@staticmethod
def Predict(request,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.unary_unary(request, target, '/userdefinedprotos.ImageClassificationService/Predict',
user__defined__protos__pb2.ImageData.SerializeToString,
user__defined__protos__pb2.ImageClass.FromString,
options, channel_credentials,
insecure, call_credentials, compression, wait_for_ready, timeout, metadata)
doc/source/serve/doc_code/http_guide/disconnects.py
+121
View File
@@ -0,0 +1,121 @@
# flake8: noqa
# fmt: off
import ray
import sys
from typing import List
from ray._common.test_utils import wait_for_condition
# Overwrite print statement to make doc code testable
@ray.remote
class PrintStorage:
def __init__(self):
self.print_storage: List[str] = []
def add(self, s: str):
self.print_storage.append(s)
def clear(self):
self.print_storage.clear()
def get(self) -> List[str]:
return self.print_storage
print_storage_handle = PrintStorage.remote()
def print(string: str):
ray.get(print_storage_handle.add.remote(string))
sys.stdout.write(f"{string}\n")
# __start_basic_disconnect__
import asyncio
from ray import serve
@serve.deployment
async def startled():
try:
print("Replica received request!")
await asyncio.sleep(10000)
except asyncio.CancelledError:
# Add custom behavior that should run
# upon cancellation here.
print("Request got cancelled!")
# __end_basic_disconnect__
serve.run(startled.bind())
import requests
from requests.exceptions import Timeout
# Intentionally time out request to test cancellation behavior
try:
requests.get("http://localhost:8000", timeout=0.5)
except Timeout:
pass
wait_for_condition(
lambda: {"Replica received request!", "Request got cancelled!"}
== set(ray.get(print_storage_handle.get.remote())),
timeout=5,
)
sys.stdout.write(f"{ray.get(print_storage_handle.get.remote())}\n")
ray.get(print_storage_handle.clear.remote())
# __start_shielded_disconnect__
import asyncio
from ray import serve
@serve.deployment
class SnoringSleeper:
async def snore(self):
await asyncio.sleep(1)
print("ZZZ")
async def __call__(self):
try:
print("SnoringSleeper received request!")
# Prevent the snore() method from being cancelled
await asyncio.shield(self.snore())
except asyncio.CancelledError:
print("SnoringSleeper's request was cancelled!")
app = SnoringSleeper.bind()
# __end_shielded_disconnect__
serve.run(app)
import requests
from requests.exceptions import Timeout
# Intentionally time out request to test cancellation behavior
try:
requests.get("http://localhost:8000", timeout=0.5)
except Timeout:
pass
wait_for_condition(
lambda: {
"SnoringSleeper received request!",
"SnoringSleeper's request was cancelled!",
"ZZZ",
}
== set(ray.get(print_storage_handle.get.remote())),
timeout=5,
)
sys.stdout.write(f"{ray.get(print_storage_handle.get.remote())}\n")
ray.get(print_storage_handle.clear.remote())
doc/source/serve/doc_code/http_guide/http_guide.py
+176
View File
@@ -0,0 +1,176 @@
# flake8: noqa
# __begin_starlette__
import starlette.requests
import requests
from ray import serve
@serve.deployment
class Counter:
def __call__(self, request: starlette.requests.Request):
return request.query_params
serve.run(Counter.bind())
resp = requests.get("http://localhost:8000?a=b&c=d")
assert resp.json() == {"a": "b", "c": "d"}
# __end_starlette__
# __begin_fastapi__
import ray
import requests
from fastapi import FastAPI
from ray import serve
app = FastAPI()
@serve.deployment
@serve.ingress(app)
class MyFastAPIDeployment:
@app.get("/")
def root(self):
return "Hello, world!"
serve.run(MyFastAPIDeployment.bind(), route_prefix="/hello")
resp = requests.get("http://localhost:8000/hello")
assert resp.json() == "Hello, world!"
# __end_fastapi__
# __begin_fastapi_multi_routes__
import ray
import requests
from fastapi import FastAPI
from ray import serve
app = FastAPI()
@serve.deployment
@serve.ingress(app)
class MyFastAPIDeployment:
@app.get("/")
def root(self):
return "Hello, world!"
@app.post("/{subpath}")
def root(self, subpath: str):
return f"Hello from {subpath}!"
serve.run(MyFastAPIDeployment.bind(), route_prefix="/hello")
resp = requests.post("http://localhost:8000/hello/Serve")
assert resp.json() == "Hello from Serve!"
# __end_fastapi_multi_routes__
# __begin_byo_fastapi__
import ray
import requests
from fastapi import FastAPI
from ray import serve
app = FastAPI()
@app.get("/")
def f():
return "Hello from the root!"
@serve.deployment
@serve.ingress(app)
class FastAPIWrapper:
pass
serve.run(FastAPIWrapper.bind(), route_prefix="/")
resp = requests.get("http://localhost:8000/")
assert resp.json() == "Hello from the root!"
# __end_byo_fastapi__
# __begin_fastapi_middleware__
import requests
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from ray import serve
app = FastAPI()
app.add_middleware(
CORSMiddleware,
allow_origins=["https://example.com"],
allow_methods=["GET", "POST"],
)
@serve.deployment
@serve.ingress(app)
class Ingress:
@app.get("/")
def root(self):
return "ok"
serve.run(Ingress.bind())
resp = requests.get(
"http://localhost:8000/",
headers={"Origin": "https://example.com"},
)
assert resp.json() == "ok"
assert resp.headers["access-control-allow-origin"] == "https://example.com"
# __end_fastapi_middleware__
# __begin_fastapi_factory_pattern__
import requests
from fastapi import FastAPI
from ray import serve
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
@serve.deployment
class ChildDeployment:
def __call__(self):
return "Hello from the child deployment!"
def fastapi_factory():
"""Factory-style FastAPI app used as Serve ingress.
We build the FastAPI app inside a factory and pass the callable to
@serve.ingress.
"""
app = FastAPI()
# In an object-based ingress (where the FastAPI app is stored on the
# deployment instance), Ray would need to serialize the app and its
# instrumentation. Some instrumentors (like FastAPIInstrumentor) are not
# picklable, which can cause serialization failures. Creating and
# instrumenting the app here sidesteps that issue.
FastAPIInstrumentor.instrument_app(app)
@app.get("/")
async def root():
# Handlers defined inside this factory don't have access to the
# ParentDeployment instance (i.e., there's no `self` here), so we
# can't call `self.child`. Instead, fetch a handle by deployment name.
handle = serve.get_deployment_handle("ChildDeployment", app_name="default")
return {"message": await handle.remote()}
return app
@serve.deployment
@serve.ingress(fastapi_factory)
class ParentDeployment:
def __init__(self, child):
self.child = child
serve.run(ParentDeployment.bind(ChildDeployment.bind()))
resp = requests.get("http://localhost:8000/")
assert resp.json() == {"message": "Hello from the child deployment!"}
# __end_fastapi_factory_pattern__
doc/source/serve/doc_code/http_guide/streaming_example.py
+82
View File
@@ -0,0 +1,82 @@
# flake8: noqa
# __begin_example__
import time
from typing import Generator
import requests
from starlette.responses import StreamingResponse
from starlette.requests import Request
from ray import serve
@serve.deployment
class StreamingResponder:
def generate_numbers(self, max: int) -> Generator[str, None, None]:
for i in range(max):
yield str(i)
time.sleep(0.1)
def __call__(self, request: Request) -> StreamingResponse:
max = request.query_params.get("max", "25")
gen = self.generate_numbers(int(max))
return StreamingResponse(gen, status_code=200, media_type="text/plain")
serve.run(StreamingResponder.bind())
r = requests.get("http://localhost:8000?max=10", stream=True)
start = time.time()
r.raise_for_status()
for chunk in r.iter_content(chunk_size=None, decode_unicode=True):
print(f"Got result {round(time.time()-start, 1)}s after start: '{chunk}'")
# __end_example__
r = requests.get("http://localhost:8000?max=10", stream=True)
r.raise_for_status()
for i, chunk in enumerate(r.iter_content(chunk_size=None, decode_unicode=True)):
assert chunk == str(i)
# __begin_cancellation__
import asyncio
import time
from typing import AsyncGenerator
import requests
from starlette.responses import StreamingResponse
from starlette.requests import Request
from ray import serve
@serve.deployment
class StreamingResponder:
async def generate_forever(self) -> AsyncGenerator[str, None]:
try:
i = 0
while True:
yield str(i)
i += 1
await asyncio.sleep(0.1)
except asyncio.CancelledError:
print("Cancelled! Exiting.")
def __call__(self, request: Request) -> StreamingResponse:
gen = self.generate_forever()
return StreamingResponse(gen, status_code=200, media_type="text/plain")
serve.run(StreamingResponder.bind())
r = requests.get("http://localhost:8000?max=10", stream=True)
start = time.time()
r.raise_for_status()
for i, chunk in enumerate(r.iter_content(chunk_size=None, decode_unicode=True)):
print(f"Got result {round(time.time()-start, 1)}s after start: '{chunk}'")
if i == 10:
print("Client disconnecting")
break
# __end_cancellation__
doc/source/serve/doc_code/http_guide/websockets_example.py
+39
View File
@@ -0,0 +1,39 @@
# flake8: noqa
# __websocket_serve_app_start__
from fastapi import FastAPI, WebSocket, WebSocketDisconnect
from ray import serve
app = FastAPI()
@serve.deployment
@serve.ingress(app)
class EchoServer:
@app.websocket("/")
async def echo(self, ws: WebSocket):
await ws.accept()
try:
while True:
text = await ws.receive_text()
await ws.send_text(text)
except WebSocketDisconnect:
print("Client disconnected.")
serve_app = serve.run(EchoServer.bind())
# __websocket_serve_app_end__
# __websocket_serve_client_start__
from websockets.sync.client import connect
with connect("ws://localhost:8000") as websocket:
websocket.send("Eureka!")
assert websocket.recv() == "Eureka!"
websocket.send("I've found it!")
assert websocket.recv() == "I've found it!"
# __websocket_serve_client_end__
doc/source/serve/doc_code/image_classifier_example.py
+98
View File
@@ -0,0 +1,98 @@
# __serve_example_begin__
import requests
import starlette
from transformers import pipeline
from io import BytesIO
from PIL import Image
from ray import serve
from ray.serve.handle import DeploymentHandle
@serve.deployment
def downloader(image_url: str):
image_bytes = requests.get(image_url).content
image = Image.open(BytesIO(image_bytes)).convert("RGB")
return image
@serve.deployment
class ImageClassifier:
def __init__(self, downloader: DeploymentHandle):
self.downloader = downloader
self.model = pipeline(
"image-classification", model="google/vit-base-patch16-224"
)
async def classify(self, image_url: str) -> str:
image = await self.downloader.remote(image_url)
results = self.model(image)
return results[0]["label"]
async def __call__(self, req: starlette.requests.Request):
req = await req.json()
return await self.classify(req["image_url"])
app = ImageClassifier.bind(downloader.bind())
# __serve_example_end__
@serve.deployment
class ModifiedImageClassifier:
def __init__(self, downloader: DeploymentHandle):
self.downloader = downloader
self.model = pipeline(
"image-classification", model="google/vit-base-patch16-224"
)
async def classify(self, image_url: str) -> str:
image = await self.downloader.remote(image_url)
results = self.model(image)
return results[0]["label"]
# __serve_example_modified_begin__
async def __call__(self, req: starlette.requests.Request):
req = await req.json()
result = await self.classify(req["image_url"])
if req.get("should_translate") is True:
handle: DeploymentHandle = serve.get_app_handle("app2")
return await handle.translate.remote(result)
return result
# __serve_example_modified_end__
serve.run(app, name="app1", route_prefix="/classify")
# __request_begin__
bear_url = "https://cdn.britannica.com/41/156441-050-A4424AEC/Grizzly-bear-Jasper-National-Park-Canada-Alberta.jpg" # noqa
resp = requests.post("http://localhost:8000/classify", json={"image_url": bear_url})
print(resp.text)
# 'brown bear, bruin, Ursus arctos'
# __request_end__
assert resp.text == "brown bear, bruin, Ursus arctos"
from translator_example import app as translator_app # noqa
serve.run(
ModifiedImageClassifier.bind(downloader.bind()),
name="app1",
route_prefix="/classify",
)
serve.run(translator_app, name="app2")
# __second_request_begin__
bear_url = "https://cdn.britannica.com/41/156441-050-A4424AEC/Grizzly-bear-Jasper-National-Park-Canada-Alberta.jpg" # noqa
resp = requests.post(
"http://localhost:8000/classify",
json={"image_url": bear_url, "should_translate": True},
)
print(resp.text)
# 'Braunbär, Bruin, Ursus arctos'
# __second_request_end__
assert resp.text == "Braunbär, Bruin, Ursus arctos"
doc/source/serve/doc_code/intel_gaudi_inference_client.py
+23
View File
@@ -0,0 +1,23 @@
# __main_code_start__
import requests
# Prompt for the model
prompt = "Once upon a time,"
# Add generation config here
config = {}
# Non-streaming response
sample_input = {"text": prompt, "config": config, "stream": False}
outputs = requests.post("http://127.0.0.1:8000/", json=sample_input, stream=False)
print(outputs.text, flush=True)
# Streaming response
sample_input["stream"] = True
outputs = requests.post("http://127.0.0.1:8000/", json=sample_input, stream=True)
outputs.raise_for_status()
for output in outputs.iter_content(chunk_size=None, decode_unicode=True):
print(output, end="", flush=True)
print()
# __main_code_end__
doc/source/serve/doc_code/intel_gaudi_inference_serve.py
+137
View File
@@ -0,0 +1,137 @@
# __model_def_start__
import asyncio
from functools import partial
from queue import Empty
from typing import Dict, Any
from starlette.requests import Request
from starlette.responses import StreamingResponse
import torch
from ray import serve
# Define the Ray Serve deployment
@serve.deployment(ray_actor_options={"num_cpus": 10, "resources": {"HPU": 1}})
class LlamaModel:
def __init__(self, model_id_or_path: str):
from transformers import AutoTokenizer, AutoModelForCausalLM, AutoConfig
from optimum.habana.transformers.modeling_utils import (
adapt_transformers_to_gaudi,
)
# Tweak transformers to optimize performance
adapt_transformers_to_gaudi()
self.device = torch.device("hpu")
self.tokenizer = AutoTokenizer.from_pretrained(
model_id_or_path, use_fast=False, use_auth_token=""
)
hf_config = AutoConfig.from_pretrained(
model_id_or_path,
torchscript=True,
use_auth_token="",
trust_remote_code=False,
)
# Load the model in Gaudi
model = AutoModelForCausalLM.from_pretrained(
model_id_or_path,
config=hf_config,
torch_dtype=torch.float32,
low_cpu_mem_usage=True,
use_auth_token="",
)
model = model.eval().to(self.device)
from habana_frameworks.torch.hpu import wrap_in_hpu_graph
# Enable hpu graph runtime
self.model = wrap_in_hpu_graph(model)
# Set pad token, etc.
self.tokenizer.pad_token_id = self.model.generation_config.pad_token_id
self.tokenizer.padding_side = "left"
# Use async loop in streaming
self.loop = asyncio.get_running_loop()
def tokenize(self, prompt: str):
"""Tokenize the input and move to HPU."""
input_tokens = self.tokenizer(prompt, return_tensors="pt", padding=True)
return input_tokens.input_ids.to(device=self.device)
def generate(self, prompt: str, **config: Dict[str, Any]):
"""Take a prompt and generate a response."""
input_ids = self.tokenize(prompt)
gen_tokens = self.model.generate(input_ids, **config)
return self.tokenizer.batch_decode(gen_tokens, skip_special_tokens=True)[0]
async def consume_streamer_async(self, streamer):
"""Consume the streamer asynchronously."""
while True:
try:
for token in streamer:
yield token
break
except Empty:
await asyncio.sleep(0.001)
def streaming_generate(self, prompt: str, streamer, **config: Dict[str, Any]):
"""Generate a streamed response given an input."""
input_ids = self.tokenize(prompt)
self.model.generate(input_ids, streamer=streamer, **config)
async def __call__(self, http_request: Request):
"""Handle HTTP requests."""
# Load fields from the request
json_request: str = await http_request.json()
text = json_request["text"]
# Config used in generation
config = json_request.get("config", {})
streaming_response = json_request["stream"]
# Prepare prompts
prompts = []
if isinstance(text, list):
prompts.extend(text)
else:
prompts.append(text)
# Process config
config.setdefault("max_new_tokens", 128)
# Enable HPU graph runtime
config["hpu_graphs"] = True
# Lazy mode should be True when using HPU graphs
config["lazy_mode"] = True
# Non-streaming case
if not streaming_response:
return self.generate(prompts, **config)
# Streaming case
from transformers import TextIteratorStreamer
streamer = TextIteratorStreamer(
self.tokenizer, skip_prompt=True, timeout=0, skip_special_tokens=True
)
# Convert the streamer into a generator
self.loop.run_in_executor(
None, partial(self.streaming_generate, prompts, streamer, **config)
)
return StreamingResponse(
self.consume_streamer_async(streamer),
status_code=200,
media_type="text/plain",
)
# Replace the model ID with path if necessary
entrypoint = LlamaModel.bind("meta-llama/Llama-2-7b-chat-hf")
# __model_def_end__
doc/source/serve/doc_code/intel_gaudi_inference_serve_deepspeed.py
+273
View File
@@ -0,0 +1,273 @@
# __worker_def_start__
import tempfile
from typing import Dict, Any
from starlette.requests import Request
from starlette.responses import StreamingResponse
import torch
from transformers import TextStreamer
import ray
from ray import serve
from ray.util.queue import Queue
from ray.runtime_env import RuntimeEnv
@ray.remote(resources={"HPU": 1})
class DeepSpeedInferenceWorker:
def __init__(self, model_id_or_path: str, world_size: int, local_rank: int):
"""An actor that runs a DeepSpeed inference engine.
Arguments:
model_id_or_path: Either a Hugging Face model ID
or a path to a cached model.
world_size: Total number of worker processes.
local_rank: Rank of this worker process.
The rank 0 worker is the head worker.
"""
from transformers import AutoTokenizer, AutoConfig
from optimum.habana.transformers.modeling_utils import (
adapt_transformers_to_gaudi,
)
# Tweak transformers for better performance on Gaudi.
adapt_transformers_to_gaudi()
self.model_id_or_path = model_id_or_path
self._world_size = world_size
self._local_rank = local_rank
self.device = torch.device("hpu")
self.model_config = AutoConfig.from_pretrained(
model_id_or_path,
torch_dtype=torch.bfloat16,
token="",
trust_remote_code=False,
)
# Load and configure the tokenizer.
self.tokenizer = AutoTokenizer.from_pretrained(
model_id_or_path, use_fast=False, token=""
)
self.tokenizer.padding_side = "left"
if self.tokenizer.pad_token is None:
self.tokenizer.pad_token = self.tokenizer.eos_token
import habana_frameworks.torch.distributed.hccl as hccl
# Initialize the distributed backend.
hccl.initialize_distributed_hpu(
world_size=world_size, rank=local_rank, local_rank=local_rank
)
torch.distributed.init_process_group(backend="hccl")
def load_model(self):
"""Load the model to HPU and initialize the DeepSpeed inference engine."""
import deepspeed
from transformers import AutoModelForCausalLM
from optimum.habana.checkpoint_utils import (
get_ds_injection_policy,
write_checkpoints_json,
)
# Construct the model with fake meta Tensors.
# Loads the model weights from the checkpoint later.
with deepspeed.OnDevice(dtype=torch.bfloat16, device="meta"):
model = AutoModelForCausalLM.from_config(
self.model_config, torch_dtype=torch.bfloat16
)
model = model.eval()
# Create a file to indicate where the checkpoint is.
checkpoints_json = tempfile.NamedTemporaryFile(suffix=".json", mode="w+")
write_checkpoints_json(
self.model_id_or_path, self._local_rank, checkpoints_json, token=""
)
# Prepare the DeepSpeed inference configuration.
kwargs = {"dtype": torch.bfloat16}
kwargs["checkpoint"] = checkpoints_json.name
kwargs["tensor_parallel"] = {"tp_size": self._world_size}
# Enable the HPU graph, similar to the cuda graph.
kwargs["enable_cuda_graph"] = True
# Specify the injection policy, required by DeepSpeed Tensor parallelism.
kwargs["injection_policy"] = get_ds_injection_policy(self.model_config)
# Initialize the inference engine.
self.model = deepspeed.init_inference(model, **kwargs).module
def tokenize(self, prompt: str):
"""Tokenize the input and move it to HPU."""
input_tokens = self.tokenizer(prompt, return_tensors="pt", padding=True)
return input_tokens.input_ids.to(device=self.device)
def generate(self, prompt: str, **config: Dict[str, Any]):
"""Take in a prompt and generate a response."""
input_ids = self.tokenize(prompt)
gen_tokens = self.model.generate(input_ids, **config)
return self.tokenizer.batch_decode(gen_tokens, skip_special_tokens=True)[0]
def streaming_generate(self, prompt: str, streamer, **config: Dict[str, Any]):
"""Generate a streamed response given an input."""
input_ids = self.tokenize(prompt)
self.model.generate(input_ids, streamer=streamer, **config)
def get_streamer(self):
"""Return a streamer.
We only need the rank 0 worker's result.
Other workers return a fake streamer.
"""
if self._local_rank == 0:
return RayTextIteratorStreamer(self.tokenizer, skip_special_tokens=True)
else:
class FakeStreamer:
def put(self, value):
pass
def end(self):
pass
return FakeStreamer()
class RayTextIteratorStreamer(TextStreamer):
def __init__(
self,
tokenizer,
skip_prompt: bool = False,
timeout: int = None,
**decode_kwargs: Dict[str, Any],
):
super().__init__(tokenizer, skip_prompt, **decode_kwargs)
self.text_queue = Queue()
self.stop_signal = None
self.timeout = timeout
def on_finalized_text(self, text: str, stream_end: bool = False):
self.text_queue.put(text, timeout=self.timeout)
if stream_end:
self.text_queue.put(self.stop_signal, timeout=self.timeout)
def __iter__(self):
return self
def __next__(self):
value = self.text_queue.get(timeout=self.timeout)
if value == self.stop_signal:
raise StopIteration()
else:
return value
# __worker_def_end__
# __deploy_def_start__
# We need to set these variables for this example.
HABANA_ENVS = {
"PT_HPU_LAZY_ACC_PAR_MODE": "0",
"PT_HPU_ENABLE_REFINE_DYNAMIC_SHAPES": "0",
"PT_HPU_ENABLE_WEIGHT_CPU_PERMUTE": "0",
"PT_HPU_ENABLE_LAZY_COLLECTIVES": "true",
"HABANA_VISIBLE_MODULES": "0,1,2,3,4,5,6,7",
}
# Define the Ray Serve deployment.
@serve.deployment
class DeepSpeedLlamaModel:
def __init__(self, world_size: int, model_id_or_path: str):
self._world_size = world_size
# Create the DeepSpeed workers
self.deepspeed_workers = []
for i in range(world_size):
self.deepspeed_workers.append(
DeepSpeedInferenceWorker.options(
runtime_env=RuntimeEnv(env_vars=HABANA_ENVS)
).remote(model_id_or_path, world_size, i)
)
# Load the model to all workers.
for worker in self.deepspeed_workers:
worker.load_model.remote()
# Get the workers' streamers.
self.streamers = ray.get(
[worker.get_streamer.remote() for worker in self.deepspeed_workers]
)
def generate(self, prompt: str, **config: Dict[str, Any]):
"""Send the prompt to workers for generation.
Return after all workers finish the generation.
Only return the rank 0 worker's result.
"""
futures = [
worker.generate.remote(prompt, **config)
for worker in self.deepspeed_workers
]
return ray.get(futures)[0]
def streaming_generate(self, prompt: str, **config: Dict[str, Any]):
"""Send the prompt to workers for streaming generation.
Only use the rank 0 worker's result.
"""
for worker, streamer in zip(self.deepspeed_workers, self.streamers):
worker.streaming_generate.remote(prompt, streamer, **config)
def consume_streamer(self, streamer):
"""Consume the streamer and return a generator."""
for token in streamer:
yield token
async def __call__(self, http_request: Request):
"""Handle received HTTP requests."""
# Load fields from the request
json_request: str = await http_request.json()
text = json_request["text"]
# Config used in generation
config = json_request.get("config", {})
streaming_response = json_request["stream"]
# Prepare prompts
prompts = []
if isinstance(text, list):
prompts.extend(text)
else:
prompts.append(text)
# Process the configuration.
config.setdefault("max_new_tokens", 128)
# Enable HPU graph runtime.
config["hpu_graphs"] = True
# Lazy mode should be True when using HPU graphs.
config["lazy_mode"] = True
# Non-streaming case
if not streaming_response:
return self.generate(prompts, **config)
# Streaming case
self.streaming_generate(prompts, **config)
return StreamingResponse(
self.consume_streamer(self.streamers[0]),
status_code=200,
media_type="text/plain",
)
# Replace the model ID with a path if necessary.
entrypoint = DeepSpeedLlamaModel.bind(8, "meta-llama/Llama-2-70b-chat-hf")
# __deploy_def_end__
doc/source/serve/doc_code/interdeployment_grpc.py
+32
View File
@@ -0,0 +1,32 @@
# flake8: noqa
from ray import serve
from ray.serve.handle import DeploymentHandle
# __start_grpc_override__
@serve.deployment
class Caller:
def __init__(self, target: DeploymentHandle):
# Override this specific handle to use actor RPC instead of gRPC.
# This is useful for large payloads (over ~1 MB) where passing
# objects by reference through Ray's object store is more efficient.
self._target = target.options(_by_reference=True)
async def __call__(self, data: bytes) -> str:
return await self._target.remote(data)
@serve.deployment
class LargePayloadProcessor:
def __call__(self, data: bytes) -> str:
return f"processed {len(data)} bytes"
processor = LargePayloadProcessor.bind()
app = Caller.bind(processor)
handle: DeploymentHandle = serve.run(app)
assert handle.remote(b"x" * 1024).result() == "processed 1024 bytes"
# __end_grpc_override__
serve.shutdown()
doc/source/serve/doc_code/key_concepts.py
+108
View File
@@ -0,0 +1,108 @@
# flake8: noqa
# __start_my_first_deployment__
from ray import serve
from ray.serve.handle import DeploymentHandle
@serve.deployment
class MyFirstDeployment:
# Take the message to return as an argument to the constructor.
def __init__(self, msg):
self.msg = msg
def __call__(self):
return self.msg
my_first_deployment = MyFirstDeployment.bind("Hello world!")
handle: DeploymentHandle = serve.run(my_first_deployment)
assert handle.remote().result() == "Hello world!"
# __end_my_first_deployment__
# __start_deployment_handle__
from ray import serve
from ray.serve.handle import DeploymentHandle
@serve.deployment
class Hello:
def __call__(self) -> str:
return "Hello"
@serve.deployment
class World:
def __call__(self) -> str:
return " world!"
@serve.deployment
class Ingress:
def __init__(self, hello_handle: DeploymentHandle, world_handle: DeploymentHandle):
self._hello_handle = hello_handle
self._world_handle = world_handle
async def __call__(self) -> str:
hello_response = self._hello_handle.remote()
world_response = self._world_handle.remote()
return (await hello_response) + (await world_response)
hello = Hello.bind()
world = World.bind()
# The deployments passed to the Ingress constructor are replaced with handles.
app = Ingress.bind(hello, world)
# Deploys Hello, World, and Ingress.
handle: DeploymentHandle = serve.run(app)
# `DeploymentHandle`s can also be used to call the ingress deployment of an application.
assert handle.remote().result() == "Hello world!"
# __end_deployment_handle__
# __start_basic_ingress__
import requests
from starlette.requests import Request
from ray import serve
@serve.deployment
class MostBasicIngress:
async def __call__(self, request: Request) -> str:
name = (await request.json())["name"]
return f"Hello {name}!"
app = MostBasicIngress.bind()
serve.run(app)
assert (
requests.get("http://127.0.0.1:8000/", json={"name": "Corey"}).text
== "Hello Corey!"
)
# __end_basic_ingress__
# __start_fastapi_ingress__
import requests
from fastapi import FastAPI
from fastapi.responses import PlainTextResponse
from ray import serve
fastapi_app = FastAPI()
@serve.deployment
@serve.ingress(fastapi_app)
class FastAPIIngress:
@fastapi_app.get("/{name}")
async def say_hi(self, name: str) -> str:
return PlainTextResponse(f"Hello {name}!")
app = FastAPIIngress.bind()
serve.run(app)
assert requests.get("http://127.0.0.1:8000/Corey").text == "Hello Corey!"
# __end_fastapi_ingress__
doc/source/serve/doc_code/load_shedding.py
+55
View File
@@ -0,0 +1,55 @@
# flake8: noqa
# fmt: off
# __example_deployment_start__
import time
from ray import serve
from starlette.requests import Request
@serve.deployment(
# Each replica will be sent 2 requests at a time.
max_ongoing_requests=2,
# Each caller queues up to 2 requests at a time.
# (beyond those that are sent to replicas).
max_queued_requests=2,
)
class SlowDeployment:
def __call__(self, request: Request) -> str:
# Emulate a long-running request, such as ML inference.
time.sleep(2)
return "Hello!"
# __example_deployment_end__
# __client_test_start__
import ray
import aiohttp
@ray.remote
class Requester:
async def do_request(self) -> int:
async with aiohttp.ClientSession("http://localhost:8000/") as session:
return (await session.get("/")).status
r = Requester.remote()
serve.run(SlowDeployment.bind())
# Send 4 requests first.
# 2 of these will be sent to the replica. These requests take a few seconds to execute.
first_refs = [r.do_request.remote() for _ in range(2)]
_, pending = ray.wait(first_refs, timeout=1)
assert len(pending) == 2
# 2 will be queued in the proxy.
queued_refs = [r.do_request.remote() for _ in range(2)]
_, pending = ray.wait(queued_refs, timeout=0.1)
assert len(pending) == 2
# Send an additional 5 requests. These will be rejected immediately because
# the replica and the proxy queue are already full.
for status_code in ray.get([r.do_request.remote() for _ in range(5)]):
assert status_code == 503
# The initial requests will finish successfully.
for ref in first_refs:
print(f"Request finished with status code {ray.get(ref)}.")
# __client_test_end__
doc/source/serve/doc_code/local_dev.py
+38
View File
@@ -0,0 +1,38 @@
# __local_dev_start__
# Filename: local_dev.py
from starlette.requests import Request
from ray import serve
from ray.serve.handle import DeploymentHandle, DeploymentResponse
@serve.deployment
class Doubler:
def double(self, s: str):
return s + " " + s
@serve.deployment
class HelloDeployment:
def __init__(self, doubler: DeploymentHandle):
self.doubler = doubler
async def say_hello_twice(self, name: str):
return await self.doubler.double.remote(f"Hello, {name}!")
async def __call__(self, request: Request):
return await self.say_hello_twice(request.query_params["name"])
app = HelloDeployment.bind(Doubler.bind())
# __local_dev_end__
# __local_dev_handle_start__
handle: DeploymentHandle = serve.run(app)
response: DeploymentResponse = handle.say_hello_twice.remote(name="Ray")
assert response.result() == "Hello, Ray! Hello, Ray!"
# __local_dev_handle_end__
# __local_dev_testing_start__
serve.run(app, _local_testing_mode=True)
# __local_dev_testing_end__
doc/source/serve/doc_code/managing_deployments.py
+80
View File
@@ -0,0 +1,80 @@
from ray import serve
import time
import os
# __updating_a_deployment_start__
@serve.deployment(name="my_deployment", num_replicas=1)
class SimpleDeployment:
pass
# Creates one initial replica.
serve.run(SimpleDeployment.bind())
# Re-deploys, creating an additional replica.
# This could be the SAME Python script, modified and re-run.
@serve.deployment(name="my_deployment", num_replicas=2)
class SimpleDeployment:
pass
serve.run(SimpleDeployment.bind())
# You can also use Deployment.options() to change options without redefining
# the class. This is useful for programmatically updating deployments.
serve.run(SimpleDeployment.options(num_replicas=2).bind())
# __updating_a_deployment_end__
# __scaling_out_start__
# Create with a single replica.
@serve.deployment(num_replicas=1)
def func(*args):
pass
serve.run(func.bind())
# Scale up to 3 replicas.
serve.run(func.options(num_replicas=3).bind())
# Scale back down to 1 replica.
serve.run(func.options(num_replicas=1).bind())
# __scaling_out_end__
# __autoscaling_start__
@serve.deployment(
autoscaling_config={
"min_replicas": 1,
"initial_replicas": 2,
"max_replicas": 5,
"target_ongoing_requests": 10,
}
)
def func(_):
time.sleep(1)
return ""
serve.run(
func.bind()
) # The func deployment will now autoscale based on requests demand.
# __autoscaling_end__
# __configure_parallism_start__
@serve.deployment
class MyDeployment:
def __init__(self, parallelism: str):
os.environ["OMP_NUM_THREADS"] = parallelism
# Download model weights, initialize model, etc.
def __call__(self):
pass
serve.run(MyDeployment.bind("12"))
# __configure_parallism_end__
doc/source/serve/doc_code/mlflow_model_registry_integration.py
+139
View File
@@ -0,0 +1,139 @@
# __train_model_start__
from sklearn.datasets import make_regression
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
import mlflow
import mlflow.sklearn
import mlflow.pyfunc
from mlflow.entities import LoggedModelStatus
from mlflow.models import infer_signature
import numpy as np
def train_and_register_model():
# Initialize model in PENDING state
logged_model = mlflow.initialize_logged_model(
name="sk-learn-random-forest-reg-model",
model_type="sklearn",
tags={"model_type": "random_forest"},
)
try:
with mlflow.start_run() as run:
X, y = make_regression(n_features=4, n_informative=2, random_state=0, shuffle=False)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)
params = {"max_depth": 2, "random_state": 42}
# Best Practice: Use sklearn Pipeline to persist preprocessing
# This ensures training and serving transformations stay aligned
pipeline = Pipeline([
("scaler", StandardScaler()),
("regressor", RandomForestRegressor(**params))
])
pipeline.fit(X_train, y_train)
# Log parameters and metrics
mlflow.log_params(params)
y_pred = pipeline.predict(X_test)
mlflow.log_metrics({"mse": mean_squared_error(y_test, y_pred)})
# Best Practice: Infer model signature for input validation
# Prevents silent failures from mismatched feature order or missing columns
signature = infer_signature(X_train, y_pred)
# Best Practice: Pin dependency versions explicitly
# Ensures identical behavior across training, evaluation, and serving
pip_requirements = [
f"scikit-learn=={__import__('sklearn').__version__}",
f"numpy=={np.__version__}",
]
# Log the sklearn pipeline with signature and dependencies
mlflow.sklearn.log_model(
sk_model=pipeline,
name="sklearn-model",
input_example=X_train[:1],
signature=signature,
pip_requirements=pip_requirements,
registered_model_name="sk-learn-random-forest-reg-model",
model_id=logged_model.model_id,
)
# Finalize model as READY
mlflow.finalize_logged_model(logged_model.model_id, LoggedModelStatus.READY)
mlflow.set_logged_model_tags(
logged_model.model_id,
tags={"production": "true"},
)
except Exception as e:
# Mark model as FAILED if issues occur
mlflow.finalize_logged_model(logged_model.model_id, LoggedModelStatus.FAILED)
raise
# Retrieve and work with the logged model
final_model = mlflow.get_logged_model(logged_model.model_id)
print(f"Model {final_model.name} is {final_model.status}")
# __train_model_end__
# __deployment_start__
from ray import serve
import mlflow.pyfunc
import numpy as np
@serve.deployment
class MLflowModelDeployment:
def __init__(self):
# Search for models with production tag
models = mlflow.search_logged_models(
filter_string="tags.production='true' AND name='sk-learn-random-forest-reg-model'",
order_by=[{"field_name": "creation_time", "ascending": False}],
)
if models.empty:
raise ValueError("No model with production tag found")
# Get the most recent production model
model_row = models.iloc[0]
artifact_location = model_row["artifact_location"]
# Best Practice: Load model once during initialization (warm-start)
# This eliminates first-request latency spikes
self.model = mlflow.pyfunc.load_model(artifact_location)
# Pre-warm the model with a dummy prediction
dummy_input = np.zeros((1, 4))
_ = self.model.predict(dummy_input)
async def __call__(self, request):
data = await request.json()
features = np.array(data["features"])
# MLflow validates input against the logged signature automatically
prediction = self.model.predict(features)
return {"prediction": prediction.tolist()}
app = MLflowModelDeployment.bind()
# __deployment_end__
if __name__ == "__main__":
import requests
from ray import serve
train_and_register_model()
serve.run(app)
# Test prediction
response = requests.post("http://localhost:8000/", json={"features": [[0.1, 0.2, 0.3, 0.4]]})
print(response.json())
doc/source/serve/doc_code/model_composition/chaining_example.py
+50
View File
@@ -0,0 +1,50 @@
# flake8: noqa
# __chaining_example_start__
# File name: chain.py
from ray import serve
from ray.serve.handle import DeploymentHandle, DeploymentResponse
@serve.deployment
class Adder:
def __init__(self, increment: int):
self._increment = increment
def __call__(self, val: int) -> int:
return val + self._increment
@serve.deployment
class Multiplier:
def __init__(self, multiple: int):
self._multiple = multiple
def __call__(self, val: int) -> int:
return val * self._multiple
@serve.deployment
class Ingress:
def __init__(self, adder: DeploymentHandle, multiplier: DeploymentHandle):
self._adder = adder
self._multiplier = multiplier
async def __call__(self, input: int) -> int:
adder_response: DeploymentResponse = self._adder.remote(input)
# Pass the adder response directly into the multiplier (no `await` needed).
multiplier_response: DeploymentResponse = self._multiplier.remote(
adder_response
)
# `await` the final chained response.
return await multiplier_response
app = Ingress.bind(
Adder.bind(increment=1),
Multiplier.bind(multiple=2),
)
handle: DeploymentHandle = serve.run(app)
response = handle.remote(5)
assert response.result() == 12, "(5 + 1) * 2 = 12"
# __chaining_example_end__
doc/source/serve/doc_code/model_composition/language_example.py
+98
View File
@@ -0,0 +1,98 @@
# flake8: noqa
import requests
# __echo_class_start__
# File name: echo.py
from starlette.requests import Request
from ray import serve
@serve.deployment
class EchoClass:
def __init__(self, echo_str: str):
self.echo_str = echo_str
def __call__(self, request: Request) -> str:
return self.echo_str
# You can create ClassNodes from the EchoClass deployment
foo_node = EchoClass.bind("foo")
bar_node = EchoClass.bind("bar")
baz_node = EchoClass.bind("baz")
# __echo_class_end__
for node, echo in [(foo_node, "foo"), (bar_node, "bar"), (baz_node, "baz")]:
serve.run(node)
assert requests.get("http://localhost:8000/").text == echo
# __echo_client_start__
# File name: echo_client.py
import requests
response = requests.get("http://localhost:8000/")
echo = response.text
print(echo)
# __echo_client_end__
# __hello_start__
# File name: hello.py
from ray import serve
from ray.serve.handle import DeploymentHandle
@serve.deployment
class LanguageClassifer:
def __init__(
self, spanish_responder: DeploymentHandle, french_responder: DeploymentHandle
):
self.spanish_responder = spanish_responder
self.french_responder = french_responder
async def __call__(self, http_request):
request = await http_request.json()
language, name = request["language"], request["name"]
if language == "spanish":
response = self.spanish_responder.say_hello.remote(name)
elif language == "french":
response = self.french_responder.say_hello.remote(name)
else:
return "Please try again."
return await response
@serve.deployment
class SpanishResponder:
def say_hello(self, name: str):
return f"Hola {name}"
@serve.deployment
class FrenchResponder:
def say_hello(self, name: str):
return f"Bonjour {name}"
spanish_responder = SpanishResponder.bind()
french_responder = FrenchResponder.bind()
language_classifier = LanguageClassifer.bind(spanish_responder, french_responder)
# __hello_end__
serve.run(language_classifier)
# __hello_client_start__
# File name: hello_client.py
import requests
response = requests.post(
"http://localhost:8000", json={"language": "spanish", "name": "Dora"}
)
greeting = response.text
print(greeting)
# __hello_client_end__
assert greeting == "Hola Dora"
doc/source/serve/doc_code/model_composition/response_to_object_ref_example.py
+37
View File
@@ -0,0 +1,37 @@
# flake8: noqa
# __response_to_object_ref_example_start__
# File name: response_to_object_ref.py
import ray
from ray import serve
from ray.serve.handle import DeploymentHandle, DeploymentResponse
@ray.remote
def say_hi_task(inp: str):
return f"Ray task got message: '{inp}'"
@serve.deployment
class SayHi:
def __call__(self) -> str:
return "Hi from Serve deployment"
@serve.deployment
class Ingress:
def __init__(self, say_hi: DeploymentHandle):
self._say_hi = say_hi
async def __call__(self):
# Make a call to the SayHi deployment and pass the result ref to
# a downstream Ray task.
response: DeploymentResponse = self._say_hi.remote()
response_obj_ref: ray.ObjectRef = await response._to_object_ref()
final_obj_ref: ray.ObjectRef = say_hi_task.remote(response_obj_ref)
return await final_obj_ref
app = Ingress.bind(SayHi.bind())
handle: DeploymentHandle = serve.run(app)
assert handle.remote().result() == "Ray task got message: 'Hi from Serve deployment'"
# __response_to_object_ref_example_end__
doc/source/serve/doc_code/model_composition/streaming_example.py
+42
View File
@@ -0,0 +1,42 @@
# flake8: noqa
# __streaming_example_start__
# File name: stream.py
from typing import AsyncGenerator, Generator
from ray import serve
from ray.serve.handle import DeploymentHandle, DeploymentResponseGenerator
@serve.deployment
class Streamer:
def __call__(self, limit: int) -> Generator[int, None, None]:
for i in range(limit):
yield i
@serve.deployment
class Caller:
def __init__(self, streamer: DeploymentHandle):
self._streamer = streamer.options(
# Must set `stream=True` on the handle, then the output will be a
# response generator.
stream=True,
)
async def __call__(self, limit: int) -> AsyncGenerator[int, None]:
# Response generator can be used in an `async for` block.
r: DeploymentResponseGenerator = self._streamer.remote(limit)
async for i in r:
yield i
app = Caller.bind(Streamer.bind())
handle: DeploymentHandle = serve.run(app).options(
stream=True,
)
# Response generator can also be used as a regular generator in a sync context.
r: DeploymentResponseGenerator = handle.remote(10)
assert list(r) == list(range(10))
# __streaming_example_end__
doc/source/serve/doc_code/monitoring/custom_metric_snippet.py
+36
View File
@@ -0,0 +1,36 @@
# __start__
from ray import serve
from ray.serve import metrics
import time
import requests
@serve.deployment
class MyDeployment:
def __init__(self):
self.num_requests = 0
self.my_counter = metrics.Counter(
"my_counter",
description=("The number of odd-numbered requests to this deployment."),
tag_keys=("model",),
)
self.my_counter.set_default_tags({"model": "123"})
def __call__(self):
self.num_requests += 1
if self.num_requests % 2 == 1:
self.my_counter.inc()
my_deployment = MyDeployment.bind()
serve.run(my_deployment)
while True:
requests.get("http://localhost:8000/")
time.sleep(1)
# __end__
break
response = requests.get("http://localhost:8000/")
assert response.status_code == 200
doc/source/serve/doc_code/monitoring/deployment_logger.py
+29
View File
@@ -0,0 +1,29 @@
# __start__
from ray import serve
import logging
import requests
logger = logging.getLogger("ray.serve")
@serve.deployment
class Counter:
def __init__(self):
self.count = 0
def __call__(self, request):
self.count += 1
logger.info(f"count: {self.count}")
return {"count": self.count}
counter = Counter.bind()
serve.run(counter)
for i in range(10):
requests.get("http://127.0.0.1:8000/")
# __end__
response = requests.get("http://127.0.0.1:8000/")
assert response.json() == {"count": 11}
doc/source/serve/doc_code/monitoring/logging_config.py
+129
View File
@@ -0,0 +1,129 @@
# flake8: noqa
# __deployment_json_start__
import requests
from ray import serve
from ray.serve.schema import LoggingConfig
@serve.deployment(logging_config=LoggingConfig(encoding="JSON"))
class Model:
def __call__(self) -> int:
return "hello world"
serve.run(Model.bind())
resp = requests.get("http://localhost:8000/")
# __deployment_json_end__
# __serve_run_json_start__
import requests
from ray import serve
from ray.serve.schema import LoggingConfig
@serve.deployment
class Model:
def __call__(self) -> int:
return "hello world"
serve.run(Model.bind(), logging_config=LoggingConfig(encoding="JSON"))
resp = requests.get("http://localhost:8000/")
# __serve_run_json_end__
# __level_start__
@serve.deployment(logging_config=LoggingConfig(log_level="DEBUG"))
class Model:
def __call__(self) -> int:
logger = logging.getLogger("ray.serve")
logger.debug("This debug message is from the router.")
return "hello world"
# __level_end__
serve.run(Model.bind())
resp = requests.get("http://localhost:8000/")
# __logs_dir_start__
@serve.deployment(logging_config=LoggingConfig(logs_dir="/my_dirs"))
class Model:
def __call__(self) -> int:
return "hello world"
# __logs_dir_end__
# __enable_access_log_start__
import requests
import logging
from ray import serve
@serve.deployment(logging_config={"enable_access_log": False})
class Model:
def __call__(self):
logger = logging.getLogger("ray.serve")
logger.info("hello world")
serve.run(Model.bind())
resp = requests.get("http://localhost:8000/")
# __enable_access_log_end__
# __application_and_deployment_start__
import requests
import logging
from ray import serve
@serve.deployment
class Router:
def __init__(self, handle):
self.handle = handle
async def __call__(self):
logger = logging.getLogger("ray.serve")
logger.debug("This debug message is from the router.")
return await self.handle.remote()
@serve.deployment(logging_config={"log_level": "INFO"})
class Model:
def __call__(self) -> int:
logger = logging.getLogger("ray.serve")
logger.debug("This debug message is from the model.")
return "hello world"
serve.run(Router.bind(Model.bind()), logging_config={"log_level": "DEBUG"})
resp = requests.get("http://localhost:8000/")
# __application_and_deployment_end__
# __configure_serve_component_start__
from ray import serve
serve.start(
logging_config={
"encoding": "JSON",
"log_level": "DEBUG",
"enable_access_log": False,
}
)
# __configure_serve_component_end__
doc/source/serve/doc_code/monitoring/metrics_snippet.py
+23
View File
@@ -0,0 +1,23 @@
# __start__
from ray import serve
import time
import requests
@serve.deployment
def sleeper():
time.sleep(1)
s = sleeper.bind()
serve.run(s)
while True:
requests.get("http://localhost:8000/")
# __end__
break
response = requests.get("http://localhost:8000/")
assert response.status_code == 200
doc/source/serve/doc_code/monitoring/monitor_deployment.py
+33
View File
@@ -0,0 +1,33 @@
# flake8: noqa
# fmt: off
# __monitor_start__
from typing import List, Dict
from ray import serve
from ray.serve.schema import ServeStatus, ApplicationStatusOverview
@serve.deployment
def get_healthy_apps() -> List[str]:
serve_status: ServeStatus = serve.status()
app_statuses: Dict[str, ApplicationStatusOverview] = serve_status.applications
running_apps = []
for app_name, app_status in app_statuses.items():
if app_status.status == "RUNNING":
running_apps.append(app_name)
return running_apps
monitoring_app = get_healthy_apps.bind()
# __monitor_end__
serve.run(monitoring_app, name="monitor")
import requests
resp = requests.get("http://localhost:8000/")
assert requests.get("http://localhost:8000/").json() == ["monitor"]
doc/source/serve/doc_code/monitoring/monitoring.py
+27
View File
@@ -0,0 +1,27 @@
# flake8: noqa
# __start__
# File name: monitoring.py
from ray import serve
import logging
from starlette.requests import Request
logger = logging.getLogger("ray.serve")
@serve.deployment
class SayHello:
async def __call__(self, request: Request) -> str:
logger.info("Hello world!")
return "hi"
say_hello = SayHello.bind()
# __end__
# serve.run(say_hello)
# import requests
# response = requests.get("http://localhost:8000/")
# assert response.text == "hi"
doc/source/serve/doc_code/monitoring/request_id.py
+15
View File
@@ -0,0 +1,15 @@
from ray import serve
import requests
@serve.deployment
class Model:
def __call__(self) -> int:
return 1
serve.run(Model.bind())
resp = requests.get("http://localhost:8000", headers={"X-Request-ID": "123-234"})
print(resp.headers["X-Request-ID"])
doc/source/serve/doc_code/multiplexed.py
+99
View File
@@ -0,0 +1,99 @@
# __serve_deployment_example_begin__
from ray import serve
import aioboto3
import torch
import starlette
@serve.deployment
class ModelInferencer:
def __init__(self):
self.bucket_name = "my_bucket"
@serve.multiplexed(max_num_models_per_replica=3)
async def get_model(self, model_id: str):
session = aioboto3.Session()
async with session.resource("s3") as s3:
obj = await s3.Bucket(self.bucket_name)
await obj.download_file(f"{model_id}/model.pt", f"model_{model_id}.pt")
return torch.load(f"model_{model_id}.pt", weights_only=False)
async def __call__(self, request: starlette.requests.Request):
model_id = serve.get_multiplexed_model_id()
model = await self.get_model(model_id)
return model.forward(torch.rand(64, 3, 512, 512))
entry = ModelInferencer.bind()
# __serve_deployment_example_end__
handle = serve.run(entry)
# __serve_request_send_example_begin__
import requests # noqa: E402
resp = requests.get(
"http://localhost:8000", headers={"serve_multiplexed_model_id": str("1")}
)
# __serve_request_send_example_end__
# __serve_handle_send_example_begin__
obj_ref = handle.options(multiplexed_model_id="1").remote("<your param>")
# __serve_handle_send_example_end__
from ray.serve.handle import DeploymentHandle # noqa: E402
# __serve_model_composition_example_begin__
@serve.deployment
class Downstream:
def __call__(self):
return serve.get_multiplexed_model_id()
@serve.deployment
class Upstream:
def __init__(self, downstream: DeploymentHandle):
self._h = downstream
async def __call__(self, request: starlette.requests.Request):
return await self._h.options(multiplexed_model_id="bar").remote()
serve.run(Upstream.bind(Downstream.bind()))
resp = requests.get("http://localhost:8000")
# __serve_model_composition_example_end__
# __serve_multiplexed_batching_example_begin__
from typing import List # noqa: E402
from starlette.requests import Request
@serve.deployment(max_ongoing_requests=15)
class BatchedMultiplexModel:
@serve.multiplexed(max_num_models_per_replica=3)
async def get_model(self, model_id: str):
# Load and return your model here
return model_id
@serve.batch(max_batch_size=10, batch_wait_timeout_s=0.1)
async def batched_predict(self, inputs: List[str]) -> List[str]:
# Get the model ID - this works correctly inside batched functions
# because all requests in the batch target the same model
model_id = serve.get_multiplexed_model_id()
model = await self.get_model(model_id)
# Process the batch with the loaded model
return [f"{model}:{inp}" for inp in inputs]
async def __call__(self, request: Request):
# Extract input from the request body
input_text = await request.body()
return await self.batched_predict(input_text.decode())
# __serve_multiplexed_batching_example_end__
doc/source/serve/doc_code/object_detection.py
+69
View File
@@ -0,0 +1,69 @@
# __example_code_start__
import torch
from PIL import Image
import numpy as np
from io import BytesIO
from fastapi.responses import Response
from fastapi import FastAPI
from ray import serve
from ray.serve.handle import DeploymentHandle
app = FastAPI()
@serve.deployment(num_replicas=1)
@serve.ingress(app)
class APIIngress:
def __init__(self, object_detection_handle: DeploymentHandle):
self.handle = object_detection_handle
@app.get(
"/detect",
responses={200: {"content": {"image/jpeg": {}}}},
response_class=Response,
)
async def detect(self, image_url: str):
image = await self.handle.detect.remote(image_url)
file_stream = BytesIO()
image.save(file_stream, "jpeg")
return Response(content=file_stream.getvalue(), media_type="image/jpeg")
@serve.deployment(
ray_actor_options={"num_gpus": 1},
autoscaling_config={"min_replicas": 1, "max_replicas": 2},
)
class ObjectDetection:
def __init__(self):
self.model = torch.hub.load("ultralytics/yolov5", "yolov5s")
self.model.cuda()
self.model.to(torch.device(0))
def detect(self, image_url: str):
result_im = self.model(image_url)
return Image.fromarray(result_im.render()[0].astype(np.uint8))
entrypoint = APIIngress.bind(ObjectDetection.bind())
# __example_code_end__
if __name__ == "__main__":
import ray
import requests
import os
ray.init(runtime_env={"pip": ["seaborn", "ultralytics"]})
serve.run(entrypoint)
image_url = "https://ultralytics.com/images/zidane.jpg"
resp = requests.get(f"http://127.0.0.1:8000/detect?image_url={image_url}")
with open("output.jpeg", "wb") as f:
f.write(resp.content)
assert os.path.exists("output.jpeg")
os.remove("output.jpeg")
doc/source/serve/doc_code/pd_dissagregation/lmcache/mooncake.yaml
+17
View File
@@ -0,0 +1,17 @@
# LMCache configuration for Mooncake store backend
chunk_size: 256
local_device: "cpu"
remote_url: "mooncakestore://storage-server:49999/"
remote_serde: "naive"
pipelined_backend: false
local_cpu: false
max_local_cpu_size: 5
extra_config:
local_hostname: "compute-node-001"
metadata_server: "etcd://metadata-server:2379"
protocol: "rdma"
device_name: "rdma0"
master_server_address: "storage-server:49999"
global_segment_size: 3355443200 # 3.125 GB
local_buffer_size: 1073741824 # 1 GB
transfer_timeout: 1

Some files were not shown because too many files have changed in this diff Show More