chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,59 @@
|
||||
.. _ray-cluster-cli:
|
||||
|
||||
Cluster Management CLI
|
||||
======================
|
||||
This section contains commands for managing Ray clusters.
|
||||
|
||||
.. _ray-start-doc:
|
||||
|
||||
.. click:: ray.scripts.scripts:start
|
||||
:prog: ray start
|
||||
:show-nested:
|
||||
|
||||
.. _ray-stop-doc:
|
||||
|
||||
.. click:: ray.scripts.scripts:stop
|
||||
:prog: ray stop
|
||||
:show-nested:
|
||||
|
||||
.. _ray-up-doc:
|
||||
|
||||
.. click:: ray.scripts.scripts:up
|
||||
:prog: ray up
|
||||
:show-nested:
|
||||
|
||||
.. _ray-down-doc:
|
||||
|
||||
.. click:: ray.scripts.scripts:down
|
||||
:prog: ray down
|
||||
:show-nested:
|
||||
|
||||
.. _ray-exec-doc:
|
||||
|
||||
.. click:: ray.scripts.scripts:exec
|
||||
:prog: ray exec
|
||||
:show-nested:
|
||||
|
||||
.. _ray-submit-doc:
|
||||
|
||||
.. click:: ray.scripts.scripts:submit
|
||||
:prog: ray submit
|
||||
:show-nested:
|
||||
|
||||
.. _ray-attach-doc:
|
||||
|
||||
.. click:: ray.scripts.scripts:attach
|
||||
:prog: ray attach
|
||||
:show-nested:
|
||||
|
||||
.. _ray-get_head_ip-doc:
|
||||
|
||||
.. click:: ray.scripts.scripts:get_head_ip
|
||||
:prog: ray get_head_ip
|
||||
:show-nested:
|
||||
|
||||
.. _ray-monitor-doc:
|
||||
|
||||
.. click:: ray.scripts.scripts:monitor
|
||||
:prog: ray monitor
|
||||
:show-nested:
|
||||
@@ -0,0 +1,284 @@
|
||||
(observability-configure-manage-dashboard)=
|
||||
# Configuring and Managing Ray Dashboard
|
||||
{ref}`Ray Dashboard<observability-getting-started>` is one of the most important tools to monitor and debug Ray applications and Clusters. This page describes how to configure Ray Dashboard on your Clusters.
|
||||
|
||||
Dashboard configurations may differ depending on how you launch Ray Clusters (e.g., local Ray Cluster vs. KubeRay). Integrations with Prometheus and Grafana are optional for enhanced Dashboard experience.
|
||||
|
||||
:::{note}
|
||||
Ray Dashboard is useful for interactive development and debugging because when clusters terminate, the dashboard UI and the underlying data are no longer accessible. For production monitoring and debugging, you should rely on [persisted logs](../cluster/kubernetes/user-guides/persist-kuberay-custom-resource-logs.md), [persisted metrics](./metrics.md), [persisted Ray states](../ray-observability/user-guides/cli-sdk.rst), and other observability tools.
|
||||
:::
|
||||
|
||||
## Changing the Ray Dashboard port
|
||||
Ray Dashboard runs on port `8265` of the head node. Follow the instructions below to customize the port if needed.
|
||||
|
||||
::::{tab-set}
|
||||
|
||||
:::{tab-item} Single-node local cluster
|
||||
**Start the cluster explicitly with CLI** <br/> Pass the ``--dashboard-port`` argument with ``ray start`` in the command line.
|
||||
|
||||
**Start the cluster implicitly with `ray.init`** <br/> Pass the keyword argument ``dashboard_port`` in your call to ``ray.init()``.
|
||||
:::
|
||||
|
||||
:::{tab-item} VM Cluster Launcher
|
||||
Include the ``--dashboard-port`` argument in the `head_start_ray_commands` section of the [Cluster Launcher's YAML file](https://github.com/ray-project/ray/blob/0574620d454952556fa1befc7694353d68c72049/python/ray/autoscaler/aws/example-full.yaml#L172).
|
||||
```yaml
|
||||
head_start_ray_commands:
|
||||
- ray stop
|
||||
# Replace ${YOUR_PORT} with the port number you need.
|
||||
- ulimit -n 65536; ray start --head --dashboard-port=${YOUR_PORT} --port=6379 --object-manager-port=8076 --autoscaling-config=~/ray_bootstrap_config.yaml
|
||||
|
||||
```
|
||||
:::
|
||||
|
||||
:::{tab-item} KubeRay
|
||||
View the [specifying non-default ports](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/config.html#specifying-non-default-ports) page for details.
|
||||
:::
|
||||
|
||||
::::
|
||||
|
||||
(dashboard-in-browser)=
|
||||
## Viewing Ray Dashboard in browsers
|
||||
When you start a single-node Ray cluster on your laptop, you can access the dashboard through a URL printed when Ray is initialized (the default URL is `http://localhost:8265`).
|
||||
|
||||
|
||||
When you start a remote Ray cluster with the {ref}`VM cluster launcher <vm-cluster-quick-start>`, {ref}`KubeRay operator <kuberay-quickstart>`, or manual configuration, the Ray Dashboard launches on the head node but the dashboard port may not be publicly exposed. You need an additional setup to access the Ray Dashboard from outside the head node.
|
||||
|
||||
:::{danger}
|
||||
For security purposes, do not expose Ray Dashboard publicly without proper authentication in place.
|
||||
:::
|
||||
|
||||
::::{tab-set}
|
||||
|
||||
:::{tab-item} VM Cluster Launcher
|
||||
**Port forwarding** <br/> You can securely port-forward local traffic to the dashboard with the ``ray dashboard`` command.
|
||||
|
||||
```shell
|
||||
$ ray dashboard [-p <port, 8265 by default>] <cluster config file>
|
||||
```
|
||||
|
||||
The dashboard is now visible at ``http://localhost:8265``.
|
||||
:::
|
||||
|
||||
:::{tab-item} KubeRay
|
||||
|
||||
The KubeRay operator makes Dashboard available via a Service targeting the Ray head pod, named ``<RayCluster name>-head-svc``. Access Dashboard from within the Kubernetes cluster at ``http://<RayCluster name>-head-svc:8265``.
|
||||
|
||||
There are two ways to expose Dashboard outside the Cluster:
|
||||
|
||||
**1. Setting up ingress** <br/> Follow the [instructions](kuberay-ingress) to set up ingress to access Ray Dashboard. **The Ingress must only allows access from trusted sources.**
|
||||
|
||||
**2. Port forwarding** <br/> You can also view the dashboard from outside the Kubernetes cluster by using port-forwarding:
|
||||
|
||||
```shell
|
||||
$ kubectl port-forward service/${RAYCLUSTER_NAME}-head-svc 8265:8265
|
||||
# Visit ${YOUR_IP}:8265 for the Dashboard (e.g. 127.0.0.1:8265 or ${YOUR_VM_IP}:8265)
|
||||
```
|
||||
|
||||
```{admonition} Note
|
||||
:class: note
|
||||
Do not use port forwarding for production environment. Follow the instructions above to expose the Dashboard with Ingress.
|
||||
```
|
||||
|
||||
For more information about configuring network access to a Ray cluster on Kubernetes, see the {ref}`networking notes <kuberay-networking>`.
|
||||
|
||||
:::
|
||||
|
||||
::::
|
||||
|
||||
## Running behind a reverse proxy
|
||||
|
||||
Ray Dashboard should work out-of-the-box when accessed via a reverse proxy. API requests don't need to be proxied individually.
|
||||
|
||||
Always access the dashboard with a trailing ``/`` at the end of the URL. For example, if your proxy is set up to handle requests to ``/ray/dashboard``, view the dashboard at ``www.my-website.com/ray/dashboard/``.
|
||||
|
||||
The dashboard sends HTTP requests with relative URL paths. Browsers handle these requests as expected when the ``window.location.href`` ends in a trailing ``/``.
|
||||
|
||||
This is a peculiarity of how many browsers handle requests with relative URLs, despite what [MDN](https://developer.mozilla.org/en-US/docs/Learn/Common_questions/What_is_a_URL#examples_of_relative_urls) defines as the expected behavior.
|
||||
|
||||
Make your dashboard visible without a trailing ``/`` by including a rule in your reverse proxy that redirects the user's browser to ``/``, i.e. ``/ray/dashboard`` --> ``/ray/dashboard/``.
|
||||
|
||||
Below is an example with a [traefik](https://doc.traefik.io/traefik/getting-started/quick-start/) TOML file that accomplishes this:
|
||||
|
||||
```yaml
|
||||
[http]
|
||||
[http.routers]
|
||||
[http.routers.to-dashboard]
|
||||
rule = "PathPrefix(`/ray/dashboard`)"
|
||||
middlewares = ["test-redirectregex", "strip"]
|
||||
service = "dashboard"
|
||||
[http.middlewares]
|
||||
[http.middlewares.test-redirectregex.redirectRegex]
|
||||
regex = "^(.*)/ray/dashboard$"
|
||||
replacement = "${1}/ray/dashboard/"
|
||||
[http.middlewares.strip.stripPrefix]
|
||||
prefixes = ["/ray/dashboard"]
|
||||
[http.services]
|
||||
[http.services.dashboard.loadBalancer]
|
||||
[[http.services.dashboard.loadBalancer.servers]]
|
||||
url = "http://localhost:8265"
|
||||
```
|
||||
|
||||
```{admonition} Warning
|
||||
:class: warning
|
||||
The Ray Dashboard provides read **and write** access to the Ray Cluster. The reverse proxy must provide authentication or network ingress controls to prevent unauthorized access to the Cluster.
|
||||
```
|
||||
|
||||
## Disabling the Dashboard
|
||||
|
||||
Dashboard is included if you use `ray[default]` or {ref}`other installation commands <installation>` and automatically started.
|
||||
|
||||
To disable the Dashboard, use the `--include-dashboard` argument.
|
||||
|
||||
::::{tab-set}
|
||||
|
||||
:::{tab-item} Single-node local cluster
|
||||
|
||||
**Start the cluster explicitly with CLI** <br/>
|
||||
|
||||
```bash
|
||||
ray start --include-dashboard=False
|
||||
```
|
||||
|
||||
**Start the cluster implicitly with `ray.init`** <br/>
|
||||
|
||||
```{testcode}
|
||||
:hide:
|
||||
|
||||
import ray
|
||||
ray.shutdown()
|
||||
```
|
||||
|
||||
```{testcode}
|
||||
import ray
|
||||
ray.init(include_dashboard=False)
|
||||
```
|
||||
|
||||
:::
|
||||
|
||||
:::{tab-item} VM Cluster Launcher
|
||||
Include the `ray start --head --include-dashboard=False` argument in the `head_start_ray_commands` section of the [Cluster Launcher's YAML file](https://github.com/ray-project/ray/blob/0574620d454952556fa1befc7694353d68c72049/python/ray/autoscaler/aws/example-full.yaml#L172).
|
||||
:::
|
||||
|
||||
:::{tab-item} KubeRay
|
||||
|
||||
```{admonition} Warning
|
||||
:class: warning
|
||||
It's not recommended to disable Dashboard because several KubeRay features like `RayJob` and `RayService` depend on it.
|
||||
```
|
||||
|
||||
Set `spec.headGroupSpec.rayStartParams.include-dashboard` to `False`. Check out this [example YAML file](https://gist.github.com/kevin85421/0e6a8dd02c056704327d949b9ec96ef9).
|
||||
|
||||
:::
|
||||
::::
|
||||
|
||||
|
||||
(observability-visualization-setup)=
|
||||
## Embed Grafana visualizations into Ray Dashboard
|
||||
|
||||
For the enhanced Ray Dashboard experience, like {ref}`viewing time-series metrics<dash-metrics-view>` together with logs, Job info, etc., set up Prometheus and Grafana and integrate them with Ray Dashboard.
|
||||
|
||||
### Setting up Prometheus
|
||||
To render Grafana visualizations, you need Prometheus to scrape metrics from Ray Clusters. Follow {ref}`the instructions <prometheus-setup>` to set up your Prometheus server and start to scrape system and application metrics from Ray Clusters.
|
||||
|
||||
|
||||
### Setting up Grafana
|
||||
Grafana is a tool that supports advanced visualizations of Prometheus metrics and allows you to create custom dashboards with your favorite metrics. Follow {ref}`the instructions <grafana>` to set up Grafana.
|
||||
|
||||
|
||||
(embed-grafana-in-dashboard)=
|
||||
### Embedding Grafana visualizations into Ray Dashboard
|
||||
To view embedded time-series visualizations in Ray Dashboard, the following must be set up:
|
||||
|
||||
1. The head node of the cluster is able to access Prometheus and Grafana.
|
||||
2. The browser of the dashboard user is able to access Grafana.
|
||||
|
||||
Configure these settings using the `RAY_GRAFANA_HOST`, `RAY_PROMETHEUS_HOST`, `RAY_PROMETHEUS_NAME`, and `RAY_GRAFANA_IFRAME_HOST` environment variables when you start the Ray Clusters.
|
||||
|
||||
* Set `RAY_GRAFANA_HOST` to an address that the head node can use to access Grafana. Head node does health checks on Grafana on the backend.
|
||||
* Set `RAY_GRAFANA_ORG_ID` to the organization ID you use in Grafana. Default is "1".
|
||||
* Set `RAY_PROMETHEUS_HOST` to an address the head node can use to access Prometheus.
|
||||
* Set `RAY_PROMETHEUS_NAME` to select a different data source to use for the Grafana dashboard panels to use. Default is "Prometheus".
|
||||
* Set `RAY_GRAFANA_IFRAME_HOST` to an address that the user's browsers can use to access Grafana and embed visualizations. If `RAY_GRAFANA_IFRAME_HOST` is not set, Ray Dashboard uses the value of `RAY_GRAFANA_HOST`.
|
||||
|
||||
For example, if the IP of the head node is 55.66.77.88 and Grafana is hosted on port 3000. Set the value to `RAY_GRAFANA_HOST=http://55.66.77.88:3000`.
|
||||
* If you start a single-node Ray Cluster manually, make sure these environment variables are set and accessible before you start the cluster or as a prefix to the `ray start ...` command, e.g., `RAY_GRAFANA_HOST=http://55.66.77.88:3000 ray start ...`
|
||||
* If you start a Ray Cluster with {ref}`VM Cluster Launcher <cloud-vm-index>`, the environment variables should be set under `head_start_ray_commands` as a prefix to the `ray start ...` command.
|
||||
* If you start a Ray Cluster with {ref}`KubeRay <kuberay-index>`, refer to this {ref}`tutorial <kuberay-prometheus-grafana>`.
|
||||
|
||||
If all the environment variables are set properly, you should see time-series metrics in {ref}`Ray Dashboard <observability-getting-started>`.
|
||||
|
||||
:::{note}
|
||||
If you use a different Prometheus server for each Ray Cluster and use the same Grafana server for all Clusters, set the `RAY_PROMETHEUS_NAME` environment variable to different values for each Ray Cluster and add these datasources in Grafana. Follow {ref}`these instructions <grafana>` to set up Grafana.
|
||||
:::
|
||||
|
||||
#### Alternate Prometheus host location
|
||||
By default, Ray Dashboard assumes Prometheus is hosted at `localhost:9090`. You can choose to run Prometheus on a non-default port or on a different machine. In this case, make sure that Prometheus can scrape the metrics from your Ray nodes following instructions {ref}`here <scrape-metrics>`.
|
||||
|
||||
Then, configure `RAY_PROMETHEUS_HOST` environment variable properly as stated above. For example, if Prometheus is hosted at port 9000 on a node with ip 55.66.77.88, set `RAY_PROMETHEUS_HOST=http://55.66.77.88:9000`.
|
||||
|
||||
|
||||
#### Customize headers for requests from the Ray dashboard to Prometheus
|
||||
|
||||
If Prometheus requires additional headers for authentication, set `RAY_PROMETHEUS_HEADERS` in one of the following JSON formats for Ray dashboard to send them to Prometheus:
|
||||
1. `{"Header1": "Value1", "Header2": "Value2"}`
|
||||
2. `[["Header1", "Value1"], ["Header2", "Value2"], ["Header2", "Value3"]]`
|
||||
|
||||
|
||||
#### Alternate Grafana host location
|
||||
By default, Ray Dashboard assumes Grafana is hosted at `localhost:3000`. You can choose to run Grafana on a non-default port or on a different machine as long as the head node and the dashboard browsers of can access it.
|
||||
|
||||
If Grafana is exposed with NGINX ingress on a Kubernetes cluster, the following line should be present in the Grafana ingress annotation:
|
||||
|
||||
```yaml
|
||||
nginx.ingress.kubernetes.io/configuration-snippet: |
|
||||
add_header X-Frame-Options SAMEORIGIN always;
|
||||
```
|
||||
|
||||
When both Grafana and the Ray Cluster are on the same Kubernetes cluster, set `RAY_GRAFANA_HOST` to the external URL of the Grafana ingress.
|
||||
|
||||
|
||||
|
||||
#### User authentication for Grafana
|
||||
When the Grafana instance requires user authentication, the following settings have to be in its [configuration file](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/) to correctly embed in Ray Dashboard:
|
||||
|
||||
```ini
|
||||
[security]
|
||||
allow_embedding = true
|
||||
cookie_secure = true
|
||||
cookie_samesite = none
|
||||
```
|
||||
|
||||
#### Troubleshooting
|
||||
|
||||
##### Dashboard message: either Prometheus or Grafana server is not detected
|
||||
If you have followed the instructions above to set up everything, run the connection checks below in your browser:
|
||||
* check Head Node connection to Prometheus server: add `api/prometheus_health` to the end of Ray Dashboard URL (for example: http://127.0.0.1:8265/api/prometheus_health)and visit it.
|
||||
* check Head Node connection to Grafana server: add `api/grafana_health` to the end of Ray Dashboard URL (for example: http://127.0.0.1:8265/api/grafana_health) and visit it.
|
||||
* check browser connection to Grafana server: visit the URL used in `RAY_GRAFANA_IFRAME_HOST`.
|
||||
|
||||
|
||||
##### Getting an error that says `RAY_GRAFANA_HOST` is not setup
|
||||
If you have set up Grafana, check that:
|
||||
* You've included the protocol in the URL (e.g., `http://your-grafana-url.com` instead of `your-grafana-url.com`).
|
||||
* The URL doesn't have a trailing slash (e.g., `http://your-grafana-url.com` instead of `http://your-grafana-url.com/`).
|
||||
|
||||
##### Certificate Authority (CA error)
|
||||
You may see a CA error if your Grafana instance is hosted behind HTTPS. Contact the Grafana service owner to properly enable HTTPS traffic.
|
||||
|
||||
|
||||
## Viewing built-in Dashboard API metrics
|
||||
|
||||
Dashboard is powered by a server that serves both the UI code and the data about the cluster via API endpoints. Ray emits basic Prometheus metrics for each API endpoint:
|
||||
|
||||
`ray_dashboard_api_requests_count_requests_total`: Collects the total count of requests. This is tagged by endpoint, method, and http_status.
|
||||
|
||||
`ray_dashboard_api_requests_duration_seconds_bucket`: Collects the duration of requests. This is tagged by endpoint and method.
|
||||
|
||||
For example, you can view the p95 duration of all requests with this query:
|
||||
|
||||
```text
|
||||
|
||||
histogram_quantile(0.95, sum(rate(ray_dashboard_api_requests_duration_seconds_bucket[5m])) by (le))
|
||||
```
|
||||
|
||||
You can query these metrics from the Prometheus or Grafana UI. Find instructions above for how to set these tools up.
|
||||
@@ -0,0 +1,32 @@
|
||||
import os
|
||||
import socket
|
||||
import sys
|
||||
import time
|
||||
|
||||
# trainer.py
|
||||
from collections import Counter
|
||||
|
||||
import ray
|
||||
|
||||
num_cpus = int(sys.argv[1])
|
||||
|
||||
ray.init(address=os.environ["ip_head"])
|
||||
|
||||
print("Nodes in the Ray cluster:")
|
||||
print(ray.nodes())
|
||||
|
||||
|
||||
@ray.remote
|
||||
def f():
|
||||
time.sleep(1)
|
||||
return socket.gethostbyname("localhost")
|
||||
|
||||
|
||||
# The following takes one second (assuming that
|
||||
# ray was able to access all of the allocated nodes).
|
||||
for i in range(60):
|
||||
start = time.time()
|
||||
ip_addresses = ray.get([f.remote() for _ in range(num_cpus)])
|
||||
print(Counter(ip_addresses))
|
||||
end = time.time()
|
||||
print(end - start)
|
||||
@@ -0,0 +1,45 @@
|
||||
#!/bin/bash
|
||||
# shellcheck disable=SC2206
|
||||
#SBATCH --job-name=test
|
||||
#SBATCH --cpus-per-task=5
|
||||
#SBATCH --mem-per-cpu=1GB
|
||||
#SBATCH --nodes=4
|
||||
#SBATCH --tasks-per-node=1
|
||||
#SBATCH --time=00:30:00
|
||||
|
||||
set -x
|
||||
|
||||
# __doc_head_address_start__
|
||||
|
||||
# Getting the node names
|
||||
nodes=$(scontrol show hostnames "$SLURM_JOB_NODELIST")
|
||||
nodes_array=($nodes)
|
||||
|
||||
head_node=${nodes_array[0]}
|
||||
|
||||
port=6379
|
||||
ip_head=$head_node:$port
|
||||
export ip_head
|
||||
echo "IP Head: $ip_head"
|
||||
# __doc_head_address_end__
|
||||
|
||||
# __doc_symmetric_run_start__
|
||||
# Start Ray cluster using symmetric_run.py on all nodes.
|
||||
# Symmetric run will automatically start Ray on all nodes and run the script ONLY the head node.
|
||||
# Use the '--' separator to separate Ray arguments and the entrypoint command.
|
||||
# The --min-nodes argument ensures all nodes join before running the script.
|
||||
|
||||
# All nodes (including head and workers) will execute this block.
|
||||
# The entrypoint (simple-trainer.py) will only run on the head node.
|
||||
srun --nodes="$SLURM_JOB_NUM_NODES" --ntasks="$SLURM_JOB_NUM_NODES" \
|
||||
ray symmetric-run \
|
||||
--address "$ip_head" \
|
||||
--min-nodes "$SLURM_JOB_NUM_NODES" \
|
||||
--num-cpus="${SLURM_CPUS_PER_TASK}" \
|
||||
--num-gpus="${SLURM_GPUS_PER_TASK}" \
|
||||
-- \
|
||||
python -u simple-trainer.py "$SLURM_CPUS_PER_TASK"
|
||||
# __doc_symmetric_run_end__
|
||||
|
||||
# __doc_script_start__
|
||||
# The entrypoint script (simple-trainer.py) will be run on the head node by symmetric_run.
|
||||
@@ -0,0 +1,109 @@
|
||||
# slurm-launch.py
|
||||
# Usage:
|
||||
# python slurm-launch.py --exp-name test \
|
||||
# --command "rllib train --run PPO --env CartPole-v0"
|
||||
|
||||
import argparse
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
template_file = Path(__file__) / "slurm-template.sh"
|
||||
JOB_NAME = "${JOB_NAME}"
|
||||
NUM_NODES = "${NUM_NODES}"
|
||||
NUM_GPUS_PER_NODE = "${NUM_GPUS_PER_NODE}"
|
||||
PARTITION_OPTION = "${PARTITION_OPTION}"
|
||||
COMMAND_PLACEHOLDER = "${COMMAND_PLACEHOLDER}"
|
||||
GIVEN_NODE = "${GIVEN_NODE}"
|
||||
LOAD_ENV = "${LOAD_ENV}"
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument(
|
||||
"--exp-name",
|
||||
type=str,
|
||||
required=True,
|
||||
help="The job name and path to logging file (exp_name.log).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--num-nodes", "-n", type=int, default=1, help="Number of nodes to use."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--node",
|
||||
"-w",
|
||||
type=str,
|
||||
help="The specified nodes to use. Same format as the "
|
||||
"return of 'sinfo'. Default: ''.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--num-gpus",
|
||||
type=int,
|
||||
default=0,
|
||||
help="Number of GPUs to use in each node. (Default: 0)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--partition",
|
||||
"-p",
|
||||
type=str,
|
||||
)
|
||||
parser.add_argument(
|
||||
"--load-env",
|
||||
type=str,
|
||||
help="The script to load your environment ('module load cuda/10.1')",
|
||||
default="",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--command",
|
||||
type=str,
|
||||
required=True,
|
||||
help="The command you wish to execute. For example: "
|
||||
" --command 'python test.py'. "
|
||||
"Note that the command must be a string.",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.node:
|
||||
# assert args.num_nodes == 1
|
||||
node_info = "#SBATCH -w {}".format(args.node)
|
||||
else:
|
||||
node_info = ""
|
||||
|
||||
job_name = "{}_{}".format(
|
||||
args.exp_name, time.strftime("%m%d-%H%M", time.localtime())
|
||||
)
|
||||
|
||||
partition_option = (
|
||||
"#SBATCH --partition={}".format(args.partition) if args.partition else ""
|
||||
)
|
||||
|
||||
# ===== Modified the template script =====
|
||||
with open(template_file, "r") as f:
|
||||
text = f.read()
|
||||
text = text.replace(JOB_NAME, job_name)
|
||||
text = text.replace(NUM_NODES, str(args.num_nodes))
|
||||
text = text.replace(NUM_GPUS_PER_NODE, str(args.num_gpus))
|
||||
text = text.replace(PARTITION_OPTION, partition_option)
|
||||
text = text.replace(COMMAND_PLACEHOLDER, str(args.command))
|
||||
text = text.replace(LOAD_ENV, str(args.load_env))
|
||||
text = text.replace(GIVEN_NODE, node_info)
|
||||
text = text.replace(
|
||||
"# THIS FILE IS A TEMPLATE AND IT SHOULD NOT BE DEPLOYED TO PRODUCTION!",
|
||||
"# THIS FILE IS MODIFIED AUTOMATICALLY FROM TEMPLATE AND SHOULD BE "
|
||||
"RUNNABLE!",
|
||||
)
|
||||
|
||||
# ===== Save the script =====
|
||||
script_file = "{}.sh".format(job_name)
|
||||
with open(script_file, "w") as f:
|
||||
f.write(text)
|
||||
|
||||
# ===== Submit the job =====
|
||||
print("Starting to submit job!")
|
||||
subprocess.Popen(["sbatch", script_file])
|
||||
print(
|
||||
"Job submitted! Script file is at: <{}>. Log file is at: <{}>".format(
|
||||
script_file, "{}.log".format(job_name)
|
||||
)
|
||||
)
|
||||
sys.exit(0)
|
||||
@@ -0,0 +1,64 @@
|
||||
#!/bin/bash
|
||||
# shellcheck disable=SC2206
|
||||
# THIS FILE IS GENERATED BY AUTOMATION SCRIPT! PLEASE REFER TO ORIGINAL SCRIPT!
|
||||
# THIS FILE IS A TEMPLATE AND IT SHOULD NOT BE DEPLOYED TO PRODUCTION!
|
||||
${PARTITION_OPTION}
|
||||
#SBATCH --job-name=${JOB_NAME}
|
||||
#SBATCH --output=${JOB_NAME}.log
|
||||
${GIVEN_NODE}
|
||||
### This script works for any number of nodes, Ray will find and manage all resources
|
||||
#SBATCH --nodes=${NUM_NODES}
|
||||
#SBATCH --exclusive
|
||||
### Give all resources to a single Ray task, ray can manage the resources internally
|
||||
#SBATCH --ntasks-per-node=1
|
||||
#SBATCH --gpus-per-task=${NUM_GPUS_PER_NODE}
|
||||
|
||||
# Load modules or your own conda environment here
|
||||
# module load pytorch/v1.4.0-gpu
|
||||
# conda activate ${CONDA_ENV}
|
||||
${LOAD_ENV}
|
||||
|
||||
# ===== DO NOT CHANGE THINGS HERE UNLESS YOU KNOW WHAT YOU ARE DOING =====
|
||||
# This script is a modification to the implementation suggest by gregSchwartz18 here:
|
||||
# https://github.com/ray-project/ray/issues/826#issuecomment-522116599
|
||||
redis_password=$(uuidgen)
|
||||
export redis_password
|
||||
|
||||
nodes=$(scontrol show hostnames "$SLURM_JOB_NODELIST") # Getting the node names
|
||||
nodes_array=($nodes)
|
||||
|
||||
node_1=${nodes_array[0]}
|
||||
ip=$(srun --nodes=1 --ntasks=1 -w "$node_1" hostname --ip-address) # making redis-address
|
||||
|
||||
# if we detect a space character in the head node IP, we'll
|
||||
# convert it to an ipv4 address. This step is optional.
|
||||
if [[ "$ip" == *" "* ]]; then
|
||||
IFS=' ' read -ra ADDR <<< "$ip"
|
||||
if [[ ${#ADDR[0]} -gt 16 ]]; then
|
||||
ip=${ADDR[1]}
|
||||
else
|
||||
ip=${ADDR[0]}
|
||||
fi
|
||||
echo "IPV6 address detected. We split the IPV4 address as $ip"
|
||||
fi
|
||||
|
||||
port=6379
|
||||
ip_head=$ip:$port
|
||||
export ip_head
|
||||
echo "IP Head: $ip_head"
|
||||
|
||||
echo "STARTING HEAD at $node_1"
|
||||
srun --nodes=1 --ntasks=1 -w "$node_1" \
|
||||
ray start --head --node-ip-address="$ip" --port=$port --redis-password="$redis_password" --block &
|
||||
sleep 30
|
||||
|
||||
worker_num=$((SLURM_JOB_NUM_NODES - 1)) #number of nodes other than the head node
|
||||
for ((i = 1; i <= worker_num; i++)); do
|
||||
node_i=${nodes_array[$i]}
|
||||
echo "STARTING WORKER $i at $node_i"
|
||||
srun --nodes=1 --ntasks=1 -w "$node_i" ray start --address "$ip_head" --redis-password="$redis_password" --block &
|
||||
sleep 5
|
||||
done
|
||||
|
||||
# ===== Call your code below =====
|
||||
${COMMAND_PLACEHOLDER}
|
||||
@@ -0,0 +1,19 @@
|
||||
from ray.job_submission import JobSubmissionClient
|
||||
|
||||
client = JobSubmissionClient("http://127.0.0.1:8265")
|
||||
|
||||
kick_off_xgboost_benchmark = (
|
||||
# Clone ray. If ray is already present, don't clone again.
|
||||
"git clone https://github.com/ray-project/ray || true; "
|
||||
# Run the benchmark.
|
||||
"python ray/release/train_tests/xgboost_lightgbm/train_batch_inference_benchmark.py"
|
||||
" xgboost --size=100G --disable-check"
|
||||
)
|
||||
|
||||
|
||||
submission_id = client.submit_job(
|
||||
entrypoint=kick_off_xgboost_benchmark,
|
||||
)
|
||||
|
||||
print("Use the following command to follow this Job's logs:")
|
||||
print(f"ray job logs '{submission_id}' --follow")
|
||||
@@ -0,0 +1,18 @@
|
||||
import skein
|
||||
import sys
|
||||
from urllib.parse import urlparse
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) < 2:
|
||||
print("Usage: python dashboard.py <dashboard-address>")
|
||||
sys.exit(1)
|
||||
address = sys.argv[1]
|
||||
# Check if the address is a valid URL
|
||||
result = urlparse(address)
|
||||
if not all([result.scheme, result.netloc]):
|
||||
print("Error: Invalid dashboard address. Please provide a valid URL.")
|
||||
sys.exit(1)
|
||||
|
||||
print("Registering dashboard " + address + " on skein.")
|
||||
app = skein.ApplicationClient.from_current()
|
||||
app.ui.add_page("ray-dashboard", address, "Ray Dashboard")
|
||||
@@ -0,0 +1,50 @@
|
||||
import sys
|
||||
import time
|
||||
from collections import Counter
|
||||
|
||||
import ray
|
||||
|
||||
|
||||
@ray.remote
|
||||
def get_host_name(x):
|
||||
import platform
|
||||
import time
|
||||
|
||||
time.sleep(0.01)
|
||||
return x + (platform.node(),)
|
||||
|
||||
|
||||
def wait_for_nodes(expected):
|
||||
# Wait for all nodes to join the cluster.
|
||||
while True:
|
||||
num_nodes = len(ray.nodes())
|
||||
if num_nodes < expected:
|
||||
print(
|
||||
"{} nodes have joined so far, waiting for {} more.".format(
|
||||
num_nodes, expected - num_nodes
|
||||
)
|
||||
)
|
||||
sys.stdout.flush()
|
||||
time.sleep(1)
|
||||
else:
|
||||
break
|
||||
|
||||
|
||||
def main():
|
||||
wait_for_nodes(4)
|
||||
|
||||
# Check that objects can be transferred from each node to each other node.
|
||||
for i in range(10):
|
||||
print("Iteration {}".format(i))
|
||||
results = [get_host_name.remote(get_host_name.remote(())) for _ in range(100)]
|
||||
print(Counter(ray.get(results)))
|
||||
sys.stdout.flush()
|
||||
|
||||
print("Success!")
|
||||
sys.stdout.flush()
|
||||
time.sleep(20)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
ray.init(address="localhost:6379")
|
||||
main()
|
||||
@@ -0,0 +1,71 @@
|
||||
name: ray
|
||||
|
||||
services:
|
||||
# Head service.
|
||||
ray-head:
|
||||
# There should only be one instance of the head node per cluster.
|
||||
instances: 1
|
||||
resources:
|
||||
# The resources for the head node.
|
||||
vcores: 1
|
||||
memory: 2048
|
||||
files:
|
||||
# ray/doc/source/cluster/doc_code/yarn/example.py
|
||||
example.py: example.py
|
||||
# ray/doc/source/cluster/doc_code/yarn/dashboard.py
|
||||
dashboard.py: dashboard.py
|
||||
# # A packaged python environment using `conda-pack`. Note that Skein
|
||||
# # doesn't require any specific way of distributing files, but this
|
||||
# # is a good one for python projects. This is optional.
|
||||
# # See https://jcrist.github.io/skein/distributing-files.html
|
||||
# environment: environment.tar.gz
|
||||
script: |
|
||||
# Activate the packaged conda environment
|
||||
# - source environment/bin/activate
|
||||
|
||||
# This gets the IP address of the head node.
|
||||
RAY_HEAD_ADDRESS=$(hostname -i)
|
||||
|
||||
# This stores the Ray head address in the Skein key-value store so that the workers can retrieve it later.
|
||||
skein kv put current --key=RAY_HEAD_ADDRESS --value=$RAY_HEAD_ADDRESS
|
||||
|
||||
# This command starts all the processes needed on the ray head node.
|
||||
# By default, we set object store memory and heap memory to roughly 200 MB. This is conservative
|
||||
# and should be set according to application needs.
|
||||
#
|
||||
ray start --head --port=6379 --object-store-memory=200000000 --memory 200000000 --num-cpus=1 --dashboard-host=$RAY_HEAD_ADDRESS
|
||||
|
||||
# This registers the Ray dashboard on Skein, which can be accessed on Skein's web UI.
|
||||
python dashboard.py "http://$RAY_HEAD_ADDRESS:8265"
|
||||
|
||||
# This executes the user script.
|
||||
python example.py
|
||||
|
||||
# After the user script has executed, all started processes should also die.
|
||||
ray stop
|
||||
skein application shutdown current
|
||||
# Worker service.
|
||||
ray-worker:
|
||||
# The number of instances to start initially. This can be scaled
|
||||
# dynamically later.
|
||||
instances: 4
|
||||
resources:
|
||||
# The resources for the worker node
|
||||
vcores: 1
|
||||
memory: 2048
|
||||
# files:
|
||||
# environment: environment.tar.gz
|
||||
depends:
|
||||
# Don't start any worker nodes until the head node is started
|
||||
- ray-head
|
||||
script: |
|
||||
# Activate the packaged conda environment
|
||||
# - source environment/bin/activate
|
||||
|
||||
# This command gets any addresses it needs (e.g. the head node) from
|
||||
# the skein key-value store.
|
||||
RAY_HEAD_ADDRESS=$(skein kv get --key=RAY_HEAD_ADDRESS current)
|
||||
|
||||
# The below command starts all the processes needed on a ray worker node, blocking until killed with sigterm.
|
||||
# After sigterm, all started processes should also die (ray stop).
|
||||
ray start --object-store-memory=200000000 --memory 200000000 --num-cpus=1 --address=$RAY_HEAD_ADDRESS:6379 --block; ray stop
|
||||
@@ -0,0 +1,103 @@
|
||||
.. _cluster-FAQ:
|
||||
|
||||
===
|
||||
FAQ
|
||||
===
|
||||
|
||||
These are some Frequently Asked Questions for Ray clusters.
|
||||
If you still have questions after reading this FAQ, reach out on the
|
||||
`Ray Discourse forum <https://discuss.ray.io/>`__.
|
||||
|
||||
Do Ray clusters support multi-tenancy?
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Yes, you can run multiple :ref:`jobs <jobs-overview>` from different users simultaneously in a Ray cluster
|
||||
but it's not recommended in production.
|
||||
Some Ray features are still missing for multi-tenancy in production:
|
||||
|
||||
* Ray doesn't provide strong resource isolation:
|
||||
Ray :ref:`resources <core-resources>` are logical and they don't limit the physical resources a task or actor can use while running.
|
||||
This means simultaneous jobs can interfere with each other and makes them less reliable to run in production.
|
||||
|
||||
* Ray doesn't support priorities: All jobs, tasks and actors have the same priority so there is no way to prioritize important jobs under load.
|
||||
|
||||
* Ray doesn't support access control: Jobs have full access to a Ray cluster and all of the resources within it.
|
||||
|
||||
On the other hand, you can run the same job multiple times using the same cluster to save the cluster startup time.
|
||||
|
||||
.. note::
|
||||
A Ray :ref:`namespace <namespaces-guide>` is just a logical grouping of jobs and named actors. Unlike a Kubernetes namespace, it doesn't provide any other multi-tenancy functions like resource quotas.
|
||||
|
||||
|
||||
I have multiple Ray users. What's the right way to deploy Ray for them?
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Start a Ray cluster for each user to isolate their workloads.
|
||||
|
||||
What's the difference between ``--node-ip-address`` and ``--address``?
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
When starting a head node on a machine with more than one network address, you
|
||||
may need to specify the externally available address so worker nodes can
|
||||
connect. Use this command:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
ray start --head --node-ip-address xx.xx.xx.xx --port nnnn
|
||||
|
||||
Then when starting the worker node, use this command to connect to the head node:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
ray start --address xx.xx.xx.xx:nnnn
|
||||
|
||||
What does a worker node failure to connect look like?
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
If the worker node can't connect to the head node, you should see this error:
|
||||
|
||||
Unable to connect to GCS at xx.xx.xx.xx:nnnn. Check that (1) Ray GCS with
|
||||
matching version started successfully at the specified address, and (2)
|
||||
there is no firewall setting preventing access.
|
||||
|
||||
The most likely cause is that the worker node can't access the IP address
|
||||
given. You can use ``ip route get xx.xx.xx.xx`` on the worker node to start
|
||||
debugging routing issues.
|
||||
|
||||
You may also see failures in the log like:
|
||||
|
||||
This node has an IP address of xx.xx.xx.xx, while we cannot find the
|
||||
matched Raylet address. This may come from when you connect the Ray
|
||||
cluster with a different IP address or connect a container.
|
||||
|
||||
The cause of this error may be the head node overloading with too many simultaneous
|
||||
connections. The solution for this problem is to start the worker nodes more slowly.
|
||||
|
||||
Problems getting a SLURM cluster to work
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
A class of issues exist with starting Ray on SLURM clusters. While the exact causes aren't understood, (as of June 2023), some Ray
|
||||
improvements mitigate some of the resource contention. Some of the issues
|
||||
reported are as follows:
|
||||
|
||||
* Using a machine with a large number of CPUs, and starting one worker per CPU
|
||||
together with OpenBLAS (as used in NumPy) may allocate too many threads. This
|
||||
issue is a `known OpenBLAS limitation`_. You can mitigate it by limiting OpenBLAS
|
||||
to one thread per process as explained in the link.
|
||||
|
||||
* Resource allocation isn't as expected: usually the configuration has too many CPUs allocated per node. The best practice is to verify the SLURM configuration without
|
||||
starting Ray to verify that the allocations are as expected. For more
|
||||
detailed information see :ref:`ray-slurm-deploy`.
|
||||
|
||||
.. _`known OpenBLAS limitation`: http://www.openmathlib.org/OpenBLAS/docs/faq/#how-can-i-use-openblas-in-multi-threaded-applications
|
||||
|
||||
Where does my Ray Job entrypoint script run? On the head node or worker nodes?
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
By default, jobs submitted using the :ref:`Ray Job API <jobs-quickstart>` run
|
||||
their `entrypoint` script on the head node. You can change this by specifying
|
||||
any of the options `--entrypoint-num-cpus`, `--entrypoint-num-gpus`,
|
||||
`--entrypoint-resources` or `--entrypoint-memory` to `ray job submit`, or the
|
||||
corresponding arguments if using the Python SDK. If these are specified, the
|
||||
job entrypoint will be scheduled on a node that has the requested resources
|
||||
available.
|
||||
@@ -0,0 +1,109 @@
|
||||
.. _cluster-index:
|
||||
|
||||
Ray Clusters Overview
|
||||
=====================
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
|
||||
Key Concepts <key-concepts>
|
||||
Deploying on Kubernetes <kubernetes/index>
|
||||
Deploying on VMs <vms/index>
|
||||
metrics
|
||||
configure-manage-dashboard
|
||||
Applications Guide <running-applications/index>
|
||||
faq
|
||||
package-overview
|
||||
usage-stats
|
||||
|
||||
|
||||
Ray enables seamless scaling of workloads from a laptop to a large cluster. While Ray
|
||||
works out of the box on single machines with just a call to ``ray.init``, to run Ray
|
||||
applications on multiple nodes you must first *deploy a Ray cluster*.
|
||||
|
||||
A Ray cluster is a set of worker nodes connected to a common :ref:`Ray head node <cluster-head-node>`.
|
||||
Ray clusters can be fixed-size, or they may :ref:`autoscale up and down <cluster-autoscaler>` according
|
||||
to the resources requested by applications running on the cluster.
|
||||
|
||||
Where can I deploy Ray clusters?
|
||||
--------------------------------
|
||||
|
||||
Ray provides native cluster deployment support on the following technology stacks:
|
||||
|
||||
* On :ref:`AWS, GCP, and Azure <cloud-vm-index>`. Community-supported Aliyun and vSphere integrations also exist.
|
||||
* On :ref:`Kubernetes <kuberay-index>`, via the officially supported KubeRay project.
|
||||
* On `Anyscale <https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=ray-doc-upsell&utm_content=ray-cluster-deployment>`_, a fully managed Ray platform by the creators of Ray. You can either bring an existing AWS, GCP, Azure and Kubernetes clusters, or use the Anyscale hosted compute layer.
|
||||
|
||||
Advanced users may want to :ref:`deploy Ray manually <on-prem>`
|
||||
or onto :ref:`platforms not listed here <ref-cluster-setup>`.
|
||||
|
||||
.. note::
|
||||
|
||||
Multi-node Ray clusters are only supported on Linux. At your own risk, you
|
||||
may deploy Windows and OSX clusters by setting the environment variable
|
||||
``RAY_ENABLE_WINDOWS_OR_OSX_CLUSTER=1`` during deployment.
|
||||
|
||||
What's next?
|
||||
------------
|
||||
|
||||
.. grid:: 1 2 2 2
|
||||
:gutter: 1
|
||||
:class-container: container pb-3
|
||||
|
||||
.. grid-item-card::
|
||||
|
||||
**I want to learn key Ray cluster concepts**
|
||||
^^^
|
||||
Understand the key concepts and main ways of interacting with a Ray cluster.
|
||||
|
||||
+++
|
||||
.. button-ref:: cluster-key-concepts
|
||||
:color: primary
|
||||
:outline:
|
||||
:expand:
|
||||
|
||||
Learn Key Concepts
|
||||
|
||||
.. grid-item-card::
|
||||
|
||||
**I want to run Ray on Kubernetes**
|
||||
^^^
|
||||
Deploy a Ray application to a Kubernetes cluster. You can run the tutorial on a
|
||||
Kubernetes cluster or on your laptop via Kind.
|
||||
|
||||
+++
|
||||
.. button-ref:: kuberay-quickstart
|
||||
:color: primary
|
||||
:outline:
|
||||
:expand:
|
||||
|
||||
Get Started with Ray on Kubernetes
|
||||
|
||||
.. grid-item-card::
|
||||
|
||||
**I want to run Ray on a cloud provider**
|
||||
^^^
|
||||
Take a sample application designed to run on a laptop and scale it up in the
|
||||
cloud. Access to an AWS or GCP account is required.
|
||||
|
||||
+++
|
||||
.. button-ref:: vm-cluster-quick-start
|
||||
:color: primary
|
||||
:outline:
|
||||
:expand:
|
||||
|
||||
Get Started with Ray on VMs
|
||||
|
||||
.. grid-item-card::
|
||||
|
||||
**I want to run my application on an existing Ray cluster**
|
||||
^^^
|
||||
Guide to submitting applications as Jobs to existing Ray clusters.
|
||||
|
||||
+++
|
||||
.. button-ref:: jobs-quickstart
|
||||
:color: primary
|
||||
:outline:
|
||||
:expand:
|
||||
|
||||
Job Submission
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 147 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 123 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.0 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 122 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 543 KiB |
@@ -0,0 +1,87 @@
|
||||
Key Concepts
|
||||
============
|
||||
|
||||
.. _cluster-key-concepts:
|
||||
|
||||
This page introduces key concepts for Ray clusters:
|
||||
|
||||
.. contents::
|
||||
:local:
|
||||
|
||||
Ray Cluster
|
||||
-----------
|
||||
A Ray cluster consists of a single :ref:`head node <cluster-head-node>`
|
||||
and any number of connected :ref:`worker nodes <cluster-worker-nodes>`:
|
||||
|
||||
.. figure:: images/ray-cluster.svg
|
||||
:align: center
|
||||
:width: 600px
|
||||
|
||||
*A Ray cluster with two worker nodes. Each node runs Ray helper processes to
|
||||
facilitate distributed scheduling and memory management. The head node runs
|
||||
additional control processes (highlighted in blue).*
|
||||
|
||||
The number of worker nodes may be *autoscaled* with application demand as specified
|
||||
by your Ray cluster configuration. The head node runs the :ref:`autoscaler <cluster-autoscaler>`.
|
||||
|
||||
.. note::
|
||||
Ray nodes are implemented as pods when :ref:`running on Kubernetes <kuberay-index>`.
|
||||
|
||||
Users can submit jobs for execution on the Ray cluster, or can interactively use the
|
||||
cluster by connecting to the head node and running `ray.init`. See
|
||||
:ref:`Ray Jobs <jobs-quickstart>` for more information.
|
||||
|
||||
.. _cluster-head-node:
|
||||
|
||||
Head Node
|
||||
---------
|
||||
Every Ray cluster has one node which is designated as the *head node* of the cluster.
|
||||
The head node is identical to other worker nodes, except that it also runs singleton processes responsible for cluster management such as the
|
||||
:ref:`autoscaler <cluster-autoscaler>`, :term:`GCS <GCS / Global Control Service>` and the Ray driver processes
|
||||
which run :ref:`Ray jobs <cluster-clients-and-jobs>`. Ray may schedule
|
||||
tasks and actors on the head node just like any other worker node, which is not desired in large-scale clusters.
|
||||
See :ref:`vms-large-cluster-configure-head-node` for the best practice in large-scale clusters.
|
||||
|
||||
.. _cluster-worker-nodes:
|
||||
|
||||
Worker Node
|
||||
------------
|
||||
*Worker nodes* do not run any head node management processes, and serve only to run user code in Ray tasks and actors. They participate in distributed scheduling, as well as the storage and distribution of Ray objects in :ref:`cluster memory <objects-in-ray>`.
|
||||
|
||||
.. _cluster-autoscaler:
|
||||
|
||||
Autoscaler
|
||||
----------
|
||||
|
||||
The *Ray autoscaler* is a process that runs on the :ref:`head node <cluster-head-node>` (or as a sidecar container in the head pod if :ref:`using Kubernetes <kuberay-index>`).
|
||||
When the resource demands of the Ray workload exceed the
|
||||
current capacity of the cluster, the autoscaler will try to increase the number of worker nodes. When worker nodes
|
||||
sit idle, the autoscaler will remove worker nodes from the cluster.
|
||||
|
||||
It is important to understand that the autoscaler only reacts to task and actor resource requests, and not application metrics or physical resource utilization.
|
||||
To learn more about autoscaling, refer to the user guides for Ray clusters on :ref:`VMs <cloud-vm-index>` and :ref:`Kubernetes <kuberay-index>`.
|
||||
|
||||
|
||||
.. note::
|
||||
Version 2.10.0 introduces the alpha release of Autoscaling V2 on KubeRay. Discover the enhancements and configuration details :ref:`here <kuberay-autoscaler-v2>`.
|
||||
|
||||
.. _cluster-clients-and-jobs:
|
||||
|
||||
Ray Jobs
|
||||
--------
|
||||
|
||||
A Ray job is a single application: it is the collection of Ray tasks, objects, and actors that originate from the same script.
|
||||
The worker that runs the Python script is known as the *driver* of the job.
|
||||
|
||||
There are two ways to run a Ray job on a Ray cluster:
|
||||
|
||||
1. (Recommended) Submit the job using the :ref:`Ray Jobs API <jobs-overview>`.
|
||||
2. Run the driver script directly on the Ray cluster, for interactive development.
|
||||
|
||||
For details on these workflows, refer to the :ref:`Ray Jobs API guide <jobs-overview>`.
|
||||
|
||||
.. figure:: images/ray-job-diagram.png
|
||||
:align: center
|
||||
:width: 650px
|
||||
|
||||
*Two ways of running a job on a Ray cluster.*
|
||||
@@ -0,0 +1,11 @@
|
||||
(kuberay-benchmarks)=
|
||||
|
||||
# KubeRay Benchmarks
|
||||
|
||||
```{toctree}
|
||||
:hidden:
|
||||
|
||||
benchmarks/memory-scalability-benchmark
|
||||
```
|
||||
|
||||
- {ref}`kuberay-mem-scalability`
|
||||
@@ -0,0 +1,81 @@
|
||||
(kuberay-mem-scalability)=
|
||||
|
||||
# KubeRay memory and scalability benchmark
|
||||
|
||||
## Architecture
|
||||
|
||||
|
||||
|
||||
This architecture is not a good practice, but it can fulfill the current requirements.
|
||||
|
||||
## Preparation
|
||||
|
||||
Clone the [KubeRay repository](https://github.com/ray-project/kuberay) and checkout the `master` branch. This tutorial requires several files in the repository.
|
||||
|
||||
## Step 1: Create a new Kubernetes cluster
|
||||
|
||||
Create a GKE cluster with autoscaling enabled. The following command creates a Kubernetes cluster named `kuberay-benchmark-cluster` on Google GKE. The cluster can scale up to 16 nodes, and each node of type `e2-highcpu-16` has 16 CPUs and 16 GB of memory. The following experiments may create up to ~150 Pods in the Kubernetes cluster, and each Ray Pod requires 1 CPU and 1 GB of memory.
|
||||
|
||||
```sh
|
||||
gcloud container clusters create kuberay-benchmark-cluster \
|
||||
--num-nodes=1 --min-nodes 0 --max-nodes 16 --enable-autoscaling \
|
||||
--zone=us-west1-b --machine-type e2-highcpu-16
|
||||
```
|
||||
|
||||
## Step 2: Install Prometheus and Grafana
|
||||
|
||||
```sh
|
||||
# Path: kuberay/
|
||||
./install/prometheus/install.sh
|
||||
```
|
||||
|
||||
Follow "Step 2: Install Kubernetes Prometheus Stack via Helm chart" in [prometheus-grafana.md](kuberay-prometheus-grafana) to install the [kube-prometheus-stack v48.2.1](https://github.com/prometheus-community/helm-charts/tree/kube-prometheus-stack-48.2.1/charts/kube-prometheus-stack) chart and related custom resources.
|
||||
|
||||
## Step 3: Install a KubeRay operator
|
||||
|
||||
Follow [this document](kuberay-operator-deploy) to install the latest stable KubeRay operator via Helm repository.
|
||||
|
||||
## Step 4: Run experiments
|
||||
|
||||
* Step 4.1: Make sure the `kubectl` CLI can connect to your GKE cluster. If not, run `gcloud auth login`.
|
||||
* Step 4.2: Run an experiment.
|
||||
```sh
|
||||
# You can modify `memory_benchmark_utils` to run the experiment you want to run.
|
||||
# (path: benchmark/memory_benchmark/scripts)
|
||||
python3 memory_benchmark_utils.py | tee benchmark_log
|
||||
```
|
||||
* Step 4.3: Follow [prometheus-grafana.md](kuberay-prometheus-grafana) to access Grafana's dashboard.
|
||||
* Sign into the Grafana dashboard.
|
||||
* Click on "Dashboards".
|
||||
* Select "Kubernetes / Compute Resources / Pod".
|
||||
* Locate the "Memory Usage" panel for the KubeRay operator Pod.
|
||||
* Select the time range, then click on "Inspect" followed by "Data" to download the memory usage data of the KubeRay operator Pod.
|
||||
* Step 4.4: Delete all RayCluster custom resources.
|
||||
```sh
|
||||
kubectl delete --all rayclusters.ray.io --namespace=default
|
||||
```
|
||||
* Step 4.5: Repeat Step 4.2 to Step 4.4 for other experiments.
|
||||
|
||||
# Experiments
|
||||
|
||||
This benchmark is based on three benchmark experiments:
|
||||
|
||||
* Experiment 1: Launch a RayCluster with 1 head and no workers. A new cluster is initiated every 20 seconds until there are a total of 150 RayCluster custom resources.
|
||||
* Experiment 2: Create a Kubernetes cluster, with only 1 RayCluster. Add 5 new worker Pods to this RayCluster every 60 seconds until the total reaches 150 Pods.
|
||||
* Experiment 3: Create a 5-node (1 head + 4 workers) RayCluster every 60 seconds up to 30 RayCluster custom resources.
|
||||
|
||||
Based on [the survey](https://forms.gle/KtMLzjXcKoeSTj359) for KubeRay users, the benchmark target is set at 150 Ray Pods to cover most use cases.
|
||||
|
||||
## Experiment results (KubeRay v0.6.0)
|
||||
|
||||
|
||||
|
||||
* You can generate the above figure by running:
|
||||
```sh
|
||||
# (path: benchmark/memory_benchmark/scripts)
|
||||
python3 experiment_figures.py
|
||||
# The output image `benchmark_result.png` will be stored in `scripts/`.
|
||||
```
|
||||
|
||||
* As shown in the figure, the memory usage of the KubeRay operator Pod is highly and positively correlated to the number of Pods in the Kubernetes cluster. In addition, the number of custom resources in the Kubernetes cluster does not have a significant impact on the memory usage.
|
||||
* Note that the x-axis "Number of Pods" is the number of Pods that are created rather than running. If the Kubernetes cluster does not have enough computing resources, the GKE Autopilot adds a new Kubernetes node into the cluster. This process may take a few minutes, so some Pods may be pending in the process. This lag may can explain why the memory usage is somewhat throttled.
|
||||
@@ -0,0 +1,46 @@
|
||||
# Fluent Bit Config
|
||||
config:
|
||||
inputs: |
|
||||
[INPUT]
|
||||
Name tail
|
||||
Path /var/log/containers/*.log
|
||||
multiline.parser docker, cri
|
||||
Tag kube.*
|
||||
Mem_Buf_Limit 5MB
|
||||
Skip_Long_Lines On
|
||||
|
||||
filters: |
|
||||
[FILTER]
|
||||
Name kubernetes
|
||||
Match kube.*
|
||||
Merge_Log On
|
||||
Keep_Log Off
|
||||
K8S-Logging.Parser On
|
||||
K8S-Logging.Exclude On
|
||||
|
||||
outputs: |
|
||||
[OUTPUT]
|
||||
Name loki
|
||||
Match *
|
||||
Host loki-gateway
|
||||
Port 80
|
||||
Labels job=fluent-bit,namespace=$kubernetes['namespace_name'],pod=$kubernetes['pod_name'],container=$kubernetes['container_name']
|
||||
Auto_Kubernetes_Labels Off
|
||||
tenant_id test
|
||||
---
|
||||
# Grafana Datasource Config
|
||||
datasources:
|
||||
datasources.yaml:
|
||||
apiVersion: 1
|
||||
datasources:
|
||||
- name: Loki
|
||||
type: loki
|
||||
access: proxy
|
||||
editable: true
|
||||
url: http://loki-gateway.default
|
||||
jsonData:
|
||||
timeout: 60
|
||||
maxLines: 1000
|
||||
httpHeaderName1: "X-Scope-OrgID"
|
||||
secureJsonData:
|
||||
httpHeaderValue1: "test"
|
||||
@@ -0,0 +1,383 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: ca-tls
|
||||
data:
|
||||
# output from cat ca.crt | base64
|
||||
ca.crt: |
|
||||
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM3RENDQWRRQ0NRQ05Yck8zQTAwbWRqQU5CZ2txaGtpRzl3MEJBUXNGQURBNE1SRXdEd1lEVlFRRERBZ3EKTG5KaGVTNXBiekVMTUFrR0ExVUVCaE1DVlZNeEZqQVVCZ05WQkFjTURWTmhiaUJHY21GdVkybHpZMjh3SGhjTgpNak13TXpJM01EZ3dNVFF4V2hjTk16TXdNekkwTURnd01UUXhXakE0TVJFd0R3WURWUVFEREFncUxuSmhlUzVwCmJ6RUxNQWtHQTFVRUJoTUNWVk14RmpBVUJnTlZCQWNNRFZOaGJpQkdjbUZ1WTJselkyOHdnZ0VpTUEwR0NTcUcKU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRQ3ZJbGNGSmZxaFNidWowQ3ZpalA0c2xXN3I3Qk1kYVJOeAp5aDhJMGNaSU5QcjQ5Rjg1dXNrY0pxbnFHNC9LeThBYnlacURBUUxsalFUa0Exb3FxVHhGdTZMSm5LOGJHN012Cm90dStjVlZLWW5SeDlLWVoyWi90THRPdzhjZHFzOURuNXVERVh0L0loZzBRc0tVRDNJN3U3QjF5bVpxTjQwWEgKWDVMRUJkN1llSm5XZExqOStLOTl6ZVR0aHlUMWtsRGsySVp2ZjVsa2xjT2hHRzA5RmNtZlF5REFlM2VvTm1IWQpVaUhVU0NORGtnWTV3U3A4V3R6RXEydHBhZEQ2eTVCNVRMS2kvV1l4ZTJLM2tXbTZnUytwQTIvdkZIaU93RHNaClNqb1ZncUtMZ0lNSnZMOGR0bitaWjNLbDlMRkZNY0JiMWJ1NCtKN2U1bno3RTRVSG4wN0pBZ01CQUFFd0RRWUoKS29aSWh2Y05BUUVMQlFBRGdnRUJBQWhSY3g2NzVJbjJVaERhMzArTkZ0UlNTcUJwK1E2WTl3VGNTL0NqM1J3MgpLSnkzUVhBU0xJUW1ESWdrVlBJeEY0V1VYUFdGdmxUL0taQ2JRejRvN2M3ck9DWEVEWnVhbExUSHRrTHVSZFNWClVHSTVSWTJXNUx6UXM2MnNtUG13OWVQYnNLek5kOEpjWkwvNndHZnNsZVQyY1RLTjliZVE2ZWdiQmdEcy91d0sKeVdOREtnaE4vaE16YmRSaFh2SFNiTW8rUkgvRG1Va1VhTXZZc3NNbzFYQkwzRXZwbmpnZXI1ZWQ5ZDVjQWYvUQpuU0VCMk13Z08rWHEwKy9sWmpiUFNWOVdWQnY1YjZlc1ZPcnZrV2o2TUFKcjUwb3BwT09KUy9TbTNEU3F5aDRBClR5c1BOblQxYStxWDRVZXljZ05VbXRoOXdONFBnc3B6ZEpORWtVdTVSSmM9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
|
||||
# output from cat ca.key | base64
|
||||
ca.key: |
|
||||
LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2Z0lCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktnd2dnU2tBZ0VBQW9JQkFRQ3ZJbGNGSmZxaFNidWoKMEN2aWpQNHNsVzdyN0JNZGFSTnh5aDhJMGNaSU5QcjQ5Rjg1dXNrY0pxbnFHNC9LeThBYnlacURBUUxsalFUawpBMW9xcVR4RnU2TEpuSzhiRzdNdm90dStjVlZLWW5SeDlLWVoyWi90THRPdzhjZHFzOURuNXVERVh0L0loZzBRCnNLVUQzSTd1N0IxeW1acU40MFhIWDVMRUJkN1llSm5XZExqOStLOTl6ZVR0aHlUMWtsRGsySVp2ZjVsa2xjT2gKR0cwOUZjbWZReURBZTNlb05tSFlVaUhVU0NORGtnWTV3U3A4V3R6RXEydHBhZEQ2eTVCNVRMS2kvV1l4ZTJLMwprV202Z1MrcEEyL3ZGSGlPd0RzWlNqb1ZncUtMZ0lNSnZMOGR0bitaWjNLbDlMRkZNY0JiMWJ1NCtKN2U1bno3CkU0VUhuMDdKQWdNQkFBRUNnZ0VCQUpYbG9XK2hveE83UlNRZmdBQkhSeUdud1NtaWhIWE93cnJKRWFqOXkyVncKRzBOTC9ka3Vld1ZpUGxwR3Z0c0hhMlVkTitkYXpUem1aMEkxY0U1RlRYWXQ5RlgxaXBaOExmRGV4cEFJOXNSVQo0bS9Ld3dRckZVdnZvWGE0YWtOMHBxQm1Kd2xNWHVPRmdOZEJLZXZWTW0xaW9JMisxTjhPb0dIVjlvdGFydks5ClUzY09CbmVBSjZmamF6ODd4RG1NY0dBcG82ZWdMOG0xaWJ1NUNwcFo2L2J2YVZYbHhFdXRtUjZYR2VKczdBRzMKVEtFYVhzTU1qdFdaM3ZXUDArMFJIMGpzRVI2a0ZMeEI3KzRHRWdPSk1WblZqbjlzT3FhVW41KzJ1REkrdkFkbAo0K2Fya3dwQnpzbGlaUVJLVW95aGwvMTRRZW9pcXpwVk9oVVpheFJOTnpVQ2dZRUE0RC83TmRudGFxa0JSdEdiClZUQTE0clA3Vy90THZSQnpBNWtLc3Q4V3crWFVGeTcvVEY2NkxpMVVtKzhhK2ttY1pMTm9mamNZc1pTMExkVXMKMlR4dk1IRWplcmdNUm5oWmUrOGJBZlZkd1RTcCtpdUpMKzRZWW1JRUZWUWlsSXhtRURzMkZQSnVZMHRDZW9ETAprVEFSeUNtMENPYUR1VXdLZjlMY3h3SFR4M3NDZ1lFQXgrNGpyOHV3aXh3WmwwUFBJb1Z1by9wSHZMT0ZxNXNBCmIrVEZnMEhFTVdIK1JKclhLRjA1YTRGNS9zc3pLZ09ZMGFZVUxlWnp3V1dJZElId0pzQnhGWktOdHRYTkhRbS8KOEFlVGRENnZ1OXlmN0tFZjhRNnFmaDRPRExvVDg0UTFWbGs4ek5ZN0FNUWZwN2p5RnpFOStvSm9tdlM0Snc1SApCZUNLZGZGR1RZc0NnWUVBaitkL0JhZTd1MTZJK3pFM1JRdVRDTkFHMVpnRm1tWWI2SXNsV25QZTRBZDBld3dsCnVKUnhWWUN4Y3YrVmlGZ0VqSHEwNjRuZnh0VnVhcHNLRkwyN2ZKS2QrZnB4cGlkRkJVc0RRZFo3TzZqWUN6bzAKNXhVYmdNYjFaOXA5OW1YQ2VWZ0Y5SnMrUzJuWVYxU2ZUYVJUUk9lK0tKZ0VuN3cwWUtLb0d1MEpRbEVDZ1lCZApZdXJnYm5Ca1NoZmFCQjU0cllMa3JUOWM4UzM2M2tmeC9CWVdIVjRiQXY3VjVNMmpXUWc5SXhsczNsVmp4cEpYCk94QXA4SDhaVXVmT0kvT2M1ajdzS0t4eFBxUzBiNTFyN04zL2FsaURrNlpQeldNeUlmdVpOVWl5d1NnWWt5U20KMU1BRm5mdXBlL0tkVVZJamF5amNIcFhsNjNFcExRNFh2SzV3TU9iNXlRS0JnR0kzSTAwSTlnbURzS1JrOFkxdQpId1l0dVdrNjFvWEhUTHorR3d6RUNCQ0VnNkZxMjZVeDZmVzBySlVwV3pOVURCNkRRRGxCTGx3S1M4Z1R3eGtGCkRkY3VrbzFHekdlQWYvazEwWktTZmFXNVcwVlloVGVjSDhyZXpReWxwUk5YT3ZNZkFwWUplcnhBZ09yK3hFajUKK2wwalU0MDBTMUx0cWhLVzZMK3kxRVd5Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: tls
|
||||
data:
|
||||
gencert_head.sh: |
|
||||
#!/bin/sh
|
||||
## Create tls.key
|
||||
openssl genrsa -out /etc/ray/tls/tls.key 2048
|
||||
|
||||
## Write CSR Config
|
||||
cat > /etc/ray/tls/csr.conf <<EOF
|
||||
[ req ]
|
||||
default_bits = 2048
|
||||
prompt = no
|
||||
default_md = sha256
|
||||
req_extensions = req_ext
|
||||
distinguished_name = dn
|
||||
|
||||
[ dn ]
|
||||
C = US
|
||||
ST = California
|
||||
L = San Fransisco
|
||||
O = ray
|
||||
OU = ray
|
||||
CN = *.ray.io
|
||||
|
||||
[ req_ext ]
|
||||
subjectAltName = @alt_names
|
||||
|
||||
[ alt_names ]
|
||||
DNS.1 = localhost
|
||||
DNS.2 = service-ray-head.default.svc.cluster.local
|
||||
IP.1 = 127.0.0.1
|
||||
IP.2 = $POD_IP
|
||||
|
||||
EOF
|
||||
|
||||
## Create CSR using tls.key
|
||||
openssl req -new -key /etc/ray/tls/tls.key -out /etc/ray/tls/ca.csr -config /etc/ray/tls/csr.conf
|
||||
|
||||
## Write cert config
|
||||
cat > /etc/ray/tls/cert.conf <<EOF
|
||||
|
||||
authorityKeyIdentifier=keyid,issuer
|
||||
basicConstraints=CA:FALSE
|
||||
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
|
||||
subjectAltName = @alt_names
|
||||
|
||||
[alt_names]
|
||||
DNS.1 = localhost
|
||||
DNS.2 = service-ray-head.default.svc.cluster.local
|
||||
IP.1 = 127.0.0.1
|
||||
IP.2 = $POD_IP
|
||||
|
||||
EOF
|
||||
|
||||
## Generate tls.cert
|
||||
openssl x509 -req \
|
||||
-in /etc/ray/tls/ca.csr \
|
||||
-CA /etc/ray/tls/ca.crt -CAkey /etc/ray/tls/ca.key \
|
||||
-CAcreateserial -out /etc/ray/tls/tls.crt \
|
||||
-days 365 \
|
||||
-sha256 -extfile /etc/ray/tls/cert.conf
|
||||
|
||||
gencert_worker.sh: |
|
||||
#!/bin/sh
|
||||
## Create tls.key
|
||||
openssl genrsa -out /etc/ray/tls/tls.key 2048
|
||||
|
||||
## Write CSR Config
|
||||
cat > /etc/ray/tls/csr.conf <<EOF
|
||||
[ req ]
|
||||
default_bits = 2048
|
||||
prompt = no
|
||||
default_md = sha256
|
||||
req_extensions = req_ext
|
||||
distinguished_name = dn
|
||||
|
||||
[ dn ]
|
||||
C = US
|
||||
ST = California
|
||||
L = San Fransisco
|
||||
O = ray
|
||||
OU = ray
|
||||
CN = *.ray.io
|
||||
|
||||
[ req_ext ]
|
||||
subjectAltName = @alt_names
|
||||
|
||||
[ alt_names ]
|
||||
DNS.1 = localhost
|
||||
IP.1 = 127.0.0.1
|
||||
IP.2 = $POD_IP
|
||||
|
||||
EOF
|
||||
|
||||
## Create CSR using tls.key
|
||||
openssl req -new -key /etc/ray/tls/tls.key -out /etc/ray/tls/ca.csr -config /etc/ray/tls/csr.conf
|
||||
|
||||
## Write cert config
|
||||
cat > /etc/ray/tls/cert.conf <<EOF
|
||||
|
||||
authorityKeyIdentifier=keyid,issuer
|
||||
basicConstraints=CA:FALSE
|
||||
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
|
||||
subjectAltName = @alt_names
|
||||
|
||||
[alt_names]
|
||||
DNS.1 = localhost
|
||||
IP.1 = 127.0.0.1
|
||||
IP.2 = $POD_IP
|
||||
|
||||
EOF
|
||||
|
||||
## Generate tls.cert
|
||||
openssl x509 -req \
|
||||
-in /etc/ray/tls/ca.csr \
|
||||
-CA /etc/ray/tls/ca.crt -CAkey /etc/ray/tls/ca.key \
|
||||
-CAcreateserial -out /etc/ray/tls/tls.crt \
|
||||
-days 365 \
|
||||
-sha256 -extfile /etc/ray/tls/cert.conf
|
||||
---
|
||||
# Ray head node service, allowing worker pods to discover the head node to perform the bidirectional communication.
|
||||
# More contexts can be found at [the Ports configurations doc](https://docs.ray.io/en/latest/ray-core/configure.html#ports-configurations).
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: service-ray-head
|
||||
labels:
|
||||
app: ray-head
|
||||
spec:
|
||||
clusterIP: None
|
||||
ports:
|
||||
- name: client
|
||||
protocol: TCP
|
||||
port: 10001
|
||||
targetPort: 10001
|
||||
- name: dashboard
|
||||
protocol: TCP
|
||||
port: 8265
|
||||
targetPort: 8265
|
||||
- name: gcs-server
|
||||
protocol: TCP
|
||||
port: 6379
|
||||
targetPort: 6379
|
||||
selector:
|
||||
app: ray-head
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: deployment-ray-head
|
||||
labels:
|
||||
app: ray-head
|
||||
spec:
|
||||
# Do not change this - Ray currently only supports one head node per cluster.
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: ray-head
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: ray-head
|
||||
spec:
|
||||
# If the head node goes down, the entire cluster (including all worker
|
||||
# nodes) goes down as well. If you want Kubernetes to bring up a new
|
||||
# head node in this case, set this to "Always," else set it to "Never."
|
||||
restartPolicy: Always
|
||||
terminationGracePeriodSeconds: 60
|
||||
|
||||
# This volume allocates shared memory for Ray to use for its plasma
|
||||
# object store. If you do not provide this, Ray falls back to
|
||||
# /tmp which cause slowdowns if it's not a shared memory volume.
|
||||
volumes:
|
||||
- name: dshm
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
- name: ca-tls
|
||||
secret:
|
||||
secretName: ca-tls
|
||||
- name: ray-tls
|
||||
emptyDir: {}
|
||||
# The gencert_head.sh can be prebaked into the docker container so the configMap is optional
|
||||
- name: gen-tls-script
|
||||
configMap:
|
||||
name: tls
|
||||
defaultMode: 0777
|
||||
# An array of keys from the ConfigMap to create as files
|
||||
items:
|
||||
- key: gencert_head.sh
|
||||
path: gencert_head.sh
|
||||
initContainers:
|
||||
- name: ray-head-tls
|
||||
image: rayproject/ray:2.3.0
|
||||
command: ["/bin/sh", "-c", "cp -R /etc/ca/tls /etc/ray && /etc/gen/tls/gencert_head.sh"]
|
||||
volumeMounts:
|
||||
- mountPath: /etc/ca/tls
|
||||
name: ca-tls
|
||||
readOnly: true
|
||||
- mountPath: /etc/ray/tls
|
||||
name: ray-tls
|
||||
- mountPath: /etc/gen/tls
|
||||
name: gen-tls-script
|
||||
env:
|
||||
- name: POD_IP
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: status.podIP
|
||||
containers:
|
||||
- name: ray-head
|
||||
image: rayproject/ray:2.3.0
|
||||
imagePullPolicy: IfNotPresent
|
||||
command: [ "/bin/sh", "-c", "--" ]
|
||||
args:
|
||||
- "ray start --head --port=6379 --num-cpus=$MY_CPU_REQUEST --dashboard-host=127.0.0.1 --object-manager-port=8076 --node-manager-port=8077 --dashboard-agent-grpc-port=8078 --dashboard-agent-listen-port=52365 --block"
|
||||
ports:
|
||||
- containerPort: 6379 # GCS server
|
||||
- containerPort: 10001 # Used by Ray Client
|
||||
- containerPort: 8265 # Used by Ray Dashboard
|
||||
# This volume allocates shared memory for Ray to use for its plasma
|
||||
# object store. If you do not provide this, Ray falls back to
|
||||
# /tmp which cause slowdowns if it's not a shared memory volume.
|
||||
volumeMounts:
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /etc/ca/tls
|
||||
name: ca-tls
|
||||
readOnly: true
|
||||
- mountPath: /etc/ray/tls
|
||||
name: ray-tls
|
||||
env:
|
||||
- name: RAY_USE_TLS
|
||||
value: "1"
|
||||
- name: RAY_TLS_SERVER_CERT
|
||||
value: "/etc/ray/tls/tls.crt"
|
||||
- name: RAY_TLS_SERVER_KEY
|
||||
value: "/etc/ray/tls/tls.key"
|
||||
- name: RAY_TLS_CA_CERT
|
||||
value: "/etc/ca/tls/ca.crt"
|
||||
- name: RAY_BACKEND_LOG_LEVEL
|
||||
value: warning
|
||||
# This is used in the ray start command so that Ray can spawn the
|
||||
# correct number of processes. Omitting this may lead to degraded
|
||||
# performance.
|
||||
- name: MY_CPU_REQUEST
|
||||
valueFrom:
|
||||
resourceFieldRef:
|
||||
resource: requests.cpu
|
||||
resources:
|
||||
limits:
|
||||
cpu: "2"
|
||||
memory: "4G"
|
||||
requests:
|
||||
# For production use-cases, we recommend specifying integer CPU requests and limits.
|
||||
# We also recommend setting requests equal to limits for both CPU and memory.
|
||||
# For this example, we use a 500m CPU request to accommodate resource-constrained local
|
||||
# Kubernetes testing environments such as Kind and minikube.
|
||||
cpu: "2"
|
||||
# The rest state memory usage of the Ray head node is around 1Gb. We do not
|
||||
# recommend allocating less than 2Gb memory for the Ray head pod.
|
||||
# For production use-cases, we recommend allocating at least 8Gb memory for each Ray container.
|
||||
memory: "4G"
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: deployment-ray-worker
|
||||
labels:
|
||||
app: ray-worker
|
||||
spec:
|
||||
# Change this to scale the number of worker nodes started in the Ray cluster.
|
||||
replicas: 2
|
||||
selector:
|
||||
matchLabels:
|
||||
app: ray-worker
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: ray-worker
|
||||
spec:
|
||||
restartPolicy: Always
|
||||
volumes:
|
||||
- name: dshm
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
- name: ca-tls
|
||||
secret:
|
||||
secretName: ca-tls
|
||||
- name: ray-tls
|
||||
emptyDir: {}
|
||||
# The gencert_worker.sh can be prebaked into the docker container so the configMap is optional
|
||||
- name: gen-tls-script
|
||||
configMap:
|
||||
name: tls
|
||||
defaultMode: 0777
|
||||
# An array of keys from the ConfigMap to create as files
|
||||
items:
|
||||
- key: gencert_worker.sh
|
||||
path: gencert_worker.sh
|
||||
initContainers:
|
||||
- name: ray-worker-tls
|
||||
image: rayproject/ray:2.3.0
|
||||
command: ["/bin/sh", "-c", "cp -R /etc/ca/tls /etc/ray && /etc/gen/tls/gencert_worker.sh"]
|
||||
volumeMounts:
|
||||
- mountPath: /etc/ca/tls
|
||||
name: ca-tls
|
||||
readOnly: true
|
||||
- mountPath: /etc/ray/tls
|
||||
name: ray-tls
|
||||
- mountPath: /etc/gen/tls
|
||||
name: gen-tls-script
|
||||
env:
|
||||
- name: POD_IP
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: status.podIP
|
||||
containers:
|
||||
- name: ray-worker
|
||||
image: rayproject/ray:2.3.0
|
||||
imagePullPolicy: IfNotPresent
|
||||
command: ["/bin/bash", "-c", "--"]
|
||||
args:
|
||||
- "ray start --num-cpus=$MY_CPU_REQUEST --address=service-ray-head.default.svc.cluster.local:6379 --object-manager-port=8076 --node-manager-port=8077 --dashboard-agent-grpc-port=8078 --dashboard-agent-listen-port=52365 --block"
|
||||
ports:
|
||||
- containerPort: 6379 # GCS server
|
||||
# This volume allocates shared memory for Ray to use for its plasma
|
||||
# object store. If you do not provide this, Ray falls back to
|
||||
# /tmp which cause slowdowns if it's not a shared memory volume.
|
||||
volumeMounts:
|
||||
- mountPath: /dev/shm
|
||||
name: dshm
|
||||
- mountPath: /etc/ca/tls
|
||||
name: ca-tls
|
||||
readOnly: true
|
||||
- mountPath: /etc/ray/tls
|
||||
name: ray-tls
|
||||
env:
|
||||
- name: RAY_USE_TLS
|
||||
value: "1"
|
||||
- name: RAY_TLS_SERVER_CERT
|
||||
value: "/etc/ray/tls/tls.crt"
|
||||
- name: RAY_TLS_SERVER_KEY
|
||||
value: "/etc/ray/tls/tls.key"
|
||||
- name: RAY_TLS_CA_CERT
|
||||
value: "/etc/ca/tls/ca.crt"
|
||||
- name: RAY_BACKEND_LOG_LEVEL
|
||||
value: warning
|
||||
# This is used in the ray start command so that Ray can spawn the
|
||||
# correct number of processes. Omitting this may lead to degraded
|
||||
# performance.
|
||||
- name: MY_CPU_REQUEST
|
||||
valueFrom:
|
||||
resourceFieldRef:
|
||||
resource: requests.cpu
|
||||
# The resource requests and limits in this config are too small for production!
|
||||
# It is better to use a few large Ray pods than many small ones.
|
||||
# For production, it is ideal to size each Ray pod to take up the
|
||||
# entire Kubernetes node on which it is scheduled.
|
||||
resources:
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: "1G"
|
||||
# For production use-cases, we recommend specifying integer CPU requests and limits.
|
||||
# We also recommend setting requests equal to limits for both CPU and memory.
|
||||
# For this example, we use a 500m CPU request to accommodate resource-constrained local
|
||||
# Kubernetes testing environments such as Kind and minikube.
|
||||
requests:
|
||||
cpu: "500m"
|
||||
memory: "1G"
|
||||
@@ -0,0 +1,39 @@
|
||||
(kuberay-examples)=
|
||||
|
||||
# Examples
|
||||
|
||||
```{toctree}
|
||||
:hidden:
|
||||
|
||||
examples/mnist-training-example
|
||||
examples/stable-diffusion-rayservice
|
||||
examples/tpu-serve-stable-diffusion
|
||||
examples/mobilenet-rayservice
|
||||
examples/text-summarizer-rayservice
|
||||
examples/rayjob-batch-inference-example
|
||||
examples/rayjob-kueue-priority-scheduling
|
||||
examples/rayjob-kueue-gang-scheduling
|
||||
examples/distributed-checkpointing-with-gcsfuse
|
||||
examples/rayserve-llm-example
|
||||
examples/rayserve-deepseek-example
|
||||
examples/verl-post-training
|
||||
examples/argocd
|
||||
examples/rayjob-agent-sandbox
|
||||
```
|
||||
|
||||
|
||||
This section presents example Ray workloads to try out on your Kubernetes cluster.
|
||||
|
||||
- {ref}`kuberay-mnist-training-example` (CPU-only)
|
||||
- {ref}`kuberay-mobilenet-rayservice-example` (CPU-only)
|
||||
- {ref}`kuberay-stable-diffusion-rayservice-example`
|
||||
- {ref}`kuberay-tpu-stable-diffusion-example`
|
||||
- {ref}`kuberay-text-summarizer-rayservice-example`
|
||||
- {ref}`kuberay-batch-inference-example`
|
||||
- {ref}`kuberay-kueue-priority-scheduling-example`
|
||||
- {ref}`kuberay-kueue-gang-scheduling-example`
|
||||
- {ref}`kuberay-distributed-checkpointing-gcsfuse`
|
||||
- {ref}`kuberay-rayservice-llm-example`
|
||||
- {ref}`kuberay-rayservice-deepseek-example`
|
||||
- {ref}`kuberay-verl`
|
||||
- {ref}`kuberay-agent-sandbox`
|
||||
@@ -0,0 +1,532 @@
|
||||
(deploying-on-argocd-example)=
|
||||
|
||||
# Deploying Ray Clusters via ArgoCD
|
||||
|
||||
This guide provides a step-by-step approach for deploying Ray clusters on Kubernetes using ArgoCD. ArgoCD is a declarative GitOps tool that enables you to manage Ray cluster configurations in Git repositories with automated synchronization, version control, and rollback capabilities. This approach is particularly valuable when managing multiple Ray clusters across different environments, implementing audit trails and approval workflows, or maintaining infrastructure-as-code practices. For simpler use cases like single-cluster development or quick experimentation, direct kubectl or Helm deployments may be sufficient. You can read more about the benefits of ArgoCD [here](https://argo-cd.readthedocs.io/en/stable/#why-argo-cd).
|
||||
|
||||
This example demonstrates how to deploy the KubeRay operator and a RayCluster with three different worker groups, leveraging ArgoCD's GitOps capabilities for automated cluster management.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before proceeding with this guide, ensure you have the following:
|
||||
|
||||
* A Kubernetes cluster with appropriate resources for running Ray workloads.
|
||||
* `kubectl` configured to access your Kubernetes cluster.
|
||||
* (Optional)[ArgoCD installed](https://argo-cd.readthedocs.io/en/stable/getting_started/) on your Kubernetes cluster.
|
||||
* (Optional)[ArgoCD CLI](https://argo-cd.readthedocs.io/en/stable/cli_installation/) installed on your local machine (recommended for easier application management. It might need [port-forwarding and login](https://argo-cd.readthedocs.io/en/stable/getting_started/#port-forwarding) depending on your environment).
|
||||
* (Optional)Access to the ArgoCD UI or API server.
|
||||
|
||||
## Step 1: Deploy KubeRay Operator CRDs
|
||||
|
||||
First, deploy the Custom Resource Definitions (CRDs) required by the KubeRay operator. Create a file named `ray-operator-crds.yaml` with the following content:
|
||||
|
||||
```yaml
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: ray-operator-crds
|
||||
namespace: argocd
|
||||
spec:
|
||||
project: default
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
namespace: ray-cluster
|
||||
source:
|
||||
repoURL: https://github.com/ray-project/kuberay
|
||||
targetRevision: v1.6.0 # update this as necessary
|
||||
path: helm-chart/kuberay-operator/crds
|
||||
syncPolicy:
|
||||
automated:
|
||||
prune: true
|
||||
selfHeal: true
|
||||
syncOptions:
|
||||
- CreateNamespace=true
|
||||
- Replace=true
|
||||
```
|
||||
|
||||
Apply the ArgoCD Application:
|
||||
|
||||
```sh
|
||||
kubectl apply -f ray-operator-crds.yaml
|
||||
```
|
||||
|
||||
Wait for the CRDs Application to sync and become healthy. You can check the status using:
|
||||
|
||||
```sh
|
||||
kubectl get application ray-operator-crds -n argocd
|
||||
```
|
||||
|
||||
Which should eventually give something like:
|
||||
```
|
||||
NAME SYNC STATUS HEALTH STATUS
|
||||
ray-operator-crds Synced Healthy
|
||||
```
|
||||
|
||||
Alternatively, if you have the ArgoCD CLI installed, you can wait for the application:
|
||||
|
||||
```sh
|
||||
argocd app wait ray-operator-crds
|
||||
```
|
||||
|
||||
## Step 2: Deploy the KubeRay Operator
|
||||
|
||||
After the CRDs are installed, deploy the KubeRay operator itself. Create a file named `ray-operator.yaml` with the following content:
|
||||
|
||||
```yaml
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: ray-operator
|
||||
namespace: argocd
|
||||
spec:
|
||||
project: default
|
||||
source:
|
||||
repoURL: https://github.com/ray-project/kuberay
|
||||
targetRevision: v1.6.0 # update this as necessary
|
||||
path: helm-chart/kuberay-operator
|
||||
helm:
|
||||
skipCrds: true # CRDs are already installed in Step 1
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
namespace: ray-cluster
|
||||
syncPolicy:
|
||||
automated:
|
||||
prune: true
|
||||
selfHeal: true
|
||||
syncOptions:
|
||||
- CreateNamespace=true
|
||||
```
|
||||
|
||||
Note the `skipCrds: true` setting in the Helm configuration. This is required because the CRDs were installed separately in Step 1.
|
||||
|
||||
Apply the ArgoCD Application:
|
||||
|
||||
```sh
|
||||
kubectl apply -f ray-operator.yaml
|
||||
```
|
||||
|
||||
Wait for the operator Application to sync and become healthy. You can check the status using:
|
||||
|
||||
```sh
|
||||
kubectl get application ray-operator -n argocd
|
||||
```
|
||||
|
||||
Which should give the following output eventually:
|
||||
```
|
||||
NAME SYNC STATUS HEALTH STATUS
|
||||
ray-operator Synced Healthy
|
||||
```
|
||||
|
||||
Alternatively, if you have the ArgoCD CLI installed:
|
||||
|
||||
```sh
|
||||
argocd app wait ray-operator
|
||||
```
|
||||
|
||||
Verify that the KubeRay operator pod is running:
|
||||
|
||||
```sh
|
||||
kubectl get pods -n ray-cluster -l app.kubernetes.io/name=kuberay-operator
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Step 3: Deploy a RayCluster
|
||||
|
||||
Now deploy a RayCluster with autoscaling enabled and three different worker groups. Create a file named `raycluster.yaml` with the following content:
|
||||
|
||||
```yaml
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: raycluster
|
||||
namespace: argocd
|
||||
spec:
|
||||
project: default
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
namespace: ray-cluster
|
||||
ignoreDifferences:
|
||||
- group: ray.io
|
||||
kind: RayCluster
|
||||
name: raycluster-kuberay
|
||||
namespace: ray-cluster
|
||||
jqPathExpressions:
|
||||
- .spec.workerGroupSpecs[].replicas
|
||||
source:
|
||||
repoURL: https://ray-project.github.io/kuberay-helm/
|
||||
chart: ray-cluster
|
||||
targetRevision: "1.6.0"
|
||||
helm:
|
||||
releaseName: raycluster
|
||||
valuesObject:
|
||||
image:
|
||||
repository: docker.io/rayproject/ray
|
||||
tag: latest
|
||||
pullPolicy: IfNotPresent
|
||||
head:
|
||||
rayStartParams:
|
||||
num-cpus: "0"
|
||||
enableInTreeAutoscaling: true
|
||||
autoscalerOptions:
|
||||
version: v2
|
||||
upscalingMode: Default
|
||||
idleTimeoutSeconds: 600 # 10 minutes
|
||||
env:
|
||||
- name: AUTOSCALER_MAX_CONCURRENT_LAUNCHES
|
||||
value: "100"
|
||||
worker:
|
||||
groupName: standard-worker
|
||||
replicas: 1
|
||||
minReplicas: 1
|
||||
maxReplicas: 200
|
||||
rayStartParams:
|
||||
resources: '"{\"standard-worker\": 1}"'
|
||||
resources:
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: "1G"
|
||||
additionalWorkerGroups:
|
||||
additional-worker-group1:
|
||||
image:
|
||||
repository: docker.io/rayproject/ray
|
||||
tag: latest
|
||||
pullPolicy: IfNotPresent
|
||||
disabled: false
|
||||
replicas: 1
|
||||
minReplicas: 1
|
||||
maxReplicas: 30
|
||||
rayStartParams:
|
||||
resources: '"{\"additional-worker-group1\": 1}"'
|
||||
resources:
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: "1G"
|
||||
additional-worker-group2:
|
||||
image:
|
||||
repository: docker.io/rayproject/ray
|
||||
tag: latest
|
||||
pullPolicy: IfNotPresent
|
||||
disabled: false
|
||||
replicas: 1
|
||||
minReplicas: 1
|
||||
maxReplicas: 200
|
||||
rayStartParams:
|
||||
resources: '"{\"additional-worker-group2\": 1}"'
|
||||
resources:
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: "1G"
|
||||
syncPolicy:
|
||||
automated:
|
||||
prune: true
|
||||
selfHeal: true
|
||||
syncOptions:
|
||||
- CreateNamespace=true
|
||||
```
|
||||
|
||||
Apply the ArgoCD Application:
|
||||
|
||||
```sh
|
||||
kubectl apply -f raycluster.yaml
|
||||
```
|
||||
|
||||
Wait for the RayCluster Application to sync and become healthy. You can check the status using:
|
||||
|
||||
```sh
|
||||
kubectl get application raycluster -n argocd
|
||||
```
|
||||
|
||||
Alternatively, if you have the ArgoCD CLI installed:
|
||||
|
||||
```sh
|
||||
argocd app wait raycluster
|
||||
```
|
||||
|
||||
Verify that the RayCluster is running:
|
||||
|
||||
```sh
|
||||
kubectl get raycluster -n ray-cluster
|
||||
```
|
||||
|
||||
Which will give something like:
|
||||
```
|
||||
NAME DESIRED WORKERS AVAILABLE WORKERS CPUS MEMORY GPUS STATUS AGE
|
||||
raycluster-kuberay 3 3 ... ... ... ready ...
|
||||
```
|
||||
|
||||
|
||||
You should see the head pod and worker pods:
|
||||
|
||||
```sh
|
||||
kubectl get pods -n ray-cluster
|
||||
```
|
||||
|
||||
Gives something like:
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
kuberay-operator-6c485bc876-28dnl 1/1 Running 0 11d
|
||||
raycluster-kuberay-additional-worker-group1-n45rc 1/1 Running 0 5d21h
|
||||
raycluster-kuberay-additional-worker-group2-b2455 1/1 Running 0 2d18h
|
||||
raycluster-kuberay-head 2/2 Running 0 5d21h
|
||||
raycluster-kuberay-standard-worker-worker-bs8t8 1/1 Running 0 5d21h
|
||||
```
|
||||
|
||||
## Understanding Ray Autoscaling with ArgoCD
|
||||
|
||||
### Determining Fields to Ignore
|
||||
|
||||
The `ignoreDifferences` section in the RayCluster Application configuration is critical for proper autoscaling. To determine which fields need to be ignored, you can inspect the RayCluster resource to identify fields that change dynamically during runtime.
|
||||
|
||||
First, describe the RayCluster resource to see its full specification:
|
||||
|
||||
```sh
|
||||
kubectl describe raycluster raycluster-kuberay -n ray-cluster
|
||||
```
|
||||
|
||||
Or, get the resource in YAML format to see the exact field paths:
|
||||
|
||||
```sh
|
||||
kubectl get raycluster raycluster-kuberay -n ray-cluster -o yaml
|
||||
```
|
||||
|
||||
Look for fields that are modified by controllers or autoscalers. In the case of Ray, the autoscaler modifies the `replicas` field under each worker group spec. You'll see output similar to:
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
workerGroupSpecs:
|
||||
- replicas: 5 # This value changes dynamically
|
||||
minReplicas: 1
|
||||
maxReplicas: 200
|
||||
groupName: standard-worker
|
||||
# ...
|
||||
```
|
||||
|
||||
When ArgoCD detects differences between the desired state (in Git) and the actual state (in the cluster), it will show these in the UI or via CLI:
|
||||
|
||||
```sh
|
||||
argocd app diff raycluster
|
||||
```
|
||||
|
||||
If you see repeated differences in fields that should be managed by controllers (like autoscalers), those are candidates for `ignoreDifferences`.
|
||||
|
||||
### Configuring ignoreDifferences
|
||||
|
||||
The `ignoreDifferences` section in the RayCluster Application configuration tells ArgoCD which fields to ignore. Without this setting, ArgoCD and the Ray Autoscaler may conflict, resulting in unexpected behavior when requesting workers dynamically (for example, using `ray.autoscaler.sdk.request_resources`). Specifically, when requesting N workers, the Autoscaler might not spin up the expected number of workers because ArgoCD could revert the replica count back to the original value defined in the Application manifest.
|
||||
|
||||
The recommended approach is to use `jqPathExpressions`, which automatically handles any number of worker groups:
|
||||
|
||||
```yaml
|
||||
ignoreDifferences:
|
||||
- group: ray.io
|
||||
kind: RayCluster
|
||||
name: raycluster-kuberay
|
||||
namespace: ray-cluster
|
||||
jqPathExpressions:
|
||||
- .spec.workerGroupSpecs[].replicas
|
||||
```
|
||||
|
||||
This configuration tells ArgoCD to ignore differences in the `replicas` field for all worker groups. The `jqPathExpressions` field uses JQ syntax with array wildcards (`[]`), which means you don't need to update the configuration when adding or removing worker groups.
|
||||
|
||||
**Note**: The `name` and `namespace` must match your RayCluster resource name and namespace. Verify these values separately if you've customized them.
|
||||
|
||||
**Alternative: Using jsonPointers**
|
||||
|
||||
If you prefer explicit configuration, you can use `jsonPointers` instead:
|
||||
|
||||
```yaml
|
||||
ignoreDifferences:
|
||||
- group: ray.io
|
||||
kind: RayCluster
|
||||
name: raycluster-kuberay
|
||||
namespace: ray-cluster
|
||||
jsonPointers:
|
||||
- /spec/workerGroupSpecs/0/replicas
|
||||
- /spec/workerGroupSpecs/1/replicas
|
||||
- /spec/workerGroupSpecs/2/replicas
|
||||
```
|
||||
|
||||
With `jsonPointers`, you must explicitly list each worker group by index:
|
||||
- `/spec/workerGroupSpecs/0/replicas` - First worker group (the default `worker` group)
|
||||
- `/spec/workerGroupSpecs/1/replicas` - Second worker group (`additional-worker-group1`)
|
||||
- `/spec/workerGroupSpecs/2/replicas` - Third worker group (`additional-worker-group2`)
|
||||
|
||||
If you add or remove worker groups, you **must** update this list accordingly. The index corresponds to the order of worker groups as they appear in the RayCluster spec, with the default `worker` group at index 0 and `additionalWorkerGroups` following in the order they are defined.
|
||||
|
||||
See the [ArgoCD diff customization documentation](https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/) for more details on both approaches.
|
||||
|
||||
By ignoring these differences, ArgoCD allows the Ray Autoscaler to dynamically manage worker replicas without interference.
|
||||
|
||||
## Step 4: Access the Ray Dashboard
|
||||
|
||||
To access the Ray Dashboard, port-forward the head service:
|
||||
|
||||
```sh
|
||||
kubectl port-forward -n ray-cluster svc/raycluster-kuberay-head-svc 8265:8265
|
||||
```
|
||||
|
||||
Navigate to `http://localhost:8265` in your browser to view the Ray Dashboard.
|
||||
|
||||
## Customizing the Configuration
|
||||
|
||||
You can customize the RayCluster configuration by modifying the `valuesObject` section in the `raycluster.yaml` file:
|
||||
|
||||
* **Image**: Change the `repository` and `tag` to use different Ray versions.
|
||||
* **Worker Groups**: Add or remove worker groups by modifying the `additionalWorkerGroups` section.
|
||||
* **Autoscaling**: Adjust `minReplicas`, `maxReplicas`, and `idleTimeoutSeconds` to control autoscaling behavior.
|
||||
* **Resources**: Modify `rayStartParams` to allocate custom resources to worker groups.
|
||||
|
||||
After making changes, commit them to your Git repository. ArgoCD will automatically sync the changes to your cluster if automated sync is enabled.
|
||||
|
||||
## Alternative: Deploy Everything in One File
|
||||
|
||||
If you prefer to deploy all components at once, you can combine all three ArgoCD Applications into a single file. Create a file named `ray-argocd-all.yaml` with the following content:
|
||||
|
||||
```yaml
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: ray-operator-crds
|
||||
namespace: argocd
|
||||
spec:
|
||||
project: default
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
namespace: ray-cluster
|
||||
source:
|
||||
repoURL: https://github.com/ray-project/kuberay
|
||||
targetRevision: v1.6.0 # update this as necessary
|
||||
path: helm-chart/kuberay-operator/crds
|
||||
syncPolicy:
|
||||
automated:
|
||||
prune: true
|
||||
selfHeal: true
|
||||
syncOptions:
|
||||
- CreateNamespace=true
|
||||
- Replace=true
|
||||
|
||||
---
|
||||
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: ray-operator
|
||||
namespace: argocd
|
||||
spec:
|
||||
project: default
|
||||
source:
|
||||
repoURL: https://github.com/ray-project/kuberay
|
||||
targetRevision: v1.6.0 # update this as necessary
|
||||
path: helm-chart/kuberay-operator
|
||||
helm:
|
||||
skipCrds: true # CRDs are installed in the first Application
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
namespace: ray-cluster
|
||||
syncPolicy:
|
||||
automated:
|
||||
prune: true
|
||||
selfHeal: true
|
||||
syncOptions:
|
||||
- CreateNamespace=true
|
||||
|
||||
---
|
||||
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Application
|
||||
metadata:
|
||||
name: raycluster
|
||||
namespace: argocd
|
||||
spec:
|
||||
project: default
|
||||
destination:
|
||||
server: https://kubernetes.default.svc
|
||||
namespace: ray-cluster
|
||||
ignoreDifferences:
|
||||
- group: ray.io
|
||||
kind: RayCluster
|
||||
name: raycluster-kuberay # ensure this is aligned with the release name
|
||||
namespace: ray-cluster # ensure this is aligned with the namespace
|
||||
jqPathExpressions:
|
||||
- .spec.workerGroupSpecs[].replicas
|
||||
source:
|
||||
repoURL: https://ray-project.github.io/kuberay-helm/
|
||||
chart: ray-cluster
|
||||
targetRevision: "1.4.1"
|
||||
helm:
|
||||
releaseName: raycluster # this affects the ignoreDifferences field
|
||||
valuesObject:
|
||||
image:
|
||||
repository: docker.io/rayproject/ray
|
||||
tag: latest
|
||||
pullPolicy: IfNotPresent
|
||||
head:
|
||||
rayStartParams:
|
||||
num-cpus: "0"
|
||||
enableInTreeAutoscaling: true
|
||||
autoscalerOptions:
|
||||
version: v2
|
||||
upscalingMode: Default
|
||||
idleTimeoutSeconds: 600 # 10 minutes
|
||||
env:
|
||||
- name: AUTOSCALER_MAX_CONCURRENT_LAUNCHES
|
||||
value: "100"
|
||||
worker:
|
||||
groupName: standard-worker
|
||||
replicas: 1
|
||||
minReplicas: 1
|
||||
maxReplicas: 200
|
||||
rayStartParams:
|
||||
resources: '"{\"standard-worker\": 1}"'
|
||||
resources:
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: "1G"
|
||||
additionalWorkerGroups:
|
||||
additional-worker-group1:
|
||||
image:
|
||||
repository: docker.io/rayproject/ray
|
||||
tag: latest
|
||||
pullPolicy: IfNotPresent
|
||||
disabled: false
|
||||
replicas: 1
|
||||
minReplicas: 1
|
||||
maxReplicas: 30
|
||||
rayStartParams:
|
||||
resources: '"{\"additional-worker-group1\": 1}"'
|
||||
resources:
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: "1G"
|
||||
additional-worker-group2:
|
||||
image:
|
||||
repository: docker.io/rayproject/ray
|
||||
tag: latest
|
||||
pullPolicy: IfNotPresent
|
||||
disabled: false
|
||||
replicas: 1
|
||||
minReplicas: 1
|
||||
maxReplicas: 200
|
||||
rayStartParams:
|
||||
resources: '"{\"additional-worker-group2\": 1}"'
|
||||
resources:
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: "1G"
|
||||
syncPolicy:
|
||||
automated:
|
||||
prune: true
|
||||
selfHeal: true
|
||||
syncOptions:
|
||||
- CreateNamespace=true
|
||||
```
|
||||
|
||||
Note in this example, the `jqPathExpressions` approach is used.
|
||||
|
||||
Apply all three Applications at once:
|
||||
|
||||
```sh
|
||||
kubectl apply -f ray-argocd-all.yaml
|
||||
```
|
||||
|
||||
This single-file approach is convenient for quick deployments, but the step-by-step approach in the earlier sections provides better visibility into the deployment process and makes it easier to troubleshoot issues.
|
||||
@@ -0,0 +1,289 @@
|
||||
(kuberay-distributed-checkpointing-gcsfuse)=
|
||||
|
||||
# Distributed checkpointing with KubeRay and GCSFuse
|
||||
|
||||
This example orchestrates distributed checkpointing with KubeRay, using the GCSFuse CSI driver and Google Cloud Storage as the remote storage system. To illustrate the concepts, this guide uses the [Finetuning a Pytorch Image Classifier with Ray Train](https://docs.ray.io/en/latest/train/examples/pytorch/pytorch_resnet_finetune.html) example.
|
||||
|
||||
## Why distributed checkpointing with GCSFuse?
|
||||
|
||||
In large-scale, high-performance machine learning, distributed checkpointing is crucial for fault tolerance, ensuring that if a node fails during training, Ray can resume the process from the latest saved checkpoint instead of starting from scratch. While it's possible to directly reference remote storage paths (e.g., `gs://my-checkpoint-bucket`), using Google Cloud Storage FUSE (GCSFuse) has distinct advantages for distributed applications. GCSFuse allows you to mount Cloud Storage buckets like local file systems, making checkpoint management more intuitive for distributed applications that rely on these semantics. Furthermore, GCSFuse is designed for high-performance workloads, delivering the performance and scalability you need for distributed checkpointing of large models.
|
||||
|
||||
[Distributed checkpointing](https://docs.ray.io/en/latest/train/user-guides/checkpoints.html), in combination with [GCSFuse](https://cloud.google.com/storage/docs/gcs-fuse), allows for larger-scale model training with increased availability and efficiency.
|
||||
|
||||
## Create a Kubernetes cluster on GKE
|
||||
|
||||
Create a GKE cluster with the [GCSFuse CSI driver](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver) and [Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) enabled, as well as a GPU node pool with 4 L4 GPUs:
|
||||
```
|
||||
export PROJECT_ID=<your project id>
|
||||
gcloud container clusters create kuberay-with-gcsfuse \
|
||||
--addons GcsFuseCsiDriver \
|
||||
--cluster-version=1.29.4 \
|
||||
--location=us-east4-c \
|
||||
--machine-type=g2-standard-8 \
|
||||
--release-channel=rapid \
|
||||
--num-nodes=4 \
|
||||
--accelerator type=nvidia-l4,count=1,gpu-driver-version=latest \
|
||||
--workload-pool=${PROJECT_ID}.svc.id.goog
|
||||
```
|
||||
|
||||
Verify the successful creation of your cluster with 4 GPUs:
|
||||
```
|
||||
$ kubectl get nodes "-o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"
|
||||
NAME GPU
|
||||
gke-kuberay-with-gcsfuse-default-pool-xxxx-0000 1
|
||||
gke-kuberay-with-gcsfuse-default-pool-xxxx-1111 1
|
||||
gke-kuberay-with-gcsfuse-default-pool-xxxx-2222 1
|
||||
gke-kuberay-with-gcsfuse-default-pool-xxxx-3333 1
|
||||
```
|
||||
|
||||
## Install the KubeRay operator
|
||||
|
||||
Follow [Deploy a KubeRay operator](kuberay-operator-deploy) to install the latest stable KubeRay operator from the Helm repository. The KubeRay operator Pod must be on the CPU node if you set up the taint for the GPU node pool correctly.
|
||||
|
||||
## Configuring the GCS Bucket
|
||||
|
||||
Create a GCS bucket that Ray uses as the remote filesystem.
|
||||
```
|
||||
BUCKET=<your GCS bucket>
|
||||
gcloud storage buckets create gs://$BUCKET --uniform-bucket-level-access
|
||||
```
|
||||
|
||||
Create a Kubernetes ServiceAccount that grants the RayCluster access to mount the GCS bucket:
|
||||
```
|
||||
kubectl create serviceaccount pytorch-distributed-training
|
||||
```
|
||||
|
||||
Bind the `roles/storage.objectUser` role to the Kubernetes service account and bucket IAM policy. See [Identifying projects](https://cloud.google.com/resource-manager/docs/creating-managing-projects#identifying_projects) to find your project ID and project number:
|
||||
```
|
||||
PROJECT_ID=<your project ID>
|
||||
PROJECT_NUMBER=<your project number>
|
||||
gcloud storage buckets add-iam-policy-binding gs://${BUCKET} --member "principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${PROJECT_ID}.svc.id.goog/subject/ns/default/sa/pytorch-distributed-training" --role "roles/storage.objectUser"
|
||||
```
|
||||
|
||||
See [Access Cloud Storage buckets with the Cloud Storage FUSE CSI driver](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver) for more details.
|
||||
|
||||
## Deploy the RayJob
|
||||
|
||||
Download the RayJob that executes all the steps documented in [Finetuning a Pytorch Image Classifier with Ray Train](https://docs.ray.io/en/latest/train/examples/pytorch/pytorch_resnet_finetune.html). The [source code](https://github.com/ray-project/kuberay/tree/master/ray-operator/config/samples/pytorch-resnet-image-classifier) is also in the KubeRay repository.
|
||||
|
||||
```
|
||||
curl -LO https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/pytorch-resnet-image-classifier/ray-job.pytorch-image-classifier.yaml
|
||||
```
|
||||
|
||||
Modify the RayJob by replacing all instances of the `GCS_BUCKET` placeholder with the Google Cloud Storage bucket you created earlier. Alternatively you can use `sed`:
|
||||
```
|
||||
sed -i "s/GCS_BUCKET/$BUCKET/g" ray-job.pytorch-image-classifier.yaml
|
||||
```
|
||||
|
||||
Deploy the RayJob:
|
||||
```
|
||||
kubectl create -f ray-job.pytorch-image-classifier.yaml
|
||||
```
|
||||
|
||||
The deployed RayJob includes the following configuration to enable distributed checkpointing to a shared filesystem:
|
||||
* 4 Ray workers, each with a single GPU.
|
||||
* All Ray nodes use the `pytorch-distributed-training` ServiceAccount, which we created earlier.
|
||||
* Includes volumes that are managed by the `gcsfuse.csi.storage.gke.io` CSI driver.
|
||||
* Mounts a shared storage path `/mnt/cluster_storage`, backed by the GCS bucket you created earlier.
|
||||
|
||||
You can configure the Pod with annotations, which allows for finer grain control of the GCSFuse sidecar container. See [Specify Pod annotations](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver#pod-annotations) for more details.
|
||||
```
|
||||
annotations:
|
||||
gke-gcsfuse/volumes: "true"
|
||||
gke-gcsfuse/cpu-limit: "0"
|
||||
gke-gcsfuse/memory-limit: 5Gi
|
||||
gke-gcsfuse/ephemeral-storage-limit: 10Gi
|
||||
```
|
||||
|
||||
You can also specify mount options when defining the GCSFuse container volume:
|
||||
```
|
||||
csi:
|
||||
driver: gcsfuse.csi.storage.gke.io
|
||||
volumeAttributes:
|
||||
bucketName: GCS_BUCKET
|
||||
mountOptions: "implicit-dirs,uid=1000,gid=100"
|
||||
```
|
||||
|
||||
See [Mount options](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver#mount-options) to learn more about mount options.
|
||||
|
||||
Logs from the Ray job should indicate the use of the shared remote filesystem in `/mnt/cluster_storage` and the checkpointing directory. For example:
|
||||
```
|
||||
Training finished iteration 10 at 2024-04-29 10:22:08. Total running time: 1min 30s
|
||||
╭─────────────────────────────────────────╮
|
||||
│ Training result │
|
||||
├─────────────────────────────────────────┤
|
||||
│ checkpoint_dir_name checkpoint_000009 │
|
||||
│ time_this_iter_s 6.47154 │
|
||||
│ time_total_s 74.5547 │
|
||||
│ training_iteration 10 │
|
||||
│ acc 0.24183 │
|
||||
│ loss 0.06882 │
|
||||
╰─────────────────────────────────────────╯
|
||||
Training saved a checkpoint for iteration 10 at: (local)/mnt/cluster_storage/finetune-resnet/TorchTrainer_cbb82_00000_0_2024-04-29_10-20-37/checkpoint_000009
|
||||
```
|
||||
|
||||
## Inspect checkpointing data
|
||||
|
||||
Once the RayJob completes, you can inspect the contents of your bucket using a tool like [gsutil](https://cloud.google.com/storage/docs/gsutil).
|
||||
```
|
||||
gsutil ls gs://my-ray-bucket/**
|
||||
gs://my-ray-bucket/finetune-resnet/
|
||||
gs://my-ray-bucket/finetune-resnet/.validate_storage_marker
|
||||
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/
|
||||
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000007/
|
||||
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000007/checkpoint.pt
|
||||
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000008/
|
||||
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000008/checkpoint.pt
|
||||
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000009/
|
||||
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000009/checkpoint.pt
|
||||
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/error.pkl
|
||||
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/error.txt
|
||||
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/events.out.tfevents.1714436502.orch-image-classifier-nc2sq-raycluster-tdrfx-head-xzcl8
|
||||
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/events.out.tfevents.1714436809.orch-image-classifier-zz4sj-raycluster-vn7kz-head-lwx8k
|
||||
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/params.json
|
||||
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/params.pkl
|
||||
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/progress.csv
|
||||
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/result.json
|
||||
gs://my-ray-bucket/finetune-resnet/basic-variant-state-2024-04-29_17-21-29.json
|
||||
gs://my-ray-bucket/finetune-resnet/basic-variant-state-2024-04-29_17-26-35.json
|
||||
gs://my-ray-bucket/finetune-resnet/experiment_state-2024-04-29_17-21-29.json
|
||||
gs://my-ray-bucket/finetune-resnet/experiment_state-2024-04-29_17-26-35.json
|
||||
gs://my-ray-bucket/finetune-resnet/trainer.pkl
|
||||
gs://my-ray-bucket/finetune-resnet/tuner.pkl
|
||||
|
||||
```
|
||||
|
||||
## Resuming from checkpoint
|
||||
|
||||
In the event of a failed job, you can use the latest checkpoint to resume training of the model. This example configures `TorchTrainer` to automatically resume from the latest checkpoint:
|
||||
```python
|
||||
experiment_path = os.path.expanduser("/mnt/cluster_storage/finetune-resnet")
|
||||
if TorchTrainer.can_restore(experiment_path):
|
||||
trainer = TorchTrainer.restore(experiment_path,
|
||||
train_loop_per_worker=train_loop_per_worker,
|
||||
train_loop_config=train_loop_config,
|
||||
scaling_config=scaling_config,
|
||||
run_config=run_config,
|
||||
)
|
||||
else:
|
||||
trainer = TorchTrainer(
|
||||
train_loop_per_worker=train_loop_per_worker,
|
||||
train_loop_config=train_loop_config,
|
||||
scaling_config=scaling_config,
|
||||
run_config=run_config,
|
||||
)
|
||||
```
|
||||
|
||||
You can verify automatic checkpoint recovery by redeploying the same RayJob:
|
||||
```
|
||||
kubectl create -f ray-job.pytorch-image-classifier.yaml
|
||||
```
|
||||
|
||||
If the previous job succeeded, the training job should restore the checkpoint state from the `checkpoint_000009` directory and then immediately complete training with 0 iterations:
|
||||
```
|
||||
2024-04-29 15:51:32,528 INFO experiment_state.py:366 -- Trying to find and download experiment checkpoint at /mnt/cluster_storage/finetune-resnet
|
||||
2024-04-29 15:51:32,651 INFO experiment_state.py:396 -- A remote experiment checkpoint was found and will be used to restore the previous experiment state.
|
||||
2024-04-29 15:51:32,652 INFO tune_controller.py:404 -- Using the newest experiment state file found within the experiment directory: experiment_state-2024-04-29_15-43-40.json
|
||||
|
||||
View detailed results here: /mnt/cluster_storage/finetune-resnet
|
||||
To visualize your results with TensorBoard, run: `tensorboard --logdir /home/ray/ray_results/finetune-resnet`
|
||||
|
||||
Result(
|
||||
metrics={'loss': 0.070047477101968, 'acc': 0.23529411764705882},
|
||||
path='/mnt/cluster_storage/finetune-resnet/TorchTrainer_ecc04_00000_0_2024-04-29_15-43-40',
|
||||
filesystem='local',
|
||||
checkpoint=Checkpoint(filesystem=local, path=/mnt/cluster_storage/finetune-resnet/TorchTrainer_ecc04_00000_0_2024-04-29_15-43-40/checkpoint_000009)
|
||||
)
|
||||
```
|
||||
|
||||
If the previous job failed at an earlier checkpoint, the job should resume from the last saved checkpoint and run until `max_epochs=10`. For example, if the last run failed at epoch 7, the training automatically resumes using `checkpoint_000006` and run 3 more iterations until epoch 10:
|
||||
```
|
||||
(TorchTrainer pid=611, ip=10.108.2.65) Restored on 10.108.2.65 from checkpoint: Checkpoint(filesystem=local, path=/mnt/cluster_storage/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000006)
|
||||
(RayTrainWorker pid=671, ip=10.108.2.65) Setting up process group for: env:// [rank=0, world_size=4]
|
||||
(TorchTrainer pid=611, ip=10.108.2.65) Started distributed worker processes:
|
||||
(TorchTrainer pid=611, ip=10.108.2.65) - (ip=10.108.2.65, pid=671) world_rank=0, local_rank=0, node_rank=0
|
||||
(TorchTrainer pid=611, ip=10.108.2.65) - (ip=10.108.1.83, pid=589) world_rank=1, local_rank=0, node_rank=1
|
||||
(TorchTrainer pid=611, ip=10.108.2.65) - (ip=10.108.0.72, pid=590) world_rank=2, local_rank=0, node_rank=2
|
||||
(TorchTrainer pid=611, ip=10.108.2.65) - (ip=10.108.3.76, pid=590) world_rank=3, local_rank=0, node_rank=3
|
||||
(RayTrainWorker pid=589, ip=10.108.1.83) Downloading: "https://download.pytorch.org/models/resnet50-0676ba61.pth" to /home/ray/.cache/torch/hub/checkpoints/resnet50-0676ba61.pth
|
||||
(RayTrainWorker pid=671, ip=10.108.2.65)
|
||||
0%| | 0.00/97.8M [00:00<?, ?B/s]
|
||||
(RayTrainWorker pid=671, ip=10.108.2.65)
|
||||
22%|██▏ | 21.8M/97.8M [00:00<00:00, 229MB/s]
|
||||
(RayTrainWorker pid=671, ip=10.108.2.65)
|
||||
92%|█████████▏| 89.7M/97.8M [00:00<00:00, 327MB/s]
|
||||
(RayTrainWorker pid=671, ip=10.108.2.65)
|
||||
100%|██████████| 97.8M/97.8M [00:00<00:00, 316MB/s]
|
||||
(RayTrainWorker pid=671, ip=10.108.2.65) Moving model to device: cuda:0
|
||||
(RayTrainWorker pid=671, ip=10.108.2.65) Wrapping provided model in DistributedDataParallel.
|
||||
(RayTrainWorker pid=671, ip=10.108.2.65) Downloading: "https://download.pytorch.org/models/resnet50-0676ba61.pth" to /home/ray/.cache/torch/hub/checkpoints/resnet50-0676ba61.pth [repeated 3x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/ray-logging.html#log-deduplication for more options.)
|
||||
(RayTrainWorker pid=590, ip=10.108.3.76)
|
||||
0%| | 0.00/97.8M [00:00<?, ?B/s] [repeated 3x across cluster]
|
||||
(RayTrainWorker pid=590, ip=10.108.0.72)
|
||||
85%|████████▍ | 82.8M/97.8M [00:00<00:00, 256MB/s]
|
||||
100%|██████████| 97.8M/97.8M [00:00<00:00, 231MB/s] [repeated 11x across cluster]
|
||||
(RayTrainWorker pid=590, ip=10.108.3.76)
|
||||
100%|██████████| 97.8M/97.8M [00:00<00:00, 238MB/s]
|
||||
(RayTrainWorker pid=671, ip=10.108.2.65) Epoch 7-train Loss: 0.0903 Acc: 0.2418
|
||||
(RayTrainWorker pid=671, ip=10.108.2.65) Epoch 7-val Loss: 0.0881 Acc: 0.2353
|
||||
(RayTrainWorker pid=590, ip=10.108.0.72) Checkpoint successfully created at: Checkpoint(filesystem=local, path=/mnt/cluster_storage/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000007)
|
||||
(RayTrainWorker pid=590, ip=10.108.0.72) Moving model to device: cuda:0 [repeated 3x across cluster]
|
||||
(RayTrainWorker pid=590, ip=10.108.0.72) Wrapping provided model in DistributedDataParallel. [repeated 3x across cluster]
|
||||
|
||||
Training finished iteration 8 at 2024-04-29 17:27:29. Total running time: 54s
|
||||
╭─────────────────────────────────────────╮
|
||||
│ Training result │
|
||||
├─────────────────────────────────────────┤
|
||||
│ checkpoint_dir_name checkpoint_000007 │
|
||||
│ time_this_iter_s 40.46113 │
|
||||
│ time_total_s 95.00043 │
|
||||
│ training_iteration 8 │
|
||||
│ acc 0.23529 │
|
||||
│ loss 0.08811 │
|
||||
╰─────────────────────────────────────────╯
|
||||
Training saved a checkpoint for iteration 8 at: (local)/mnt/cluster_storage/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000007
|
||||
(RayTrainWorker pid=671, ip=10.108.2.65) Epoch 8-train Loss: 0.0893 Acc: 0.2459
|
||||
(RayTrainWorker pid=671, ip=10.108.2.65) Epoch 8-val Loss: 0.0859 Acc: 0.2353
|
||||
(RayTrainWorker pid=589, ip=10.108.1.83) Checkpoint successfully created at: Checkpoint(filesystem=local, path=/mnt/cluster_storage/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000008) [repeated 4x across cluster]
|
||||
|
||||
Training finished iteration 9 at 2024-04-29 17:27:36. Total running time: 1min 1s
|
||||
╭─────────────────────────────────────────╮
|
||||
│ Training result │
|
||||
├─────────────────────────────────────────┤
|
||||
│ checkpoint_dir_name checkpoint_000008 │
|
||||
│ time_this_iter_s 5.99923 │
|
||||
│ time_total_s 100.99965 │
|
||||
│ training_iteration 9 │
|
||||
│ acc 0.23529 │
|
||||
│ loss 0.08592 │
|
||||
╰─────────────────────────────────────────╯
|
||||
Training saved a checkpoint for iteration 9 at: (local)/mnt/cluster_storage/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000008
|
||||
2024-04-29 17:27:37,170 WARNING util.py:202 -- The `process_trial_save` operation took 0.540 s, which may be a performance bottleneck.
|
||||
(RayTrainWorker pid=671, ip=10.108.2.65) Epoch 9-train Loss: 0.0866 Acc: 0.2377
|
||||
(RayTrainWorker pid=671, ip=10.108.2.65) Epoch 9-val Loss: 0.0833 Acc: 0.2353
|
||||
(RayTrainWorker pid=589, ip=10.108.1.83) Checkpoint successfully created at: Checkpoint(filesystem=local, path=/mnt/cluster_storage/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000009) [repeated 4x across cluster]
|
||||
|
||||
Training finished iteration 10 at 2024-04-29 17:27:43. Total running time: 1min 8s
|
||||
╭─────────────────────────────────────────╮
|
||||
│ Training result │
|
||||
├─────────────────────────────────────────┤
|
||||
│ checkpoint_dir_name checkpoint_000009 │
|
||||
│ time_this_iter_s 6.71457 │
|
||||
│ time_total_s 107.71422 │
|
||||
│ training_iteration 10 │
|
||||
│ acc 0.23529 │
|
||||
│ loss 0.08333 │
|
||||
╰─────────────────────────────────────────╯
|
||||
Training saved a checkpoint for iteration 10 at: (local)/mnt/cluster_storage/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000009
|
||||
|
||||
Training completed after 10 iterations at 2024-04-29 17:27:45. Total running time: 1min 9s
|
||||
2024-04-29 17:27:46,236 WARNING experiment_state.py:323 -- Experiment checkpoint syncing has been triggered multiple times in the last 30.0 seconds. A sync will be triggered whenever a trial has checkpointed more than `num_to_keep` times since last sync or if 300 seconds have passed since last sync. If you have set `num_to_keep` in your `CheckpointConfig`, consider increasing the checkpoint frequency or keeping more checkpoints. You can suppress this warning by changing the `TUNE_WARN_EXCESSIVE_EXPERIMENT_CHECKPOINT_SYNC_THRESHOLD_S` environment variable.
|
||||
|
||||
Result(
|
||||
metrics={'loss': 0.08333033206416111, 'acc': 0.23529411764705882},
|
||||
path='/mnt/cluster_storage/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29',
|
||||
filesystem='local',
|
||||
checkpoint=Checkpoint(filesystem=local, path=/mnt/cluster_storage/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000009)
|
||||
)
|
||||
(RayTrainWorker pid=590, ip=10.108.3.76) Checkpoint successfully created at: Checkpoint(filesystem=local, path=/mnt/cluster_storage/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000009) [repeated 3x across cluster]
|
||||
```
|
||||
@@ -0,0 +1,121 @@
|
||||
(kuberay-mnist-training-example)=
|
||||
|
||||
# Train a PyTorch model on Fashion MNIST with CPUs on Kubernetes
|
||||
|
||||
This example runs distributed training of a PyTorch model on Fashion MNIST with Ray Train. See [Train a PyTorch model on Fashion MNIST](train-pytorch-fashion-mnist) for more details.
|
||||
|
||||
## Step 1: Create a Kubernetes cluster
|
||||
|
||||
This step creates a local Kubernetes cluster using [Kind](https://kind.sigs.k8s.io/). If you already have a Kubernetes cluster, you can skip this step.
|
||||
|
||||
```sh
|
||||
kind create cluster --image=kindest/node:v1.26.0
|
||||
```
|
||||
|
||||
## Step 2: Install KubeRay operator
|
||||
|
||||
Follow [this document](kuberay-operator-deploy) to install the latest stable KubeRay operator from the Helm repository.
|
||||
|
||||
## Step 3: Create a RayJob
|
||||
|
||||
A RayJob consists of a RayCluster custom resource and a job that can you can submit to the RayCluster. With RayJob, KubeRay creates a RayCluster and submits a job when the cluster is ready. The following is a CPU-only RayJob description YAML file for MNIST training on a PyTorch model.
|
||||
|
||||
```sh
|
||||
# Download `ray-job.pytorch-mnist.yaml`
|
||||
curl -LO https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/pytorch-mnist/ray-job.pytorch-mnist.yaml
|
||||
```
|
||||
|
||||
You might need to adjust some fields in the RayJob description YAML file so that it can run in your environment:
|
||||
* `replicas` under `workerGroupSpecs` in `rayClusterSpec`: This field specifies the number of worker Pods that KubeRay schedules to the Kubernetes cluster. Each worker Pod requires 3 CPUs, and the head Pod requires 1 CPU, as described in the `template` field. A RayJob submitter Pod requires 1 CPU. For example, if your machine has 8 CPUs, the maximum `replicas` value is 2 to allow all Pods to reach the `Running` status.
|
||||
* `NUM_WORKERS` under `runtimeEnvYAML` in `spec`: This field indicates the number of Ray actors to launch (see `ScalingConfig` in this [Document](ray-train-configs-api) for more information). Each Ray actor must be served by a worker Pod in the Kubernetes cluster. Therefore, `NUM_WORKERS` must be less than or equal to `replicas`.
|
||||
* `CPUS_PER_WORKER`: This must be set to less than or equal to `(CPU resource request per worker Pod) - 1`. For example, in the sample YAML file, the CPU resource request per worker Pod is 3 CPUs, so `CPUS_PER_WORKER` must be set to 2 or less.
|
||||
|
||||
```sh
|
||||
# `replicas` and `NUM_WORKERS` set to 2.
|
||||
# Create a RayJob.
|
||||
kubectl apply -f ray-job.pytorch-mnist.yaml
|
||||
|
||||
# Check existing Pods: According to `replicas`, there should be 2 worker Pods.
|
||||
# Make sure all the Pods are in the `Running` status.
|
||||
kubectl get pods
|
||||
# NAME READY STATUS RESTARTS AGE
|
||||
# kuberay-operator-6dddd689fb-ksmcs 1/1 Running 0 6m8s
|
||||
# rayjob-pytorch-mnist-raycluster-rkdmq-small-group-worker-c8bwx 1/1 Running 0 5m32s
|
||||
# rayjob-pytorch-mnist-raycluster-rkdmq-small-group-worker-s7wvm 1/1 Running 0 5m32s
|
||||
# rayjob-pytorch-mnist-nxmj2 1/1 Running 0 4m17s
|
||||
# rayjob-pytorch-mnist-raycluster-rkdmq-head-m4dsl 1/1 Running 0 5m32s
|
||||
```
|
||||
|
||||
Check that the RayJob is in the `RUNNING` status:
|
||||
|
||||
```sh
|
||||
kubectl get rayjob
|
||||
# NAME JOB STATUS DEPLOYMENT STATUS START TIME END TIME AGE
|
||||
# rayjob-pytorch-mnist RUNNING Running 2024-06-17T04:08:25Z 11m
|
||||
```
|
||||
|
||||
## Step 4: Wait until the RayJob completes and check the training results
|
||||
|
||||
Wait until the RayJob completes. It might take several minutes.
|
||||
|
||||
```sh
|
||||
kubectl get rayjob
|
||||
# NAME JOB STATUS DEPLOYMENT STATUS START TIME END TIME AGE
|
||||
# rayjob-pytorch-mnist SUCCEEDED Complete 2024-06-17T04:08:25Z 2024-06-17T04:22:21Z 16m
|
||||
```
|
||||
|
||||
After seeing `JOB_STATUS` marked as `SUCCEEDED`, you can check the training logs:
|
||||
|
||||
```sh
|
||||
# Check Pods name.
|
||||
kubectl get pods
|
||||
# NAME READY STATUS RESTARTS AGE
|
||||
# kuberay-operator-6dddd689fb-ksmcs 1/1 Running 0 113m
|
||||
# rayjob-pytorch-mnist-raycluster-rkdmq-small-group-worker-c8bwx 1/1 Running 0 38m
|
||||
# rayjob-pytorch-mnist-raycluster-rkdmq-small-group-worker-s7wvm 1/1 Running 0 38m
|
||||
# rayjob-pytorch-mnist-nxmj2 0/1 Completed 0 38m
|
||||
# rayjob-pytorch-mnist-raycluster-rkdmq-head-m4dsl 1/1 Running 0 38m
|
||||
|
||||
# Check training logs.
|
||||
kubectl logs -f rayjob-pytorch-mnist-nxmj2
|
||||
|
||||
# 2024-06-16 22:23:01,047 INFO cli.py:36 -- Job submission server address: http://rayjob-pytorch-mnist-raycluster-rkdmq-head-svc.default.svc.cluster.local:8265
|
||||
# 2024-06-16 22:23:01,844 SUCC cli.py:60 -- -------------------------------------------------------
|
||||
# 2024-06-16 22:23:01,844 SUCC cli.py:61 -- Job 'rayjob-pytorch-mnist-l6ccc' submitted successfully
|
||||
# 2024-06-16 22:23:01,844 SUCC cli.py:62 -- -------------------------------------------------------
|
||||
# ...
|
||||
# (RayTrainWorker pid=1138, ip=10.244.0.18)
|
||||
# 0%| | 0/26421880 [00:00<?, ?it/s]
|
||||
# (RayTrainWorker pid=1138, ip=10.244.0.18)
|
||||
# 0%| | 32768/26421880 [00:00<01:27, 301113.97it/s]
|
||||
# ...
|
||||
# Training finished iteration 10 at 2024-06-16 22:33:05. Total running time: 7min 9s
|
||||
# ╭───────────────────────────────╮
|
||||
# │ Training result │
|
||||
# ├───────────────────────────────┤
|
||||
# │ checkpoint_dir_name │
|
||||
# │ time_this_iter_s 28.2635 │
|
||||
# │ time_total_s 423.388 │
|
||||
# │ training_iteration 10 │
|
||||
# │ accuracy 0.8748 │
|
||||
# │ loss 0.35477 │
|
||||
# ╰───────────────────────────────╯
|
||||
|
||||
# Training completed after 10 iterations at 2024-06-16 22:33:06. Total running time: 7min 10s
|
||||
|
||||
# Training result: Result(
|
||||
# metrics={'loss': 0.35476621258825347, 'accuracy': 0.8748},
|
||||
# path='/home/ray/ray_results/TorchTrainer_2024-06-16_22-25-55/TorchTrainer_122aa_00000_0_2024-06-16_22-25-55',
|
||||
# filesystem='local',
|
||||
# checkpoint=None
|
||||
# )
|
||||
# ...
|
||||
```
|
||||
|
||||
## Clean up
|
||||
|
||||
Delete your RayJob with the following command:
|
||||
|
||||
```sh
|
||||
kubectl delete -f ray-job.pytorch-mnist.yaml
|
||||
```
|
||||
@@ -0,0 +1,52 @@
|
||||
(kuberay-mobilenet-rayservice-example)=
|
||||
|
||||
# Serve a MobileNet image classifier on Kubernetes
|
||||
|
||||
> **Note:** The Python files for the Ray Serve application and its client are in the repository [ray-project/serve_config_examples](https://github.com/ray-project/serve_config_examples).
|
||||
|
||||
## Step 1: Create a Kubernetes cluster with Kind
|
||||
|
||||
```sh
|
||||
kind create cluster --image=kindest/node:v1.26.0
|
||||
```
|
||||
|
||||
## Step 2: Install KubeRay operator
|
||||
|
||||
Follow [this document](kuberay-operator-deploy) to install the latest stable KubeRay operator from the Helm repository. Note that the YAML file in this example uses `serveConfigV2`. You need KubeRay version v0.6.0 or later to use this feature.
|
||||
|
||||
## Step 3: Install a RayService
|
||||
|
||||
```sh
|
||||
# Create a RayService
|
||||
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-service.mobilenet.yaml
|
||||
```
|
||||
|
||||
* The [mobilenet.py](https://github.com/ray-project/serve_config_examples/blob/master/mobilenet/mobilenet.py) file needs `tensorflow` as a dependency. Hence, the YAML file uses `rayproject/ray-ml` image instead of `rayproject/ray` image.
|
||||
* The request parsing function `starlette.requests.form()` needs `python-multipart`, so the YAML file includes `python-multipart` in the runtime environment.
|
||||
|
||||
## Step 4: Forward the port for Ray Serve
|
||||
|
||||
```sh
|
||||
# Wait for the RayService to be ready to serve requests
|
||||
kubectl describe rayservice/rayservice-mobilenet
|
||||
# Conditions:
|
||||
# Last Transition Time: 2025-02-13T02:29:26Z
|
||||
# Message: Number of serve endpoints is greater than 0
|
||||
# Observed Generation: 1
|
||||
# Reason: NonZeroServeEndpoints
|
||||
# Status: True
|
||||
# Type: Ready
|
||||
|
||||
# Forward the port for Ray Serve service
|
||||
kubectl port-forward svc/rayservice-mobilenet-serve-svc 8000
|
||||
```
|
||||
|
||||
## Step 5: Send a request to the ImageClassifier
|
||||
|
||||
* Step 5.1: Prepare an image file.
|
||||
* Step 5.2: Update `image_path` in [mobilenet_req.py](https://github.com/ray-project/serve_config_examples/blob/master/mobilenet/mobilenet_req.py)
|
||||
* Step 5.3: Send a request to the `ImageClassifier`.
|
||||
```sh
|
||||
python mobilenet_req.py
|
||||
# sample output: {"prediction":["n02099601","golden_retriever",0.17944198846817017]}
|
||||
```
|
||||
@@ -0,0 +1,170 @@
|
||||
(kuberay-agent-sandbox)=
|
||||
|
||||
# Sandboxed Code Execution with Ray and Agent Sandbox
|
||||
|
||||
This example shows how to use the [Agent Sandbox](https://github.com/kubernetes-sigs/agent-sandbox) with Ray and KubeRay to orchestrate code execution in a secure sandboxed environment. This example uses GKE and gVisor but can be modified to work on other sandbox runtimes.
|
||||
|
||||
---
|
||||
|
||||
## What is Agent Sandbox?
|
||||
|
||||
[Agent Sandbox](https://github.com/kubernetes-sigs/agent-sandbox) is a Kubernetes project to streamline the management of sandboxes on Kubernetes. Agent sandbox provides declarative Kubernetes APIs that can be used with KubeRay to manage sandbox environments that can be invoked from a Ray cluster.
|
||||
|
||||
Agent sandbox is compatible with multiple runtimes that offer strong isolation guarantees such as [gVisor](https://github.com/google/gvisor) and [Kata containers](https://github.com/kata-containers/kata-containers). Consider using Ray and Agent Sandbox for agentic RL use-cases where you need to securely execute code generated from a model during its post-training phase.
|
||||
|
||||
Agent Sandbox provides a collection of declarative Kubernetes APIs to easily manage Sandbox runtimes. This example uses the following custom resources provided by Agent Sandbox:
|
||||
- `Sandbox`: This is the foundational unit, it manages a single Pod with a stable hostname and network identity. Unlike standard Pods, a Sandbox can be configured with persistent storage via volumeClaimTemplates that survives restarts
|
||||
- `SandboxClaim`: Allows users to create `Sandboxes` from a `SandboxTemplate`, abstracting away the details of the underlying Sandbox configuration.
|
||||
- `SandboxTemplate`: Provides a way to define reusable templates for creating `Sandboxes`, making it easier to manage large numbers of similar `Sandboxes`.
|
||||
- `SandboxWarmPool`: Manages a pool of pre-warmed `Sandboxes` that can be quickly (<200ms) allocated to users, reducing the time it takes to get a new Sandbox up and running.
|
||||
|
||||
The Agent Sandbox project also provides a [Python SDK](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/clients/python/agentic-sandbox-client) which can be used from within Ray actors to invoke Sandbox creation and secure code execution on sandboxes.
|
||||
|
||||
## Deploying KubeRay with Agent Sandbox
|
||||
|
||||
The following example creates a KubeRay RayJob, which runs a Ray job that uses the Agent Sandbox SDK to invoke code execution in a secure sandbox. It is highly recommended to keep Pods used for sandboxing decoupled from the Ray cluster itself.
|
||||
|
||||
### Step 1: Create a GKE cluster and Node Pools
|
||||
|
||||
Run the following command to create a GKE cluster. In this example we will create two separate node pools, one for KubeRay provisioned Pods and one for Sandbox pods using the gVisor runtime:
|
||||
|
||||
```bash
|
||||
gcloud container node-pools create ray-worker-pool \
|
||||
--cluster=<YOUR_CLUSTER_NAME> \
|
||||
--machine-type=e2-standard-4 \
|
||||
--num-nodes=2
|
||||
|
||||
gcloud container node-pools create ray-gvisor-pool \
|
||||
--cluster=<YOUR_CLUSTER_NAME> \
|
||||
--sandbox type=gvisor \
|
||||
--machine-type=e2-standard-4 \
|
||||
--num-nodes=1
|
||||
```
|
||||
|
||||
### Step 2: Install KubeRay operator
|
||||
|
||||
Follow the instructions in [KubeRay operator](kuberay-operator-deploy) to install the KubeRay operator.
|
||||
|
||||
### Step 3: Deploy Agent Sandbox
|
||||
|
||||
Install the Custom Resource Definitions (CRDs), controllers, and extensions from the official Agent Sandbox release.
|
||||
|
||||
```bash
|
||||
export VERSION="v0.5.0"
|
||||
|
||||
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${VERSION}/manifest.yaml
|
||||
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${VERSION}/extensions.yaml
|
||||
```
|
||||
|
||||
### Step 4: Apply RBAC Permissions for the Ray Workers
|
||||
|
||||
The Agent Sandbox Python SDK running inside Ray Workers needs to talk to the Kubernetes API to claim and delete sandboxes. In this example we will use the default service account token in the default namespace to grant Ray workers the ability to spawn Sandboxes:
|
||||
|
||||
Create a file named `rbac.yaml` with the following content:
|
||||
```yaml
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: ray-sandbox-manager-role
|
||||
namespace: default
|
||||
rules:
|
||||
- apiGroups: ["extensions.agents.x-k8s.io"]
|
||||
resources: ["sandboxclaims"]
|
||||
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
|
||||
- apiGroups: ["agents.x-k8s.io"]
|
||||
resources: ["sandboxes"]
|
||||
verbs: ["get", "list", "watch"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: ray-sandbox-manager-binding
|
||||
namespace: default
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: default
|
||||
namespace: default
|
||||
roleRef:
|
||||
kind: Role
|
||||
name: ray-sandbox-manager-role
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
```
|
||||
|
||||
Apply the RBAC configurations:
|
||||
```bash
|
||||
kubectl apply -f rbac.yaml
|
||||
```
|
||||
|
||||
### Step 5: Deploy Sandbox Infrastructure
|
||||
|
||||
Run the following command to create sandbox infrastructure using Agent Sandbox:
|
||||
|
||||
```bash
|
||||
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/agent-sandbox/sandbox.yaml
|
||||
```
|
||||
|
||||
The following resources are created:
|
||||
- **`SandboxTemplate`**: defines the per-sandbox podSpec. The Pod is configured to use gVisor, sets `automountServiceAccountToken: false` so untrusted code inside the sandbox cannot read a Kubernetes ServiceAccount token, and sets `networkPolicyManagement: Unmanaged` because the NetworkPolicy below is stricter than the controller's Secure Default. The template also labels every sandbox pod with `app: python-runtime-pool` so other selectors (the NetworkPolicy podSelector, your own `kubectl get` queries) can target them by a stable, human-readable label.
|
||||
- **`SandboxWarmPool`** (`python-runtime-pool`) — keeps 6 pre-booted sandbox pods ready so the Ray actors' claims complete in under 200ms.
|
||||
- **`NetworkPolicy`** (`python-runtime-pool-restrict-egress`) — default-denies egress for every sandbox pod except DNS. This is what provides concrete containment, ensuring packets are dropped by the CNI at the node rather than relying on cluster-default policies.
|
||||
|
||||
Verify the warm pool pods are running:
|
||||
|
||||
```bash
|
||||
kubectl get pods -l app=python-runtime-pool
|
||||
```
|
||||
|
||||
Based on the configuration of the SandboxWarmpool, we expect 6 gVisor Pods to be running:
|
||||
|
||||
```bash
|
||||
GVISOR_POD=$(kubectl get pod -l app=python-runtime-pool -o jsonpath='{.items[0].metadata.name}')
|
||||
kubectl get pod "$GVISOR_POD" -o jsonpath='{.spec.automountServiceAccountToken}{"\n"}' # expect: false
|
||||
kubectl exec "$GVISOR_POD" -- ls /var/run/secrets/kubernetes.io/serviceaccount/ 2>&1 # expect: No such file or directory
|
||||
```
|
||||
|
||||
### Step 6: Create the RayJob
|
||||
|
||||
Run the following command to create a RayJob resource:
|
||||
|
||||
```sh
|
||||
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/agent-sandbox/ray-cluster.yaml
|
||||
```
|
||||
|
||||
The RayJob is configured to do the following:
|
||||
1. Create a RayCluster
|
||||
2. Submit a Ray job that runs `sandboxed_code_execution.py` on the Ray cluster
|
||||
3. The driver script will run Ray actors that use the Agent Sandbox Python SDK to invoke Sandbox creation.
|
||||
4. Once the Sandbox environments are created, the actor will execute some code in the sandboxed environment and verify its output.
|
||||
|
||||
### Step 7: Verify the output
|
||||
|
||||
Monitor the status and query the execution logs of the submitted RayJob:
|
||||
|
||||
```sh
|
||||
# List running job pods
|
||||
kubectl get pods -l job-name=agent-sandbox-code-execution-demo
|
||||
|
||||
# Stream the demo logs
|
||||
kubectl logs -f -l job-name=agent-sandbox-code-execution-demo
|
||||
```
|
||||
|
||||
Once the job starts, two `SandboxExecutor` Ray actors each claim one pod from `python-runtime-pool` (the SDK reports per-actor adoption latency — sub-200ms when the warm pool is healthy). Every Python snippet that follows runs **inside the sandbox pod, never on the Ray worker**: gVisor isolates the syscall surface, the `python-runtime-pool-restrict-egress` NetworkPolicy applied in Step 5 default-denies all egress except DNS, and `sandbox.commands.run(..., timeout=5)` bounds wall-clock blast radius per call.
|
||||
|
||||
Expected output (abridged):
|
||||
|
||||
```
|
||||
Starting 2 SandboxExecutors...
|
||||
Dispatching 2 code executors...
|
||||
(SandboxExecutor pid=457, ip=10.72.5.24) [executor-1] claimed sandbox 'sandbox-claim-6d4504d8' in 0.257s
|
||||
|
||||
--- Execution Results ---
|
||||
|
||||
[compute_fib.py] (Exit Code: 0)
|
||||
Stdout: fib(20) = 6765
|
||||
|
||||
[json_aggregation.py] (Exit Code: 0)
|
||||
Stdout: {"mean": 11.0, "max": 25}
|
||||
|
||||
Cleaning up sandboxes...
|
||||
(SandboxExecutor pid=342, ip=10.72.1.10) [executor-0] claimed sandbox 'sandbox-claim-3a93b626' in 0.212s
|
||||
```
|
||||
@@ -0,0 +1,137 @@
|
||||
(kuberay-batch-inference-example)=
|
||||
|
||||
# RayJob Batch Inference Example
|
||||
|
||||
This example demonstrates how to use the RayJob custom resource to run a batch inference job for an image classification workload on a Ray cluster. See [Image Classification Batch Inference with HuggingFace Vision Transformer](https://docs.ray.io/en/latest/data/examples/huggingface_vit_batch_prediction.html) for a full explanation of the code.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
You must have a Kubernetes cluster running,`kubectl` configured to use it, and GPUs available. This example provides a brief tutorial for setting up the necessary GPUs on Google Kubernetes Engine (GKE), but you can use any Kubernetes cluster with GPUs.
|
||||
|
||||
## Step 0: Create a Kubernetes cluster on GKE (Optional)
|
||||
|
||||
If you already have a Kubernetes cluster with GPUs, you can skip this step.
|
||||
|
||||
|
||||
Otherwise, follow [this tutorial](kuberay-gke-gpu-cluster-setup), but substitute the following GPU node pool creation command to create a Kubernetes cluster on GKE with four NVIDIA T4 GPUs:
|
||||
|
||||
```sh
|
||||
gcloud container node-pools create gpu-node-pool \
|
||||
--accelerator type=nvidia-tesla-t4,count=4,gpu-driver-version=default \
|
||||
--zone us-west1-b \
|
||||
--cluster kuberay-gpu-cluster \
|
||||
--num-nodes 1 \
|
||||
--min-nodes 0 \
|
||||
--max-nodes 1 \
|
||||
--enable-autoscaling \
|
||||
--machine-type n1-standard-64
|
||||
```
|
||||
|
||||
This example uses four [NVIDIA T4](https://cloud.google.com/compute/docs/gpus#nvidia_t4_gpus) GPUs. The machine type is `n1-standard-64`, which has [64 vCPUs and 240 GB RAM](https://cloud.google.com/compute/docs/general-purpose-machines#n1_machine_types).
|
||||
|
||||
## Step 1: Install the KubeRay Operator
|
||||
|
||||
Follow [this document](kuberay-operator-deploy) to install the latest stable KubeRay operator from the Helm repository. The KubeRay operator Pod must be on the CPU node if you have set up the taint for the GPU node pool correctly.
|
||||
|
||||
## Step 2: Submit the RayJob
|
||||
|
||||
Create the RayJob custom resource with [ray-job.batch-inference.yaml](https://github.com/ray-project/kuberay/blob/v1.6.0/ray-operator/config/samples/ray-job.batch-inference.yaml).
|
||||
|
||||
Download the file with `curl`:
|
||||
|
||||
```bash
|
||||
curl -LO https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-job.batch-inference.yaml
|
||||
```
|
||||
|
||||
Note that the `RayJob` spec contains a spec for the `RayCluster`. This tutorial uses a single-node cluster with 4 GPUs. For production use cases, use a multi-node cluster where the head node doesn't have GPUs, so that Ray can automatically schedule GPU workloads on worker nodes which won't interfere with critical Ray processes on the head node.
|
||||
|
||||
Note the following fields in the `RayJob` spec, which specify the Ray image and the GPU resources for the Ray node:
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
containers:
|
||||
- name: ray-head
|
||||
image: rayproject/ray-ml:2.6.3-gpu
|
||||
resources:
|
||||
limits:
|
||||
nvidia.com/gpu: "4"
|
||||
cpu: "54"
|
||||
memory: "54Gi"
|
||||
requests:
|
||||
nvidia.com/gpu: "4"
|
||||
cpu: "54"
|
||||
memory: "54Gi"
|
||||
volumeMounts:
|
||||
- mountPath: /home/ray/samples
|
||||
name: code-sample
|
||||
nodeSelector:
|
||||
cloud.google.com/gke-accelerator: nvidia-tesla-t4 # This is the GPU type we used in the GPU node pool.
|
||||
```
|
||||
|
||||
To submit the job, run the following command:
|
||||
|
||||
```bash
|
||||
kubectl apply -f ray-job.batch-inference.yaml
|
||||
```
|
||||
|
||||
Check the status with `kubectl describe rayjob rayjob-sample`.
|
||||
|
||||
Sample output:
|
||||
|
||||
```
|
||||
[...]
|
||||
Status:
|
||||
Dashboard URL: rayjob-sample-raycluster-j6t8n-head-svc.default.svc.cluster.local:8265
|
||||
End Time: ...
|
||||
Job Deployment Status: Complete
|
||||
Job Id: rayjob-sample-ft8lh
|
||||
Job Status: SUCCEEDED
|
||||
Message: Job finished successfully.
|
||||
Observed Generation: 2
|
||||
...
|
||||
```
|
||||
|
||||
To view the logs, first find the name of the pod running the job with `kubectl get pods`.
|
||||
|
||||
Sample output:
|
||||
|
||||
```bash
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
kuberay-operator-8b86754c-r4rc2 1/1 Running 0 25h
|
||||
rayjob-sample-raycluster-j6t8n-head-kx2gz 1/1 Running 0 35m
|
||||
rayjob-sample-w98c7 0/1 Completed 0 30m
|
||||
```
|
||||
|
||||
The Ray cluster is still running because `shutdownAfterJobFinishes` isn't set in the `RayJob` spec. If you set `shutdownAfterJobFinishes` to `true`, the cluster is shut down after the job finishes.
|
||||
|
||||
Next, run:
|
||||
|
||||
```text
|
||||
kubectl logs rayjob-sample-w98c7
|
||||
```
|
||||
|
||||
to get the standard output of the `entrypoint` command for the `RayJob`. Sample output:
|
||||
|
||||
```text
|
||||
[...]
|
||||
Running: 62.0/64.0 CPU, 4.0/4.0 GPU, 955.57 MiB/12.83 GiB object_store_memory: 0%| | 0/200 [00:05<?, ?it/s]
|
||||
Running: 61.0/64.0 CPU, 4.0/4.0 GPU, 999.41 MiB/12.83 GiB object_store_memory: 0%| | 0/200 [00:05<?, ?it/s]
|
||||
Running: 61.0/64.0 CPU, 4.0/4.0 GPU, 999.41 MiB/12.83 GiB object_store_memory: 0%| | 1/200 [00:05<17:04, 5.15s/it]
|
||||
Running: 61.0/64.0 CPU, 4.0/4.0 GPU, 1008.68 MiB/12.83 GiB object_store_memory: 0%| | 1/200 [00:05<17:04, 5.15s/it]
|
||||
Running: 61.0/64.0 CPU, 4.0/4.0 GPU, 1008.68 MiB/12.83 GiB object_store_memory: 100%|██████████| 1/1 [00:05<00:00, 5.15s/it]
|
||||
|
||||
2023-08-22 15:48:33,905 WARNING actor_pool_map_operator.py:267 -- To ensure full parallelization across an actor pool of size 4, the specified batch size should be at most 5. Your configured batch size for this operator was 16.
|
||||
<PIL.Image.Image image mode=RGB size=500x375 at 0x7B37546CF7F0>
|
||||
Label: tench, Tinca tinca
|
||||
<PIL.Image.Image image mode=RGB size=500x375 at 0x7B37546AE430>
|
||||
Label: tench, Tinca tinca
|
||||
<PIL.Image.Image image mode=RGB size=500x375 at 0x7B37546CF430>
|
||||
Label: tench, Tinca tinca
|
||||
<PIL.Image.Image image mode=RGB size=500x375 at 0x7B37546AE430>
|
||||
Label: tench, Tinca tinca
|
||||
<PIL.Image.Image image mode=RGB size=500x375 at 0x7B37546CF7F0>
|
||||
Label: tench, Tinca tinca
|
||||
2023-08-22 15:48:36,522 SUCC cli.py:33 -- -----------------------------------
|
||||
2023-08-22 15:48:36,522 SUCC cli.py:34 -- Job 'rayjob-sample-ft8lh' succeeded
|
||||
2023-08-22 15:48:36,522 SUCC cli.py:35 -- -----------------------------------
|
||||
```
|
||||
@@ -0,0 +1,225 @@
|
||||
(kuberay-kueue-gang-scheduling-example)=
|
||||
|
||||
# Gang Scheduling with RayJob and Kueue
|
||||
|
||||
This guide demonstrates how to use Kueue for gang scheduling RayJob resources, taking advantage of dynamic resource provisioning and queueing on Kubernetes. To illustrate the concepts, this guide uses the [Fine-tune a PyTorch Lightning Text Classifier with Ray Data](https://docs.ray.io/en/master/train/examples/lightning/lightning_cola_advanced.html) example.
|
||||
|
||||
## Gang scheduling
|
||||
|
||||
Gang scheduling in Kubernetes ensures that a group of related Pods, such as those in a Ray cluster, only start when all required resources are available. Having this requirement is crucial when working with expensive, limited resources like GPUs.
|
||||
|
||||
## Kueue
|
||||
|
||||
[Kueue](https://kueue.sigs.k8s.io/) is a Kubernetes-native system that manages quotas and how jobs consume them. Kueue decides when:
|
||||
* To make a job wait.
|
||||
* To admit a job to start, which triggers Kubernetes to create Pods.
|
||||
* To preempt a job, which triggers Kubernetes to delete active Pods.
|
||||
|
||||
Kueue has native support for some KubeRay APIs. Specifically, you can use Kueue to manage resources that RayJob, RayCluster, and RayService consume. See the [Kueue documentation](https://kueue.sigs.k8s.io/docs/overview/) to learn more.
|
||||
|
||||
## Why use gang scheduling
|
||||
|
||||
Gang scheduling is essential when working with expensive, limited hardware accelerators like GPUs. It prevents RayJobs from partially provisioning Ray clusters and claiming but not using the GPUs. Kueue suspends a RayJob until the Kubernetes cluster and the underlying cloud provider can guarantee the capacity that the RayJob needs to execute. This approach greatly improves GPU utilization and cost, especially when GPU availability is limited.
|
||||
|
||||
## Create a Kubernetes cluster on GKE
|
||||
|
||||
Create a GKE cluster with the `enable-autoscaling` option:
|
||||
```bash
|
||||
gcloud container clusters create kuberay-gpu-cluster \
|
||||
--num-nodes=1 --min-nodes 0 --max-nodes 1 --enable-autoscaling \
|
||||
--zone=us-east4-c --machine-type e2-standard-4
|
||||
```
|
||||
|
||||
Create a GPU node pool with the `enable-queued-provisioning` option enabled:
|
||||
```bash
|
||||
gcloud container node-pools create gpu-node-pool \
|
||||
--accelerator type=nvidia-l4,count=1,gpu-driver-version=latest \
|
||||
--enable-queued-provisioning \
|
||||
--reservation-affinity=none \
|
||||
--zone us-east4-c \
|
||||
--cluster kuberay-gpu-cluster \
|
||||
--num-nodes 0 \
|
||||
--min-nodes 0 \
|
||||
--max-nodes 10 \
|
||||
--enable-autoscaling \
|
||||
--machine-type g2-standard-4
|
||||
```
|
||||
|
||||
This command creates a node pool, which initially has zero nodes. The `--enable-queued-provisioning` flag enables "queued provisioning" in the Kubernetes node autoscaler using the ProvisioningRequest API. More details are below. You need to use the `--reservation-affinity=none` flag because GKE doesn't support Node Reservations with ProvisioningRequest.
|
||||
|
||||
|
||||
## Install the KubeRay operator
|
||||
|
||||
Follow [Deploy a KubeRay operator](kuberay-operator-deploy) to install the latest stable KubeRay operator from the Helm repository. The KubeRay operator Pod must be on the CPU node if you set up the taint for the GPU node pool correctly.
|
||||
|
||||
## Install Kueue
|
||||
|
||||
Install the latest released version of Kueue.
|
||||
```
|
||||
kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.13.4/manifests.yaml
|
||||
```
|
||||
|
||||
See [Kueue Installation](https://kueue.sigs.k8s.io/docs/installation/#install-a-released-version) for more details on installing Kueue.
|
||||
|
||||
## Configure Kueue for gang scheduling
|
||||
|
||||
Next, configure Kueue for gang scheduling. Kueue leverages the ProvisioningRequest API for two key tasks:
|
||||
1. Dynamically adding new nodes to the cluster when a job needs more resources.
|
||||
2. Blocking the admission of new jobs that are waiting for sufficient resources to become available.
|
||||
|
||||
See [How ProvisioningRequest works](https://cloud.google.com/kubernetes-engine/docs/how-to/provisioningrequest#how-provisioningrequest-works) for more details.
|
||||
|
||||
### Create Kueue resources
|
||||
|
||||
This manifest creates the following resources:
|
||||
* [ClusterQueue](https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/): Defines quotas and fair sharing rules
|
||||
* [LocalQueue](https://kueue.sigs.k8s.io/docs/concepts/local_queue/): A namespaced queue, belonging to a tenant, that references a ClusterQueue
|
||||
* [ResourceFlavor](https://kueue.sigs.k8s.io/docs/concepts/resource_flavor/): Defines what resources are available in the cluster, typically from Nodes
|
||||
* [AdmissionCheck](https://kueue.sigs.k8s.io/docs/concepts/admission_check/): A mechanism allowing components to influence the timing of a workload admission
|
||||
|
||||
```yaml
|
||||
# kueue-resources.yaml
|
||||
apiVersion: kueue.x-k8s.io/v1beta1
|
||||
kind: ResourceFlavor
|
||||
metadata:
|
||||
name: "default-flavor"
|
||||
---
|
||||
apiVersion: kueue.x-k8s.io/v1beta1
|
||||
kind: AdmissionCheck
|
||||
metadata:
|
||||
name: rayjob-gpu
|
||||
spec:
|
||||
controllerName: kueue.x-k8s.io/provisioning-request
|
||||
parameters:
|
||||
apiGroup: kueue.x-k8s.io
|
||||
kind: ProvisioningRequestConfig
|
||||
name: rayjob-gpu-config
|
||||
---
|
||||
apiVersion: kueue.x-k8s.io/v1beta1
|
||||
kind: ProvisioningRequestConfig
|
||||
metadata:
|
||||
name: rayjob-gpu-config
|
||||
spec:
|
||||
provisioningClassName: queued-provisioning.gke.io
|
||||
managedResources:
|
||||
- nvidia.com/gpu
|
||||
---
|
||||
apiVersion: kueue.x-k8s.io/v1beta1
|
||||
kind: ClusterQueue
|
||||
metadata:
|
||||
name: "cluster-queue"
|
||||
spec:
|
||||
namespaceSelector: {} # match all
|
||||
resourceGroups:
|
||||
- coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
|
||||
flavors:
|
||||
- name: "default-flavor"
|
||||
resources:
|
||||
- name: "cpu"
|
||||
nominalQuota: 10000 # infinite quotas
|
||||
- name: "memory"
|
||||
nominalQuota: 10000Gi # infinite quotas
|
||||
- name: "nvidia.com/gpu"
|
||||
nominalQuota: 10000 # infinite quotas
|
||||
admissionChecks:
|
||||
- rayjob-gpu
|
||||
---
|
||||
apiVersion: kueue.x-k8s.io/v1beta1
|
||||
kind: LocalQueue
|
||||
metadata:
|
||||
namespace: "default"
|
||||
name: "user-queue"
|
||||
spec:
|
||||
clusterQueue: "cluster-queue"
|
||||
```
|
||||
|
||||
Create the Kueue resources:
|
||||
```bash
|
||||
kubectl apply -f kueue-resources.yaml
|
||||
```
|
||||
|
||||
:::{note}
|
||||
This example configures Kueue to orchestrate the gang scheduling of GPUs. However, you can use other resources such as CPU and memory.
|
||||
:::
|
||||
|
||||
## Deploy a RayJob
|
||||
|
||||
Download the RayJob that executes all the steps documented in [Fine-tune a PyTorch Lightning Text Classifier](https://docs.ray.io/en/master/train/examples/lightning/lightning_cola_advanced.html). The [source code](https://github.com/ray-project/kuberay/tree/master/ray-operator/config/samples/pytorch-text-classifier) is also in the KubeRay repository.
|
||||
|
||||
```bash
|
||||
curl -LO https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/pytorch-text-classifier/ray-job.pytorch-distributed-training.yaml
|
||||
```
|
||||
|
||||
Before creating the RayJob, modify the RayJob metadata with a label to assign the RayJob to the LocalQueue that you created earlier:
|
||||
```yaml
|
||||
metadata:
|
||||
generateName: pytorch-text-classifier-
|
||||
labels:
|
||||
kueue.x-k8s.io/queue-name: user-queue
|
||||
```
|
||||
|
||||
Deploy the RayJob:
|
||||
```bash
|
||||
$ kubectl create -f ray-job.pytorch-distributed-training.yaml
|
||||
rayjob.ray.io/dev-pytorch-text-classifier-r6d4p created
|
||||
```
|
||||
|
||||
## Gang scheduling with RayJob
|
||||
|
||||
Following is the expected behavior when you deploy a GPU-requiring RayJob to a cluster that initially lacks GPUs:
|
||||
* Kueue suspends the RayJob due to insufficient GPU resources in the cluster.
|
||||
* Kueue creates a ProvisioningRequest, specifying the GPU requirements for the RayJob.
|
||||
* The Kubernetes node autoscaler monitors ProvisioningRequests and adds nodes with GPUs as needed.
|
||||
* Once the required GPU nodes are available, the ProvisioningRequest is satisfied.
|
||||
* Kueue admits the RayJob, allowing Kubernetes to schedule the Ray nodes on the newly provisioned nodes, and the RayJob execution begins.
|
||||
|
||||
If GPUs are unavailable, Kueue keeps suspending the RayJob. In addition, the node autoscaler avoids provisioning new nodes until it can fully satisfy the RayJob's GPU requirements.
|
||||
|
||||
Upon creating a RayJob, notice that the RayJob status is immediately `suspended` despite the ClusterQueue having GPU quotas available.
|
||||
```bash
|
||||
$ kubectl get rayjob pytorch-text-classifier-rj4sg -o yaml
|
||||
apiVersion: ray.io/v1
|
||||
kind: RayJob
|
||||
metadata:
|
||||
name: pytorch-text-classifier-rj4sg
|
||||
labels:
|
||||
kueue.x-k8s.io/queue-name: user-queue
|
||||
...
|
||||
...
|
||||
...
|
||||
status:
|
||||
jobDeploymentStatus: Suspended # RayJob suspended
|
||||
jobId: pytorch-text-classifier-rj4sg-pj9hx
|
||||
jobStatus: PENDING
|
||||
```
|
||||
|
||||
Kueue keeps suspending this RayJob until its corresponding ProvisioningRequest is satisfied. List ProvisioningRequest resources and their status with this command:
|
||||
```bash
|
||||
$ kubectl get provisioningrequest
|
||||
NAME ACCEPTED PROVISIONED FAILED AGE
|
||||
rayjob-pytorch-text-classifier-nv77q-e95ec-rayjob-gpu-1 True False False 22s
|
||||
```
|
||||
|
||||
Note the two columns in the output: `ACCEPTED` and `PROVISIONED`. `ACCEPTED=True` means that Kueue and the Kubernetes node autoscaler have acknowledged the request. `PROVISIONED=True` means that the Kubernetes node autoscaler has completed provisioning nodes. Once both of these conditions are true, the ProvisioningRequest is satisfied.
|
||||
```bash
|
||||
$ kubectl get provisioningrequest
|
||||
NAME ACCEPTED PROVISIONED FAILED AGE
|
||||
rayjob-pytorch-text-classifier-nv77q-e95ec-rayjob-gpu-1 True True False 57s
|
||||
```
|
||||
|
||||
Because the example RayJob requires 1 GPU for fine-tuning, the ProvisioningRequest is satisfied by the addition of a single GPU node in the `gpu-node-pool` Node Pool.
|
||||
```bash
|
||||
$ kubectl get nodes
|
||||
NAME STATUS ROLES AGE VERSION
|
||||
gke-kuberay-gpu-cluster-default-pool-8d883840-fd6d Ready <none> 14m v1.29.0-gke.1381000
|
||||
gke-kuberay-gpu-cluster-gpu-node-pool-b176212e-g3db Ready <none> 46s v1.29.0-gke.1381000 # new node with GPUs
|
||||
```
|
||||
|
||||
Once the ProvisioningRequest is satisfied, Kueue admits the RayJob. The Kubernetes scheduler then immediately places the head and worker nodes onto the newly provisioned resources. The ProvisioningRequest ensures a seamless Ray cluster start up, with no scheduling delays for any Pods.
|
||||
|
||||
```bash
|
||||
$ kubectl get pods
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
pytorch-text-classifier-nv77q-g6z57 1/1 Running 0 13s
|
||||
torch-text-classifier-nv77q-raycluster-gstrk-head-phnfl 1/1 Running 0 6m43s
|
||||
```
|
||||
@@ -0,0 +1,250 @@
|
||||
(kuberay-kueue-priority-scheduling-example)=
|
||||
|
||||
# Priority Scheduling with RayJob and Kueue
|
||||
|
||||
This guide shows how to run [Fine-tune a PyTorch Lightning Text Classifier with Ray Data](https://docs.ray.io/en/master/train/examples/lightning/lightning_cola_advanced.html) example as a RayJob and leverage Kueue to orchestrate priority scheduling and quota management.
|
||||
|
||||
## What's Kueue?
|
||||
|
||||
[Kueue](https://kueue.sigs.k8s.io/) is a Kubernetes-native job queueing system that manages quotas and how jobs consume them. Kueue decides when:
|
||||
* To make a job wait
|
||||
* To admit a job to start, meaning that Kubernetes creates pods.
|
||||
* To preempt a job, meaning that Kubernetes deletes active pods.
|
||||
|
||||
Kueue has native support for some KubeRay APIs. Specifically, you can use Kueue to manage resources consumed by RayJob and RayCluster. See the [Kueue documentation](https://kueue.sigs.k8s.io/docs/overview/) to learn more.
|
||||
|
||||
## Step 0: Create a Kubernetes cluster on GKE (Optional)
|
||||
|
||||
If you already have a Kubernetes cluster with GPUs, you can skip this step. Otherwise, follow [Start Google Cloud GKE Cluster with GPUs for KubeRay](kuberay-gke-gpu-cluster-setup) to set up a Kubernetes cluster on GKE.
|
||||
|
||||
## Step 1: Install the KubeRay operator
|
||||
|
||||
Follow [Deploy a KubeRay operator](kuberay-operator-deploy) to install the latest stable KubeRay operator from the Helm repository. The KubeRay operator Pod must be on the CPU node if you set up the taint for the GPU node pool correctly.
|
||||
|
||||
## Step 2: Install Kueue
|
||||
|
||||
```bash
|
||||
VERSION=v0.13.4
|
||||
kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/$VERSION/manifests.yaml
|
||||
```
|
||||
|
||||
See [Kueue Installation](https://kueue.sigs.k8s.io/docs/installation/#install-a-released-version) for more details on installing Kueue.
|
||||
|
||||
## Step 3: Configure Kueue with priority scheduling
|
||||
|
||||
To understand this tutorial, it's important to understand the following Kueue concepts:
|
||||
* [ResourceFlavor](https://kueue.sigs.k8s.io/docs/concepts/resource_flavor/)
|
||||
* [ClusterQueue](https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/)
|
||||
* [LocalQueue](https://kueue.sigs.k8s.io/docs/concepts/local_queue/)
|
||||
* [WorkloadPriorityClass](https://kueue.sigs.k8s.io/docs/concepts/workload_priority_class/)
|
||||
|
||||
```yaml
|
||||
# kueue-resources.yaml
|
||||
apiVersion: kueue.x-k8s.io/v1beta1
|
||||
kind: ResourceFlavor
|
||||
metadata:
|
||||
name: "default-flavor"
|
||||
---
|
||||
apiVersion: kueue.x-k8s.io/v1beta1
|
||||
kind: ClusterQueue
|
||||
metadata:
|
||||
name: "cluster-queue"
|
||||
spec:
|
||||
preemption:
|
||||
withinClusterQueue: LowerPriority
|
||||
namespaceSelector: {} # Match all namespaces.
|
||||
resourceGroups:
|
||||
- coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
|
||||
flavors:
|
||||
- name: "default-flavor"
|
||||
resources:
|
||||
- name: "cpu"
|
||||
nominalQuota: 2
|
||||
- name: "memory"
|
||||
nominalQuota: 8G
|
||||
- name: "nvidia.com/gpu" # ClusterQueue only has quota for a single GPU.
|
||||
nominalQuota: 1
|
||||
---
|
||||
apiVersion: kueue.x-k8s.io/v1beta1
|
||||
kind: LocalQueue
|
||||
metadata:
|
||||
namespace: "default"
|
||||
name: "user-queue"
|
||||
spec:
|
||||
clusterQueue: "cluster-queue"
|
||||
---
|
||||
apiVersion: kueue.x-k8s.io/v1beta1
|
||||
kind: WorkloadPriorityClass
|
||||
metadata:
|
||||
name: prod-priority
|
||||
value: 1000
|
||||
description: "Priority class for prod jobs"
|
||||
---
|
||||
apiVersion: kueue.x-k8s.io/v1beta1
|
||||
kind: WorkloadPriorityClass
|
||||
metadata:
|
||||
name: dev-priority
|
||||
value: 100
|
||||
description: "Priority class for development jobs"
|
||||
```
|
||||
|
||||
The YAML manifest configures:
|
||||
|
||||
* **ResourceFlavor**
|
||||
* The ResourceFlavor `default-flavor` is an empty ResourceFlavor because the compute resources in the Kubernetes cluster are homogeneous. In other words, users can request 1 GPU without considering whether it's an NVIDIA A100 or a T4 GPU.
|
||||
* **ClusterQueue**
|
||||
* The ClusterQueue `cluster-queue` only has 1 ResourceFlavor `default-flavor` with quotas for 2 CPUs, 8G memory, and 1 GPU. It exactly matches the resources requested by 1 RayJob custom resource. ***Hence, only 1 RayJob can run at a time.***
|
||||
* The ClusterQueue `cluster-queue` has a preemption policy `withinClusterQueue: LowerPriority`. This policy allows the pending RayJob that doesn’t fit within the nominal quota for its ClusterQueue to preempt active RayJob custom resources in the ClusterQueue that have lower priority.
|
||||
* **LocalQueue**
|
||||
* The LocalQueue `user-queue` is a namespaced object in the `default` namespace which belongs to a ClusterQueue. A typical practice is to assign a namespace to a tenant, team or user, of an organization. Users submit jobs to a LocalQueue, instead of to a ClusterQueue directly.
|
||||
* **WorkloadPriorityClass**
|
||||
* The WorkloadPriorityClass `prod-priority` has a higher value than the WorkloadPriorityClass `dev-priority`. This means that RayJob custom resources with the `prod-priority` priority class take precedence over RayJob custom resources with the `dev-priority` priority class.
|
||||
|
||||
Create the Kueue resources:
|
||||
```bash
|
||||
kubectl apply -f kueue-resources.yaml
|
||||
```
|
||||
|
||||
## Step 4: Deploy a RayJob
|
||||
|
||||
Download the RayJob that executes all the steps documented in [Fine-tune a PyTorch Lightning Text Classifier](https://docs.ray.io/en/master/train/examples/lightning/lightning_cola_advanced.html). The [source code](https://github.com/ray-project/kuberay/tree/master/ray-operator/config/samples/pytorch-text-classifier) is also in the KubeRay repository.
|
||||
|
||||
```bash
|
||||
curl -LO https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/pytorch-text-classifier/ray-job.pytorch-distributed-training.yaml
|
||||
```
|
||||
|
||||
Before creating the RayJob, modify the RayJob metadata with:
|
||||
|
||||
```yaml
|
||||
metadata:
|
||||
generateName: dev-pytorch-text-classifier-
|
||||
labels:
|
||||
kueue.x-k8s.io/queue-name: user-queue
|
||||
kueue.x-k8s.io/priority-class: dev-priority
|
||||
```
|
||||
|
||||
* `kueue.x-k8s.io/queue-name: user-queue`: As the previous step mentioned, users submit jobs to a LocalQueue instead of directly to a ClusterQueue.
|
||||
* `kueue.x-k8s.io/priority-class: dev-priority`: Assign the RayJob with the `dev-priority` WorkloadPriorityClass.
|
||||
* A modified name to indicate that this job is for development.
|
||||
|
||||
Also note the resources required for this RayJob by looking at the resources that the Ray head Pod requests:
|
||||
```yaml
|
||||
resources:
|
||||
limits:
|
||||
memory: "8G"
|
||||
nvidia.com/gpu: "1"
|
||||
requests:
|
||||
cpu: "2"
|
||||
memory: "8G"
|
||||
nvidia.com/gpu: "1"
|
||||
```
|
||||
|
||||
Now deploy the RayJob:
|
||||
```bash
|
||||
$ kubectl create -f ray-job.pytorch-distributed-training.yaml
|
||||
rayjob.ray.io/dev-pytorch-text-classifier-r6d4p created
|
||||
```
|
||||
|
||||
Verify that the RayCluster and the submitter Kubernetes Job are running:
|
||||
```bash
|
||||
$ kubectl get pod
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
dev-pytorch-text-classifier-r6d4p-4nczg 1/1 Running 0 4s # Submitter Kubernetes Job
|
||||
torch-text-classifier-r6d4p-raycluster-br45j-head-8bbwt 1/1 Running 0 34s # Ray head Pod
|
||||
```
|
||||
|
||||
Delete the RayJob after verifying that the job has completed successfully.
|
||||
```bash
|
||||
$ kubectl get rayjobs.ray.io dev-pytorch-text-classifier-r6d4p -o jsonpath='{.status.jobStatus}'
|
||||
SUCCEEDED
|
||||
$ kubectl get rayjobs.ray.io dev-pytorch-text-classifier-r6d4p -o jsonpath='{.status.jobDeploymentStatus}'
|
||||
Complete
|
||||
$ kubectl delete rayjob dev-pytorch-text-classifier-r6d4p
|
||||
rayjob.ray.io "dev-pytorch-text-classifier-r6d4p" deleted
|
||||
```
|
||||
|
||||
## Step 5: Queuing multiple RayJob resources
|
||||
|
||||
Create 3 RayJob custom resources to see how Kueue interacts with KubeRay to implement job queueing.
|
||||
|
||||
```bash
|
||||
$ kubectl create -f ray-job.pytorch-distributed-training.yaml
|
||||
rayjob.ray.io/dev-pytorch-text-classifier-8vg2c created
|
||||
$ kubectl create -f ray-job.pytorch-distributed-training.yaml
|
||||
rayjob.ray.io/dev-pytorch-text-classifier-n5k89 created
|
||||
$ kubectl create -f ray-job.pytorch-distributed-training.yaml
|
||||
rayjob.ray.io/dev-pytorch-text-classifier-ftcs9 created
|
||||
```
|
||||
|
||||
Because each RayJob requests 1 GPU and the ClusterQueue has quotas for only 1 GPU, Kueue automatically suspends new RayJob resources until GPU quotas become available.
|
||||
|
||||
You can also inspect the `ClusterQueue` to see available and used quotas:
|
||||
```bash
|
||||
$ kubectl get clusterqueue
|
||||
NAME COHORT PENDING WORKLOADS
|
||||
cluster-queue 2
|
||||
$ kubectl get clusterqueue cluster-queue -o yaml
|
||||
apiVersion: kueue.x-k8s.io/v1beta1
|
||||
kind: ClusterQueue
|
||||
...
|
||||
...
|
||||
...
|
||||
status:
|
||||
admittedWorkloads: 1 # Workloads admitted by queue.
|
||||
flavorsReservation:
|
||||
- name: default-flavor
|
||||
resources:
|
||||
- borrowed: "0"
|
||||
name: cpu
|
||||
total: "8"
|
||||
- borrowed: "0"
|
||||
name: memory
|
||||
total: 19531250Ki
|
||||
- borrowed: "0"
|
||||
name: nvidia.com/gpu
|
||||
total: "2"
|
||||
flavorsUsage:
|
||||
- name: default-flavor
|
||||
resources:
|
||||
- borrowed: "0"
|
||||
name: cpu
|
||||
total: "8"
|
||||
- borrowed: "0"
|
||||
name: memory
|
||||
total: 19531250Ki
|
||||
- borrowed: "0"
|
||||
name: nvidia.com/gpu
|
||||
total: "2"
|
||||
pendingWorkloads: 2 # Queued workloads waiting for quotas.
|
||||
reservingWorkloads: 1 # Running workloads that are using quotas.
|
||||
```
|
||||
|
||||
## Step 6: Deploy a RayJob with higher priority
|
||||
|
||||
At this point there are multiple RayJob custom resources queued up but only enough quota to run a single RayJob. Now you can create a new RayJob with higher priority to preempt the already queued RayJob resources. Modify the RayJob with:
|
||||
|
||||
```yaml
|
||||
metadata:
|
||||
generateName: prod-pytorch-text-classifier-
|
||||
labels:
|
||||
kueue.x-k8s.io/queue-name: user-queue
|
||||
kueue.x-k8s.io/priority-class: prod-priority
|
||||
```
|
||||
|
||||
* `kueue.x-k8s.io/queue-name: user-queue`: As the previous step mentioned, users submit jobs to a LocalQueue instead of directly to a ClusterQueue.
|
||||
* `kueue.x-k8s.io/priority-class: dev-priority`: Assign the RayJob with the `prod-priority` WorkloadPriorityClass.
|
||||
* A modified name to indicate that this job is for production.
|
||||
|
||||
Create the new RayJob:
|
||||
```sh
|
||||
$ kubectl create -f ray-job.pytorch-distributed-training.yaml
|
||||
rayjob.ray.io/prod-pytorch-text-classifier-gkp9b created
|
||||
```
|
||||
|
||||
Note that higher priority jobs preempt lower priority jobs when there aren't enough quotas for both:
|
||||
```bash
|
||||
$ kubectl get pods
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
prod-pytorch-text-classifier-gkp9b-r9k5r 1/1 Running 0 5s
|
||||
torch-text-classifier-gkp9b-raycluster-s2f65-head-hfvht 1/1 Running 0 35s
|
||||
```
|
||||
@@ -0,0 +1,198 @@
|
||||
(kuberay-rayservice-deepseek-example)=
|
||||
|
||||
# Serve Deepseek R1 using Ray Serve LLM
|
||||
|
||||
This guide provides a step-by-step guide for deploying a Large Language Model (LLM) using Ray Serve LLM on Kubernetes. Leveraging KubeRay, Ray Serve, and vLLM, this guide deploys the `deepseek-ai/DeepSeek-R1` model from Hugging Face, enabling scalable, efficient, and OpenAI-compatible LLM serving within a Kubernetes environment. See [Serving LLMs](serving-llms) for information on Ray Serve LLM.
|
||||
|
||||
## Prerequisites
|
||||
A DeepSeek model requires 2 nodes, each equipped with 8 H100 80 GB GPUs. It should be deployable on Kubernetes clusters that meet this requirement. This guide provides instructions for setting up a GKE cluster using [A3 High](https://cloud.google.com/compute/docs/gpus#a3-high) or [A3 Mega](https://cloud.google.com/compute/docs/gpus#a3-mega) machine types.
|
||||
|
||||
Before creating the cluster, ensure that your project has sufficient [quota](https://console.cloud.google.com/iam-admin/quotas) for the required accelerators.
|
||||
|
||||
## Step 1: Create a Kubernetes cluster on GKE
|
||||
Run this command and all following commands on your local machine or on the [Google Cloud Shell](https://cloud.google.com/shell). If running from your local machine, you need to install the [Google Cloud SDK](https://cloud.google.com/sdk/docs/install). The following command creates a Kubernetes cluster named `kuberay-gpu-cluster` with 1 default CPU node in the `us-east5-a` zone. This example uses the `e2-standard-16` machine type, which has 16 vCPUs and 64 GB memory.
|
||||
|
||||
```sh
|
||||
gcloud container clusters create kuberay-gpu-cluster \
|
||||
--location=us-east5-a \
|
||||
--machine-type=e2-standard-16 \
|
||||
--num-nodes=1 \
|
||||
--enable-image-streaming
|
||||
```
|
||||
|
||||
Run the following command to create an on-demand GPU node pool for Ray GPU workers.
|
||||
|
||||
```sh
|
||||
gcloud beta container node-pools create gpu-node-pool \
|
||||
--cluster kuberay-gpu-cluster \
|
||||
--machine-type a3-highgpu-8g \
|
||||
--num-nodes 2 \
|
||||
--accelerator "type=nvidia-h100-80gb,count=8" \
|
||||
--zone us-east5-a \
|
||||
--node-locations us-east5-a \
|
||||
--host-maintenance-interval=PERIODIC
|
||||
```
|
||||
|
||||
The `--accelerator` flag specifies the type and number of GPUs for each node in the node pool. This example uses the [A3 High](https://cloud.google.com/compute/docs/gpus#a3-high) GPU. The machine type `a3-highgpu-8g` has 8 GPU, 640 GB GPU Memory, 208 vCPUs, and 1872 GB RAM.
|
||||
|
||||
|
||||
```{admonition} Note
|
||||
:class: note
|
||||
|
||||
To create a node pool that uses reservations, you can specify the following parameters:
|
||||
* `--reservation-affinity=specific`
|
||||
* `--reservation=RESERVATION_NAME`
|
||||
* `--placement-policy=PLACEMENT_POLICY_NAME` (Optional)
|
||||
```
|
||||
|
||||
Run the following `gcloud` command to configure `kubectl` to communicate with your cluster:
|
||||
|
||||
```sh
|
||||
gcloud container clusters get-credentials kuberay-gpu-cluster --zone us-east5-a
|
||||
```
|
||||
|
||||
## Step 2: Install the KubeRay operator
|
||||
|
||||
Install the most recent stable KubeRay operator from the Helm repository by following [Deploy a KubeRay operator](kuberay-operator-deploy). The Kubernetes `NoSchedule` taint in the example config prevents the KubeRay operator Pod from running on a GPU node.
|
||||
|
||||
## Step 3: Deploy a RayService
|
||||
|
||||
Deploy DeepSeek-R1 as a RayService custom resource by running the following command:
|
||||
|
||||
```sh
|
||||
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-service.deepseek.yaml
|
||||
```
|
||||
|
||||
This step sets up a custom Ray Serve application to serve the `deepseek-ai/DeepSeek-R1` model on two worker nodes. You can inspect and modify the `serveConfigV2` section in the YAML file to learn more about the Serve application:
|
||||
```yaml
|
||||
serveConfigV2: |
|
||||
applications:
|
||||
- args:
|
||||
llm_configs:
|
||||
- model_loading_config:
|
||||
model_id: "deepseek"
|
||||
model_source: "deepseek-ai/DeepSeek-R1"
|
||||
accelerator_type: "H100"
|
||||
deployment_config:
|
||||
autoscaling_config:
|
||||
min_replicas: 1
|
||||
max_replicas: 1
|
||||
runtime_env:
|
||||
env_vars:
|
||||
VLLM_USE_V1: "1"
|
||||
engine_kwargs:
|
||||
tensor_parallel_size: 8
|
||||
pipeline_parallel_size: 2
|
||||
gpu_memory_utilization: 0.92
|
||||
dtype: "auto"
|
||||
max_num_seqs: 40
|
||||
max_model_len: 16384
|
||||
enable_chunked_prefill: true
|
||||
enable_prefix_caching: true
|
||||
import_path: ray.serve.llm:build_openai_app
|
||||
name: llm_app
|
||||
route_prefix: "/"
|
||||
```
|
||||
|
||||
In particular, this configuration loads the model from `deepseek-ai/DeepSeek-R1` and sets its `model_id` to `deepseek`. The `LLMDeployment` initializes the underlying LLM engine using the `engine_kwargs` field, which includes key performance tuning parameters:
|
||||
|
||||
- `tensor_parallel_size: 8`
|
||||
|
||||
This setting enables tensor parallelism, splitting individual large layers of the model across 8 GPUs. Adjust this variable according to the number of GPUs used by cluster nodes.
|
||||
|
||||
- `pipeline_parallel_size: 2`
|
||||
|
||||
This setting enables pipeline parallelism, dividing the model's entire set of layers into 2 sequential stages. Adjust this variable according to cluster worker node numbers.
|
||||
|
||||
|
||||
The `deployment_config` section sets the desired number of engine replicas. See [Serving LLMs](serving-llms) and the [Ray Serve config documentation](serve-in-production-config-file) for more information.
|
||||
|
||||
Wait for the RayService resource to become healthy. You can confirm its status by running the following command:
|
||||
```sh
|
||||
kubectl get rayservice deepseek-r1 -o yaml
|
||||
```
|
||||
|
||||
After a few minutes, the result should be similar to the following:
|
||||
```
|
||||
status:
|
||||
activeServiceStatus:
|
||||
applicationStatuses:
|
||||
llm_app:
|
||||
serveDeploymentStatuses:
|
||||
LLMDeployment:deepseek:
|
||||
status: HEALTHY
|
||||
LLMRouter:
|
||||
status: HEALTHY
|
||||
status: RUNNING
|
||||
```
|
||||
|
||||
```{admonition} Note
|
||||
:class: note
|
||||
|
||||
The model download and deployment will typically take 20-30 minutes. While this is in progress, use the Ray Dashboard (Step 4) Cluster tab to monitor the download progress as disk fills up.
|
||||
```
|
||||
|
||||
## Step 4: View the Ray dashboard
|
||||
```sh
|
||||
# Forward the service port
|
||||
kubectl port-forward svc/deepseek-r1-head-svc 8265:8265
|
||||
```
|
||||
|
||||
Once forwarded, navigate to the Serve tab on the dashboard to review application status, deployments, routers, logs, and other relevant features. 
|
||||
|
||||
## Step 5: Send a request
|
||||
|
||||
To send requests to the Ray Serve deployment, port-forward port 8000 from the Serve app service:
|
||||
```sh
|
||||
kubectl port-forward svc/deepseek-r1-serve-svc 8000
|
||||
```
|
||||
|
||||
Note that this Kubernetes service comes up only after Ray Serve apps are running and ready.
|
||||
|
||||
Test the service with the following command:
|
||||
```sh
|
||||
$ curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
|
||||
"model": "deepseek",
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "I have four boxes. I put the red box on the bottom and put the blue box on top. Then I put the yellow box on top the blue. Then I take the blue box out and put it on top. And finally I put the green box on the top. Give me the final order of the boxes from bottom to top. Show your reasoning but be brief"}
|
||||
],
|
||||
"temperature": 0.7
|
||||
}'
|
||||
```
|
||||
|
||||
The output should be in the following format:
|
||||
|
||||
```
|
||||
{
|
||||
"id": "deepseek-653881a7-18f3-493b-a43f-adc8501f01f8",
|
||||
"object": "chat.completion",
|
||||
"created": 1753345252,
|
||||
"model": "deepseek",
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"reasoning_content": null,
|
||||
"content": "Okay, let's break this down step by step. The user has four boxes: red, blue, yellow, and green. The starting point is putting the red box on the bottom. Then blue is placed on top of red. Next, yellow goes on top of blue. At this point, the order is red (bottom), blue, yellow. \n\nThen the instruction says to take the blue box out and put it on top. Wait, when they take the blue box out from where? The current stack is red, blue, yellow. If we remove blue from between red and yellow, that leaves red and yellow. Then placing blue on top would make the stack red, yellow, blue. But the problem is, when you remove a box from the middle, the boxes above it should fall down, right? So after removing blue, yellow would be on top of red. Then putting blue on top of that stack would make it red, yellow, blue.\n\nThen the final step is putting the green box on top. So the final order would be red (bottom), yellow, blue, green. Let me verify again to make sure I didn't miss anything. Start with red at bottom. Blue on top of red: red, blue. Yellow on top of blue: red, blue, yellow. Remove blue from the middle, so yellow moves down to be on red, then put blue on top: red, yellow, blue. Finally, add green on top: red, yellow, blue, green. Yes, that seems right.\n</think>\n\nThe final order from bottom to top is: red, yellow, blue, green.\n\n1. Start with red at the bottom. \n2. Add blue on top: red → blue. \n3. Add yellow on top: red → blue → yellow. \n4. **Remove blue** from between red and yellow; yellow drops to second position. Now: red → yellow. \n5. Place blue back on top: red → yellow → blue. \n6. Add green on top: red → yellow → blue → green.",
|
||||
"tool_calls": []
|
||||
},
|
||||
"logprobs": null,
|
||||
"finish_reason": "stop",
|
||||
"stop_reason": null
|
||||
}
|
||||
],
|
||||
"usage": {
|
||||
"prompt_tokens": 81,
|
||||
"total_tokens": 505,
|
||||
"completion_tokens": 424,
|
||||
"prompt_tokens_details": null
|
||||
},
|
||||
"prompt_logprobs": null
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,164 @@
|
||||
(kuberay-rayservice-llm-example)=
|
||||
|
||||
# Serve a Large Language Model using Ray Serve LLM on Kubernetes
|
||||
|
||||
This guide provides a step-by-step guide for deploying a Large Language Model (LLM) using Ray Serve LLM on Kubernetes. Leveraging KubeRay, Ray Serve, and vLLM, this guide deploys the `Qwen/Qwen2.5-7B-Instruct` model from Hugging Face, enabling scalable, efficient, and OpenAI-compatible LLM serving within a Kubernetes environment. See [Serving LLMs](serving-llms) for information on Ray Serve LLM.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
This example downloads model weights from the [`Qwen/Qwen2.5-7B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) Hugging Face repository. To completely finish this guide, you must fulfill the following requirements:
|
||||
* A [Hugging Face account](https://huggingface.co/) and a Hugging Face [access token](https://huggingface.co/settings/tokens) with read access to gated repositories.
|
||||
* In your RayService custom resource, set the `HUGGING_FACE_HUB_TOKEN` environment variable to the Hugging Face token to enable model downloads.
|
||||
* A Kubernetes cluster with GPUs.
|
||||
|
||||
## Step 1: Create a Kubernetes cluster with GPUs
|
||||
|
||||
Refer to the Kubernetes cluster setup [instructions](../user-guides/k8s-cluster-setup.md) for guides on creating a Kubernetes cluster.
|
||||
|
||||
## Step 2: Install the KubeRay operator
|
||||
|
||||
Install the most recent stable KubeRay operator from the Helm repository by following [Deploy a KubeRay operator](../getting-started/kuberay-operator-installation.md). The Kubernetes `NoSchedule` taint in the example config prevents the KubeRay operator pod from running on a GPU node.
|
||||
|
||||
## Step 3: Create a Kubernetes Secret containing your Hugging Face access token
|
||||
|
||||
For additional security, instead of passing the HF access token directly as an environment variable, create a Kubernetes secret containing your Hugging Face access token. Download the Ray Serve LLM service config .yaml file using the following command:
|
||||
|
||||
```sh
|
||||
curl -o ray-service.llm-serve.yaml https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-service.llm-serve.yaml
|
||||
```
|
||||
|
||||
After downloading, update the value for `hf_token` to your private access token in the `Secret`.
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: hf-token
|
||||
type: Opaque
|
||||
stringData:
|
||||
hf_token: <your-hf-access-token-value>
|
||||
```
|
||||
|
||||
## Step 4: Deploy a RayService
|
||||
|
||||
After adding the Hugging Face access token, create a RayService custom resource using the config file:
|
||||
|
||||
```sh
|
||||
kubectl apply -f ray-service.llm-serve.yaml
|
||||
```
|
||||
|
||||
This step sets up a custom Ray Serve app to serve the `Qwen/Qwen2.5-7B-Instruct` model, creating an OpenAI-compatible server. You can inspect and modify the `serveConfigV2` section in the YAML file to learn more about the Serve app:
|
||||
```yaml
|
||||
serveConfigV2: |
|
||||
applications:
|
||||
- name: llms
|
||||
import_path: ray.serve.llm:build_openai_app
|
||||
route_prefix: "/"
|
||||
args:
|
||||
llm_configs:
|
||||
- model_loading_config:
|
||||
model_id: qwen2.5-7b-instruct
|
||||
model_source: Qwen/Qwen2.5-7B-Instruct
|
||||
engine_kwargs:
|
||||
dtype: bfloat16
|
||||
max_model_len: 1024
|
||||
device: auto
|
||||
gpu_memory_utilization: 0.75
|
||||
deployment_config:
|
||||
autoscaling_config:
|
||||
min_replicas: 1
|
||||
max_replicas: 4
|
||||
target_ongoing_requests: 64
|
||||
max_ongoing_requests: 128
|
||||
```
|
||||
|
||||
In particular, this configuration loads the model from `Qwen/Qwen2.5-7B-Instruct` and sets its `model_id` to `qwen2.5-7b-instruct`. The `LLMDeployment` initializes the underlying LLM engine using the `engine_kwargs` field. The `deployment_config` section sets the desired number of engine replicas. By default, each replica requires one GPU. See [Serving LLMs](serving-llms) and the [Ray Serve config documentation](serve-in-production-config-file) for more information.
|
||||
|
||||
Wait for the RayService resource to become healthy. You can confirm its status by running the following command:
|
||||
```sh
|
||||
kubectl get rayservice ray-serve-llm -o yaml
|
||||
```
|
||||
|
||||
After a few minutes, the result should be similar to the following:
|
||||
```
|
||||
status:
|
||||
activeServiceStatus:
|
||||
applicationStatuses:
|
||||
llms:
|
||||
serveDeploymentStatuses:
|
||||
LLMDeployment:qwen2_5-7b-instruct:
|
||||
status: HEALTHY
|
||||
LLMRouter:
|
||||
status: HEALTHY
|
||||
status: RUNNING
|
||||
```
|
||||
|
||||
## Step 5: Send a request
|
||||
|
||||
To send requests to the Ray Serve deployment, port-forward port 8000 from the Serve app service:
|
||||
```sh
|
||||
kubectl port-forward ray-serve-llm-serve-svc 8000
|
||||
```
|
||||
|
||||
|
||||
Note that this Kubernetes service comes up only after Ray Serve apps are running and ready.
|
||||
|
||||
Test the service with the following command:
|
||||
```sh
|
||||
curl --location 'http://localhost:8000/v1/chat/completions' --header 'Content-Type: application/json'
|
||||
--data '{
|
||||
"model": "qwen2.5-7b-instruct",
|
||||
"messages": [
|
||||
{
|
||||
"role": "system",
|
||||
"content": "You are a helpful assistant."
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Provide steps to serve an LLM using Ray Serve."
|
||||
}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
The output should be in the following format:
|
||||
|
||||
```
|
||||
{
|
||||
"id": "qwen2.5-7b-instruct-550d3fd491890a7e7bca74e544d3479e",
|
||||
"object": "chat.completion",
|
||||
"created": 1746595284,
|
||||
"model": "qwen2.5-7b-instruct",
|
||||
"choices": [
|
||||
{
|
||||
"index": 0,
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"reasoning_content": null,
|
||||
"content": "Sure! Ray Serve is a library built on top of Ray...",
|
||||
"tool_calls": []
|
||||
},
|
||||
"logprobs": null,
|
||||
"finish_reason": "stop",
|
||||
"stop_reason": null
|
||||
}
|
||||
],
|
||||
"usage": {
|
||||
"prompt_tokens": 30,
|
||||
"total_tokens": 818,
|
||||
"completion_tokens": 788,
|
||||
"prompt_tokens_details": null
|
||||
},
|
||||
"prompt_logprobs": null
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
## Step 6: View the Ray dashboard
|
||||
|
||||
|
||||
```sh
|
||||
kubectl port-forward svc/ray-serve-llm-head-svc 8265
|
||||
```
|
||||
|
||||
Once forwarded, navigate to the Serve tab on the dashboard to review application status, deployments, routers, logs, and other relevant features. 
|
||||
@@ -0,0 +1,75 @@
|
||||
(kuberay-stable-diffusion-rayservice-example)=
|
||||
|
||||
# Serve a StableDiffusion text-to-image model on Kubernetes
|
||||
|
||||
> **Note:** The Python files for the Ray Serve application and its client are in the [ray-project/serve_config_examples](https://github.com/ray-project/serve_config_examples) repository
|
||||
and [the Ray documentation](https://docs.ray.io/en/latest/serve/tutorials/stable-diffusion.html).
|
||||
|
||||
## Step 1: Create a Kubernetes cluster with GPUs
|
||||
|
||||
See [aws-eks-gpu-cluster.md](kuberay-eks-gpu-cluster-setup) or [gcp-gke-gpu-cluster.md](kuberay-gke-gpu-cluster-setup) or [ack-gpu-cluster.md](kuberay-ack-gpu-cluster-setup) to create a Kubernetes cluster with 1 CPU node and 1 GPU node.
|
||||
|
||||
## Step 2: Install KubeRay operator
|
||||
|
||||
Follow [this document](kuberay-operator-deploy) to install the latest stable KubeRay operator using the Helm repository. Note that the YAML file in this example uses `serveConfigV2`. This feature requires KubeRay v0.6.0 or later.
|
||||
|
||||
## Step 3: Install a RayService
|
||||
|
||||
```sh
|
||||
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-service.stable-diffusion.yaml
|
||||
```
|
||||
|
||||
This RayService configuration contains some important settings:
|
||||
|
||||
* In the RayService, the head Pod doesn't have any `tolerations`. Meanwhile, the worker Pods use the following `tolerations` so the scheduler won't assign the head Pod to the GPU node.
|
||||
```yaml
|
||||
# Please add the following taints to the GPU node.
|
||||
tolerations:
|
||||
- key: "ray.io/node-type"
|
||||
operator: "Equal"
|
||||
value: "worker"
|
||||
effect: "NoSchedule"
|
||||
```
|
||||
* It includes `diffusers` in `runtime_env` since this package isn't included by default in the `ray-ml` image.
|
||||
|
||||
## Step 4: Forward the port of Serve
|
||||
|
||||
First get the service name from this command.
|
||||
|
||||
```sh
|
||||
kubectl get services
|
||||
```
|
||||
|
||||
Then, port forward to the serve.
|
||||
|
||||
```sh
|
||||
# Wait until the RayService `Ready` condition is `True`. This means the RayService is ready to serve.
|
||||
kubectl describe rayservices.ray.io stable-diffusion
|
||||
|
||||
# [Example output]
|
||||
# Conditions:
|
||||
# Last Transition Time: 2025-02-13T07:10:34Z
|
||||
# Message: Number of serve endpoints is greater than 0
|
||||
# Observed Generation: 1
|
||||
# Reason: NonZeroServeEndpoints
|
||||
# Status: True
|
||||
# Type: Ready
|
||||
|
||||
# Forward the port of Serve
|
||||
kubectl port-forward svc/stable-diffusion-serve-svc 8000
|
||||
```
|
||||
|
||||
## Step 5: Send a request to the text-to-image model
|
||||
|
||||
```sh
|
||||
# Step 5.1: Download `stable_diffusion_req.py`
|
||||
curl -LO https://raw.githubusercontent.com/ray-project/serve_config_examples/master/stable_diffusion/stable_diffusion_req.py
|
||||
|
||||
# Step 5.2: Set your `prompt` in `stable_diffusion_req.py`.
|
||||
|
||||
# Step 5.3: Send a request to the Stable Diffusion model.
|
||||
python stable_diffusion_req.py
|
||||
# Check output.png
|
||||
```
|
||||
|
||||
* You can refer to the document ["Serving a Stable Diffusion Model"](https://docs.ray.io/en/latest/serve/tutorials/stable-diffusion.html) for an example output image.
|
||||
@@ -0,0 +1,69 @@
|
||||
(kuberay-text-summarizer-rayservice-example)=
|
||||
|
||||
# Serve a text summarizer on Kubernetes
|
||||
|
||||
> **Note:** The Python files for the Ray Serve application and its client are in the [ray-project/serve_config_examples](https://github.com/ray-project/serve_config_examples) repository.
|
||||
|
||||
## Step 1: Create a Kubernetes cluster with GPUs
|
||||
|
||||
See [aws-eks-gpu-cluster.md](kuberay-eks-gpu-cluster-setup) or [gcp-gke-gpu-cluster.md](kuberay-gke-gpu-cluster-setup) or [ack-gpu-cluster.md](kuberay-ack-gpu-cluster-setup) to create a Kubernetes cluster with 1 CPU node and 1 GPU node.
|
||||
|
||||
## Step 2: Install KubeRay operator
|
||||
|
||||
Follow [this document](kuberay-operator-deploy) to install the latest stable KubeRay operator using the Helm repository.
|
||||
|
||||
## Step 3: Install a RayService
|
||||
|
||||
```sh
|
||||
# Create a RayService
|
||||
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-service.text-summarizer.yaml
|
||||
```
|
||||
|
||||
* In the RayService, the head Pod doesn't have any `tolerations`. Meanwhile, the worker Pods use the following `tolerations` so the scheduler won't assign the head Pod to the GPU node.
|
||||
```yaml
|
||||
# Please add the following taints to the GPU node.
|
||||
tolerations:
|
||||
- key: "ray.io/node-type"
|
||||
operator: "Equal"
|
||||
value: "worker"
|
||||
effect: "NoSchedule"
|
||||
```
|
||||
|
||||
## Step 4: Forward the port of Serve
|
||||
|
||||
```sh
|
||||
# Step 4.1: Wait until the RayService is ready to serve requests.
|
||||
kubectl describe rayservices text-summarizer
|
||||
|
||||
# Step 4.2: Get the service name.
|
||||
kubectl get services
|
||||
|
||||
# [Example output]
|
||||
# text-summarizer-head-svc ClusterIP None <none> 10001/TCP,8265/TCP,6379/TCP,8080/TCP,8000/TCP 31s
|
||||
# text-summarizer-raycluster-tb9zf-head-svc ClusterIP None <none> 10001/TCP,8265/TCP,6379/TCP,8080/TCP,8000/TCP 108s
|
||||
# text-summarizer-serve-svc ClusterIP 34.118.226.139 <none> 8000/TCP 31s
|
||||
|
||||
# Step 4.3: Forward the port of Serve.
|
||||
kubectl port-forward svc/text-summarizer-serve-svc 8000
|
||||
```
|
||||
|
||||
## Step 5: Send a request to the text summarizer model
|
||||
|
||||
```sh
|
||||
# Step 5.1: Download `text_summarizer_req.py`
|
||||
curl -LO https://raw.githubusercontent.com/ray-project/serve_config_examples/master/text_summarizer/text_summarizer_req.py
|
||||
|
||||
# Step 5.2: Send a request to the Summarizer model.
|
||||
python text_summarizer_req.py
|
||||
# Check printed to console
|
||||
```
|
||||
|
||||
## Step 6: Delete your service
|
||||
|
||||
```sh
|
||||
kubectl delete -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-service.text-summarizer.yaml
|
||||
```
|
||||
|
||||
## Step 7: Uninstall your KubeRay operator
|
||||
|
||||
Follow [this document](https://github.com/ray-project/kuberay/tree/master/helm-chart/kuberay-operator) to uninstall the latest stable KubeRay operator using the Helm repository.
|
||||
@@ -0,0 +1,77 @@
|
||||
(kuberay-tpu-stable-diffusion-example)=
|
||||
|
||||
# Serve a Stable Diffusion model on GKE with TPUs
|
||||
|
||||
> **Note:** The Python files for the Ray Serve app and its client are in the [ray-project/serve_config_examples](https://github.com/ray-project/serve_config_examples). This guide adapts the [tensorflow/tpu](https://github.com/tensorflow/tpu/tree/master/tools/ray_tpu/src/serve) example.
|
||||
|
||||
## Step 1: Create a Kubernetes cluster with TPUs
|
||||
|
||||
Follow [Creating a GKE Cluster with TPUs for KubeRay](kuberay-gke-tpu-cluster-setup) to create a GKE cluster with 1 CPU node and 1 TPU node.
|
||||
|
||||
## Step 2: Install the KubeRay operator
|
||||
|
||||
Skip this step if the [Ray Operator Addon](https://cloud.google.com/kubernetes-engine/docs/add-on/ray-on-gke/concepts/overview) is enabled in your GKE cluster. Follow [Deploy a KubeRay operator](kuberay-operator-deploy) instructions to install the latest stable KubeRay operator from the Helm repository. Multi-host TPU support is available in KubeRay v1.1.0+. Note that the YAML file in this example uses `serveConfigV2`, which KubeRay supports starting from v0.6.0.
|
||||
|
||||
## Step 3: Install the RayService CR
|
||||
|
||||
```sh
|
||||
# Creates a RayCluster with a single-host v4 TPU worker group of 2x2x1 topology.
|
||||
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-service.tpu-single-host.yaml
|
||||
```
|
||||
|
||||
KubeRay operator v1.1.0 adds a new `NumOfHosts` field to the RayCluster CR, supporting multi-host worker groups. This field specifies the number of workers to create per replica, with each replica representing a multi-host Pod slice. The value for `NumOfHosts` should match the number of TPU VM hosts that the given `cloud.google.com/gke-tpu-topology` node selector expects. For this example, the Stable Diffusion model is small enough to run on a single TPU host, so `numOfHosts` is set to 1 in the RayService manifest.
|
||||
|
||||
## Step 4: View the Serve deployment in the Ray Dashboard
|
||||
|
||||
Verify that you deployed the RayService CR and it's running:
|
||||
|
||||
```sh
|
||||
kubectl get rayservice
|
||||
|
||||
# NAME SERVICE STATUS NUM SERVE ENDPOINTS
|
||||
# stable-diffusion-tpu-serve-svc Running 2
|
||||
```
|
||||
|
||||
Port-forward the Ray Dashboard from the Ray head service. To view the dashboard, open http://localhost:8265/ on your local machine.
|
||||
|
||||
```sh
|
||||
kubectl port-forward svc/stable-diffusion-tpu-head-svc 8265:8265 &
|
||||
```
|
||||
|
||||
Monitor the status of the RayService CR in the Ray Dashboard from the 'Serve' tab. The installed RayService CR should create a running app with the name 'stable_diffusion'. The app should have two deployments, the API ingress, which receives input prompts, and the Stable Diffusion model server.
|
||||
|
||||
|
||||
|
||||
|
||||
## Step 5: Send text-to-image prompts to the model server
|
||||
|
||||
Port forward the Ray Serve service:
|
||||
```sh
|
||||
kubectl port-forward svc/stable-diffusion-tpu-serve-svc 8000
|
||||
```
|
||||
|
||||
In a separate terminal, download the Python prompt script:
|
||||
|
||||
```sh
|
||||
curl -LO https://raw.githubusercontent.com/ray-project/serve_config_examples/master/stable_diffusion/stable_diffusion_tpu_req.py
|
||||
```
|
||||
|
||||
Install the required dependencies to run the Python script locally:
|
||||
|
||||
```sh
|
||||
# Create a Python virtual environment.
|
||||
python3 -m venv myenv
|
||||
source myenv/bin/activate
|
||||
|
||||
pip install numpy pillow requests tqdm
|
||||
```
|
||||
|
||||
|
||||
Submit a text-to-image prompt to the Stable Diffusion model server:
|
||||
```sh
|
||||
python stable_diffusion_tpu_req.py --save_pictures
|
||||
```
|
||||
|
||||
* The Python prompt script saves the results of the Stable Diffusion inference to a file named diffusion_results.png.
|
||||
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
(kuberay-verl)=
|
||||
# Reinforcement Learning with Human Feedback (RLHF) for LLMs with verl on KubeRay
|
||||
|
||||
[verl](https://github.com/volcengine/verl) is an open-source framework that provides a flexible, efficient, and production-ready RL training library for large language models (LLMs). This guide demonstrates Proximal Policy Optimization (PPO) training on the GSM8K dataset with verl for `Qwen2.5-0.5B-Instruct` on KubeRay.
|
||||
|
||||
* To make it easier to follow, this guide launches a single-node RayCluster with 4 GPUs. You can easily use KubeRay to launch a multi-node RayCluster to train larger models.
|
||||
* You can also use the [RayJob CRD](kuberay-rayjob-quickstart) for production use cases.
|
||||
|
||||
# Step 1: Create a Kubernetes cluster with GPUs
|
||||
|
||||
Follow the instructions in [Managed Kubernetes services](kuberay-k8s-setup) to create a Kubernetes cluster with GPUs.
|
||||
|
||||
This guide uses a Kubernetes cluster with 4 L4 GPUs.
|
||||
|
||||
For GKE, you can follow the instructions in [this tutorial](kuberay-gke-gpu-cluster-setup) and use the following command to create a GPU node pool with 4 L4 GPUs per Kubernetes node:
|
||||
|
||||
```bash
|
||||
gcloud container node-pools create gpu-node-pool \
|
||||
--accelerator type=nvidia-l4-vws,count=4 \
|
||||
--zone us-west1-b \
|
||||
--cluster kuberay-gpu-cluster \
|
||||
--num-nodes 1 \
|
||||
--min-nodes 0 \
|
||||
--max-nodes 1 \
|
||||
--enable-autoscaling \
|
||||
--machine-type g2-standard-48
|
||||
```
|
||||
|
||||
# Step 2: Install KubeRay operator
|
||||
|
||||
Follow the instructions in [KubeRay operator](kuberay-operator-deploy) to install the KubeRay operator.
|
||||
|
||||
# Step 3: Create a RayCluster
|
||||
|
||||
```sh
|
||||
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-cluster.verl.yaml
|
||||
```
|
||||
|
||||
# Step 4: Install verl in the head Pod
|
||||
|
||||
Log in to the head Pod and install verl. The verl community doesn't provide images with verl installed ([verl#2222](https://github.com/volcengine/verl/issues/2222)) at the moment.
|
||||
|
||||
```sh
|
||||
# Log in to the head Pod.
|
||||
export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)
|
||||
kubectl exec -it $HEAD_POD -- bash
|
||||
|
||||
# Follow the instructions in https://verl.readthedocs.io/en/latest/start/install.html#install-from-docker-image to install verl.
|
||||
git clone https://github.com/volcengine/verl && cd verl
|
||||
pip3 install -e .[vllm]
|
||||
```
|
||||
|
||||
# Step 5: Prepare the dataset and download `Qwen2.5-0.5B-Instruct` model
|
||||
|
||||
Run the following commands in the head Pod's verl root directory to prepare the dataset and download the `Qwen2.5-0.5B-Instruct` model.
|
||||
|
||||
```sh
|
||||
# Prepare the dataset.
|
||||
python3 examples/data_preprocess/gsm8k.py --local_dir ~/data/gsm8k
|
||||
|
||||
# Download the `Qwen2.5-0.5B-Instruct` model.
|
||||
python3 -c "import transformers; transformers.pipeline('text-generation', model='Qwen/Qwen2.5-0.5B-Instruct')"
|
||||
```
|
||||
|
||||
# Step 6: Run a PPO training job
|
||||
|
||||
Run the following command to start a PPO training job. This differs slightly from the instructions in [verl's documentation](https://verl.readthedocs.io/en/latest/start/quickstart.html#step-3-perform-ppo-training-with-the-instruct-model). The main differences are the following:
|
||||
* Set `n_gpus_per_node` to `4` because the head Pod has 4 GPUs.
|
||||
* Set `save_freq` to `-1` to avoid disk pressure caused by checkpointing.
|
||||
|
||||
```sh
|
||||
PYTHONUNBUFFERED=1 python3 -m verl.trainer.main_ppo \
|
||||
data.train_files=$HOME/data/gsm8k/train.parquet \
|
||||
data.val_files=$HOME/data/gsm8k/test.parquet \
|
||||
data.train_batch_size=256 \
|
||||
data.max_prompt_length=512 \
|
||||
data.max_response_length=256 \
|
||||
actor_rollout_ref.model.path=Qwen/Qwen2.5-0.5B-Instruct \
|
||||
actor_rollout_ref.actor.optim.lr=1e-6 \
|
||||
actor_rollout_ref.actor.ppo_mini_batch_size=64 \
|
||||
actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=4 \
|
||||
actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=8 \
|
||||
actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
|
||||
actor_rollout_ref.rollout.gpu_memory_utilization=0.4 \
|
||||
actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=4 \
|
||||
critic.optim.lr=1e-5 \
|
||||
critic.model.path=Qwen/Qwen2.5-0.5B-Instruct \
|
||||
critic.ppo_micro_batch_size_per_gpu=4 \
|
||||
algorithm.kl_ctrl.kl_coef=0.001 \
|
||||
trainer.logger=['console'] \
|
||||
trainer.val_before_train=False \
|
||||
trainer.default_hdfs_dir=null \
|
||||
trainer.n_gpus_per_node=4 \
|
||||
trainer.nnodes=1 \
|
||||
trainer.save_freq=-1 \
|
||||
trainer.test_freq=10 \
|
||||
trainer.total_epochs=15 2>&1 | tee verl_demo.log
|
||||
```
|
||||
|
||||
This job takes 5 hours to complete. While it's running, you can check the Ray dashboard to see more details about the PPO job and the Ray cluster. Additionally, you can follow the next step to check the PPO job logs to see how the model improves.
|
||||
|
||||
```sh
|
||||
# Port forward the Ray dashboard to your local machine's port 8265.
|
||||
kubectl port-forward $HEAD_POD 8265:8265
|
||||
```
|
||||
|
||||
Open `127.0.0.1:8265` in your browser to view the Ray dashboard and check whether all GPUs are in use.
|
||||
|
||||
|
||||
|
||||
# Step 7: Check the PPO job logs
|
||||
|
||||
Check `verl_demo.log` in the head Pod to see the PPO job's logs. For every 10 steps, verl validates the model with a simple math problem.
|
||||
|
||||
* Math problem:
|
||||
```
|
||||
Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market? Let's think step by step and output the final answer after
|
||||
```
|
||||
* Answer: `(16 - 3 - 4) * 2 = 18`
|
||||
|
||||
You should be able to see the model becomes gradually better at this question after several steps.
|
||||
|
||||
In this example run, the model first got the correct answer after 130 steps, and the following is the log. Throughout the entire process, the validation ran 44 times and got the correct answer 20 times. It may vary depending on the random seed.
|
||||
|
||||
```
|
||||
(TaskRunner pid=21297) [response] First, we calculate the number of eggs Janet's ducks lay in a day. Since there are 16 eggs per day and Janet lays these eggs every day, the number of eggs laid in a day is 16.
|
||||
(TaskRunner pid=21297)
|
||||
(TaskRunner pid=21297) Next, we calculate the number of eggs Janet eats in a day. She eats 3 eggs for breakfast and bakes 4 muffins, so the total number of eggs she eats in a day is 3 + 4 = 7.
|
||||
(TaskRunner pid=21297)
|
||||
(TaskRunner pid=21297) The number of eggs she sells in a day is the total number of eggs laid minus the number of eggs she eats, which is 16 - 7 = 9 eggs.
|
||||
(TaskRunner pid=21297)
|
||||
(TaskRunner pid=21297) She sells each egg for $2, so the total amount she makes every day is 9 * 2 = 18 dollars.
|
||||
(TaskRunner pid=21297)
|
||||
(TaskRunner pid=21297) #### 18
|
||||
(TaskRunner pid=21297) #### 18 dollars
|
||||
```
|
||||
|
||||
It's not necessary to wait for all steps to complete. You can stop the job if you observe the process of the model improving.
|
||||
|
||||
# Step 8: Clean up
|
||||
|
||||
```sh
|
||||
kubectl delete -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-cluster.verl.yaml
|
||||
```
|
||||
@@ -0,0 +1,55 @@
|
||||
(kuberay-quickstart)=
|
||||
|
||||
# Getting Started with KubeRay
|
||||
|
||||
```{toctree}
|
||||
:hidden:
|
||||
|
||||
getting-started/kuberay-operator-installation
|
||||
getting-started/raycluster-quick-start
|
||||
getting-started/rayjob-quick-start
|
||||
getting-started/rayservice-quick-start
|
||||
getting-started/raycronjob-quick-start
|
||||
```
|
||||
|
||||
|
||||
## Custom Resource Definitions (CRDs)
|
||||
|
||||
[KubeRay](https://github.com/ray-project/kuberay) is a powerful, open-source Kubernetes operator that simplifies the deployment and management of Ray applications on Kubernetes. It offers 3 custom resource definitions (CRDs):
|
||||
|
||||
* **RayCluster**: KubeRay fully manages the lifecycle of RayCluster, including cluster creation/deletion, autoscaling, and ensuring fault tolerance.
|
||||
|
||||
* **RayJob**: With RayJob, KubeRay automatically creates a RayCluster and submits a job when the cluster is ready. You can also configure RayJob to automatically delete the RayCluster once the job finishes.
|
||||
|
||||
* **RayService**: RayService is made up of two parts: a RayCluster and Ray Serve deployment graphs. RayService offers zero-downtime upgrades for RayCluster and high availability.
|
||||
|
||||
* **RayCronJob**: RayCronJob is used to run RayJobs on a recurring schedule. It automatically creates new RayJob resources based on a cron expression, making it easy to run periodic workloads such as batch jobs or scheduled tasks.
|
||||
|
||||
## Which CRD should you choose?
|
||||
|
||||
Using [RayService](kuberay-rayservice-quickstart) to serve models and using [RayCluster](kuberay-raycluster-quickstart) to develop Ray applications are no-brainer recommendations from us. However, if the use case is not model serving or prototyping, how do you choose between [RayCluster](kuberay-raycluster-quickstart), [RayJob](kuberay-rayjob-quickstart), and [RayCronJob](kuberay-raycronjob-quickstart)?
|
||||
|
||||
### Q: Is downtime acceptable during a cluster upgrade (e.g. Upgrade Ray version)?
|
||||
|
||||
If not, use RayJob. RayJob can be configured to automatically delete the RayCluster once the job is completed. You can switch between Ray versions and configurations for each job submission using RayJob.
|
||||
|
||||
If yes, use RayCluster. Ray doesn't natively support rolling upgrades; thus, you'll need to manually shut down and create a new RayCluster.
|
||||
|
||||
### Q: Do you need to run workloads on a recurring schedule?
|
||||
|
||||
If yes, use RayCronJob. RayCronJob automatically creates RayJob resources on a cron schedule, allowing you to run periodic workloads such as batch processing or scheduled inference.
|
||||
|
||||
### Q: Are you deploying on public cloud providers (e.g. AWS, GCP, Azure)?
|
||||
|
||||
If yes, use RayJob. It allows automatic deletion of the RayCluster upon job completion, helping you reduce costs.
|
||||
|
||||
### Q: Do you care about the latency introduced by spinning up a RayCluster?
|
||||
|
||||
If yes, use RayCluster. Unlike RayJob and RayCronJob, which create a new RayCluster every time a job is submitted, RayCluster creates the cluster just once and can be used multiple times.
|
||||
|
||||
## Run your first Ray application on Kubernetes!
|
||||
|
||||
* [RayCluster Quick Start](kuberay-raycluster-quickstart)
|
||||
* [RayJob Quick Start](kuberay-rayjob-quickstart)
|
||||
* [RayService Quick Start](kuberay-rayservice-quickstart)
|
||||
* [RayCronJob Quick Start](kuberay-raycronjob-quickstart)
|
||||
@@ -0,0 +1,42 @@
|
||||
(kuberay-operator-deploy)=
|
||||
|
||||
# KubeRay Operator Installation
|
||||
|
||||
## Step 1: Create a Kubernetes cluster
|
||||
|
||||
This step creates a local Kubernetes cluster using [Kind](https://kind.sigs.k8s.io/). If you already have a Kubernetes cluster, you can skip this step.
|
||||
|
||||
```sh
|
||||
kind create cluster --image=kindest/node:v1.26.0
|
||||
```
|
||||
|
||||
## Step 2: Install KubeRay operator
|
||||
|
||||
### Method 1: Helm (Recommended)
|
||||
|
||||
```sh
|
||||
helm repo add kuberay https://ray-project.github.io/kuberay-helm/
|
||||
helm repo update
|
||||
# Install both CRDs and KubeRay operator v1.6.0.
|
||||
helm install kuberay-operator kuberay/kuberay-operator --version 1.6.0
|
||||
```
|
||||
|
||||
### Method 2: Kustomize
|
||||
|
||||
```sh
|
||||
# Install CRD and KubeRay operator.
|
||||
kubectl create -k "github.com/ray-project/kuberay/ray-operator/config/default?ref=v1.6.0"
|
||||
```
|
||||
|
||||
## Step 3: Validate Installation
|
||||
|
||||
Confirm that the operator is running in the namespace `default`.
|
||||
|
||||
```sh
|
||||
kubectl get pods
|
||||
```
|
||||
|
||||
```text
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
kuberay-operator-6bc45dd644-gwtqv 1/1 Running 0 24s
|
||||
```
|
||||
@@ -0,0 +1,167 @@
|
||||
(kuberay-raycluster-quickstart)=
|
||||
|
||||
# RayCluster Quickstart
|
||||
|
||||
This guide shows you how to manage and interact with Ray clusters on Kubernetes.
|
||||
|
||||
## Preparation
|
||||
|
||||
* Install [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) (>= 1.23), [Helm](https://helm.sh/docs/intro/install/) (>= v3.4) if needed, [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation), and [Docker](https://docs.docker.com/engine/install/).
|
||||
* Make sure your Kubernetes cluster has at least 4 CPU and 4 GB RAM.
|
||||
|
||||
## Step 1: Create a Kubernetes cluster
|
||||
|
||||
This step creates a local Kubernetes cluster using [Kind](https://kind.sigs.k8s.io/). If you already have a Kubernetes cluster, you can skip this step.
|
||||
|
||||
```sh
|
||||
kind create cluster --image=kindest/node:v1.26.0
|
||||
```
|
||||
|
||||
## Step 2: Deploy a KubeRay operator
|
||||
|
||||
Follow [this document](kuberay-operator-deploy) to install the latest stable KubeRay operator from the Helm repository.
|
||||
|
||||
(raycluster-deploy)=
|
||||
## Step 3: Deploy a RayCluster custom resource
|
||||
|
||||
Once the KubeRay operator is running, you're ready to deploy a RayCluster. Create a RayCluster Custom Resource (CR) in the `default` namespace.
|
||||
|
||||
```sh
|
||||
# Deploy a sample RayCluster CR from the KubeRay Helm chart repo:
|
||||
helm install raycluster kuberay/ray-cluster --version 1.6.0
|
||||
```
|
||||
|
||||
|
||||
Once the RayCluster CR has been created, you can view it by running:
|
||||
|
||||
```sh
|
||||
# Once the RayCluster CR has been created, you can view it by running:
|
||||
kubectl get rayclusters
|
||||
```
|
||||
|
||||
```sh
|
||||
NAME DESIRED WORKERS AVAILABLE WORKERS CPUS MEMORY GPUS STATUS AGE
|
||||
raycluster-kuberay 1 1 2 3G 0 ready 55s
|
||||
```
|
||||
|
||||
The KubeRay operator detects the RayCluster object and starts your Ray cluster by creating head and worker pods. To view Ray cluster's pods, run the following command:
|
||||
|
||||
```sh
|
||||
# View the pods in the RayCluster named "raycluster-kuberay"
|
||||
kubectl get pods --selector=ray.io/cluster=raycluster-kuberay
|
||||
```
|
||||
|
||||
```sh
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
raycluster-kuberay-head 1/1 Running 0 XXs
|
||||
raycluster-kuberay-worker-workergroup-xvfkr 1/1 Running 0 XXs
|
||||
```
|
||||
|
||||
Wait for the pods to reach `Running` state. This may take a few minutes, downloading the Ray images takes most of this time. If your pods stick in the `Pending` state, you can check for errors using `kubectl describe pod raycluster-kuberay-xxxx-xxxxx` and ensure your Docker resource limits meet the requirements.
|
||||
|
||||
## Step 4: Run an application on a RayCluster
|
||||
|
||||
Now, interact with the RayCluster deployed.
|
||||
|
||||
### Method 1: Execute a Ray job in the head Pod
|
||||
|
||||
The most straightforward way to experiment with your RayCluster is to exec directly into the head pod. First, identify your RayCluster's head pod:
|
||||
|
||||
```sh
|
||||
export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)
|
||||
echo $HEAD_POD
|
||||
```
|
||||
|
||||
```sh
|
||||
raycluster-kuberay-head
|
||||
```
|
||||
|
||||
```sh
|
||||
# Print the cluster resources.
|
||||
kubectl exec -it $HEAD_POD -- python -c "import ray; ray.init(); print(ray.cluster_resources())"
|
||||
```
|
||||
|
||||
```sh
|
||||
2023-04-07 10:57:46,472 INFO worker.py:1243 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS
|
||||
2023-04-07 10:57:46,472 INFO worker.py:1364 -- Connecting to existing Ray cluster at address: 10.244.0.6:6379...
|
||||
2023-04-07 10:57:46,482 INFO worker.py:1550 -- Connected to Ray cluster. View the dashboard at http://10.244.0.6:8265
|
||||
{'CPU': 2.0,
|
||||
'memory': 3000000000.0,
|
||||
'node:10.244.0.6': 1.0,
|
||||
'node:10.244.0.7': 1.0,
|
||||
'node:__internal_head__': 1.0,
|
||||
'object_store_memory': 749467238.0}
|
||||
```
|
||||
|
||||
### Method 2: Submit a Ray job to the RayCluster using [ray job submission SDK](jobs-quickstart)
|
||||
|
||||
Unlike Method 1, this method doesn't require you to execute commands in the Ray head pod. Instead, you can use the [Ray job submission SDK](jobs-quickstart) to submit Ray jobs to the RayCluster through the Ray Dashboard port where Ray listens for Job requests. The KubeRay operator configures a [Kubernetes service](https://kubernetes.io/docs/concepts/services-networking/service/) targeting the Ray head Pod.
|
||||
|
||||
```sh
|
||||
kubectl get service raycluster-kuberay-head-svc
|
||||
```
|
||||
|
||||
```sh
|
||||
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
raycluster-kuberay-head-svc ClusterIP None <none> 10001/TCP,8265/TCP,6379/TCP,8080/TCP,8000/TCP 57s
|
||||
```
|
||||
|
||||
Now that the service name is available, use port-forwarding to access the Ray Dashboard port which is 8265 by default.
|
||||
|
||||
```sh
|
||||
# Execute this in a separate shell.
|
||||
kubectl port-forward service/raycluster-kuberay-head-svc 8265:8265 > /dev/null &
|
||||
```
|
||||
|
||||
Now that the Dashboard port is accessible, submit jobs to the RayCluster:
|
||||
|
||||
```sh
|
||||
# The following job's logs will show the Ray cluster's total resource capacity, including 2 CPUs.
|
||||
ray job submit --address http://localhost:8265 -- python -c "import ray; ray.init(); print(ray.cluster_resources())"
|
||||
```
|
||||
|
||||
```sh
|
||||
Job submission server address: http://localhost:8265
|
||||
|
||||
-------------------------------------------------------
|
||||
Job 'raysubmit_8vJ7dKqYrWKbd17i' submitted successfully
|
||||
-------------------------------------------------------
|
||||
|
||||
Next steps
|
||||
Query the logs of the job:
|
||||
ray job logs raysubmit_8vJ7dKqYrWKbd17i
|
||||
Query the status of the job:
|
||||
ray job status raysubmit_8vJ7dKqYrWKbd17i
|
||||
Request the job to be stopped:
|
||||
ray job stop raysubmit_8vJ7dKqYrWKbd17i
|
||||
|
||||
Tailing logs until the job exits (disable with --no-wait):
|
||||
2025-03-18 01:27:51,014 INFO job_manager.py:530 -- Runtime env is setting up.
|
||||
2025-03-18 01:27:51,744 INFO worker.py:1514 -- Using address 10.244.0.6:6379 set in the environment variable RAY_ADDRESS
|
||||
2025-03-18 01:27:51,744 INFO worker.py:1654 -- Connecting to existing Ray cluster at address: 10.244.0.6:6379...
|
||||
2025-03-18 01:27:51,750 INFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at 10.244.0.6:8265
|
||||
{'CPU': 2.0,
|
||||
'memory': 3000000000.0,
|
||||
'node:10.244.0.6': 1.0,
|
||||
'node:10.244.0.7': 1.0,
|
||||
'node:__internal_head__': 1.0,
|
||||
'object_store_memory': 749467238.0}
|
||||
|
||||
------------------------------------------
|
||||
Job 'raysubmit_8vJ7dKqYrWKbd17i' succeeded
|
||||
------------------------------------------
|
||||
```
|
||||
|
||||
## Step 5: Access the Ray Dashboard
|
||||
|
||||
Visit `${YOUR_IP}:8265` in your browser for the Dashboard. For example, `127.0.0.1:8265`. See the job you submitted in Step 4 in the **Recent jobs** pane as shown below.
|
||||
|
||||
|
||||
|
||||
## Step 6: Cleanup
|
||||
|
||||
```sh
|
||||
# Kill the `kubectl port-forward` background job in the earlier step
|
||||
killall kubectl
|
||||
kind delete cluster
|
||||
```
|
||||
@@ -0,0 +1,140 @@
|
||||
(kuberay-raycronjob-quickstart)=
|
||||
|
||||
# RayCronJob Quickstart
|
||||
|
||||
## Prerequisites
|
||||
|
||||
* This feature requires KubeRay version 1.6.0 or newer, and it's in alpha testing.
|
||||
* A running Kubernetes cluster.
|
||||
* The KubeRay operator installed and running in your cluster.
|
||||
|
||||
## What is a RayCronJob?
|
||||
|
||||
A `RayCronJob` is a Custom Resource (CR) that allows you to run `RayJob` workloads on a recurring, time-based schedule. It is heavily inspired by the native Kubernetes `CronJob` and brings automated, scheduled execution to your distributed Ray applications.
|
||||
|
||||
|
||||
This is particularly useful for recurring tasks such as scheduled model retraining, nightly batch inferences, or regular data processing pipelines.
|
||||
|
||||
|
||||
## RayCronJob Configuration
|
||||
|
||||
The `RayCronJob` CRD acts as an automated scheduler specifically designed to create and manage **RayJob** custom resources on a recurring basis. It does not execute workloads directly. Instead, it acts as a controller that creates a new `RayJob` each time the schedule triggers.
|
||||
|
||||
* `schedule` - The cron schedule string defining when a new Ray job should be created and run (e.g., `* * * * *` for every minute).
|
||||
* `jobTemplate` - Wraps a standard **RayJob** spec that the controller will use for each scheduled run. It supports the same fields as a RayJob spec. See the standard [RayJob Configuration](kuberay-rayjob-quickstart) documentation for the complete list of supported fields within the `jobTemplate`.
|
||||
* `suspend` (Optional): If `suspend` is true, the controller suspends the scheduling of future jobs. This does not apply to or interrupt any `RayJob`s that have already been created and are currently running.
|
||||
* `timeZone` (Optional): The time zone for the `schedule`, in [IANA Time Zone Database](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) format (e.g., `America/Los_Angeles`). If omitted, the schedule uses the local time zone of the KubeRay operator. Do not set `TZ` or `CRON_TZ` in the `schedule` string, use this field instead.
|
||||
|
||||
## How to Configure a RayCronJob
|
||||
|
||||
Configuring a `RayCronJob` requires wrapping a standard `RayJob` specification inside a `jobTemplate`, alongside your scheduling parameters.
|
||||
|
||||
The high-level structure looks like this:
|
||||
|
||||
```yaml
|
||||
apiVersion: ray.io/v1
|
||||
kind: RayCronJob
|
||||
metadata:
|
||||
name: example-raycronjob
|
||||
spec:
|
||||
schedule: "*/5 * * * *" # Run every 5 minutes
|
||||
timeZone: "America/Los_Angeles" # Optional, defaults to the operator's local time zone
|
||||
jobTemplate:
|
||||
# Everything below here is a standard RayJob spec
|
||||
entrypoint: python /home/ray/samples/sample_code.py
|
||||
# ... (RayCluster spec, runtimeEnv, etc.)
|
||||
```
|
||||
|
||||
## How to Run a Simple RayCronJob
|
||||
|
||||
Let's deploy a simple `RayCronJob` that executes a short Python script every minute.
|
||||
|
||||
### Step 1: Create a Kubernetes cluster with Kind
|
||||
|
||||
```sh
|
||||
kind create cluster --image=kindest/node:v1.26.0
|
||||
```
|
||||
|
||||
### Step 2: Install the KubeRay operator
|
||||
Install the KubeRay operator, following [these instructions](https://docs.ray.io/en/latest/cluster/kubernetes/getting-started/kuberay-operator-installation.html). The minimum version for this guide is v1.6.0. To use this feature, the `RayCronJob` feature gate must be enabled. To enable the feature gate when installing the kuberay operator, run the following command:
|
||||
|
||||
```sh
|
||||
helm repo add kuberay https://ray-project.github.io/kuberay-helm/
|
||||
helm repo update
|
||||
|
||||
# Install KubeRay operator v1.6.0 with the RayCronJob feature gate enabled
|
||||
helm install kuberay-operator kuberay/kuberay-operator \
|
||||
--version 1.6.0 \
|
||||
--set "featureGates[0].name=RayCronJob" \
|
||||
--set "featureGates[0].enabled=true"
|
||||
```
|
||||
|
||||
### Step 3: Install a RayCronJob
|
||||
|
||||
```sh
|
||||
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-cronjob.sample.yaml
|
||||
```
|
||||
|
||||
### Step 4: Monitor the RayCronJob
|
||||
|
||||
Check the status of your `RayCronJob`. The `SCHEDULE` field should be visible, while `LAST SCHEDULE` may be empty until the first run.
|
||||
|
||||
```sh
|
||||
kubectl get raycronjob raycronjob-sample
|
||||
|
||||
#You should see output listing the RayCronJob created
|
||||
# [Example output]
|
||||
|
||||
# NAME SCHEDULE LAST SCHEDULE AGE SUSPEND
|
||||
# raycronjob-sample * * * * * 10s
|
||||
```
|
||||
|
||||
Because our schedule is `* * * * *`, a new `RayJob` will be generated at the start of the next minute. You can watch the `RayJob` instances being created:
|
||||
|
||||
```shell
|
||||
kubectl get rayjob -w
|
||||
# [Example output]
|
||||
# NAME JOB STATUS DEPLOYMENT STATUS RAY CLUSTER NAME START TIME END TIME AGE
|
||||
# raycronjob-sample-l76h8 Initializing raycronjob-sample-l76h8-hjtrs 2026-04-03T05:57:00Z 2s
|
||||
# raycronjob-sample-l76h8 RUNNING Running raycronjob-sample-l76h8-hjtrs 2026-04-03T05:57:00Z 48s
|
||||
# raycronjob-sample-l76h8 SUCCEEDED Complete raycronjob-sample-l76h8-hjtrs 2026-04-03T05:57:00Z 2026-04-03T05:58:02Z 62s
|
||||
# raycronjob-sample-pct47 Initializing raycronjob-sample-pct47-bdspj 2026-04-03T05:58:00Z 0s
|
||||
# (Press Ctrl+C to stop watching once the job completes)
|
||||
```
|
||||
|
||||
### Step 5: Check the output of the RayCronJob
|
||||
|
||||
```shell
|
||||
# From the previous step, note the RayJob name label (e.g., raycronjob-sample-l76h8)
|
||||
# Use it to fetch the submitter pod logs directly:
|
||||
kubectl logs -l=job-name=<rayjob-name>
|
||||
# Example:
|
||||
# kubectl logs -l=job-name=raycronjob-sample-l76h8
|
||||
|
||||
# [Example output]
|
||||
# /home/ray/anaconda3/lib/python3.10/site-packages/ray/_private/worker.py:2062: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
|
||||
# warnings.warn(
|
||||
# test_counter got 1
|
||||
# test_counter got 2
|
||||
# test_counter got 3
|
||||
# test_counter got 4
|
||||
# test_counter got 5
|
||||
# 2026-04-02 22:57:58,968 SUCC cli.py:65 -- ---------------------------------------------
|
||||
# 2026-04-02 22:57:58,968 SUCC cli.py:66 -- Job 'raycronjob-sample-l76h8-hmjz2' succeeded
|
||||
# 2026-04-02 22:57:58,968 SUCC cli.py:67 -- ---------------------------------------------
|
||||
```
|
||||
|
||||
### Step 6: Clean Up
|
||||
|
||||
To stop the recurring jobs and delete the resource, run:
|
||||
|
||||
```bash
|
||||
# Step 6.1: Delete the RayCronJob
|
||||
kubectl delete -f https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-cronjob.sample.yaml
|
||||
|
||||
# Step 6.2: Delete the KubeRay operator
|
||||
helm uninstall kuberay-operator
|
||||
|
||||
# Step 6.3: Delete the Kubernetes cluster
|
||||
kind delete cluster
|
||||
```
|
||||
@@ -0,0 +1,208 @@
|
||||
(kuberay-rayjob-quickstart)=
|
||||
|
||||
# RayJob Quickstart
|
||||
|
||||
## Prerequisites
|
||||
|
||||
* KubeRay v0.6.0 or higher
|
||||
* KubeRay v0.6.0 or v1.0.0: Ray 1.10 or higher.
|
||||
* KubeRay v1.1.1 or newer is highly recommended: Ray 2.8.0 or higher.
|
||||
|
||||
## What's a RayJob?
|
||||
|
||||
A RayJob manages two aspects:
|
||||
|
||||
* **RayCluster**: A RayCluster custom resource manages all Pods in a Ray cluster, including a head Pod and multiple worker Pods.
|
||||
* **Job**: A Kubernetes Job runs `ray job submit` to submit a Ray job to the RayCluster.
|
||||
|
||||
## What does the RayJob provide?
|
||||
|
||||
With RayJob, KubeRay automatically creates a RayCluster and submits a job when the cluster is ready. You can also configure RayJob to automatically delete the RayCluster once the Ray job finishes.
|
||||
|
||||
To understand the following content better, you should understand the difference between:
|
||||
* RayJob: A Kubernetes custom resource definition provided by KubeRay.
|
||||
* Ray job: A Ray job is a packaged Ray application that can run on a remote Ray cluster. See [this document](jobs-overview) for more details.
|
||||
* Submitter: The submitter is a Kubernetes Job that runs `ray job submit` to submit a Ray job to the RayCluster.
|
||||
|
||||
## RayJob Configuration
|
||||
|
||||
* RayCluster configuration
|
||||
* `rayClusterSpec` - Defines the **RayCluster** custom resource to run the Ray job on.
|
||||
* `clusterSelector` - Use existing **RayCluster** custom resources to run the Ray job instead of creating a new one. See [ray-job.use-existing-raycluster.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-job.use-existing-raycluster.yaml) for example configurations.
|
||||
* Ray job configuration
|
||||
* `entrypoint` - The submitter runs `ray job submit --address ... --submission-id ... -- $entrypoint` to submit a Ray job to the RayCluster.
|
||||
* `runtimeEnvYAML` (Optional): A runtime environment that describes the dependencies the Ray job needs to run, including files, packages, environment variables, and more. Provide the configuration as a multi-line YAML string. Example:
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
runtimeEnvYAML: |
|
||||
pip:
|
||||
- requests==2.26.0
|
||||
- pendulum==2.1.2
|
||||
env_vars:
|
||||
KEY: "VALUE"
|
||||
```
|
||||
|
||||
See {ref}`Runtime Environments <runtime-environments>` for more details. _(New in KubeRay version 1.0.0)_
|
||||
* `jobId` (Optional): Defines the submission ID for the Ray job. If not provided, KubeRay generates one automatically. See {ref}`Ray Jobs CLI API Reference <ray-job-submission-cli-ref>` for more details about the submission ID.
|
||||
* `metadata` (Optional): See {ref}`Ray Jobs CLI API Reference <ray-job-submission-cli-ref>` for more details about the `--metadata-json` option.
|
||||
* `entrypointNumCpus` / `entrypointNumGpus` / `entrypointResources` (Optional): See {ref}`Ray Jobs CLI API Reference <ray-job-submission-cli-ref>` for more details.
|
||||
* `backoffLimit` (Optional, added in version 1.2.0): Specifies the number of retries before marking this RayJob failed. Each retry creates a new RayCluster. The default value is 0.
|
||||
* Submission configuration
|
||||
* `submissionMode` (Optional): Specifies how RayJob submits the Ray job to the RayCluster. There are three possible values, with the default being `K8sJobMode`.
|
||||
* `K8sJobMode`: The KubeRay operator creates a submitter Kubernetes Job to submit the Ray job.
|
||||
* `HTTPMode`: The KubeRay operator sends a request to the RayCluster to create a Ray job.
|
||||
* `InteractiveMode`: The KubeRay operator waits for the user to submit a job to the RayCluster. This mode is currently in alpha and the [KubeRay kubectl plugin](kubectl-plugin) relies on it.
|
||||
* `SidecarMode`: The KubeRay operator injects a container into the Ray head Pod to submit the Ray job. This mode does not support `clusterSelector`, `submitterPodTemplate`, and `submitterConfig`, and requires the head Pod's restart policy to be `Never`.
|
||||
* `submitterPodTemplate` (Optional): Defines the Pod template for the submitter Kubernetes Job. This field is only effective when `submissionMode` is "K8sJobMode".
|
||||
* `RAY_DASHBOARD_ADDRESS` - The KubeRay operator injects this environment variable to the submitter Pod. The value is `$HEAD_SERVICE:$DASHBOARD_PORT`.
|
||||
* `RAY_JOB_SUBMISSION_ID` - The KubeRay operator injects this environment variable to the submitter Pod. The value is the `RayJob.Status.JobId` of the RayJob.
|
||||
* Example: `ray job submit --address=http://$RAY_DASHBOARD_ADDRESS --submission-id=$RAY_JOB_SUBMISSION_ID ...`
|
||||
* See [ray-job.sample.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-job.sample.yaml) for more details.
|
||||
* `submitterConfig` (Optional): Additional configurations for the submitter Kubernetes Job.
|
||||
* `backoffLimit` (Optional, added in version 1.2.0): The number of retries before marking the submitter Job as failed. The default value is 2.
|
||||
* Automatic resource cleanup
|
||||
* `preRunningDeadlineSeconds` (Optional): If the RayJob doesn't transition the `JobDeploymentStatus` to `Running` within `preRunningDeadlineSeconds` seconds, the KubeRay operator transitions the `JobDeploymentStatus` to `Failed` with reason `PreRunningDeadlineExceeded`. The default value is 0 (no pre-running deadline is enforced).
|
||||
* `shutdownAfterJobFinishes` (Optional): Determines whether to recycle the RayCluster after the Ray job finishes. The default value is false.
|
||||
* `ttlSecondsAfterFinished` (Optional): Only works if `shutdownAfterJobFinishes` is true. The KubeRay operator deletes the RayCluster and the submitter `ttlSecondsAfterFinished` seconds after the Ray job finishes. The default value is 0.
|
||||
* `activeDeadlineSeconds` (Optional): If the RayJob doesn't transition the `JobDeploymentStatus` to `Complete` or `Failed` within `activeDeadlineSeconds`, the KubeRay operator transitions the `JobDeploymentStatus` to `Failed`, citing `DeadlineExceeded` as the reason.
|
||||
* `DELETE_RAYJOB_CR_AFTER_JOB_FINISHES` (Optional, added in version 1.2.0): Set this environment variable for the KubeRay operator, not the RayJob resource. If you set this environment variable to true, the RayJob custom resource itself is deleted if you also set `shutdownAfterJobFinishes` to true. Note that KubeRay deletes all resources created by the RayJob, including the Kubernetes Job.
|
||||
* Others
|
||||
* `suspend` (Optional): If `suspend` is true, KubeRay deletes both the RayCluster and the submitter. Note that Kueue also implements scheduling strategies by mutating this field. Avoid manually updating this field if you use Kueue to schedule RayJob.
|
||||
* `deletionStrategy` (alpha in v1.5.1, beta in v1.6.0): Configures automated cleanup after the RayJob reaches a terminal state. This field requires the `RayJobDeletionPolicy` feature gate to be enabled. Two mutually exclusive styles are supported:
|
||||
* **Rules-based** (Recommended): Define `deletionRules` as a list of deletion actions triggered by specific conditions. Each rule specifies:
|
||||
* `policy`: The deletion action to perform — `DeleteCluster` (delete the entire RayCluster and its Pods), `DeleteWorkers` (delete only worker Pods), `DeleteSelf` (delete the RayJob and all associated resources), or `DeleteNone` (no deletion).
|
||||
* `condition`: When to trigger the deletion, based on `jobStatus` (`SUCCEEDED` or `FAILED`) and an optional `ttlSeconds` delay.
|
||||
* This approach enables flexible, multi-stage cleanup strategies (e.g., delete workers immediately on success, then delete the cluster after 300 seconds).
|
||||
* Rules-based mode is incompatible with `shutdownAfterJobFinishes` and the global `ttlSecondsAfterFinished`. Use per-rule `condition.ttlSeconds` instead.
|
||||
* See [ray-job.deletion-rules.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-job.deletion-rules.yaml) for example configurations.
|
||||
* **Legacy** (Deprecated): Define both `onSuccess` and `onFailure` policies. This approach is deprecated and will be removed in v1.6.0. Migration to `deletionRules` is strongly encouraged.
|
||||
* Legacy mode can be combined with `shutdownAfterJobFinishes` and the global `ttlSecondsAfterFinished`.
|
||||
* For detailed API specifications, see the [KubeRay API Reference](https://ray-project.github.io/kuberay/reference/api/).
|
||||
|
||||
|
||||
## Example: Run a simple Ray job with RayJob
|
||||
|
||||
## Step 1: Create a Kubernetes cluster with Kind
|
||||
|
||||
```sh
|
||||
kind create cluster --image=kindest/node:v1.26.0
|
||||
```
|
||||
|
||||
## Step 2: Install the KubeRay operator
|
||||
|
||||
Follow the [KubeRay Operator Installation](kuberay-operator-deploy) to install the latest stable KubeRay operator by Helm repository.
|
||||
|
||||
## Step 3: Install a RayJob
|
||||
|
||||
```sh
|
||||
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-job.sample.yaml
|
||||
```
|
||||
|
||||
## Step 4: Verify the Kubernetes cluster status
|
||||
|
||||
```shell
|
||||
# Step 4.1: List all RayJob custom resources in the `default` namespace.
|
||||
kubectl get rayjob
|
||||
|
||||
# [Example output]
|
||||
# NAME JOB STATUS DEPLOYMENT STATUS RAY CLUSTER NAME START TIME END TIME AGE
|
||||
# rayjob-sample SUCCEEDED Complete rayjob-sample-qnftt 2025-06-25T16:21:21Z 2025-06-25T16:22:35Z 6m53s
|
||||
|
||||
# Step 4.2: List all RayCluster custom resources in the `default` namespace.
|
||||
kubectl get raycluster
|
||||
|
||||
# [Example output]
|
||||
# NAME DESIRED WORKERS AVAILABLE WORKERS CPUS MEMORY GPUS STATUS AGE
|
||||
# rayjob-sample-qnftt 1 1 400m 0 0 ready 7m48s
|
||||
|
||||
# Step 4.3: List all Pods in the `default` namespace.
|
||||
# The Pod created by the Kubernetes Job will be terminated after the Kubernetes Job finishes.
|
||||
kubectl get pods
|
||||
|
||||
# [Example output]
|
||||
# kuberay-operator-755f666c4b-wbcm4 1/1 Running 0 8m32s
|
||||
# rayjob-sample-n2vj5 0/1 Completed 0 7m18ss => Pod created by a Kubernetes Job
|
||||
# rayjob-sample-qnftt-head 1/1 Running 0 8m14s
|
||||
# rayjob-sample-qnftt-small-group-worker-4f5wz 1/1 Running 0 8m14s
|
||||
|
||||
# Step 4.4: Check the status of the RayJob.
|
||||
# The field `jobStatus` in the RayJob custom resource will be updated to `SUCCEEDED` and `jobDeploymentStatus`
|
||||
# should be `Complete` once the job finishes.
|
||||
kubectl get rayjobs.ray.io rayjob-sample -o jsonpath='{.status.jobStatus}'
|
||||
# [Expected output]: "SUCCEEDED"
|
||||
|
||||
kubectl get rayjobs.ray.io rayjob-sample -o jsonpath='{.status.jobDeploymentStatus}'
|
||||
# [Expected output]: "Complete"
|
||||
```
|
||||
|
||||
The KubeRay operator creates a RayCluster custom resource based on the `rayClusterSpec` and a submitter Kubernetes Job to submit a Ray job to the RayCluster. In this example, the `entrypoint` is `python /home/ray/samples/sample_code.py`, and `sample_code.py` is a Python script stored in a Kubernetes ConfigMap mounted to the head Pod of the RayCluster. Because the default value of `shutdownAfterJobFinishes` is false, the KubeRay operator doesn't delete the RayCluster or the submitter when the Ray job finishes.
|
||||
|
||||
## Step 5: Check the output of the Ray job
|
||||
|
||||
```sh
|
||||
kubectl logs -l=job-name=rayjob-sample
|
||||
|
||||
# [Example output]
|
||||
# 2025-06-25 09:22:27,963 INFO worker.py:1654 -- Connecting to existing Ray cluster at address: 10.244.0.6:6379...
|
||||
# 2025-06-25 09:22:27,977 INFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at 10.244.0.6:8265
|
||||
# test_counter got 1
|
||||
# test_counter got 2
|
||||
# test_counter got 3
|
||||
# test_counter got 4
|
||||
# test_counter got 5
|
||||
# 2025-06-25 09:22:31,719 SUCC cli.py:63 -- -----------------------------------
|
||||
# 2025-06-25 09:22:31,719 SUCC cli.py:64 -- Job 'rayjob-sample-zdxm6' succeeded
|
||||
# 2025-06-25 09:22:31,719 SUCC cli.py:65 -- -----------------------------------
|
||||
```
|
||||
|
||||
The Python script `sample_code.py` used by `entrypoint` is a simple Ray script that executes a counter's increment function 5 times.
|
||||
|
||||
## Step 6: Delete the RayJob
|
||||
|
||||
```sh
|
||||
kubectl delete -f https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-job.sample.yaml
|
||||
```
|
||||
|
||||
## Step 7: Create a RayJob with `shutdownAfterJobFinishes` set to true
|
||||
|
||||
```sh
|
||||
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-job.shutdown.yaml
|
||||
```
|
||||
|
||||
The `ray-job.shutdown.yaml` defines a RayJob custom resource with `shutdownAfterJobFinishes: true` and `ttlSecondsAfterFinished: 10`. Hence, the KubeRay operator deletes the RayCluster 10 seconds after the Ray job finishes. Note that the submitter job isn't deleted because it contains the ray job logs and doesn't use any cluster resources once completed. In addition, the RayJob cleans up the submitter job when the RayJob is eventually deleted due to its owner reference back to the RayJob.
|
||||
|
||||
## Step 8: Check the RayJob status
|
||||
|
||||
```sh
|
||||
# Wait until `jobStatus` is `SUCCEEDED` and `jobDeploymentStatus` is `Complete`.
|
||||
kubectl get rayjobs.ray.io rayjob-sample-shutdown -o jsonpath='{.status.jobDeploymentStatus}'
|
||||
kubectl get rayjobs.ray.io rayjob-sample-shutdown -o jsonpath='{.status.jobStatus}'
|
||||
```
|
||||
|
||||
## Step 9: Check if the KubeRay operator deletes the RayCluster
|
||||
|
||||
```sh
|
||||
# List the RayCluster custom resources in the `default` namespace. The RayCluster
|
||||
# associated with the RayJob `rayjob-sample-shutdown` should be deleted.
|
||||
kubectl get raycluster
|
||||
```
|
||||
|
||||
## Step 10: Clean up
|
||||
|
||||
```sh
|
||||
# Step 10.1: Delete the RayJob
|
||||
kubectl delete -f https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-job.shutdown.yaml
|
||||
|
||||
# Step 10.2: Delete the KubeRay operator
|
||||
helm uninstall kuberay-operator
|
||||
|
||||
# Step 10.3: Delete the Kubernetes cluster
|
||||
kind delete cluster
|
||||
```
|
||||
|
||||
## Next steps
|
||||
|
||||
* [RayJob Batch Inference Example](kuberay-batch-inference-example)
|
||||
* [Priority Scheduling with RayJob and Kueue](kuberay-kueue-priority-scheduling-example)
|
||||
* [Gang Scheduling with RayJob and Kueue](kuberay-kueue-gang-scheduling-example)
|
||||
@@ -0,0 +1,142 @@
|
||||
(kuberay-rayservice-quickstart)=
|
||||
# RayService Quickstart
|
||||
|
||||
## Prerequisites
|
||||
|
||||
This guide mainly focuses on the behavior of KubeRay v1.6.0 and Ray 2.46.0.
|
||||
|
||||
## What's a RayService?
|
||||
|
||||
A RayService manages these components:
|
||||
|
||||
* **RayCluster**: Manages resources in a Kubernetes cluster.
|
||||
* **Ray Serve Applications**: Manages users' applications.
|
||||
|
||||
## What does the RayService provide?
|
||||
|
||||
* **Kubernetes-native support for Ray clusters and Ray Serve applications:** After using a Kubernetes configuration to define a Ray cluster and its Ray Serve applications, you can use `kubectl` to create the cluster and its applications.
|
||||
* **In-place updating for Ray Serve applications:** See [RayService](kuberay-rayservice) for more details.
|
||||
* **Zero downtime upgrading for Ray clusters:** See [RayService](kuberay-rayservice) for more details.
|
||||
* **High-availabilable services:** See [RayService high availability](kuberay-rayservice-ha) for more details.
|
||||
|
||||
## Example: Serve two simple Ray Serve applications using RayService
|
||||
|
||||
## Step 1: Create a Kubernetes cluster with Kind
|
||||
|
||||
```sh
|
||||
kind create cluster --image=kindest/node:v1.26.0
|
||||
```
|
||||
|
||||
## Step 2: Install the KubeRay operator
|
||||
|
||||
Follow [this document](kuberay-operator-deploy) to install the latest stable KubeRay operator from the Helm repository. Note that the YAML file in this example uses `serveConfigV2` to specify a multi-application Serve configuration, available starting from KubeRay v0.6.0.
|
||||
|
||||
## Step 3: Install a RayService
|
||||
|
||||
```sh
|
||||
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-service.sample.yaml
|
||||
```
|
||||
|
||||
## Step 4: Verify the Kubernetes cluster status
|
||||
|
||||
```sh
|
||||
# Step 4.1: List all RayService custom resources in the `default` namespace.
|
||||
kubectl get rayservice
|
||||
|
||||
# [Example output]
|
||||
# NAME SERVICE STATUS NUM SERVE ENDPOINTS
|
||||
# rayservice-sample Running 2
|
||||
|
||||
# Step 4.2: List all RayCluster custom resources in the `default` namespace.
|
||||
kubectl get raycluster
|
||||
|
||||
# [Example output]
|
||||
# NAME DESIRED WORKERS AVAILABLE WORKERS CPUS MEMORY GPUS STATUS AGE
|
||||
# rayservice-sample-cxm7t 1 1 2500m 4Gi 0 ready 79s
|
||||
|
||||
# Step 4.3: List all Ray Pods in the `default` namespace.
|
||||
kubectl get pods -l=ray.io/is-ray-node=yes
|
||||
|
||||
# [Example output]
|
||||
# NAME READY STATUS RESTARTS AGE
|
||||
# rayservice-sample-cxm7t-head 1/1 Running 0 3m5s
|
||||
# rayservice-sample-cxm7t-small-group-worker-8hrgg 1/1 Running 0 3m5s
|
||||
|
||||
# Step 4.4: Check the `Ready` condition of the RayService.
|
||||
# The RayService is ready to serve requests when the condition is `True`.
|
||||
kubectl describe rayservices.ray.io rayservice-sample
|
||||
|
||||
# [Example output]
|
||||
# Conditions:
|
||||
# Last Transition Time: 2025-06-26T13:23:06Z
|
||||
# Message: Number of serve endpoints is greater than 0
|
||||
# Observed Generation: 1
|
||||
# Reason: NonZeroServeEndpoints
|
||||
# Status: True
|
||||
# Type: Ready
|
||||
|
||||
# Step 4.5: List services in the `default` namespace.
|
||||
kubectl get services
|
||||
|
||||
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
# ...
|
||||
# rayservice-sample-cxm7t-head-svc ClusterIP None <none> 10001/TCP,8265/TCP,6379/TCP,8080/TCP,8000/TCP 71m
|
||||
# rayservice-sample-head-svc ClusterIP None <none> 10001/TCP,8265/TCP,6379/TCP,8080/TCP,8000/TCP 70m
|
||||
# rayservice-sample-serve-svc ClusterIP 10.96.125.107 <none> 8000/TCP 70m
|
||||
```
|
||||
|
||||
When the Ray Serve applications are healthy and ready, KubeRay creates a head service and a Ray Serve service for the RayService custom resource. For example, `rayservice-sample-head-svc` and `rayservice-sample-serve-svc` in Step 4.5.
|
||||
> **What do these services do?**
|
||||
|
||||
- **`rayservice-sample-head-svc`**
|
||||
This service points to the **head pod** of the active RayCluster and is typically used to view the **Ray Dashboard** (port `8265`).
|
||||
|
||||
- **`rayservice-sample-serve-svc`**
|
||||
This service exposes the **HTTP interface** of Ray Serve, typically on port `8000`.
|
||||
Use this service to send HTTP requests to your deployed Serve applications (e.g., REST API, ML inference, etc.).
|
||||
|
||||
|
||||
## Step 5: Verify the status of the Serve applications
|
||||
|
||||
```sh
|
||||
# (1) Forward the dashboard port to localhost.
|
||||
# (2) Check the Serve page in the Ray dashboard at http://localhost:8265/#/serve.
|
||||
kubectl port-forward svc/rayservice-sample-head-svc 8265:8265
|
||||
```
|
||||
|
||||
* Refer to [rayservice-troubleshooting.md](kuberay-raysvc-troubleshoot) for more details on RayService observability. Below is a screenshot example of the Serve page in the Ray dashboard. 
|
||||
|
||||
## Step 6: Send requests to the Serve applications by the Kubernetes serve service
|
||||
|
||||
```sh
|
||||
# Step 6.1: Run a curl Pod.
|
||||
# If you already have a curl Pod, you can use `kubectl exec -it <curl-pod> -- sh` to access the Pod.
|
||||
kubectl run curl --image=curlimages/curl:latest -i --tty -- sh
|
||||
|
||||
# Step 6.2: Send a request to the fruit stand app.
|
||||
curl -X POST -H 'Content-Type: application/json' rayservice-sample-serve-svc:8000/fruit/ -d '["MANGO", 2]'
|
||||
# [Expected output]: 6
|
||||
|
||||
# Step 6.3: Send a request to the calculator app.
|
||||
curl -X POST -H 'Content-Type: application/json' rayservice-sample-serve-svc:8000/calc/ -d '["MUL", 3]'
|
||||
# [Expected output]: "15 pizzas please!"
|
||||
```
|
||||
|
||||
## Step 7: Clean up the Kubernetes cluster
|
||||
|
||||
```sh
|
||||
# Delete the RayService.
|
||||
kubectl delete -f https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-service.sample.yaml
|
||||
|
||||
# Uninstall the KubeRay operator.
|
||||
helm uninstall kuberay-operator
|
||||
|
||||
# Delete the curl Pod.
|
||||
kubectl delete pod curl
|
||||
```
|
||||
|
||||
## Next steps
|
||||
|
||||
* See [RayService](kuberay-rayservice) document for the full list of RayService features, including in-place update, zero downtime upgrade, and high-availability.
|
||||
* See [RayService troubleshooting guide](kuberay-raysvc-troubleshoot) if you encounter any issues.
|
||||
* See [Examples](kuberay-examples) for more RayService examples. The [MobileNet example](kuberay-mobilenet-rayservice-example) is a good example to start with because it doesn't require GPUs and is easy to run on a local machine.
|
||||
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 135 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 30 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 68 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 120 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 560 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 475 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 123 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 121 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 138 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 594 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 519 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 174 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 100 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 187 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 275 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 193 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 273 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 434 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 902 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 484 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 290 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 133 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 458 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 199 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 134 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 183 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 67 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 414 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 263 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 54 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 66 KiB |
@@ -0,0 +1,143 @@
|
||||
# Ray on Kubernetes
|
||||
|
||||
```{toctree}
|
||||
:hidden:
|
||||
|
||||
getting-started
|
||||
user-guides
|
||||
examples
|
||||
k8s-ecosystem
|
||||
benchmarks
|
||||
troubleshooting
|
||||
references
|
||||
```
|
||||
|
||||
(kuberay-index)=
|
||||
## Overview
|
||||
|
||||
In this section we cover how to execute your distributed Ray programs on a Kubernetes cluster.
|
||||
|
||||
Using the [KubeRay operator](https://github.com/ray-project/kuberay) is the recommended way to do so. The operator provides a Kubernetes-native way to manage Ray clusters. Each Ray cluster consists of a head node pod and a collection of worker node pods. Optional autoscaling support allows the KubeRay operator to size your Ray clusters according to the requirements of your Ray workload, adding and removing Ray pods as needed. KubeRay supports heterogenous compute nodes (including GPUs) as well as running multiple Ray clusters with different Ray versions in the same Kubernetes cluster.
|
||||
|
||||
```{eval-rst}
|
||||
.. image:: images/ray_on_kubernetes.png
|
||||
:align: center
|
||||
..
|
||||
Find source document here: https://docs.google.com/drawings/d/1E3FQgWWLuj8y2zPdKXjoWKrfwgYXw6RV_FWRwK8dVlg/edit
|
||||
```
|
||||
|
||||
KubeRay introduces three distinct Kubernetes Custom Resource Definitions (CRDs): **RayCluster**, **RayJob**, and **RayService**. These CRDs assist users in efficiently managing Ray clusters tailored to various use cases.
|
||||
|
||||
See [Getting Started](kuberay-quickstart) to learn the basics of KubeRay and follow the quickstart guides to run your first Ray application on Kubernetes with KubeRay.
|
||||
|
||||
* [RayCluster Quick Start](kuberay-raycluster-quickstart)
|
||||
* [RayJob Quick Start](kuberay-rayjob-quickstart)
|
||||
* [RayService Quick Start](kuberay-rayservice-quickstart)
|
||||
* [RayCronJob Quick Start](kuberay-raycronjob-quickstart)
|
||||
|
||||
Additionally, [Anyscale](https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=ray-doc-upsell&utm_content=deploy-ray-on-k8s) is the managed Ray platform developed by the creators of Ray. It offers an easy path to deploy Ray clusters on your existing Kubernetes infrastructure, including EKS, GKE, AKS, or self-hosted Kubernetes.
|
||||
|
||||
## Learn More
|
||||
|
||||
The Ray docs present all the information you need to start running Ray workloads on Kubernetes.
|
||||
|
||||
```{eval-rst}
|
||||
.. grid:: 1 2 2 2
|
||||
:gutter: 1
|
||||
:class-container: container pb-3
|
||||
|
||||
.. grid-item-card::
|
||||
|
||||
**Getting Started**
|
||||
^^^
|
||||
|
||||
Learn how to start a Ray cluster and deploy Ray applications on Kubernetes.
|
||||
|
||||
+++
|
||||
.. button-ref:: kuberay-quickstart
|
||||
:color: primary
|
||||
:outline:
|
||||
:expand:
|
||||
|
||||
Get Started with Ray on Kubernetes
|
||||
|
||||
.. grid-item-card::
|
||||
|
||||
**User Guides**
|
||||
^^^
|
||||
|
||||
Learn best practices for configuring Ray clusters on Kubernetes.
|
||||
|
||||
+++
|
||||
.. button-ref:: kuberay-guides
|
||||
:color: primary
|
||||
:outline:
|
||||
:expand:
|
||||
|
||||
Read the User Guides
|
||||
|
||||
.. grid-item-card::
|
||||
|
||||
**Examples**
|
||||
^^^
|
||||
|
||||
Try example Ray workloads on Kubernetes.
|
||||
|
||||
+++
|
||||
.. button-ref:: kuberay-examples
|
||||
:color: primary
|
||||
:outline:
|
||||
:expand:
|
||||
|
||||
Try example workloads
|
||||
|
||||
.. grid-item-card::
|
||||
|
||||
**Ecosystem**
|
||||
^^^
|
||||
|
||||
Integrate KubeRay with third party Kubernetes ecosystem tools.
|
||||
|
||||
+++
|
||||
.. button-ref:: kuberay-ecosystem-integration
|
||||
:color: primary
|
||||
:outline:
|
||||
:expand:
|
||||
|
||||
Ecosystem Guides
|
||||
|
||||
.. grid-item-card::
|
||||
|
||||
**Benchmarks**
|
||||
^^^
|
||||
|
||||
Check the KubeRay benchmark results.
|
||||
|
||||
+++
|
||||
.. button-ref:: kuberay-benchmarks
|
||||
:color: primary
|
||||
:outline:
|
||||
:expand:
|
||||
|
||||
Benchmark results
|
||||
|
||||
.. grid-item-card::
|
||||
|
||||
**Troubleshooting**
|
||||
^^^
|
||||
|
||||
Consult the KubeRay troubleshooting guides.
|
||||
|
||||
+++
|
||||
.. button-ref:: kuberay-troubleshooting
|
||||
:color: primary
|
||||
:outline:
|
||||
:expand:
|
||||
|
||||
Troubleshooting guides
|
||||
```
|
||||
## About KubeRay
|
||||
|
||||
Ray's Kubernetes support is developed at the [KubeRay GitHub repository](https://github.com/ray-project/kuberay), under the broader [Ray project](https://github.com/ray-project/). KubeRay is used by several companies to run production Ray deployments.
|
||||
|
||||
- Visit the [KubeRay GitHub repo](https://github.com/ray-project/kuberay) to track progress, report bugs, propose new features, or contribute to the project.
|
||||
@@ -0,0 +1,29 @@
|
||||
(kuberay-ecosystem-integration)=
|
||||
|
||||
# KubeRay Ecosystem
|
||||
|
||||
```{toctree}
|
||||
:hidden:
|
||||
|
||||
k8s-ecosystem/ingress
|
||||
k8s-ecosystem/metrics-references
|
||||
k8s-ecosystem/prometheus-grafana
|
||||
k8s-ecosystem/pyspy
|
||||
k8s-ecosystem/kai-scheduler
|
||||
k8s-ecosystem/volcano
|
||||
k8s-ecosystem/yunikorn
|
||||
k8s-ecosystem/kueue
|
||||
k8s-ecosystem/istio
|
||||
k8s-ecosystem/scheduler-plugins
|
||||
```
|
||||
|
||||
* {ref}`kuberay-ingress`
|
||||
* {ref}`kuberay-metrics-references`
|
||||
* {ref}`kuberay-prometheus-grafana`
|
||||
* {ref}`kuberay-pyspy-integration`
|
||||
* {ref}`kuberay-kai-scheduler`
|
||||
* {ref}`kuberay-volcano`
|
||||
* {ref}`kuberay-yunikorn`
|
||||
* {ref}`kuberay-kueue`
|
||||
* {ref}`kuberay-istio`
|
||||
* {ref}`kuberay-scheduler-plugins`
|
||||
@@ -0,0 +1,473 @@
|
||||
(kuberay-ingress)=
|
||||
|
||||
# Ingress
|
||||
|
||||
The following examples show how to use Ingress or Gateway to access your Ray clusters:
|
||||
|
||||
* [AWS Application Load Balancer (ALB) Ingress support on AWS EKS](kuberay-aws-alb)
|
||||
* [GKE Ingress support](kuberay-gke-ingress)
|
||||
* [GKE Gateway API support](kuberay-gke-gateway)
|
||||
* [Manually setting up NGINX Ingress on Kind](kuberay-nginx)
|
||||
* [Azure Application Gateway for Containers Gateway API support on AKS](kuberay-aks-agc)
|
||||
|
||||
|
||||
```{admonition} Warning
|
||||
:class: warning
|
||||
**Only expose Ingresses or Gateways to authorized users.** The Ray Dashboard provides read and write access to the Ray Cluster. Anyone with access to this Ingress or Gateway can execute arbitrary code on the Ray Cluster.
|
||||
```
|
||||
|
||||
|
||||
(kuberay-aws-alb)=
|
||||
## AWS Application Load Balancer (ALB) Ingress support on AWS EKS
|
||||
|
||||
### Prerequisites
|
||||
* Create an EKS cluster. See [Getting started with Amazon EKS – AWS Management Console and AWS CLI](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-console.html#eks-configure-kubectl).
|
||||
|
||||
* Set up the [AWS Load Balancer controller](https://github.com/kubernetes-sigs/aws-load-balancer-controller), see [installation instructions](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/deploy/installation/). Note that the repository maintains a webpage for each release. Confirm that you are using the latest installation instructions.
|
||||
|
||||
* (Optional) Try the [echo server example](https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/docs/examples/echo_server.md) in the [aws-load-balancer-controller](https://github.com/kubernetes-sigs/aws-load-balancer-controller) repository.
|
||||
|
||||
* (Optional) Read [how-it-works.md](https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/docs/how-it-works.md) to understand the [aws-load-balancer-controller](https://github.com/kubernetes-sigs/aws-load-balancer-controller) mechanism.
|
||||
|
||||
### Instructions
|
||||
```sh
|
||||
# Step 1: Install KubeRay operator and CRD
|
||||
helm repo add kuberay https://ray-project.github.io/kuberay-helm/
|
||||
helm repo update
|
||||
helm install kuberay-operator kuberay/kuberay-operator --version 1.6.0
|
||||
|
||||
# Step 2: Install a RayCluster
|
||||
helm install raycluster kuberay/ray-cluster --version 1.6.0
|
||||
|
||||
# Step 3: Edit the `ray-operator/config/samples/ray-cluster-alb-ingress.yaml`
|
||||
#
|
||||
# (1) Annotation `alb.ingress.kubernetes.io/subnets`
|
||||
# 1. Please include at least two subnets.
|
||||
# 2. One Availability Zone (ex: us-west-2a) can only have at most 1 subnet.
|
||||
# 3. In this example, you need to select public subnets (subnets that "Auto-assign public IPv4 address" is Yes on AWS dashboard)
|
||||
#
|
||||
# (2) Set the name of head pod service to `spec...backend.service.name`
|
||||
eksctl get cluster ${YOUR_EKS_CLUSTER} # Check subnets on the EKS cluster
|
||||
|
||||
# Step 4: Check ingress created by Step 4.
|
||||
kubectl describe ingress ray-cluster-ingress
|
||||
|
||||
# [Example]
|
||||
# Name: ray-cluster-ingress
|
||||
# Labels: <none>
|
||||
# Namespace: default
|
||||
# Address: k8s-default-rayclust-....${REGION_CODE}.elb.amazonaws.com
|
||||
# Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
|
||||
# Rules:
|
||||
# Host Path Backends
|
||||
# ---- ---- --------
|
||||
# *
|
||||
# / ray-cluster-kuberay-head-svc:8265 (192.168.185.157:8265)
|
||||
# Annotations: alb.ingress.kubernetes.io/scheme: internal
|
||||
# alb.ingress.kubernetes.io/subnets: ${SUBNET_1},${SUBNET_2}
|
||||
# alb.ingress.kubernetes.io/tags: Environment=dev,Team=test
|
||||
# alb.ingress.kubernetes.io/target-type: ip
|
||||
# Events:
|
||||
# Type Reason Age From Message
|
||||
# ---- ------ ---- ---- -------
|
||||
# Normal SuccessfullyReconciled 39m ingress Successfully reconciled
|
||||
|
||||
# Step 6: Check ALB on AWS (EC2 -> Load Balancing -> Load Balancers)
|
||||
# The name of the ALB should be like "k8s-default-rayclust-......".
|
||||
|
||||
# Step 7: Check Ray Dashboard by ALB DNS Name. The name of the DNS Name should be like
|
||||
# "k8s-default-rayclust-.....us-west-2.elb.amazonaws.com"
|
||||
|
||||
# Step 8: Delete the ingress, and AWS Load Balancer controller will remove ALB.
|
||||
# Check ALB on AWS to make sure it is removed.
|
||||
kubectl delete ingress ray-cluster-ingress
|
||||
```
|
||||
|
||||
(kuberay-gke-ingress)=
|
||||
|
||||
## GKE Ingress support
|
||||
|
||||
### Prerequisites
|
||||
|
||||
* Create a GKE cluster and ensure that you have the kubectl tool installed and authenticated to communicate with your GKE cluster. See [this tutorial](kuberay-gke-gpu-cluster-setup) for an example of how to create a GKE cluster with GPUs. (GPUs are not necessary for this section.)
|
||||
|
||||
* If you are using a `gce-internal` ingress, create a [Proxy-Only subnet](https://docs.cloud.google.com/load-balancing/docs/proxy-only-subnets#proxy_only_subnet_create) in the same region as your GKE cluster.
|
||||
|
||||
* It may be helpful to understand the concepts at <https://docs.cloud.google.com/kubernetes-engine/docs/concepts/ingress>.
|
||||
|
||||
### Instructions
|
||||
Save the following file as `ray-cluster-gclb-ingress.yaml`:
|
||||
|
||||
```yaml
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: ray-cluster-ingress
|
||||
annotations:
|
||||
kubernetes.io/ingress.class: "gce-internal"
|
||||
spec:
|
||||
rules:
|
||||
- http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: raycluster-kuberay-head-svc # Update this line with your head service in Step 3 below.
|
||||
port:
|
||||
number: 8265
|
||||
```
|
||||
|
||||
Now run the following commands:
|
||||
|
||||
```bash
|
||||
# Step 1: Install KubeRay operator and CRD
|
||||
helm repo add kuberay https://ray-project.github.io/kuberay-helm/
|
||||
helm repo update
|
||||
helm install kuberay-operator kuberay/kuberay-operator --version 1.6.0
|
||||
|
||||
# Step 2: Install a RayCluster. GKE Ingress requires the backend service to be of type NodePort.
|
||||
helm install raycluster kuberay/ray-cluster --version 1.6.0 --set service.type=NodePort
|
||||
|
||||
# Step 3: Edit ray-cluster-gclb-ingress.yaml to replace the service name with the name of the head service from the RayCluster. (Output of `kubectl get svc`)
|
||||
|
||||
# Step 4: Apply the Ingress configuration
|
||||
kubectl apply -f ray-cluster-gclb-ingress.yaml
|
||||
|
||||
# Step 5: Check ingress created by Step 4.
|
||||
kubectl describe ingress ray-cluster-ingress
|
||||
|
||||
# Step 6: After a few minutes, GKE allocates an internal IP for the ingress. Check it using:
|
||||
kubectl get ingress ray-cluster-ingress
|
||||
|
||||
# Example output:
|
||||
# NAME CLASS HOSTS ADDRESS PORTS AGE
|
||||
# ray-cluster-ingress gce-internal * 10.0.1.15 80 54m
|
||||
|
||||
# Step 7: Check Ray Dashboard. Since this is an internal Ingress, the IP is only accessible from within the VPC. To access the Ingress from your local machine, you must use a VPN, a proxy, or a VM inside the same VPC network.
|
||||
|
||||
# Step 8: Delete the ingress.
|
||||
kubectl delete ingress ray-cluster-ingress
|
||||
```
|
||||
|
||||
(kuberay-gke-gateway)=
|
||||
## GKE Gateway API support
|
||||
|
||||
### Prerequisites
|
||||
|
||||
* Create a [GKE cluster with Gateway API enabled](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/deploying-gateways#enable-gateway). Ensure that you have the `kubectl` tool installed and authenticated to communicate with your GKE cluster.
|
||||
* Gateway API is enabled by default for GKE Autopilot. For GKE Standard, you may need to enable it. See [Enabling Gateway API](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/deploying-gateways#enable-gateway) for instructions.
|
||||
|
||||
* If you are using the `gke-l7-rilb` Gateway Class for a private internal-only Gateway, create a [Proxy-Only subnet](https://cloud.google.com/load-balancing/docs/proxy-only-subnets#proxy_only_subnet_create) in the same region as your GKE cluster in the VPC network.
|
||||
|
||||
* It may be helpful to understand the concepts at <https://cloud.google.com/kubernetes-engine/docs/concepts/gateway-api>.
|
||||
|
||||
### Instructions
|
||||
Save the following file as `ray-cluster-gke-gateway.yaml`:
|
||||
|
||||
```yaml
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
kind: Gateway
|
||||
metadata:
|
||||
name: ray-cluster-gateway
|
||||
spec:
|
||||
gatewayClassName: gke-l7-rilb # Use "gke-l7-global-external-managed" instead if you want to create a public, external Gateway.
|
||||
listeners:
|
||||
- name: http
|
||||
protocol: HTTP
|
||||
port: 80
|
||||
allowedRoutes:
|
||||
namespaces:
|
||||
from: Same
|
||||
---
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
kind: HTTPRoute
|
||||
metadata:
|
||||
name: ray-cluster-http-route
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: ray-cluster-gateway
|
||||
rules:
|
||||
- backendRefs:
|
||||
- name: raycluster-kuberay-head-svc # Update this line with your head service in Step 3 below.
|
||||
port: 8265
|
||||
```
|
||||
|
||||
```{admonition} Warning
|
||||
:class: warning
|
||||
Exposing the Ray Dashboard provides cluster access, which allows executing arbitrary code. If you configure a public, external Gateway (using `gke-l7-global-external-managed`), ensure that you configure proper authentication and authorization. For details on setting up SSL/TLS, Google Cloud Armor, and Identity-Aware Proxy (IAP), see the Google Cloud guide on [Securing a Gateway](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/secure-gateway).
|
||||
```
|
||||
|
||||
Now run the following commands:
|
||||
|
||||
```bash
|
||||
# Step 1: Install KubeRay operator and CRD
|
||||
helm repo add kuberay https://ray-project.github.io/kuberay-helm/
|
||||
helm repo update
|
||||
helm install kuberay-operator kuberay/kuberay-operator --version 1.6.0
|
||||
|
||||
# Step 2: Install a RayCluster
|
||||
helm install raycluster kuberay/ray-cluster --version 1.6.0
|
||||
|
||||
# Step 3: Edit ray-cluster-gke-gateway.yaml to replace the service name with the name of the head service from the RayCluster. (Output of `kubectl get svc`)
|
||||
|
||||
# Step 4: Apply the Gateway and HTTPRoute configuration
|
||||
kubectl apply -f ray-cluster-gke-gateway.yaml
|
||||
|
||||
# Step 5: Check Gateway created by Step 4.
|
||||
kubectl describe gateway ray-cluster-gateway
|
||||
|
||||
# Step 6: Wait for GKE to allocate an internal IP and program the Gateway.
|
||||
kubectl wait --for=condition=Programmed gateway/ray-cluster-gateway --timeout=5m
|
||||
kubectl get gateway ray-cluster-gateway
|
||||
|
||||
# Example output:
|
||||
# NAME CLASS ADDRESS PROGRAMMED AGE
|
||||
# ray-cluster-gateway gke-l7-rilb 10.0.1.15 True 54m
|
||||
|
||||
# Step 7: Check Ray Dashboard. Since this is an internal Gateway, the IP is only accessible from within the VPC. To access the Gateway from your local machine, you must use a VPN, a proxy, or a VM inside the same VPC network.
|
||||
|
||||
# Step 8: Delete the gateway and HTTPRoute.
|
||||
kubectl delete -f ray-cluster-gke-gateway.yaml
|
||||
```
|
||||
|
||||
```{note}
|
||||
This guide focuses on exposing the Ray Dashboard and API on a single GKE cluster. For deploying multi-cluster Ray serving architectures, see the Google Cloud guide on [serving multi-cluster Ray inference using a Gateway](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/serve-multi-cluster-ray-inference-gateway).
|
||||
```
|
||||
|
||||
(kuberay-nginx)=
|
||||
## Manually setting up NGINX Ingress on Kind
|
||||
|
||||
```sh
|
||||
# Step 1: Create a Kind cluster with `extraPortMappings` and `node-labels`
|
||||
# Reference for the setting up of Kind cluster: https://kind.sigs.k8s.io/docs/user/ingress/
|
||||
cat <<EOF | kind create cluster --config=-
|
||||
kind: Cluster
|
||||
apiVersion: kind.x-k8s.io/v1alpha4
|
||||
nodes:
|
||||
- role: control-plane
|
||||
kubeadmConfigPatches:
|
||||
- |
|
||||
kind: InitConfiguration
|
||||
nodeRegistration:
|
||||
kubeletExtraArgs:
|
||||
node-labels: "ingress-ready=true"
|
||||
extraPortMappings:
|
||||
- containerPort: 80
|
||||
hostPort: 80
|
||||
protocol: TCP
|
||||
- containerPort: 443
|
||||
hostPort: 443
|
||||
protocol: TCP
|
||||
EOF
|
||||
|
||||
# Step 2: Install NGINX ingress controller
|
||||
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml
|
||||
sleep 10 # Wait for the Kubernetes API Server to create the related resources
|
||||
kubectl wait --namespace ingress-nginx \
|
||||
--for=condition=ready pod \
|
||||
--selector=app.kubernetes.io/component=controller \
|
||||
--timeout=90s
|
||||
|
||||
# Step 3: Install KubeRay operator and CRD
|
||||
helm repo add kuberay https://ray-project.github.io/kuberay-helm/
|
||||
helm repo update
|
||||
helm install kuberay-operator kuberay/kuberay-operator --version 1.6.0
|
||||
|
||||
# Step 4: Install RayCluster and create an ingress separately.
|
||||
# More information about change of setting was documented in https://github.com/ray-project/kuberay/pull/699
|
||||
# and `ray-operator/config/samples/ray-cluster.separate-ingress.yaml`
|
||||
curl -LO https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-cluster.separate-ingress.yaml
|
||||
kubectl apply -f ray-cluster.separate-ingress.yaml
|
||||
|
||||
# Step 5: Check the ingress created in Step 4.
|
||||
kubectl describe ingress raycluster-ingress-head-ingress
|
||||
|
||||
# [Example]
|
||||
# ...
|
||||
# Rules:
|
||||
# Host Path Backends
|
||||
# ---- ---- --------
|
||||
# *
|
||||
# /raycluster-ingress/(.*) raycluster-ingress-head-svc:8265 (10.244.0.11:8265)
|
||||
# Annotations: nginx.ingress.kubernetes.io/rewrite-target: /$1
|
||||
|
||||
# Step 6: Check `<ip>/raycluster-ingress/` on your browser. You will see the Ray Dashboard.
|
||||
# [Note] The forward slash at the end of the address is necessary. `<ip>/raycluster-ingress`
|
||||
# will report "404 Not Found".
|
||||
```
|
||||
|
||||
(kuberay-aks-agc)=
|
||||
## Azure Application Gateway for Containers Gateway API support on AKS
|
||||
|
||||
### Prerequisites
|
||||
* Create an AKS cluster. See [Quickstart: Deploy an Azure Kubernetes Service (AKS) cluster using Azure CLI](https://learn.microsoft.com/azure/aks/learn/quick-kubernetes-deploy-cli).
|
||||
|
||||
* Deploy Application Gateway for Containers ALB Controller [Quickstart: Deploy Application Gateway for Containers ALB Controller](https://learn.microsoft.com/azure/application-gateway/for-containers/quickstart-deploy-application-gateway-for-containers-alb-controller?tabs=install-helm-windows).
|
||||
|
||||
* Deploy Application Gateway for Containers [Quickstart: Create Application Gateway for Containers managed by ALB Controller](https://learn.microsoft.com/azure/application-gateway/for-containers/quickstart-create-application-gateway-for-containers-managed-by-alb-controller?tabs=new-subnet-aks-vnet)
|
||||
|
||||
* (Optional) Read [What is Application Gateway for Containers](https://learn.microsoft.com/azure/application-gateway/for-containers/overview).
|
||||
|
||||
* (Optional) Read [Secure your web applications with Azure Web Application Firewall on Application Gateway for Containers](https://learn.microsoft.com/azure/application-gateway/for-containers/web-application-firewall)
|
||||
|
||||
### Instructions
|
||||
```sh
|
||||
# Step 1: Install KubeRay operator and CRD
|
||||
helm repo add kuberay https://ray-project.github.io/kuberay-helm/
|
||||
helm repo update
|
||||
helm install kuberay-operator kuberay/kuberay-operator --version 1.6.0
|
||||
|
||||
# Step 2: Install a RayCluster
|
||||
helm install raycluster kuberay/ray-cluster --version 1.6.0
|
||||
|
||||
# Step 3: Edit the `ray-operator/config/samples/ray-cluster-agc-gatewayapi.yaml`
|
||||
#
|
||||
# (1) Annotation `alb.networking.azure.io/alb-namespace`
|
||||
# 1. Please update this to the namespace of your alb custom resource.
|
||||
#
|
||||
# (2) Annotation `alb.networking.azure.io/alb-name`
|
||||
# 1. Please update this to the name of your alb custom resource.
|
||||
|
||||
# Step 4: Check gateway and http route created by Step 3.
|
||||
kubectl describe gateway ray-cluster-gateway
|
||||
|
||||
# [Example]
|
||||
# Name: ray-cluster-gateway
|
||||
# Namespace: default
|
||||
# Labels: <none>
|
||||
# Annotations:
|
||||
# alb.networking.azure.io/alb-namespace: alb-test-infra
|
||||
# alb.networking.azure.io/alb-name: alb-test
|
||||
# API Version: gateway.networking.k8s.io/v1
|
||||
# Kind: Gateway
|
||||
# Metadata:
|
||||
# Creation Timestamp: 2025-09-12T04:44:18Z
|
||||
# Generation: 1
|
||||
# Resource Version: 247986
|
||||
# UID: 88c40c06-83fe-4ef3-84e1-7bc36c9b5b43
|
||||
# Spec:
|
||||
# Gateway Class Name: azure-alb-external
|
||||
# Listeners:
|
||||
# Allowed Routes:
|
||||
# Namespaces:
|
||||
# From: Same
|
||||
# Name: http
|
||||
# Port: 80
|
||||
# Protocol: HTTP
|
||||
# Status:
|
||||
# Addresses:
|
||||
# Type: Hostname
|
||||
# Value: xxxx.yyyy.alb.azure.com
|
||||
# Conditions:
|
||||
# Last Transition Time: 2025-09-12T04:49:30Z
|
||||
# Message: Valid Gateway
|
||||
# Observed Generation: 1
|
||||
# Reason: Accepted
|
||||
# Status: True
|
||||
# Type: Accepted
|
||||
# Last Transition Time: 2025-09-12T04:49:30Z
|
||||
# Message: Application Gateway for Containers resource has been successfully updated.
|
||||
# Observed Generation: 1
|
||||
# Reason: Programmed
|
||||
# Status: True
|
||||
# Type: Programmed
|
||||
# Listeners:
|
||||
# Attached Routes: 1
|
||||
# Conditions:
|
||||
# Last Transition Time: 2025-09-12T04:49:30Z
|
||||
# Message:
|
||||
# Observed Generation: 1
|
||||
# Reason: ResolvedRefs
|
||||
# Status: True
|
||||
# Type: ResolvedRefs
|
||||
# Last Transition Time: 2025-09-12T04:49:30Z
|
||||
# Message: Listener is Accepted
|
||||
# Observed Generation: 1
|
||||
# Reason: Accepted
|
||||
# Status: True
|
||||
# Type: Accepted
|
||||
# Last Transition Time: 2025-09-12T04:49:30Z
|
||||
# Message: Application Gateway for Containers resource has been successfully updated.
|
||||
# Observed Generation: 1
|
||||
# Reason: Programmed
|
||||
# Status: True
|
||||
# Type: Programmed
|
||||
# Name: http
|
||||
# Supported Kinds:
|
||||
# Group: gateway.networking.k8s.io
|
||||
# Kind: HTTPRoute
|
||||
# Group: gateway.networking.k8s.io
|
||||
# Kind: GRPCRoute
|
||||
# Events: <none>
|
||||
|
||||
kubectl describe httproutes ray-cluster-http-route
|
||||
|
||||
# [Example]
|
||||
# Name: ray-cluster-http-route
|
||||
# Namespace: default
|
||||
# Labels: <none>
|
||||
# Annotations: <none>
|
||||
# API Version: gateway.networking.k8s.io/v1
|
||||
# Kind: HTTPRoute
|
||||
# Metadata:
|
||||
# Creation Timestamp: 2025-09-12T04:44:43Z
|
||||
# Generation: 2
|
||||
# Resource Version: 247982
|
||||
# UID: 54bbd1e6-bd28-4cae-a469-e15105f077b8
|
||||
# Spec:
|
||||
# Parent Refs:
|
||||
# Group: gateway.networking.k8s.io
|
||||
# Kind: Gateway
|
||||
# Name: ray-cluster-gateway
|
||||
# Rules:
|
||||
# Backend Refs:
|
||||
# Group:
|
||||
# Kind: Service
|
||||
# Name: raycluster-kuberay-head-svc
|
||||
# Port: 8265
|
||||
# Weight: 1
|
||||
# Matches:
|
||||
# Path:
|
||||
# Type: PathPrefix
|
||||
# Value: /
|
||||
# Status:
|
||||
# Parents:
|
||||
# Conditions:
|
||||
# Last Transition Time: 2025-09-12T04:49:30Z
|
||||
# Message:
|
||||
# Observed Generation: 2
|
||||
# Reason: ResolvedRefs
|
||||
# Status: True
|
||||
# Type: ResolvedRefs
|
||||
# Last Transition Time: 2025-09-12T04:49:30Z
|
||||
# Message: Route is Accepted
|
||||
# Observed Generation: 2
|
||||
# Reason: Accepted
|
||||
# Status: True
|
||||
# Type: Accepted
|
||||
# Last Transition Time: 2025-09-12T04:49:30Z
|
||||
# Message: Application Gateway for Containers resource has been successfully updated.
|
||||
# Observed Generation: 2
|
||||
# Reason: Programmed
|
||||
# Status: True
|
||||
# Type: Programmed
|
||||
# Controller Name: alb.networking.azure.io/alb-controller
|
||||
# Parent Ref:
|
||||
# Group: gateway.networking.k8s.io
|
||||
# Kind: Gateway
|
||||
# Name: ray-cluster-gateway
|
||||
# Events: <none>
|
||||
|
||||
# Step 5: Check Ray Dashboard by visiting the FQDN assigned to your gateway object in your browser
|
||||
# FQDN can be obtained by the command:
|
||||
# kubectl get gateway ray-cluster-gateway -o jsonpath='{.status.addresses[0].value}'
|
||||
|
||||
# Step 6: Delete the gateway and http route
|
||||
kubectl delete gateway ray-cluster-gateway
|
||||
kubectl delete httproutes ray-cluster-http-route
|
||||
|
||||
# Step 7: Delete Application Gateway for containers
|
||||
kubectl delete applicationloadbalancer alb-test -n alb-test-infra
|
||||
kubectl delete ns alb-test-infra
|
||||
```
|
||||
@@ -0,0 +1,272 @@
|
||||
(kuberay-istio)=
|
||||
# mTLS and L7 observability with Istio
|
||||
|
||||
This integration guide for KubeRay and Istio enables mTLS and L7 traffic observability in a RayCluster on a local Kind cluster.
|
||||
|
||||
## Istio
|
||||
|
||||
[Istio](https://istio.io/) is an open-source service mesh that provides a uniform and more efficient way to secure, connect, and monitor services. Some features of its powerful control plane include:
|
||||
* Secure network traffic in a Kubernetes cluster with TLS encryption.
|
||||
* Automatic metrics, logs, and traces for all traffic within a cluster.
|
||||
|
||||
See the [Istio documentation](https://istio.io/latest/docs/) to learn more.
|
||||
|
||||
## Step 0: Create a Kind cluster
|
||||
|
||||
Create a Kind cluster with the following command:
|
||||
|
||||
```bash
|
||||
kind create cluster
|
||||
```
|
||||
|
||||
## Step 1: Install Istio
|
||||
|
||||
```bash
|
||||
# Download Istioctl and its manifests.
|
||||
export ISTIO_VERSION=1.21.1
|
||||
curl -L https://istio.io/downloadIstio | sh -
|
||||
cd istio-1.21.1
|
||||
export PATH=$PWD/bin:$PATH
|
||||
|
||||
# Install Istio with:
|
||||
# 1. 100% trace sampling for demo purposes.
|
||||
# 2. "sanitize_te" disabled for proper gRPC interception. This is required by Istio 1.21.0 (https://github.com/istio/istio/issues/49685).
|
||||
# 3. TLS 1.3 enabled.
|
||||
istioctl install -y -f - <<EOF
|
||||
apiVersion: install.istio.io/v1alpha1
|
||||
kind: IstioOperator
|
||||
spec:
|
||||
meshConfig:
|
||||
defaultConfig:
|
||||
tracing:
|
||||
sampling: 100
|
||||
runtimeValues:
|
||||
envoy.reloadable_features.sanitize_te: "false"
|
||||
meshMTLS:
|
||||
minProtocolVersion: TLSV1_3
|
||||
EOF
|
||||
|
||||
# Install Istio addons, including the Kiali and Jaeger dashboards.
|
||||
kubectl apply -f samples/addons
|
||||
# Enable the Istio sidecar auto injection.
|
||||
kubectl label namespace default istio-injection=enabled
|
||||
```
|
||||
|
||||
See [Istio Getting Started](https://istio.io/latest/docs/setup/getting-started/) for more details on installing Istio.
|
||||
|
||||
## Step 2: Install the KubeRay operator
|
||||
|
||||
Follow [Deploy a KubeRay operator](kuberay-operator-deploy) to install the latest stable KubeRay operator from the Helm repository.
|
||||
|
||||
## Step 3: (Optional) Enable Istio mTLS STRICT mode
|
||||
|
||||
This optional step enables Istio mTLS in STRICT mode, which provides the best security of service mesh by rejecting all undefined traffic.
|
||||
|
||||
In this mode, you _must_ disable the KubeRay init container injection by setting `ENABLE_INIT_CONTAINER_INJECTION=false` on the KubeRay controller. This setting is necessary because the init container starts before the `istio-proxy`, resulting in the rejection of all of its network traffic in STRICT mode.
|
||||
|
||||
```bash
|
||||
# Set ENABLE_INIT_CONTAINER_INJECTION=false on the KubeRay operator.
|
||||
helm upgrade kuberay-operator kuberay/kuberay-operator --version 1.6.0 \
|
||||
--set env\[0\].name=ENABLE_INIT_CONTAINER_INJECTION \
|
||||
--set-string env\[0\].value=false
|
||||
|
||||
# Apply mTLS STRICT mode on Istio.
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: security.istio.io/v1beta1
|
||||
kind: PeerAuthentication
|
||||
metadata:
|
||||
name: "default"
|
||||
namespace: "default"
|
||||
spec:
|
||||
mtls:
|
||||
mode: STRICT
|
||||
EOF
|
||||
```
|
||||
|
||||
See [Istio Mutual TLS Migration](https://istio.io/latest/docs/tasks/security/authentication/mtls-migration/) for more information about STRICT mode.
|
||||
|
||||
## Step 4: Apply a Headless service for the upcoming RayCluster
|
||||
|
||||
To let Istio learn the L7 information of the upcoming RayCluster, you must apply a Headless service for it.
|
||||
|
||||
```bash
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
labels:
|
||||
ray.io/headless-worker-svc: raycluster-istio
|
||||
name: raycluster-istio-headless-svc
|
||||
namespace: default
|
||||
spec:
|
||||
clusterIP: None
|
||||
selector:
|
||||
ray.io/cluster: raycluster-istio
|
||||
publishNotReadyAddresses: true
|
||||
ports:
|
||||
- name: node-manager-port
|
||||
port: 6380
|
||||
appProtocol: grpc
|
||||
- name: object-manager-port
|
||||
port: 6381
|
||||
appProtocol: grpc
|
||||
- name: runtime-env-agent-port
|
||||
port: 6382
|
||||
appProtocol: grpc
|
||||
- name: dashboard-agent-grpc-port
|
||||
port: 6383
|
||||
appProtocol: grpc
|
||||
- name: dashboard-agent-listen-port
|
||||
port: 52365
|
||||
appProtocol: http
|
||||
- name: metrics-export-port
|
||||
port: 8080
|
||||
appProtocol: http
|
||||
- name: p10002
|
||||
port: 10002
|
||||
appProtocol: grpc
|
||||
- name: p10003
|
||||
port: 10003
|
||||
appProtocol: grpc
|
||||
- name: p10004
|
||||
port: 10004
|
||||
appProtocol: grpc
|
||||
- name: p10005
|
||||
port: 10005
|
||||
appProtocol: grpc
|
||||
- name: p10006
|
||||
port: 10006
|
||||
appProtocol: grpc
|
||||
- name: p10007
|
||||
port: 10007
|
||||
appProtocol: grpc
|
||||
- name: p10008
|
||||
port: 10008
|
||||
appProtocol: grpc
|
||||
- name: p10009
|
||||
port: 10009
|
||||
appProtocol: grpc
|
||||
- name: p10010
|
||||
port: 10010
|
||||
appProtocol: grpc
|
||||
- name: p10011
|
||||
port: 10011
|
||||
appProtocol: grpc
|
||||
- name: p10012
|
||||
port: 10012
|
||||
appProtocol: grpc
|
||||
EOF
|
||||
```
|
||||
|
||||
Note that this Headless Service manifest _must_ list _all_ the ports used by Ray explicitly, including _all_ worker ports. See [Configuring Ray](https://docs.ray.io/en/latest/ray-core/configure.html#all-nodes) for more details on the ports required by Ray.
|
||||
|
||||
:::{note}
|
||||
Kubernetes Service doesn't support specifying ports in ranges. You _must_ set them one by one.
|
||||
:::
|
||||
|
||||
:::{warning}
|
||||
The default Ray worker port range, from 10002 to 19999, is too large to specify in the service manifest and can cause memory issues in Kubernetes. Set a smaller `max-worker-port` to work with Istio. Note that by default every sidecar in the service mesh caches these ports, which could lead to sidecar OOMs if you create too many headless services.
|
||||
:::
|
||||
|
||||
## Step 4: Create the RayCluster
|
||||
|
||||
The upcoming RayCluster _must_ use exactly the same ports listed in the previous Headless Service, including the `max-worker-port`. In addition, the `node-ip-address` _must_ be set to the Pod FQDN of the Headless Service to enable Istio L7 observability.
|
||||
|
||||
```bash
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: ray.io/v1
|
||||
kind: RayCluster
|
||||
metadata:
|
||||
name: raycluster-istio
|
||||
spec:
|
||||
rayVersion: '2.46.0'
|
||||
headGroupSpec:
|
||||
rayStartParams:
|
||||
num-cpus: '1'
|
||||
node-manager-port: '6380'
|
||||
object-manager-port: '6381'
|
||||
runtime-env-agent-port: '6382'
|
||||
dashboard-agent-grpc-port: '6383'
|
||||
dashboard-agent-listen-port: '52365'
|
||||
metrics-export-port: '8080'
|
||||
max-worker-port: '10012'
|
||||
node-ip-address: \$(hostname -I | tr -d ' ' | sed 's/\./-/g').raycluster-istio-headless-svc.default.svc.cluster.local
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: ray-head
|
||||
image: rayproject/ray:2.46.0
|
||||
workerGroupSpecs:
|
||||
- replicas: 1
|
||||
minReplicas: 1
|
||||
maxReplicas: 1
|
||||
groupName: small-group
|
||||
rayStartParams:
|
||||
num-cpus: '1'
|
||||
node-manager-port: '6380'
|
||||
object-manager-port: '6381'
|
||||
runtime-env-agent-port: '6382'
|
||||
dashboard-agent-grpc-port: '6383'
|
||||
dashboard-agent-listen-port: '52365'
|
||||
metrics-export-port: '8080'
|
||||
max-worker-port: '10012'
|
||||
node-ip-address: \$(hostname -I | tr -d ' ' | sed 's/\./-/g').raycluster-istio-headless-svc.default.svc.cluster.local
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: ray-worker
|
||||
image: rayproject/ray:2.46.0
|
||||
EOF
|
||||
```
|
||||
|
||||
:::{note}
|
||||
The Pod FQDN of the Headless service should be in the format of `pod-ipv4-address.service.namespace.svc.zone.` or the format of `pod-hostname.service.namespace.svc.zone.` depending on your implementation of the [Kubernetes DNS specification](https://github.com/kubernetes/dns/blob/master/docs/specification.md).
|
||||
:::
|
||||
|
||||
## Step 5: Run your Ray app to generate traffic
|
||||
|
||||
After the RayCluster is ready, use the following script to generate internal traffic for visualization.
|
||||
|
||||
```bash
|
||||
export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)
|
||||
kubectl exec -it $HEAD_POD -- python -c "import ray; ray.get([ray.remote(lambda x: print(x)).remote(i) for i in range(5000)])"
|
||||
```
|
||||
|
||||
## Step 6: Verify the auto mTLS and L7 observability
|
||||
|
||||
Run the following command to start the Kiali dashboard:
|
||||
|
||||
```bash
|
||||
istioctl dashboard kiali
|
||||
```
|
||||
|
||||
Go to the `raycluster-istio` workload at: [http://localhost:20001/kiali/console/namespaces/default/workloads/raycluster-istio?duration=60&refresh=60000&tab=info](http://localhost:20001/kiali/console/namespaces/default/workloads/raycluster-istio?duration=60&refresh=60000&tab=info)
|
||||
|
||||
|
||||
|
||||
Go to the `Traffic` tab. You can see that mTLS protects all of the traffic.
|
||||
|
||||
|
||||
|
||||
Run the following command to start the Jaeger dashboard:
|
||||
|
||||
```bash
|
||||
istioctl dashboard jaeger
|
||||
```
|
||||
|
||||
<!-- TODO: Change the following link to markdown syntax after this issue https://github.com/executablebooks/MyST-Parser/issues/760 is resolved -->
|
||||
Go to the Jaeger dashboard with the `service=raycluster-istio.default` query: <a class="reference external" href="http://localhost:16686/jaeger/search?limit=1000&lookback=1h&maxDuration&minDuration&service=raycluster-istio.default">http://localhost:16686/jaeger/search?limit=1000&lookback=1h&maxDuration&minDuration&service=raycluster-istio.default</a>
|
||||
|
||||
|
||||
|
||||
You can click on any trace of the internal gRPC calls and view their details, such as `grpc.path` and `status code`.
|
||||
|
||||
|
||||
|
||||
## Step 7: Clean up
|
||||
|
||||
Run the following command to delete your cluster.
|
||||
|
||||
```bash
|
||||
kind delete cluster
|
||||
```
|
||||
@@ -0,0 +1,244 @@
|
||||
(kuberay-kai-scheduler)=
|
||||
# Gang scheduling, queue priority, and GPU sharing for RayClusters using KAI Scheduler
|
||||
|
||||
This guide demonstrates how to use KAI Scheduler for setting up hierarchical queues with quotas, gang scheduling, and GPU sharing using RayClusters.
|
||||
|
||||
|
||||
## KAI Scheduler
|
||||
|
||||
[KAI Scheduler](https://github.com/NVIDIA/KAI-Scheduler) is a high-performance, scalable Kubernetes scheduler built for AI/ML workloads. Designed to orchestrate GPU clusters at massive scale, KAI optimizes GPU allocation and supports the full AI lifecycle - from interactive development to large distributed training and inference. Some of the key features are:
|
||||
- **Bin packing and spread scheduling**: Optimize node usage either by minimizing fragmentation using bin packing or increasing resiliency and load balancing using spread scheduling.
|
||||
- **GPU sharing**: Allow KAI to consolidate multiple Ray workloads from across teams on the same GPU, letting your organization fit more work onto your existing hardware and reducing idle GPU time.
|
||||
- **Workload autoscaling**: Scale Ray replicas or workers within min/max while respecting gang constraints
|
||||
- **Cluster autoscaling**: Compatible with dynamic cloud infrastructures (including auto-scalers like Karpenter)
|
||||
- **Workload priorities**: Prioritize Ray workloads effectively within queues
|
||||
- **Hierarchical queues and fairness**: Two-level queues with quotas, over-quota weights, limits, and equitable resource distribution between queues using DRF and many more. For more details and key features, see [the documentation](https://github.com/NVIDIA/KAI-Scheduler?tab=readme-ov-file#key-features).
|
||||
|
||||
### Core components
|
||||
|
||||
1. **PodGroups**: PodGroups are atomic units for scheduling and represent one or more interdependent pods that the scheduler execute as a single unit, also known as gang scheduling. They're vital for distributed workloads. KAI Scheduler includes a **PodGrouper** that handles gang scheduling automatically.
|
||||
|
||||
**How PodGrouper works:**
|
||||
```
|
||||
RayCluster "distributed-training":
|
||||
├── Head Pod: 1 GPU
|
||||
└── Worker Group: 4 × 0.5 GPU = 2 GPUs
|
||||
Total Group Requirement: 3 GPUs
|
||||
|
||||
PodGrouper schedules all 5 pods (1 head + 4 workers) together or none at all.
|
||||
```
|
||||
|
||||
2. **Queues**: Queues enforce fairness in resource distribution using:
|
||||
|
||||
- Quota: The baseline amount of resources guaranteed to the queue. The scheduler allocates quotas first to ensure fairness.
|
||||
- Queue priority: Determines the order in which queues receive resources beyond their quota. The scheduler serves the higher-priority queues first.
|
||||
- Over-quota weight: Controls how the scheduler divides surplus resources among queues within the same priority level. Queues with higher weights receive a larger share of the extra resources.
|
||||
- Limit: Defines the maximum resources that the queue can consume.
|
||||
|
||||
You can arrange queues hierarchically for organizations with multiple teams, for example, departments with multiple teams.
|
||||
|
||||
## [Prerequisites](https://github.com/NVIDIA/KAI-Scheduler?tab=readme-ov-file#prerequisites)
|
||||
|
||||
* Kubernetes cluster with GPU nodes
|
||||
* NVIDIA GPU Operator
|
||||
* kubectl configured to access your cluster
|
||||
* Install KAI Scheduler with GPU-sharing enabled. Choose the desired release version from [KAI Scheduler releases](https://github.com/NVIDIA/KAI-Scheduler/releases) and replace the `<KAI_SCHEDULER_VERSION>` in the following command. It's recommended to choose v0.10.0 or higher version.
|
||||
|
||||
```bash
|
||||
# Install KAI Scheduler
|
||||
helm upgrade -i kai-scheduler oci://ghcr.io/nvidia/kai-scheduler/kai-scheduler -n kai-scheduler --create-namespace --version <KAI_SCHEDULER_VERSION> --set "global.gpuSharing=true"
|
||||
```
|
||||
|
||||
## Step 1: Install the KubeRay operator with KAI Scheduler as the batch scheduler
|
||||
|
||||
Follow the official KubeRay operator [installation documentation](https://docs.ray.io/en/master/cluster/kubernetes/getting-started/kuberay-operator-installation.html#kuberay-operator-installation) and add the following configuration to enable KAI Scheduler integration:
|
||||
|
||||
```bash
|
||||
--set batchScheduler.name=kai-scheduler
|
||||
```
|
||||
|
||||
## Step 2: Create KAI Scheduler Queues
|
||||
|
||||
Create a basic queue structure for department-1 and its child team-a. For demo reasons, this example doesn't enforce any quota, overQuotaWeight, or limit. You can configure these parameters depending on your needs:
|
||||
|
||||
```yaml
|
||||
apiVersion: scheduling.run.ai/v2
|
||||
kind: Queue
|
||||
metadata:
|
||||
name: department-1
|
||||
spec:
|
||||
#priority: 100 (optional)
|
||||
resources:
|
||||
cpu:
|
||||
quota: -1
|
||||
limit: -1
|
||||
overQuotaWeight: 1
|
||||
gpu:
|
||||
quota: -1
|
||||
limit: -1
|
||||
overQuotaWeight: 1
|
||||
memory:
|
||||
quota: -1
|
||||
limit: -1
|
||||
overQuotaWeight: 1
|
||||
---
|
||||
apiVersion: scheduling.run.ai/v2
|
||||
kind: Queue
|
||||
metadata:
|
||||
name: team-a
|
||||
spec:
|
||||
#priority: 200 (optional)
|
||||
parentQueue: department-1
|
||||
resources:
|
||||
cpu:
|
||||
quota: -1
|
||||
limit: -1
|
||||
overQuotaWeight: 1
|
||||
gpu:
|
||||
quota: -1
|
||||
limit: -1
|
||||
overQuotaWeight: 1
|
||||
memory:
|
||||
quota: -1
|
||||
limit: -1
|
||||
overQuotaWeight: 1
|
||||
|
||||
```
|
||||
|
||||
Note: To make this demo easier to follow, it combined these queue definitions with the RayCluster example in the next step. You can use the single combined YAML file and apply both queues and workloads at once.
|
||||
|
||||
## Step 3: Gang scheduling with KAI Scheduler
|
||||
|
||||
The key pattern is to add the queue label to your RayCluster. [Here's a basic example](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.kai-scheduler.yaml) from the KubeRay repository:
|
||||
|
||||
```yaml
|
||||
metadata:
|
||||
name: raycluster-sample
|
||||
labels:
|
||||
kai.scheduler/queue: team-a # This is the essential configuration.
|
||||
```
|
||||
|
||||
Apply this RayCluster with queues:
|
||||
|
||||
```bash
|
||||
curl -LO https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-cluster.kai-scheduler.yaml
|
||||
|
||||
kubectl apply -f ray-cluster.kai-scheduler.yaml
|
||||
|
||||
#Verify queues are created
|
||||
kubectl get queues
|
||||
# NAME PRIORITY PARENT CHILDREN DISPLAYNAME
|
||||
# department-1 ["team-a"]
|
||||
# team-a department-1
|
||||
|
||||
# Watch the pods get scheduled
|
||||
kubectl get pods -w
|
||||
# NAME READY STATUS RESTARTS AGE
|
||||
# kuberay-operator-7d86f4f46b-dq22x 1/1 Running 0 50s
|
||||
# raycluster-sample-head-rvrkz 0/1 ContainerCreating 0 13s
|
||||
# raycluster-sample-worker-worker-mlvtz 0/1 Init:0/1 0 13s
|
||||
# raycluster-sample-worker-worker-rcb54 0/1 Init:0/1 0 13s
|
||||
# raycluster-sample-worker-worker-mlvtz 0/1 Init:0/1 0 40s
|
||||
# raycluster-sample-worker-worker-rcb54 0/1 Init:0/1 0 41s
|
||||
# raycluster-sample-head-rvrkz 0/1 Running 0 42s
|
||||
# raycluster-sample-head-rvrkz 1/1 Running 0 54s
|
||||
# raycluster-sample-worker-worker-rcb54 0/1 PodInitializing 0 59s
|
||||
# raycluster-sample-worker-worker-mlvtz 0/1 PodInitializing 0 59s
|
||||
# raycluster-sample-worker-worker-rcb54 0/1 Running 0 60s
|
||||
# raycluster-sample-worker-worker-mlvtz 0/1 Running 0 60s
|
||||
# raycluster-sample-worker-worker-rcb54 1/1 Running 0 71s
|
||||
# raycluster-sample-worker-worker-mlvtz 1/1 Running 0 71s
|
||||
```
|
||||
Note: Starting from KubeRay v1.6, KAI Scheduler supports **RayCluster**, **RayService**, and **RayJob with SidecarMode, HTTPMode, or InteractiveMode**. The following mode is not supported:
|
||||
|
||||
- **K8sJobMode**: The submitter pod is created after RayCluster is ready, preventing proper gang scheduling.
|
||||
|
||||
## Set priorities for workloads
|
||||
|
||||
In Kubernetes, assigning different priorities to workloads ensures efficient resource management, minimizes service disruption, and supports better scaling. By prioritizing workloads, KAI Scheduler schedules jobs according to their assigned priority. When sufficient resources aren't available for a workload, the scheduler can preempt lower-priority workloads to free up resources for higher-priority ones. This approach ensures the scheduler always prioritizes that mission-critical services in resource allocation.
|
||||
|
||||
KAI scheduler deployment comes with several predefined priority classes:
|
||||
|
||||
- train (50) - use for preemptible training workloads
|
||||
- build-preemptible (75) - use for preemptible build/interactive workloads
|
||||
- build (100) - use for build/interactive workloads (non-preemptible)
|
||||
- inference (125) - use for inference workloads (non-preemptible)
|
||||
|
||||
You can submit the same workload preceding with a specific priority. Modify the preceding example into a build class workload:
|
||||
|
||||
```yaml
|
||||
labels:
|
||||
kai.scheduler/queue: team-a # This is the essential configuration.
|
||||
priorityClassName: build # Here you can specify the priority class in metadata.labels (optional)
|
||||
```
|
||||
See the [documentation](https://github.com/NVIDIA/KAI-Scheduler/tree/main/docs/priority) for more information.
|
||||
|
||||
## Step 4: Submitting Ray workers with GPU sharing
|
||||
|
||||
This example creates two workers that share a single GPU, 0.5 each with time-slicing, within a RayCluster. See the [YAML file](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.kai-gpu-sharing.yaml)):
|
||||
|
||||
```bash
|
||||
curl -LO https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-cluster.kai-gpu-sharing.yaml
|
||||
|
||||
kubectl apply -f ray-cluster.kai-gpu-sharing.yaml
|
||||
|
||||
# Watch the pods get scheduled
|
||||
kubectl get pods -w
|
||||
# NAME READY STATUS RESTARTS AGE
|
||||
# kuberay-operator-7d86f4f46b-dq22x 1/1 Running 0 4m9s
|
||||
# raycluster-half-gpu-head-9rtxf 0/1 Running 0 4s
|
||||
# raycluster-half-gpu-shared-gpu-worker-5l7cn 0/1 Pending 0 4s
|
||||
# raycluster-half-gpu-shared-gpu-worker-98tzh 0/1 Pending 0 4s
|
||||
# ... (skip for brevity)
|
||||
# raycluster-half-gpu-shared-gpu-worker-5l7cn 0/1 Init:0/1 0 6s
|
||||
# raycluster-half-gpu-shared-gpu-worker-5l7cn 0/1 Init:0/1 0 7s
|
||||
# raycluster-half-gpu-shared-gpu-worker-98tzh 0/1 Init:0/1 0 8s
|
||||
# raycluster-half-gpu-head-9rtxf 1/1 Running 0 19s
|
||||
# raycluster-half-gpu-shared-gpu-worker-5l7cn 0/1 PodInitializing 0 19s
|
||||
# raycluster-half-gpu-shared-gpu-worker-98tzh 0/1 PodInitializing 0 19s
|
||||
# raycluster-half-gpu-shared-gpu-worker-5l7cn 0/1 Running 0 20s
|
||||
# raycluster-half-gpu-shared-gpu-worker-98tzh 0/1 Running 0 20s
|
||||
# raycluster-half-gpu-shared-gpu-worker-5l7cn 1/1 Running 0 31s
|
||||
# raycluster-half-gpu-shared-gpu-worker-98tzh 1/1 Running 0 31s
|
||||
```
|
||||
|
||||
Note: GPU sharing with time slicing in this example occurs only at the Kubernetes layer, allowing multiple pods to share a single GPU device. The scheduler doesn't enforce memory isolation, so applications must manage their own usage to prevent interference. For other GPU sharing approaches, for example, MPS, see [the KAI documentation](https://github.com/NVIDIA/KAI-Scheduler/tree/main/docs/gpu-sharing).
|
||||
|
||||
### Verify GPU sharing is working
|
||||
|
||||
To confirm that GPU sharing is working correctly, use these commands:
|
||||
|
||||
```bash
|
||||
# 1. Check GPU fraction annotations and shared GPU groups
|
||||
kubectl get pods -l ray.io/cluster=raycluster-half-gpu -o custom-columns="NAME:.metadata.name,NODE:.spec.nodeName,GPU-FRACTION:.metadata.annotations.gpu-fraction,GPU-GROUP:.metadata.labels.runai-gpu-group"
|
||||
```
|
||||
|
||||
You should see both worker pods on the same node with `GPU-FRACTION: 0.5` and the same `GPU-GROUP` ID:
|
||||
|
||||
```bash
|
||||
NAME NODE GPU-FRACTION GPU-GROUP
|
||||
raycluster-half-gpu-head ip-xxx-xx-xx-xxx <none> <none>
|
||||
raycluster-half-gpu-shared-gpu-worker-67tvw ip-xxx-xx-xx-xxx 0.5 3e456911-a6ea-4b1a-8f55-e90fba89ad76
|
||||
raycluster-half-gpu-shared-gpu-worker-v5tpp ip-xxx-xx-xx-xxx 0.5 3e456911-a6ea-4b1a-8f55-e90fba89ad76
|
||||
```
|
||||
|
||||
This shows that both workers have the same `NVIDIA_VISIBLE_DEVICES` (same physical GPU) and `GPU-FRACTION: 0.50`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Check for missing queue labels
|
||||
|
||||
If pods remain in `Pending` state, the most common issue is missing queue labels.
|
||||
|
||||
Check operator logs for KAI Scheduler errors and look for error messages like:
|
||||
|
||||
```bash
|
||||
"Queue label missing from RayCluster; pods will remain pending"
|
||||
```
|
||||
**Solution**: Ensure your RayCluster has the queue label that exists in the cluster:
|
||||
|
||||
```yaml
|
||||
metadata:
|
||||
labels:
|
||||
kai.scheduler/queue: default # Add this label
|
||||
```
|
||||
@@ -0,0 +1,745 @@
|
||||
(kuberay-kueue)=
|
||||
# Gang scheduling, Priority scheduling, and Autoscaling for KubeRay CRDs with Kueue
|
||||
|
||||
This guide demonstrates how to integrate KubeRay with [Kueue](https://kueue.sigs.k8s.io/) to enable advanced scheduling capabilities including gang scheduling and priority scheduling for Ray applications on Kubernetes.
|
||||
|
||||
For real-world use cases with RayJob, see [Priority Scheduling with RayJob and Kueue](kuberay-kueue-priority-scheduling-example) and [Gang Scheduling with RayJob and Kueue](kuberay-kueue-gang-scheduling-example).
|
||||
|
||||
## What's Kueue?
|
||||
[Kueue](https://kueue.sigs.k8s.io/) is a Kubernetes-native job queueing system that manages resource quotas and job lifecycle. Kueue decides when:
|
||||
* To make a job wait.
|
||||
* To admit a job to start, which triggers Kubernetes to create pods.
|
||||
* To preempt a job, which triggers Kubernetes to delete active pods.
|
||||
|
||||
## Supported KubeRay CRDs
|
||||
Kueue has native support for the following KubeRay APIs:
|
||||
- **RayJob**: Ideal for batch processing and model training workloads (covered in this guide)
|
||||
- **RayCluster**: Perfect for managing long-running Ray clusters
|
||||
- **RayService**: Designed for serving models and applications
|
||||
|
||||
*Note: This guide focuses on a detailed RayJob example on a kind cluster. For RayCluster and RayService examples, see the ["Working with RayCluster and RayService"](#working-with-raycluster-and-rayservice) section.*
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before you begin, ensure you have a Kubernetes cluster. This guide uses a local Kind cluster.
|
||||
|
||||
## Step 0: Create a Kind cluster
|
||||
|
||||
```bash
|
||||
kind create cluster
|
||||
```
|
||||
|
||||
## Step 1: Install the KubeRay operator
|
||||
|
||||
Follow [Deploy a KubeRay operator](kuberay-operator-deploy) to install the latest stable KubeRay operator from the Helm repository.
|
||||
|
||||
## Step 2: Install Kueue
|
||||
|
||||
```bash
|
||||
VERSION=v0.13.4
|
||||
kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/$VERSION/manifests.yaml
|
||||
```
|
||||
|
||||
See [Kueue Installation](https://kueue.sigs.k8s.io/docs/installation/#install-a-released-version) for more details on installing Kueue. **Note**: Some limitations exist between Kueue and RayJob. See the [limitations of Kueue](https://kueue.sigs.k8s.io/docs/tasks/run_rayjobs/#c-limitations) for more details.
|
||||
|
||||
|
||||
## Step 3: Create Kueue Resources
|
||||
|
||||
This manifest creates the necessary Kueue resources to manage scheduling and resource allocation.
|
||||
|
||||
```yaml
|
||||
# kueue-resources.yaml
|
||||
apiVersion: kueue.x-k8s.io/v1beta1
|
||||
kind: ResourceFlavor
|
||||
metadata:
|
||||
name: "default-flavor"
|
||||
---
|
||||
apiVersion: kueue.x-k8s.io/v1beta1
|
||||
kind: ClusterQueue
|
||||
metadata:
|
||||
name: "cluster-queue"
|
||||
spec:
|
||||
preemption:
|
||||
withinClusterQueue: LowerPriority
|
||||
namespaceSelector: {} # Match all namespaces.
|
||||
resourceGroups:
|
||||
- coveredResources: ["cpu", "memory"]
|
||||
flavors:
|
||||
- name: "default-flavor"
|
||||
resources:
|
||||
- name: "cpu"
|
||||
nominalQuota: 3
|
||||
- name: "memory"
|
||||
nominalQuota: 6G
|
||||
---
|
||||
apiVersion: kueue.x-k8s.io/v1beta1
|
||||
kind: LocalQueue
|
||||
metadata:
|
||||
namespace: "default"
|
||||
name: "user-queue"
|
||||
spec:
|
||||
clusterQueue: "cluster-queue"
|
||||
---
|
||||
apiVersion: kueue.x-k8s.io/v1beta1
|
||||
kind: WorkloadPriorityClass
|
||||
metadata:
|
||||
name: prod-priority
|
||||
value: 1000
|
||||
description: "Priority class for prod jobs"
|
||||
---
|
||||
apiVersion: kueue.x-k8s.io/v1beta1
|
||||
kind: WorkloadPriorityClass
|
||||
metadata:
|
||||
name: dev-priority
|
||||
value: 100
|
||||
description: "Priority class for development jobs"
|
||||
```
|
||||
|
||||
The YAML manifest configures:
|
||||
|
||||
* **ResourceFlavor**
|
||||
* The ResourceFlavor `default-flavor` is an empty ResourceFlavor because the compute resources in the Kubernetes cluster are homogeneous. In other words, users can request 1 CPU without considering whether it's an ARM chip or an x86 chip.
|
||||
* **ClusterQueue**
|
||||
* The ClusterQueue `cluster-queue` only has 1 ResourceFlavor `default-flavor` with quotas for 3 CPUs and 6G memory.
|
||||
* The ClusterQueue `cluster-queue` has a preemption policy `withinClusterQueue: LowerPriority`. This policy allows the pending RayJob that doesn’t fit within the nominal quota for its ClusterQueue to preempt active RayJob custom resources in the ClusterQueue that have lower priority.
|
||||
* **LocalQueue**
|
||||
* The LocalQueue `user-queue` is a namespace-scoped object in the `default` namespace which belongs to a ClusterQueue. A typical practice is to assign a namespace to a tenant, team, or user of an organization. Users submit jobs to a LocalQueue, instead of to a ClusterQueue directly.
|
||||
* **WorkloadPriorityClass**
|
||||
* The WorkloadPriorityClass `prod-priority` has a higher value than the WorkloadPriorityClass `dev-priority`. RayJob custom resources with the `prod-priority` priority class take precedence over RayJob custom resources with the `dev-priority` priority class.
|
||||
|
||||
Create the Kueue resources:
|
||||
```bash
|
||||
kubectl apply -f kueue-resources.yaml
|
||||
```
|
||||
|
||||
## Step 4: Gang scheduling with Kueue
|
||||
|
||||
Kueue always admits workloads in “gang” mode. Kueue admits workloads on an “all or nothing” basis, ensuring that Kubernetes never partially provisions a RayJob or RayCluster. Use gang scheduling strategy to avoid wasting compute resources caused by inefficient scheduling of workloads.
|
||||
|
||||
Download the RayJob YAML manifest from the KubeRay repository.
|
||||
|
||||
```bash
|
||||
curl -LO https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-job.kueue-toy-sample.yaml
|
||||
```
|
||||
|
||||
Before creating the RayJob, modify the RayJob metadata with the following:
|
||||
|
||||
```yaml
|
||||
metadata:
|
||||
generateName: rayjob-sample-
|
||||
labels:
|
||||
kueue.x-k8s.io/queue-name: user-queue
|
||||
kueue.x-k8s.io/priority-class: dev-priority
|
||||
```
|
||||
|
||||
Create two RayJob custom resources with the same priority `dev-priority`. Note these important points for RayJob custom resources:
|
||||
* The RayJob custom resource includes 1 head Pod and 1 worker Pod, with each Pod requesting 1 CPU and 2G of memory.
|
||||
* The RayJob runs a simple Python script that demonstrates a loop running 600 iterations, printing the iteration number and sleeping for 1 second per iteration. Hence, the RayJob runs for about 600 seconds after the submitted Kubernetes Job starts.
|
||||
* Set `shutdownAfterJobFinishes` to true for RayJob to enable automatic cleanup. This setting triggers KubeRay to delete the RayCluster after the RayJob finishes.
|
||||
* Kueue doesn't handle the RayJob custom resource with the `shutdownAfterJobFinishes` set to false. See the [limitations of Kueue](https://kueue.sigs.k8s.io/docs/tasks/run_rayjobs/#c-limitations) for more details.
|
||||
|
||||
```yaml
|
||||
kubectl create -f ray-job.kueue-toy-sample.yaml
|
||||
```
|
||||
|
||||
Each RayJob custom resource requests 2 CPUs and 4G of memory in total. However, the ClusterQueue only has 3 CPUs and 6G of memory in total. Therefore, the second RayJob custom resource remains pending, and KubeRay doesn't create Pods from the pending RayJob, even though the remaining resources are sufficient for a Pod. You can also inspect the `ClusterQueue` to see available and used quotas:
|
||||
|
||||
```bash
|
||||
$ kubectl get clusterqueues.kueue.x-k8s.io
|
||||
NAME COHORT PENDING WORKLOADS
|
||||
cluster-queue 1
|
||||
|
||||
$ kubectl get clusterqueues.kueue.x-k8s.io cluster-queue -o yaml
|
||||
Status:
|
||||
Admitted Workloads: 1 # Workloads admitted by queue.
|
||||
Conditions:
|
||||
Last Transition Time: 2024-02-28T22:41:28Z
|
||||
Message: Can admit new workloads
|
||||
Reason: Ready
|
||||
Status: True
|
||||
Type: Active
|
||||
Flavors Reservation:
|
||||
Name: default-flavor
|
||||
Resources:
|
||||
Borrowed: 0
|
||||
Name: cpu
|
||||
Total: 2
|
||||
Borrowed: 0
|
||||
Name: memory
|
||||
Total: 4Gi
|
||||
Flavors Usage:
|
||||
Name: default-flavor
|
||||
Resources:
|
||||
Borrowed: 0
|
||||
Name: cpu
|
||||
Total: 2
|
||||
Borrowed: 0
|
||||
Name: memory
|
||||
Total: 4Gi
|
||||
Pending Workloads: 1
|
||||
Reserving Workloads: 1
|
||||
```
|
||||
|
||||
Kueue admits the pending RayJob custom resource when the first RayJob custom resource finishes. Check the status of the RayJob custom resources and delete them after they finish:
|
||||
|
||||
```bash
|
||||
$ kubectl get rayjobs.ray.io
|
||||
NAME JOB STATUS DEPLOYMENT STATUS START TIME END TIME AGE
|
||||
rayjob-sample-ckvq4 SUCCEEDED Complete xxxxx xxxxx xxx
|
||||
rayjob-sample-p5msp SUCCEEDED Complete xxxxx xxxxx xxx
|
||||
|
||||
$ kubectl delete rayjob rayjob-sample-ckvq4
|
||||
$ kubectl delete rayjob rayjob-sample-p5msp
|
||||
```
|
||||
|
||||
## Step 5: Priority scheduling with Kueue
|
||||
|
||||
This step creates a RayJob with a lower priority class `dev-priority` first and a RayJob with a higher priority class `prod-priority` later. The RayJob with higher priority class `prod-priority` takes precedence over the RayJob with lower priority class `dev-priority`. Kueue preempts the RayJob with a lower priority to admit the RayJob with a higher priority.
|
||||
|
||||
If you followed the previous step, the RayJob YAML manifest `ray-job.kueue-toy-sample.yaml` should already be set to the `dev-priority` priority class. Create a RayJob with the lower priority class `dev-priority`:
|
||||
|
||||
```bash
|
||||
kubectl create -f ray-job.kueue-toy-sample.yaml
|
||||
```
|
||||
|
||||
Before creating the RayJob with the higher priority class `prod-priority`, modify the RayJob metadata with the following:
|
||||
|
||||
```yaml
|
||||
metadata:
|
||||
generateName: rayjob-sample-
|
||||
labels:
|
||||
kueue.x-k8s.io/queue-name: user-queue
|
||||
kueue.x-k8s.io/priority-class: prod-priority
|
||||
```
|
||||
|
||||
Create a RayJob with the higher priority class `prod-priority`:
|
||||
|
||||
```bash
|
||||
kubectl create -f ray-job.kueue-toy-sample.yaml
|
||||
```
|
||||
|
||||
You can see that KubeRay operator deletes the Pods belonging to the RayJob with the lower priority class `dev-priority` and creates the Pods belonging to the RayJob with the higher priority class `prod-priority`.
|
||||
|
||||
## Working with RayCluster and RayService
|
||||
|
||||
### RayCluster with Kueue
|
||||
For gang scheduling with RayCluster resources, Kueue ensures that all cluster components (head and worker nodes) are provisioned together. This prevents partial cluster creation and resource waste. **For detailed RayCluster integration**: See the [Kueue documentation for RayCluster](https://kueue.sigs.k8s.io/docs/tasks/run/rayclusters/).
|
||||
### RayService with Kueue
|
||||
RayService integration with Kueue enables gang scheduling for model serving workloads, ensuring consistent resource allocation for serving infrastructure. **For detailed RayService integration**: See the [Kueue documentation for RayService](https://kueue.sigs.k8s.io/docs/tasks/run/rayservices/).
|
||||
|
||||
## Ray Autoscaler with Kueue
|
||||
|
||||
Kueue can treat a **RayCluster** or the underlying cluster of a **RayService** as an **elastic workload**. Kueue manages queueing and quota for the entire cluster, while the in‑tree Ray autoscaler scales worker Pods up and down based on the resource demand. This section shows how to enable autoscaling for Ray workloads managed by Kueue using a step‑by‑step approach similar to the existing Kueue integration guides.
|
||||
|
||||
> **Supported resources** – At the time of writing, the Kueue
|
||||
> autoscaler integration supports `RayCluster` and `RayService`.
|
||||
> `RayJob` autoscaling is supported starting from Kueue v0.15.2,
|
||||
> and is covered below.
|
||||
|
||||
|
||||
### Prerequisites
|
||||
|
||||
Make sure you have already:
|
||||
- Installed the [KubeRay operator](kuberay-operator-deploy).
|
||||
- Installed **Kueue** (See [Kueue Installation](https://kueue.sigs.k8s.io/docs/installation/#install-a-released-version) for more details, note that it's recommended to install Kueue version >= v0.13)
|
||||
|
||||
---
|
||||
|
||||
### Step 1: Create Kueue resources
|
||||
|
||||
Define a **ResourceFlavor**, **ClusterQueue**, and **LocalQueue** so that Kueue knows how many CPUs and how much memory it can allocate. The manifest below creates an 8‑CPU/16‑GiB pool called `default-flavor`, registers it in a `ClusterQueue` named `ray-cq`, and defines a `LocalQueue` named `ray-lq`:
|
||||
|
||||
```yaml
|
||||
apiVersion: kueue.x-k8s.io/v1beta1
|
||||
kind: ResourceFlavor
|
||||
metadata:
|
||||
name: default-flavor
|
||||
---
|
||||
apiVersion: kueue.x-k8s.io/v1beta1
|
||||
kind: ClusterQueue
|
||||
metadata:
|
||||
name: ray-cq
|
||||
spec:
|
||||
cohort: ray-example
|
||||
namespaceSelector: {}
|
||||
resourceGroups:
|
||||
- coveredResources: ["cpu", "memory"]
|
||||
flavors:
|
||||
- name: default-flavor
|
||||
resources:
|
||||
- name: cpu
|
||||
nominalQuota: 8
|
||||
- name: memory
|
||||
nominalQuota: 16Gi
|
||||
---
|
||||
apiVersion: kueue.x-k8s.io/v1beta1
|
||||
kind: LocalQueue
|
||||
metadata:
|
||||
name: ray-lq
|
||||
namespace: default
|
||||
spec:
|
||||
clusterQueue: ray-cq
|
||||
```
|
||||
|
||||
Apply the resources:
|
||||
|
||||
```bash
|
||||
kubectl apply -f kueue-resources.yaml
|
||||
```
|
||||
|
||||
### Step 2: Enable elastic workloads in Kueue
|
||||
|
||||
Autoscaling only works when Kueue’s `ElasticJobsViaWorkloadSlices` feature gate is enabled.
|
||||
|
||||
Run the following command to add the feature gate flag to the `kueue-controller-manager` Deployment:
|
||||
|
||||
```bash
|
||||
kubectl -n kueue-system patch deploy kueue-controller-manager \
|
||||
--type='json' \
|
||||
-p='[
|
||||
{
|
||||
"op": "add",
|
||||
"path": "/spec/template/spec/containers/0/args/-",
|
||||
"value": "--feature-gates=ElasticJobsViaWorkloadSlices=true"
|
||||
}
|
||||
]'
|
||||
```
|
||||
### Autoscaling with RayCluster
|
||||
#### Step 1: Configure an elastic RayCluster
|
||||
|
||||
An elastic RayCluster is one that can change its worker count at runtime. Kueue requires three changes to recognize a RayCluster as elastic:
|
||||
|
||||
1. **Queue label** – set `metadata.labels.kueue.x-k8s.io/queue-name: <localqueue>` so that Kueue queues this cluster.
|
||||
2. **Elastic-job annotation** – add `metadata.annotations.kueue.x-k8s.io/elastic-job: "true"` to mark this cluster as elastic. Kueue creates **WorkloadSlices** for scaling up and down.
|
||||
3. **Enable the Ray autoscaler** – set `spec.enableInTreeAutoscaling: true` in the `RayCluster` spec and optionally configure `autoscalerOptions` such as `idleTimeoutSeconds`.
|
||||
|
||||
Here is a minimal manifest for an elastic RayCluster:
|
||||
|
||||
```yaml
|
||||
apiVersion: ray.io/v1
|
||||
kind: RayCluster
|
||||
metadata:
|
||||
name: raycluster-kueue-autoscaler
|
||||
namespace: default
|
||||
labels:
|
||||
kueue.x-k8s.io/queue-name: ray-lq
|
||||
annotations:
|
||||
kueue.x-k8s.io/elastic-job: "true" # Mark as elastic
|
||||
spec:
|
||||
rayVersion: "2.46.0"
|
||||
enableInTreeAutoscaling: true # Turn on the Ray autoscaler
|
||||
autoscalerOptions:
|
||||
idleTimeoutSeconds: 60 # Delete idle workers after 60 s
|
||||
headGroupSpec:
|
||||
serviceType: ClusterIP
|
||||
rayStartParams:
|
||||
dashboard-host: "0.0.0.0"
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: ray-head
|
||||
image: rayproject/ray:2.46.0
|
||||
resources:
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: "2Gi"
|
||||
limits:
|
||||
cpu: "2"
|
||||
memory: "5Gi"
|
||||
workerGroupSpecs:
|
||||
- groupName: workers
|
||||
replicas: 0 # start with no workers; autoscaler will add them
|
||||
minReplicas: 0 # lower bound
|
||||
maxReplicas: 4 # upper bound
|
||||
rayStartParams: {}
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: ray-worker
|
||||
image: rayproject/ray:2.46.0
|
||||
resources:
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: "1Gi"
|
||||
limits:
|
||||
cpu: "2"
|
||||
memory: "5Gi"
|
||||
```
|
||||
|
||||
Apply this manifest and verify that Kueue admits the associated Workload:
|
||||
|
||||
```bash
|
||||
kubectl apply -f raycluster-kueue-autoscaler.yaml
|
||||
kubectl get workloads.kueue.x-k8s.io -A
|
||||
```
|
||||
|
||||
The `ADMITTED` column should show `True` once the `RayCluster` has been scheduled by Kueue.
|
||||
```bash
|
||||
NAMESPACE NAME QUEUE RESERVED IN ADMITTED FINISHED AGE
|
||||
default raycluster-raycluster-kueue-autoscaler-21c46 ray-lq ray-cq True 26s
|
||||
```
|
||||
|
||||
(step-2-verify-autoscaling-for-a-raycluster)=
|
||||
#### Step 2: Verify autoscaling for a RayCluster
|
||||
|
||||
To observe autoscaling, create load on the cluster and watch worker Pods appear. The following procedure runs a CPU‑bound workload from inside the head Pod and monitors scaling:
|
||||
|
||||
1. **Enter the head Pod:**
|
||||
|
||||
```bash
|
||||
HEAD_POD=$(kubectl get pod -l ray.io/node-type=head,ray.io/cluster=raycluster-kueue-autoscaler \
|
||||
-o jsonpath='{.items[0].metadata.name}')
|
||||
kubectl exec -it "$HEAD_POD" -- bash
|
||||
```
|
||||
|
||||
2. **Run a workload:** execute the following Python script inside the head container. It submits 20 tasks that each consume a full CPU for about one minute.
|
||||
|
||||
```bash
|
||||
python << 'EOF'
|
||||
import ray, time
|
||||
|
||||
ray.init(address="auto")
|
||||
|
||||
@ray.remote(num_cpus=1)
|
||||
def busy():
|
||||
end = time.time() + 60
|
||||
while time.time() < end:
|
||||
x = 0
|
||||
for i in range(100_000):
|
||||
x += i * i
|
||||
return 1
|
||||
|
||||
tasks = [busy.remote() for _ in range(20)]
|
||||
print(sum(ray.get(tasks)))
|
||||
EOF
|
||||
```
|
||||
|
||||
Because the head Pod has a single CPU, the tasks queue up and the autoscaler raises the worker replicas toward the `maxReplicas`.
|
||||
|
||||
3. **Monitor worker Pods:** in another terminal, watch the worker Pods scale up and down:
|
||||
|
||||
```bash
|
||||
kubectl get pods -w \
|
||||
-l ray.io/cluster=raycluster-kueue-autoscaler,ray.io/node-type=worker
|
||||
```
|
||||
|
||||
New worker Pods should appear as the tasks run and vanish once the workload finishes and the idle timeout elapses.
|
||||
|
||||
### Autoscaling with RayJob
|
||||
#### Step 1: Create the RayJob code ConfigMap
|
||||
|
||||
This RayJob example fans out CPU-bound tasks, which triggers the Ray autoscaler to scale the worker group within the admitted Kueue quota. If you used a different LocalQueue name when creating Kueue resources, replace `user-queue` in the RayJob metadata with your queue name.
|
||||
|
||||
Create a ConfigMap that includes the Ray workload:
|
||||
|
||||
```yaml
|
||||
# ray-job-code-sample.yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: ray-job-autoscaling-code-sample
|
||||
namespace: default
|
||||
data:
|
||||
sample_code.py: |
|
||||
import ray
|
||||
import time
|
||||
|
||||
ray.init()
|
||||
|
||||
@ray.remote
|
||||
def busy_task(duration):
|
||||
end = time.time() + duration
|
||||
while time.time() < end:
|
||||
x = 0
|
||||
for _ in range(100_000):
|
||||
x += 1
|
||||
return duration
|
||||
|
||||
tasks = [busy_task.remote(60) for _ in range(10)]
|
||||
print(sum(ray.get(tasks)))
|
||||
```
|
||||
|
||||
#### Step 2: Create an elastic RayJob
|
||||
|
||||
This RayJob uses the elastic job annotation, enables in-tree autoscaling, and mounts the ConfigMap with the Python entrypoint.
|
||||
|
||||
```yaml
|
||||
# ray-job-autoscaling-sample.yaml
|
||||
apiVersion: ray.io/v1
|
||||
kind: RayJob
|
||||
metadata:
|
||||
name: rayjob-autoscaling-sample
|
||||
labels:
|
||||
kueue.x-k8s.io/queue-name: user-queue
|
||||
annotations:
|
||||
kueue.x-k8s.io/elastic-job: "true"
|
||||
spec:
|
||||
shutdownAfterJobFinishes: true
|
||||
entrypoint: python /home/ray/samples/sample_code.py
|
||||
runtimeEnvYAML: |
|
||||
pip:
|
||||
- requests==2.26.0
|
||||
- pendulum==2.1.2
|
||||
env_vars:
|
||||
counter_name: "test_counter"
|
||||
rayClusterSpec:
|
||||
rayVersion: "2.46.0"
|
||||
autoscalerOptions:
|
||||
idleTimeoutSeconds: 30
|
||||
upscalingMode: Aggressive
|
||||
enableInTreeAutoscaling: true
|
||||
headGroupSpec:
|
||||
rayStartParams:
|
||||
dashboard-host: "0.0.0.0"
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: ray-head
|
||||
image: rayproject/ray:2.46.0
|
||||
ports:
|
||||
- containerPort: 6379
|
||||
name: gcs-server
|
||||
- containerPort: 8265
|
||||
name: dashboard
|
||||
- containerPort: 10001
|
||||
name: client
|
||||
resources:
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: "2Gi"
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: "2Gi"
|
||||
volumeMounts:
|
||||
- mountPath: /home/ray/samples
|
||||
name: code-sample
|
||||
volumes:
|
||||
- name: code-sample
|
||||
configMap:
|
||||
name: ray-job-autoscaling-code-sample
|
||||
items:
|
||||
- key: sample_code.py
|
||||
path: sample_code.py
|
||||
workerGroupSpecs:
|
||||
- replicas: 1
|
||||
minReplicas: 1
|
||||
maxReplicas: 5
|
||||
groupName: small-group
|
||||
rayStartParams: {}
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: ray-worker
|
||||
image: rayproject/ray:2.46.0
|
||||
lifecycle:
|
||||
preStop:
|
||||
exec:
|
||||
command: ["/bin/sh", "-c", "ray stop"]
|
||||
resources:
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: "2Gi"
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: "2Gi"
|
||||
```
|
||||
|
||||
Apply the manifests:
|
||||
|
||||
```bash
|
||||
kubectl apply -f ray-job-code-sample.yaml
|
||||
kubectl apply -f ray-job-autoscaling-sample.yaml
|
||||
```
|
||||
|
||||
#### Step 3: Verify autoscaling for a RayJob
|
||||
|
||||
Find the RayCluster created by the RayJob and watch the worker Pods:
|
||||
|
||||
```bash
|
||||
kubectl get rayclusters
|
||||
kubectl get pods -w -l ray.io/cluster=<raycluster-name>,ray.io/node-type=worker
|
||||
```
|
||||
|
||||
During the workload, the worker group scales from 1 to 5 Pods, then scales back to 1 after the tasks finish and the cluster becomes idle.
|
||||
|
||||
### Autoscaling with RayService
|
||||
#### Step 1: Configure an elastic RayService
|
||||
|
||||
A `RayService` deploys a Ray Serve application by materializing the `spec.rayClusterConfig` into a managed `RayCluster`. Kueue doesn't interact with the RayService object directly. Instead, the KubeRay operator propagates relevant metadata from the RayService to the managed RayCluster, and **Kueue queues and admits that RayCluster**.
|
||||
|
||||
To make a RayService work with Kueue and the Ray autoscaler:
|
||||
|
||||
1. **Queue label** `metadata.labels.kueue.x-k8s.io/queue-name` on the `RayService`. KubeRay passes service labels to the underlying `RayCluster`, allowing Kueue to queue it.
|
||||
2. **Elastic-job annotation** `metadata.annotations.kueue.x-k8s.io/elastic-job: "true"`. This annotation propagates to the `RayCluster` and instructs Kueue to treat it as an elastic workload.
|
||||
3. **Enable the Ray autoscaler** `spec.rayClusterConfig`, set `enableInTreeAutoscaling: true` and specify worker `minReplicas`/`maxReplicas`.
|
||||
|
||||
The following manifest deploys a simple Ray Serve app with autoscaling. The Serve application (`demo_app`) and deployment names (`ServiceA` and `ServiceB`) are placeholders to avoid implying a specific KubeRay example. Adjust the deployments and resources for your own application.
|
||||
|
||||
```yaml
|
||||
apiVersion: ray.io/v1
|
||||
kind: RayService
|
||||
metadata:
|
||||
name: rayservice-kueue-autoscaler
|
||||
namespace: default
|
||||
labels:
|
||||
kueue.x-k8s.io/queue-name: ray-lq # copy to RayCluster
|
||||
annotations:
|
||||
kueue.x-k8s.io/elastic-job: "true" # mark as elastic
|
||||
spec:
|
||||
# A simple Serve config with two deployments
|
||||
serveConfigV2: |
|
||||
applications:
|
||||
- name: fruit_app
|
||||
import_path: fruit.deployment_graph
|
||||
route_prefix: /fruit
|
||||
runtime_env:
|
||||
working_dir: "https://github.com/ray-project/test_dag/archive/78b4a5da38796123d9f9ffff59bab2792a043e95.zip"
|
||||
deployments:
|
||||
- name: MangoStand
|
||||
num_replicas: 2
|
||||
max_replicas_per_node: 1
|
||||
user_config:
|
||||
price: 3
|
||||
ray_actor_options:
|
||||
num_cpus: 0.1
|
||||
- name: OrangeStand
|
||||
num_replicas: 1
|
||||
user_config:
|
||||
price: 2
|
||||
ray_actor_options:
|
||||
num_cpus: 0.1
|
||||
- name: PearStand
|
||||
num_replicas: 1
|
||||
user_config:
|
||||
price: 1
|
||||
ray_actor_options:
|
||||
num_cpus: 0.1
|
||||
- name: FruitMarket
|
||||
num_replicas: 1
|
||||
ray_actor_options:
|
||||
num_cpus: 0.1
|
||||
- name: math_app
|
||||
import_path: conditional_dag.serve_dag
|
||||
route_prefix: /calc
|
||||
runtime_env:
|
||||
working_dir: "https://github.com/ray-project/test_dag/archive/78b4a5da38796123d9f9ffff59bab2792a043e95.zip"
|
||||
deployments:
|
||||
- name: Adder
|
||||
num_replicas: 1
|
||||
user_config:
|
||||
increment: 3
|
||||
ray_actor_options:
|
||||
num_cpus: 0.1
|
||||
- name: Multiplier
|
||||
num_replicas: 1
|
||||
user_config:
|
||||
factor: 5
|
||||
ray_actor_options:
|
||||
num_cpus: 0.1
|
||||
- name: Router
|
||||
num_replicas: 1
|
||||
rayClusterConfig:
|
||||
rayVersion: "2.46.0"
|
||||
enableInTreeAutoscaling: true
|
||||
autoscalerOptions:
|
||||
idleTimeoutSeconds: 60
|
||||
headGroupSpec:
|
||||
serviceType: ClusterIP
|
||||
rayStartParams:
|
||||
dashboard-host: "0.0.0.0"
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: ray-head
|
||||
image: rayproject/ray:2.46.0
|
||||
resources:
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: "2Gi"
|
||||
limits:
|
||||
cpu: "2"
|
||||
memory: "5Gi"
|
||||
workerGroupSpecs:
|
||||
- groupName: workers
|
||||
replicas: 1 # initial workers
|
||||
minReplicas: 1 # lower bound
|
||||
maxReplicas: 5 # upper bound
|
||||
rayStartParams: {}
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: ray-worker
|
||||
image: rayproject/ray:2.46.0
|
||||
resources:
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: "1Gi"
|
||||
limits:
|
||||
cpu: "2"
|
||||
memory: "5Gi"
|
||||
```
|
||||
|
||||
Apply the manifest and verify that the service's RayCluster is admitted by Kueue:
|
||||
|
||||
```bash
|
||||
kubectl apply -f rayservice-kueue-autoscaler.yaml
|
||||
kubectl get workloads.kueue.x-k8s.io -A
|
||||
```
|
||||
|
||||
|
||||
The `ADMITTED` column should show `True` once the `RayService` has been scheduled by Kueue.
|
||||
```bash
|
||||
NAMESPACE NAME QUEUE RESERVED IN ADMITTED FINISHED AGE
|
||||
default raycluster-rayservice-kueue-autoscaler-9xvcr-d7add ray-lq ray-cq True 21s
|
||||
```
|
||||
|
||||
|
||||
#### Step 2: Verify autoscaling for a RayService
|
||||
|
||||
Autoscaling for a `RayService` is ultimately driven by load on the managed RayCluster. The verification procedure is the same as for a plain `RayCluster`.
|
||||
|
||||
To verify autoscaling:
|
||||
|
||||
1. Follow the steps in [Step 2: Verify autoscaling for a RayCluster](#step-2-verify-autoscaling-for-a-raycluster), but use the RayService name in the label selector. Concretely:
|
||||
- when selecting the head Pod, use (remember to replace your cluster name):
|
||||
```bash
|
||||
HEAD_POD=$(kubectl get pod \
|
||||
-l ray.io/node-type=head,ray.io/cluster=rayservice-kueue-autoscaler-9xvcr \
|
||||
-o jsonpath='{.items[0].metadata.name}')
|
||||
kubectl exec -it "$HEAD_POD" -- bash
|
||||
```
|
||||
- inside the head container, run the same CPU-bound Python script used in the RayCluster example.
|
||||
```bash
|
||||
python << 'EOF'
|
||||
import ray, time
|
||||
|
||||
ray.init(address="auto")
|
||||
|
||||
@ray.remote(num_cpus=1)
|
||||
def busy():
|
||||
end = time.time() + 60
|
||||
while time.time() < end:
|
||||
x = 0
|
||||
for i in range(100_000):
|
||||
x += i * i
|
||||
return 1
|
||||
|
||||
tasks = [busy.remote() for _ in range(20)]
|
||||
print(sum(ray.get(tasks)))
|
||||
EOF
|
||||
```
|
||||
- in another terminal, watch the worker Pods with:
|
||||
```bash
|
||||
kubectl get pods -w \
|
||||
-l ray.io/cluster=rayservice-kueue-autoscaler,ray.io/node-type=worker
|
||||
```
|
||||
|
||||
As in the RayCluster case, the worker Pods scale up toward `maxReplicas` while the CPU-bound tasks are running and scale back down toward `minReplicas` after the tasks finish and the idle timeout elapses. The only difference is that the `ray.io/cluster` label now matches the RayService name (`rayservice-kueue-autoscaler-9xvcr`) instead of the stand-alone `RayCluster` name (`raycluster-kueue-autoscaler`).
|
||||
|
||||
### Limitations
|
||||
|
||||
* **Feature status** – The `ElasticJobsViaWorkloadSlices` feature gate is currently **alpha**. Elastic autoscaling only applies to RayClusters that are annotated with `kueue.x-k8s.io/elastic-job: "true"` and configured with `enableInTreeAutoscaling: true` when ray image < 2.47.0.
|
||||
|
||||
* **RayJob support** – `RayJob` in-tree autoscaling requires Kueue v0.15.2+.
|
||||
|
||||
* **Kueue versions prior to v0.13** – If you are using a Kueue version earlier than v0.13, restart the Kueue controller once after installation to ensure RayCluster management works correctly.
|
||||
@@ -0,0 +1,51 @@
|
||||
(kuberay-metrics-references)=
|
||||
|
||||
# KubeRay metrics references
|
||||
|
||||
## `controller-runtime` metrics
|
||||
KubeRay exposes metrics provided by [kubernetes-sigs/controller-runtime](https://github.com/kubernetes-sigs/controller-runtime), including information about reconciliation, work queues, and more, to help users operate the KubeRay operator in production environments.
|
||||
|
||||
For more details about the default metrics provided by [kubernetes-sigs/controller-runtime](https://github.com/kubernetes-sigs/controller-runtime), see [Default Exported Metrics References](https://book.kubebuilder.io/reference/metrics-reference).
|
||||
|
||||
## KubeRay custom metrics
|
||||
|
||||
Starting with KubeRay 1.4.0, KubeRay provides metrics for its custom resources to help users better understand Ray clusters and Ray applications.
|
||||
|
||||
You can view these metrics by following the instructions below:
|
||||
```sh
|
||||
# Forward a local port to the KubeRay operator service.
|
||||
kubectl port-forward service/kuberay-operator 8080
|
||||
|
||||
# View the metrics.
|
||||
curl localhost:8080/metrics
|
||||
|
||||
# You should see metrics like the following if a RayCluster already exists:
|
||||
# kuberay_cluster_info{name="raycluster-kuberay",namespace="default",owner_kind="None"} 1
|
||||
```
|
||||
|
||||
### RayCluster metrics
|
||||
|
||||
|
||||
| Metric name | Type | Description | Labels |
|
||||
|--------------------------------------------------|-------|----------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------|
|
||||
| `kuberay_cluster_info` | Gauge | Metadata information about RayCluster custom resources. | `namespace`: <RayCluster-namespace><br/> `name`: <RayCluster-name><br/> `owner_kind`: <RayJob\|RayService\|None><br/> `uid`: <RayCluster-uid> |
|
||||
| `kuberay_cluster_condition_provisioned` | Gauge | Indicates whether the RayCluster is provisioned. See [RayClusterProvisioned](https://github.com/ray-project/kuberay/blob/7c6aedff5b4106281f50e87a7e9e177bf1237ec7/ray-operator/apis/ray/v1/raycluster_types.go#L214) for more information. | `namespace`: <RayCluster-namespace><br/> `name`: <RayCluster-name><br/> `condition`: <true\|false><br/> `uid`: <RayCluster-uid> |
|
||||
| `kuberay_cluster_provisioned_duration_seconds` | Gauge | The time, in seconds, when a RayCluster's `RayClusterProvisioned` status transitions from false (or unset) to true. | `namespace`: <RayCluster-namespace><br/> `name`: <RayCluster-name><br/> `uid`: <RayCluster-uid> |
|
||||
|
||||
### RayService metrics
|
||||
|
||||
| Metric name | Type | Description | Labels |
|
||||
|--------------------------------------------------|-------|------------------------------------------------------------|--------------------------------------------------------------------|
|
||||
| `kuberay_service_info` | Gauge | Metadata information about RayService custom resources. | `namespace`: <RayService-namespace><br/> `name`: <RayService-name><br/> `uid`: <RayService-uid> |
|
||||
| `kuberay_service_condition_ready` | Gauge | Describes whether the RayService is ready. Ready means users can send requests to the underlying cluster and the number of serve endpoints is greater than 0. See [RayServiceReady](https://github.com/ray-project/kuberay/blob/33ee6724ca2a429c77cb7ff5821ba9a3d63f7c34/ray-operator/apis/ray/v1/rayservice_types.go#L135) for more information. | `namespace`: <RayService-namespace><br/> `name`: <RayService-name><br/> `uid`: <RayService-uid> |
|
||||
| `kuberay_service_condition_upgrade_in_progress` | Gauge | Describes whether the RayService is performing a zero-downtime upgrade. See [UpgradeInProgress](https://github.com/ray-project/kuberay/blob/33ee6724ca2a429c77cb7ff5821ba9a3d63f7c34/ray-operator/apis/ray/v1/rayservice_types.go#L137) for more information. | `namespace`: <RayService-namespace><br/> `name`: <RayService-name><br/> `uid`: <RayService-uid> |
|
||||
|
||||
### RayJob metrics
|
||||
|
||||
| Metric name | Type | Description | Labels |
|
||||
|--------------------------------------------------|-------|------------------------------------------------------------|---------------------------------------------------------------------------|
|
||||
| `kuberay_job_info` | Gauge | Metadata information about RayJob custom resources. | `namespace`: <RayJob-namespace><br/> `name`: <RayJob-name><br/> `uid`: <RayJob-uid> |
|
||||
| `kuberay_job_deployment_status` | Gauge | The RayJob's current deployment status. | `namespace`: <RayJob-namespace><br/> `name`: <RayJob-name><br/> `deployment_status`: <New\|Initializing\|Running\|Complete\|Failed\|Suspending\|Suspended\|Retrying\|Waiting><br/> `uid`: <RayJob-uid> |
|
||||
| `kuberay_job_execution_duration_seconds` | Gauge | Duration of the RayJob CR’s JobDeploymentStatus transition from `Initializing` to either the `Retrying` state or a terminal state, such as `Complete` or `Failed`. The `Retrying` state indicates that the CR previously failed and that spec.backoffLimit is enabled. | `namespace`: <RayJob-namespace><br/> `name`: <RayJob-name><br/> `job_deployment_status`: <Complete\|Failed><br/> `retry_count`: <count><br/> `uid`: <RayJob-uid> |
|
||||
|
||||
|
||||
@@ -0,0 +1,419 @@
|
||||
(kuberay-prometheus-grafana)=
|
||||
|
||||
# Using Prometheus and Grafana
|
||||
|
||||
This section will describe how to monitor Ray Clusters in Kubernetes using Prometheus & Grafana.
|
||||
|
||||
If you do not have any experience with Prometheus and Grafana on Kubernetes, watch this [YouTube playlist](https://youtube.com/playlist?list=PLy7NrYWoggjxCF3av5JKwyG7FFF9eLeL4).
|
||||
|
||||
## Preparation
|
||||
|
||||
Clone the [KubeRay repository](https://github.com/ray-project/kuberay) and checkout the `master` branch. This tutorial requires several files in the repository.
|
||||
|
||||
## Step 1: Create a Kubernetes cluster with Kind
|
||||
|
||||
```sh
|
||||
kind create cluster
|
||||
```
|
||||
|
||||
## Step 2: Install Kubernetes Prometheus Stack via Helm chart
|
||||
|
||||
```sh
|
||||
# Path: kuberay/
|
||||
./install/prometheus/install.sh --auto-load-dashboard true
|
||||
|
||||
# Check the installation
|
||||
kubectl get all -n prometheus-system
|
||||
|
||||
# (part of the output)
|
||||
# NAME READY UP-TO-DATE AVAILABLE AGE
|
||||
# deployment.apps/prometheus-grafana 1/1 1 1 46s
|
||||
# deployment.apps/prometheus-kube-prometheus-operator 1/1 1 1 46s
|
||||
# deployment.apps/prometheus-kube-state-metrics 1/1 1 1 46s
|
||||
```
|
||||
|
||||
* KubeRay provides an [install.sh script](https://github.com/ray-project/kuberay/blob/master/install/prometheus/install.sh) to:
|
||||
* Install the [kube-prometheus-stack v48.2.1](https://github.com/prometheus-community/helm-charts/tree/kube-prometheus-stack-48.2.1/charts/kube-prometheus-stack) chart and related custom resources, including **PodMonitor** for Ray Pods and **PrometheusRule**, in the namespace `prometheus-system` automatically.
|
||||
* Import Ray Dashboard's [Grafana JSON files](https://github.com/ray-project/kuberay/tree/master/config/grafana) into Grafana using the `--auto-load-dashboard true` flag. If the flag isn't set, the following step also provides instructions for manual import. See [Step 12: Import Grafana dashboards manually (optional)](#step-12-import-grafana-dashboards-manually-optional) for more details.
|
||||
|
||||
* We made some modifications to the original `values.yaml` in kube-prometheus-stack chart to allow embedding Grafana panels in Ray Dashboard. See [overrides.yaml](https://github.com/ray-project/kuberay/blob/master/install/prometheus/overrides.yaml) for more details.
|
||||
```yaml
|
||||
grafana:
|
||||
grafana.ini:
|
||||
security:
|
||||
allow_embedding: true
|
||||
auth.anonymous:
|
||||
enabled: true
|
||||
org_role: Viewer
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Step 3: Install a KubeRay operator
|
||||
|
||||
* Follow [this document](kuberay-operator-deploy) to install the latest stable KubeRay operator via Helm repository.
|
||||
* Set `metrics.serviceMonitor.enabled=true` when installing the KubeRay operator with Helm to create a ServiceMonitor that scrapes metrics exposed by the KubeRay operator's service.
|
||||
```sh
|
||||
# Enable the ServiceMonitor and set the label `release: prometheus` to the ServiceMonitor so that Prometheus can discover it
|
||||
helm install kuberay-operator kuberay/kuberay-operator --version 1.6.0 \
|
||||
--set metrics.serviceMonitor.enabled=true \
|
||||
--set metrics.serviceMonitor.selector.release=prometheus
|
||||
```
|
||||
You can verify the ServiceMonitor creation with:
|
||||
```sh
|
||||
kubectl get servicemonitor
|
||||
# NAME AGE
|
||||
# kuberay-operator 11s
|
||||
```
|
||||
|
||||
## Step 4: Install a RayCluster
|
||||
|
||||
```sh
|
||||
# path: ray-operator/config/samples/
|
||||
kubectl apply -f ray-cluster.embed-grafana.yaml
|
||||
|
||||
# Check there's a Service that specifies port 8080 for the metrics endpoint.
|
||||
# There may be a slight delay between RayCluster and Service creation.
|
||||
kubectl get service -l ray.io/cluster=raycluster-embed-grafana
|
||||
|
||||
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
# raycluster-embed-grafana-head-svc ClusterIP None <none> 44217/TCP,10001/TCP,44227/TCP,8265/TCP,6379/TCP,8080/TCP 13m
|
||||
|
||||
# Wait until all Ray Pods are ready.
|
||||
kubectl wait pods -l ray.io/cluster=raycluster-embed-grafana --timeout 2m --for condition=Ready
|
||||
|
||||
# pod/raycluster-embed-grafana-head-2jk7c condition met
|
||||
# pod/raycluster-embed-grafana-small-group-worker-8g2vv condition met
|
||||
|
||||
# Forward the port of the Prometheus metrics endpoint.
|
||||
kubectl port-forward service/raycluster-embed-grafana-head-svc metrics
|
||||
|
||||
# Check metrics in a new terminal.
|
||||
curl localhost:8080
|
||||
|
||||
# Example output (Prometheus metrics format):
|
||||
# # HELP ray_spill_manager_request_total Number of {spill, restore} requests.
|
||||
# # TYPE ray_spill_manager_request_total gauge
|
||||
# ray_spill_manager_request_total{Component="raylet", NodeAddress="10.244.0.13", SessionName="session_2025-01-02_07-58-21_419367_11", Type="FailedDeletion", Version="2.9.0", container="ray-head", endpoint="metrics", instance="10.244.0.13:8080", job="prometheus-system/ray-head-monitor", namespace="default", pod="raycluster-embed-grafana-head-98fqt", ray_io_cluster="raycluster-embed-grafana"} 0
|
||||
```
|
||||
|
||||
* KubeRay exposes a Prometheus metrics endpoint in port **8080** via a built-in exporter by default. Hence, we do not need to install any external exporter.
|
||||
* If you want to configure the metrics endpoint to a different port, see [kuberay/#954](https://github.com/ray-project/kuberay/pull/954) for more details.
|
||||
* Prometheus metrics format:
|
||||
* `# HELP`: Describe the meaning of this metric.
|
||||
* `# TYPE`: See [this document](https://prometheus.io/docs/concepts/metric_types/) for more details.
|
||||
|
||||
* Three required environment variables are defined in [ray-cluster.embed-grafana.yaml](https://github.com/ray-project/kuberay/blob/v1.6.0/ray-operator/config/samples/ray-cluster.embed-grafana.yaml). See [Configuring and Managing Ray Dashboard](https://docs.ray.io/en/latest/cluster/configure-manage-dashboard.html) for more details about these environment variables.
|
||||
```yaml
|
||||
env:
|
||||
- name: RAY_GRAFANA_IFRAME_HOST
|
||||
value: http://127.0.0.1:3000
|
||||
- name: RAY_GRAFANA_HOST
|
||||
value: http://prometheus-grafana.prometheus-system.svc:80
|
||||
- name: RAY_PROMETHEUS_HOST
|
||||
value: http://prometheus-kube-prometheus-prometheus.prometheus-system.svc:9090
|
||||
```
|
||||
* Note that we do not deploy Grafana in the head Pod, so we need to set both `RAY_GRAFANA_IFRAME_HOST` and `RAY_GRAFANA_HOST`. `RAY_GRAFANA_HOST` is used by the head Pod to send health-check requests to Grafana in the backend. `RAY_GRAFANA_IFRAME_HOST` is used by your browser to fetch the Grafana panels from the Grafana server rather than from the head Pod. Because we forward the port of Grafana to `127.0.0.1:3000` in this example, we set `RAY_GRAFANA_IFRAME_HOST` to `http://127.0.0.1:3000`.
|
||||
* `http://` is required.
|
||||
|
||||
## Step 5: Collect Head Node metrics with a PodMonitor
|
||||
|
||||
RayService creates two Kubernetes services for the head Pod; one managed by the RayService and the other by the underlying RayCluster. Therefore, it's recommended to use a PodMonitor to monitor the metrics for head Pods to avoid misconfigurations that could result in double counting the same metrics when using a ServiceMonitor.
|
||||
|
||||
```yaml
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PodMonitor
|
||||
metadata:
|
||||
labels:
|
||||
# `release: $HELM_RELEASE`: Prometheus can only detect PodMonitor with this label.
|
||||
release: prometheus
|
||||
name: ray-head-monitor
|
||||
namespace: prometheus-system
|
||||
spec:
|
||||
jobLabel: ray-head
|
||||
# Only select Kubernetes Pods in the "default" namespace.
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- default
|
||||
# Only select Kubernetes Pods with "matchLabels".
|
||||
selector:
|
||||
matchLabels:
|
||||
ray.io/node-type: head
|
||||
# A list of endpoints allowed as part of this PodMonitor.
|
||||
podMetricsEndpoints:
|
||||
- port: metrics
|
||||
relabelings:
|
||||
- action: replace
|
||||
sourceLabels:
|
||||
- __meta_kubernetes_pod_label_ray_io_cluster
|
||||
targetLabel: ray_io_cluster
|
||||
- port: as-metrics # autoscaler metrics
|
||||
relabelings:
|
||||
- action: replace
|
||||
sourceLabels:
|
||||
- __meta_kubernetes_pod_label_ray_io_cluster
|
||||
targetLabel: ray_io_cluster
|
||||
- port: dash-metrics # dashboard metrics
|
||||
relabelings:
|
||||
- action: replace
|
||||
sourceLabels:
|
||||
- __meta_kubernetes_pod_label_ray_io_cluster
|
||||
targetLabel: ray_io_cluster
|
||||
```
|
||||
|
||||
* The **install.sh** script creates the above YAML example, [podMonitor.yaml](https://github.com/ray-project/kuberay/blob/master/config/prometheus/podMonitor.yaml#L26-L63) so you don't need to create anything.
|
||||
* See the official [PodMonitor doc](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api-reference/api.md#monitoring.coreos.com/v1.PodMonitor) for more details about configurations.
|
||||
* `release: $HELM_RELEASE`: Prometheus can only detect PodMonitor with this label. See [here](#prometheus-can-only-detect-this-label) for more details.
|
||||
|
||||
(prometheus-can-only-detect-this-label)=
|
||||
```sh
|
||||
helm ls -n prometheus-system
|
||||
# ($HELM_RELEASE is "prometheus".)
|
||||
# NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
|
||||
# prometheus prometheus-system 1 2023-02-06 06:27:05.530950815 +0000 UTC deployed kube-prometheus-stack-44.3.1 v0.62.0
|
||||
|
||||
kubectl get prometheuses.monitoring.coreos.com -n prometheus-system -oyaml
|
||||
# podMonitorSelector:
|
||||
# matchLabels:
|
||||
# release: prometheus
|
||||
# ruleSelector:
|
||||
# matchLabels:
|
||||
# release: prometheus
|
||||
```
|
||||
|
||||
* Prometheus uses `namespaceSelector` and `selector` to select Kubernetes Pods.
|
||||
```sh
|
||||
kubectl get pod -n default -l ray.io/node-type=head
|
||||
# NAME READY STATUS RESTARTS AGE
|
||||
# raycluster-embed-grafana-head-khfs4 1/1 Running 0 4m38s
|
||||
```
|
||||
|
||||
* `relabelings`: This configuration renames the label `__meta_kubernetes_pod_label_ray_io_cluster` to `ray_io_cluster` in the scraped metrics. It ensures that each metric includes the name of the RayCluster to which the Pod belongs. This configuration is especially useful for distinguishing metrics when deploying multiple RayClusters. For example, a metric with the `ray_io_cluster` label might look like this:
|
||||
|
||||
```
|
||||
ray_node_cpu_count{SessionName="session_2025-01-02_07-58-21_419367_11", container="ray-head", endpoint="metrics", instance="10.244.0.13:8080", ip="10.244.0.13", job="raycluster-embed-grafana-head-svc", namespace="default", pod="raycluster-embed-grafana-head-98fqt", ray_io_cluster="raycluster-embed-grafana", service="raycluster-embed-grafana-head-svc"}
|
||||
```
|
||||
|
||||
In this example, `raycluster-embed-grafana` is the name of the RayCluster.
|
||||
|
||||
## Step 6: Collect Worker Node metrics with PodMonitors
|
||||
|
||||
Similar to the head Pod, this tutorial also uses a PodMonitor to collect metrics from worker Pods. The reason for using separate PodMonitors for head Pods and worker Pods is that the head Pod exposes multiple metric endpoints, whereas a worker Pod exposes only one.
|
||||
|
||||
**Note**: You could create a Kubernetes service with selectors a common label subset from our worker pods, however, this configuration is not ideal because the workers are independent from each other, that is, they aren't a collection of replicas spawned by replicaset controller. Due to this behavior, avoid using a Kubernetes service for grouping them together.
|
||||
|
||||
```yaml
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PodMonitor
|
||||
metadata:
|
||||
name: ray-workers-monitor
|
||||
namespace: prometheus-system
|
||||
labels:
|
||||
# `release: $HELM_RELEASE`: Prometheus can only detect PodMonitor with this label.
|
||||
release: prometheus
|
||||
spec:
|
||||
jobLabel: ray-workers
|
||||
# Only select Kubernetes Pods in the "default" namespace.
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- default
|
||||
# Only select Kubernetes Pods with "matchLabels".
|
||||
selector:
|
||||
matchLabels:
|
||||
ray.io/node-type: worker
|
||||
# A list of endpoints allowed as part of this PodMonitor.
|
||||
podMetricsEndpoints:
|
||||
- port: metrics
|
||||
relabelings:
|
||||
- sourceLabels: [__meta_kubernetes_pod_label_ray_io_cluster]
|
||||
targetLabel: ray_io_cluster
|
||||
```
|
||||
|
||||
* **PodMonitor** in `namespaceSelector` and `selector` are used to select Kubernetes Pods.
|
||||
```sh
|
||||
kubectl get pod -n default -l ray.io/node-type=worker
|
||||
# NAME READY STATUS RESTARTS AGE
|
||||
# raycluster-kuberay-worker-workergroup-5stpm 1/1 Running 0 3h16m
|
||||
```
|
||||
|
||||
## Step 7: Scrape KubeRay metrics with ServiceMonitor
|
||||
|
||||
* See the official [ServiceMonitor doc](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api-reference/api.md#servicemonitor) for more details about configurations.
|
||||
* KubeRay operator provides metrics for RayCluster, RayService, and RayJob. See {ref}`kuberay-metrics-references` for more details.
|
||||
* Prometheus uses `namespaceSelector` and `selector` to select Kubernetes Service.
|
||||
```sh
|
||||
kubectl get service -n default -l app.kubernetes.io/name=kuberay-operator
|
||||
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
kuberay-operator ClusterIP 10.96.205.229 <none> 8080/TCP 53m
|
||||
```
|
||||
|
||||
## Step 8: Collect custom metrics with recording rules
|
||||
|
||||
[Recording Rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) allow KubeRay to precompute frequently needed or computationally expensive [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) expressions and save their result as custom metrics. Note that this behavior is different from [Custom application-level metrics](application-level-metrics), which are for the visibility of Ray applications.
|
||||
|
||||
```yaml
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: ray-cluster-gcs-rules
|
||||
namespace: prometheus-system
|
||||
labels:
|
||||
# `release: $HELM_RELEASE`: Prometheus can only detect Recording Rules with this label.
|
||||
release: prometheus
|
||||
spec:
|
||||
groups:
|
||||
- # Rules within a group are run periodically with the same evaluation interval(30s in this example).
|
||||
name: ray-cluster-main-staging-gcs.rules
|
||||
# How often rules in the group are evaluated.
|
||||
interval: 30s
|
||||
rules:
|
||||
- # The name of the custom metric.
|
||||
# Also see best practices for naming metrics created by recording rules:
|
||||
# https://prometheus.io/docs/practices/rules/#recording-rules
|
||||
record: ray_gcs_availability_30d
|
||||
# PromQL expression.
|
||||
expr: |
|
||||
(
|
||||
100 * (
|
||||
sum(rate(ray_gcs_update_resource_usage_time_bucket{container="ray-head", le="20.0"}[30d]))
|
||||
/
|
||||
sum(rate(ray_gcs_update_resource_usage_time_count{container="ray-head"}[30d]))
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
* The PromQL expression above is:
|
||||
$$\frac{ number\ of\ update\ resource\ usage\ RPCs\ that\ have\ RTT\ smaller\ then\ 20ms\ in\ last\ 30\ days\ }{total\ number\ of\ update\ resource\ usage\ RPCs\ in\ last\ 30\ days\ } \times 100 $$
|
||||
|
||||
|
||||
* The recording rule above is one of rules defined in [prometheusRules.yaml](https://github.com/ray-project/kuberay/blob/master/config/prometheus/rules/prometheusRules.yaml), and it is created by **install.sh**. Hence, no need to create anything here.
|
||||
|
||||
* See the official [PrometheusRule document](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api-reference/api.md#monitoring.coreos.com/v1.PrometheusRule) for more details about configurations.
|
||||
|
||||
* `release: $HELM_RELEASE`: Prometheus can only detect PrometheusRule with this label. See [here](#prometheus-can-only-detect-this-label) for more details.
|
||||
|
||||
* PrometheusRule can be reloaded at runtime. Use `kubectl apply {modified prometheusRules.yaml}` to reconfigure the rules if needed.
|
||||
|
||||
|
||||
## Step 9: Define Alert Conditions with alerting rules (optional)
|
||||
|
||||
[Alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) allow us to define alert conditions based on [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/) expressions and to send notifications about firing alerts to [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager) which adds summarization, notification rate limiting, silencing and alert dependencies on top of the simple alert definitions.
|
||||
|
||||
```yaml
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: ray-cluster-gcs-rules
|
||||
namespace: prometheus-system
|
||||
labels:
|
||||
# `release: $HELM_RELEASE`: Prometheus can only detect Alerting Rules with this label.
|
||||
release: prometheus
|
||||
spec:
|
||||
groups:
|
||||
- name: ray-cluster-main-staging-gcs.rules
|
||||
# How often rules in the group are evaluated.
|
||||
interval: 30s
|
||||
rules:
|
||||
- alert: MissingMetricRayGlobalControlStore
|
||||
# A set of informational labels. Annotations can be used to store longer additional information compared to rules.0.labels.
|
||||
annotations:
|
||||
description: Ray GCS is not emitting any metrics for Resource Update requests
|
||||
summary: Ray GCS is not emitting metrics anymore
|
||||
# PromQL expression.
|
||||
expr: |
|
||||
(
|
||||
absent(ray_gcs_update_resource_usage_time_bucket) == 1
|
||||
)
|
||||
# Time that Prometheus will wait and check if the alert continues to be active during each evaluation before firing the alert.
|
||||
# firing alerts may be due to false positives or noise if the setting value is too small.
|
||||
# On the other hand, if the value is too big, the alerts may not be handled in time.
|
||||
for: 5m
|
||||
# A set of additional labels to be attached to the alert.
|
||||
# It is possible to overwrite the labels in metadata.labels, so make sure one of the labels match the label in ruleSelector.matchLabels.
|
||||
labels:
|
||||
severity: critical
|
||||
```
|
||||
|
||||
* The PromQL expression above checks if there is no time series exist for `ray_gcs_update_resource_usage_time_bucket` metric. See [absent()](https://prometheus.io/docs/prometheus/latest/querying/functions/#absent) for more detail.
|
||||
|
||||
* The alerting rule above is one of rules defined in [prometheusRules.yaml](https://github.com/ray-project/kuberay/blob/master/config/prometheus/rules/prometheusRules.yaml), and it is created by **install.sh**. Hence, no need to create anything here.
|
||||
|
||||
* Alerting rules are configured in the same way as recording rules.
|
||||
|
||||
## Step 10: Access Prometheus Web UI
|
||||
|
||||
```sh
|
||||
# Forward the port of Prometheus Web UI in the Prometheus server Pod.
|
||||
kubectl port-forward -n prometheus-system service/prometheus-kube-prometheus-prometheus http-web
|
||||
```
|
||||
|
||||
- Go to `${YOUR_IP}:9090/targets` (e.g. `127.0.0.1:9090/targets`). You should be able to see:
|
||||
- `podMonitor/prometheus-system/ray-workers-monitor/0 (1/1 up)`
|
||||
- `serviceMonitor/prometheus-system/ray-head-monitor/0 (1/1 up)`
|
||||
|
||||
|
||||
|
||||
- Go to `${YOUR_IP}:9090/graph`. You should be able to query:
|
||||
- [System Metrics](https://docs.ray.io/en/latest/ray-observability/ray-metrics.html#system-metrics)
|
||||
- [Application Level Metrics](https://docs.ray.io/en/latest/ray-observability/ray-metrics.html#application-level-metrics)
|
||||
- Custom Metrics defined in Recording Rules (e.g. `ray_gcs_availability_30d`)
|
||||
|
||||
- Go to `${YOUR_IP}:9090/alerts`. You should be able to see:
|
||||
- Alerting Rules (e.g. `MissingMetricRayGlobalControlStore`).
|
||||
|
||||
## Step 11: Access Grafana
|
||||
|
||||
```sh
|
||||
# Forward the Grafana port
|
||||
kubectl port-forward -n prometheus-system service/prometheus-grafana 3000:http-web
|
||||
# Note: You need to update `RAY_GRAFANA_IFRAME_HOST` if you expose Grafana to a different port.
|
||||
|
||||
# Check ${YOUR_IP}:3000/login for the Grafana login page (e.g. 127.0.0.1:3000/login).
|
||||
# The default username is "admin" and the password is "prom-operator".
|
||||
```
|
||||
|
||||
> Note: `kubectl port-forward` is not recommended for production use.
|
||||
Refer to [this Grafana document](https://grafana.com/tutorials/run-grafana-behind-a-proxy/) for exposing Grafana behind a reverse proxy.
|
||||
|
||||
* The default password is defined by `grafana.adminPassword` in the [values.yaml](https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml) of the kube-prometheus-stack chart.
|
||||
|
||||
## Step 12: Import Grafana dashboards manually (optional)
|
||||
|
||||
If `--auto-load-dashboard true` is set when running `install.sh`, you can skip this step.
|
||||
|
||||
* Import Grafana dashboards manually
|
||||
* Click "Dashboards" icon in the left panel.
|
||||
* Click "New".
|
||||
* Click "Import".
|
||||
* Click "Upload JSON file".
|
||||
* Choose a JSON file.
|
||||
* Case 1: If you are using Ray 2.41.0, you can use [the sample config files in GitHub repository](https://github.com/ray-project/kuberay/tree/master/config/grafana). The file names have a pattern of `xxx_grafana_dashboard.json`.
|
||||
* Case 2: Otherwise, import the JSON files from the head Pod's `/tmp/ray/session_latest/metrics/grafana/dashboards/` directory. You can use `kubectl cp` to copy the files from the head Pod to your local machine. `kubectl cp $(kubectl get pods --selector ray.io/node-type=head,ray.io/cluster=raycluster-embed-grafana -o jsonpath={..metadata.name}):/tmp/ray/session_latest/metrics/grafana/dashboards/ /tmp/`
|
||||
* Click "Import".
|
||||
|
||||
## Step 13: View metrics from different RayCluster CRs
|
||||
|
||||
Once the Ray Dashboard is imported into Grafana, you can filter metrics by using the `Cluster` variable. Ray Dashboard automatically applies this variable by default when you use the provided `PodMonitor` configuration. You don't need any additional setup for this labeling.
|
||||
|
||||
If you have multiple RayCluster custom resources, the `Cluster` variable allows you to filter metrics specific to a particular cluster. This feature ensures that you can easily monitor or debug individual RayCluster instances without being overwhelmed by the data from all clusters.
|
||||
|
||||
For example, in the following figures, one selects the metrics from the RayCluster `raycluster-embed-grafana`, and the other selects metrics from the RayCluster `raycluster-embed-grafana-2`.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## Step 14: View the KubeRay operator dashboard
|
||||
|
||||
After importing the KubeRay operator dashboard into Grafana, you can monitor metrics from the KubeRay operator. The dashboard includes a dropdown menu that lets you filter and view controller runtime metrics for specific Ray custom resources CRs: `RayCluster`, `RayJob`, and `RayService`.
|
||||
|
||||
The KubeRay operator dashboard should look like this: 
|
||||
|
||||
## Step 15: Embed Grafana panels in the Ray dashboard (optional)
|
||||
|
||||
```sh
|
||||
kubectl port-forward service/raycluster-embed-grafana-head-svc dashboard
|
||||
# Visit http://127.0.0.1:8265/#/metrics in your browser.
|
||||
```
|
||||
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
(kuberay-pyspy-integration)=
|
||||
|
||||
# Profiling with py-spy
|
||||
|
||||
## Stack trace and CPU profiling
|
||||
[py-spy](https://github.com/benfred/py-spy/tree/master) is a sampling profiler for Python programs. It lets you visualize what your Python program is spending time on without restarting the program or modifying the code in any way. This section describes how to configure RayCluster YAML file to enable py-spy and see Stack Trace and CPU Flame Graph on Ray Dashboard.
|
||||
|
||||
## Prerequisite
|
||||
py-spy requires the `SYS_PTRACE` capability to read process memory. However, Kubernetes omits this capability by default. To enable profiling, add the following to the `template.spec.containers` for both the head and worker Pods.
|
||||
|
||||
```bash
|
||||
securityContext:
|
||||
capabilities:
|
||||
add:
|
||||
- SYS_PTRACE
|
||||
```
|
||||
**Notes:**
|
||||
- The `baseline` and `restricted` Pod Security Standards forbid adding `SYS_PTRACE`. See [Pod Security Standards](https://kubernetes.io/docs/concepts/security/pod-security-standards/) for more details.
|
||||
|
||||
## Check CPU flame graph and stack trace on Ray Dashboard
|
||||
|
||||
### Step 1: Create a Kind cluster
|
||||
|
||||
```bash
|
||||
kind create cluster
|
||||
```
|
||||
|
||||
### Step 2: Install the KubeRay operator
|
||||
|
||||
Follow [this document](kuberay-operator-deploy) to install the latest stable KubeRay operator using Helm repository.
|
||||
|
||||
### Step 3: Create a RayCluster with `SYS_PTRACE` capability
|
||||
|
||||
```bash
|
||||
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-cluster.py-spy.yaml
|
||||
```
|
||||
|
||||
### Step 4: Forward the dashboard port
|
||||
|
||||
```bash
|
||||
kubectl port-forward svc/raycluster-py-spy-head-svc 8265:8265
|
||||
```
|
||||
|
||||
### Step 5: Run a sample job within the head Pod
|
||||
|
||||
```bash
|
||||
# Log in to the head Pod
|
||||
kubectl exec -it ${YOUR_HEAD_POD} -- bash
|
||||
|
||||
# (Head Pod) Run a sample job in the Pod
|
||||
# `long_running_task` includes a `while True` loop to ensure the task remains actively running indefinitely.
|
||||
# This allows you ample time to view the Stack Trace and CPU Flame Graph via Ray Dashboard.
|
||||
python3 samples/long_running_task.py
|
||||
```
|
||||
|
||||
**Notes:**
|
||||
- If you're running your own examples and encounter the error `Failed to write flamegraph: I/O error: No stack counts found` when viewing CPU Flame Graph, it might be due to the process being idle. Notably, using the `sleep` function can lead to this state. In such situations, py-spy filters out the idle stack traces. Refer to this [issue](https://github.com/benfred/py-spy/issues/321#issuecomment-731848950) for more information.
|
||||
|
||||
### Step 6: Profile using Ray Dashboard
|
||||
|
||||
- Visit http://localhost:8265/#/cluster.
|
||||
- Click `Stack Trace` for `ray::long_running_task`. 
|
||||
- Click `CPU Flame Graph` for `ray::long_running_task`. 
|
||||
- For additional details on using the profiler, See [Python CPU profiling in the Dashboard](https://docs.ray.io/en/latest/ray-observability/user-guides/debug-apps/optimize-performance.html#python-cpu-profiling-in-the-dashboard).
|
||||
|
||||
### Step 7: Clean up
|
||||
|
||||
```bash
|
||||
kubectl delete -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-cluster.py-spy.yaml
|
||||
helm uninstall kuberay-operator
|
||||
```
|
||||
@@ -0,0 +1,57 @@
|
||||
(kuberay-scheduler-plugins)=
|
||||
# KubeRay integration with scheduler plugins
|
||||
|
||||
The [kubernetes-sigs/scheduler-plugins](https://github.com/kubernetes-sigs/scheduler-plugins) repository provides out-of-tree scheduler plugins based on the scheduler framework.
|
||||
|
||||
Starting with KubeRay v1.4.0, KubeRay integrates with the [PodGroup API](https://github.com/kubernetes-sigs/scheduler-plugins/blob/93126eabdf526010bf697d5963d849eab7e8e898/site/content/en/docs/plugins/coscheduling.md) provided by scheduler plugins to support gang scheduling for RayCluster custom resources.
|
||||
|
||||
## Step 1: Create a Kubernetes cluster with Kind
|
||||
|
||||
```sh
|
||||
kind create cluster --image=kindest/node:v1.26.0
|
||||
```
|
||||
|
||||
## Step 2: Install scheduler plugins
|
||||
|
||||
Follow the [installation guide](https://scheduler-plugins.sigs.k8s.io/docs/user-guide/installation/) in the scheduler-plugins repository to install the scheduler plugins.
|
||||
|
||||
:::{note}
|
||||
|
||||
There are two modes for installing the scheduler plugins: *single scheduler mode* and *second scheduler mode*.
|
||||
|
||||
KubeRay v1.4.0 only supports the *single scheduler mode*. You need to have the access to configure Kubernetes control plane to replace the default scheduler with the scheduler plugins.
|
||||
|
||||
:::
|
||||
|
||||
## Step 3: Install KubeRay operator with scheduler plugins enabled
|
||||
|
||||
KubeRay v1.4.0 and later versions support scheduler plugins.
|
||||
|
||||
```sh
|
||||
helm install kuberay-operator kuberay/kuberay-operator --version 1.6.0 --set batchScheduler.name=scheduler-plugins
|
||||
```
|
||||
|
||||
## Step 4: Deploy a RayCluster with gang scheduling
|
||||
|
||||
```sh
|
||||
# Configure the RayCluster with label `ray.io/gang-scheduling-enabled: "true"`
|
||||
# to enable gang scheduling.
|
||||
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/release-1.4/ray-operator/config/samples/ray-cluster.scheduler-plugins.yaml
|
||||
```
|
||||
|
||||
## Step 5: Verify Ray Pods and PodGroup
|
||||
|
||||
Note that if you use "second scheduler mode," which KubeRay currently doesn't support, the following commands still show similar results. However, the Ray Pods don't get scheduled in a gang scheduling manner. Make sure to use "single scheduler mode" to enable gang scheduling.
|
||||
|
||||
```sh
|
||||
kubectl get podgroups.scheduling.x-k8s.io
|
||||
# NAME PHASE MINMEMBER RUNNING SUCCEEDED FAILED AGE
|
||||
# test-podgroup-0 Running 3 3 2m25s
|
||||
|
||||
# All Ray Pods (1 head and 2 workers) belong to the same PodGroup.
|
||||
kubectl get pods -L scheduling.x-k8s.io/pod-group
|
||||
# NAME READY STATUS RESTARTS AGE POD-GROUP
|
||||
# test-podgroup-0-head 1/1 Running 0 3m30s test-podgroup-0
|
||||
# test-podgroup-0-worker-worker-4vc6j 1/1 Running 0 3m30s test-podgroup-0
|
||||
# test-podgroup-0-worker-worker-ntm9f 1/1 Running 0 3m30s test-podgroup-0
|
||||
```
|
||||
@@ -0,0 +1,476 @@
|
||||
(kuberay-volcano)=
|
||||
# KubeRay integration with Volcano
|
||||
|
||||
[Volcano](https://github.com/volcano-sh/volcano) is a batch scheduling system built on Kubernetes, providing gang scheduling, job queues, fair scheduling policies, and network topology-aware scheduling. KubeRay integrates natively with Volcano for RayCluster, RayJob, and RayService, enabling more efficient scheduling of Ray pods in multi-tenant Kubernetes environments.
|
||||
|
||||
This guide covers [setup instructions](#setup), [configuration options](#step-4-install-a-raycluster-with-the-volcano-scheduler), and [examples](#example) demonstrating gang scheduling for both RayCluster and RayJob.
|
||||
|
||||
## Setup
|
||||
|
||||
### Step 1: Create a Kubernetes cluster with KinD
|
||||
Run the following command in a terminal:
|
||||
|
||||
```shell
|
||||
kind create cluster
|
||||
```
|
||||
|
||||
### Step 2: Install Volcano
|
||||
|
||||
You need to successfully install Volcano on your Kubernetes cluster before enabling Volcano integration with KubeRay. See [Quick Start Guide](https://github.com/volcano-sh/volcano#quick-start-guide) for Volcano installation instructions.
|
||||
|
||||
### Step 3: Install the KubeRay Operator with batch scheduling
|
||||
|
||||
Deploy the KubeRay Operator with the `--batch-scheduler=volcano` flag to enable Volcano batch scheduling support.
|
||||
|
||||
When installing KubeRay Operator using Helm, you should use one of these two options:
|
||||
|
||||
* Set `batchScheduler.name` to `volcano` in your [`values.yaml`](https://github.com/ray-project/kuberay/blob/753dc05dbed5f6fe61db3a43b34a1b350f26324c/helm-chart/kuberay-operator/values.yaml#L48) file:
|
||||
```shell
|
||||
# values.yaml file
|
||||
batchScheduler:
|
||||
name: volcano
|
||||
```
|
||||
|
||||
* Pass the `--set batchScheduler.name=volcano` flag when running on the command line:
|
||||
```shell
|
||||
# Install the Helm chart with the --batch-scheduler=volcano flag
|
||||
helm install kuberay-operator kuberay/kuberay-operator --version 1.6.0 --set batchScheduler.name=volcano
|
||||
```
|
||||
|
||||
### Step 4: Install a RayCluster with the Volcano scheduler
|
||||
```shell
|
||||
# Path: kuberay/ray-operator/config/samples
|
||||
curl -LO https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-cluster.volcano-scheduler.yaml
|
||||
kubectl apply -f ray-cluster.volcano-scheduler.yaml
|
||||
|
||||
# Check the RayCluster
|
||||
kubectl get pod -l ray.io/cluster=test-cluster-0
|
||||
# NAME READY STATUS RESTARTS AGE
|
||||
# test-cluster-0-head-jj9bg 1/1 Running 0 36s
|
||||
```
|
||||
|
||||
You can also provide the following labels in the RayCluster, RayJob and RayService metadata:
|
||||
|
||||
- `ray.io/priority-class-name`: The cluster priority class as defined by [Kubernetes](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass)
|
||||
- This label only works after you create a `PriorityClass` resource
|
||||
- ```shell
|
||||
labels:
|
||||
ray.io/priority-class-name: <replace with correct PriorityClass resource name>
|
||||
```
|
||||
- `volcano.sh/queue-name`: The Volcano [queue](https://volcano.sh/en/docs/queue/) name the cluster submits to.
|
||||
- This label only works after you create a `Queue` resource
|
||||
- ```shell
|
||||
labels:
|
||||
volcano.sh/queue-name: <replace with correct Queue resource name>
|
||||
```
|
||||
- `volcano.sh/network-topology-mode`: Enables [network topology-aware scheduling](https://volcano.sh/en/docs/network_topology_aware_scheduling/) to optimize pod placement based on network proximity, reducing inter-node communication latency for distributed workloads. Valid values are `soft` (best-effort) or `hard` (strict enforcement).
|
||||
- This label only works after you create a `HyperNode` resource
|
||||
- ```shell
|
||||
labels:
|
||||
volcano.sh/network-topology-mode: "soft" # or "hard"
|
||||
```
|
||||
- `volcano.sh/network-topology-highest-tier-allowed`: Specifies the highest network topology tier for pod placement, restricting pods to be scheduled within the specified tier boundary. The value must match a tier defined in your `HyperNode` resource. Must be used together with `volcano.sh/network-topology-mode`.
|
||||
- This label only works after you create a `HyperNode` resource
|
||||
- ```shell
|
||||
labels:
|
||||
volcano.sh/network-topology-highest-tier-allowed: <tier>
|
||||
```
|
||||
|
||||
**Note**:
|
||||
- Starting from KubeRay v1.3.0, you **no** longer need to add the `ray.io/scheduler-name: volcano` label to your RayCluster/RayJob. The batch scheduler is now configured at the operator level using the `--batch-scheduler=volcano` flag.
|
||||
- When autoscaling is enabled, KubeRay uses `minReplicas` to calculate the minimum resources required for gang scheduling. Otherwise, it uses the `desired` replicas value.
|
||||
|
||||
### Step 5: Use Volcano for batch scheduling
|
||||
|
||||
For guidance, see [examples](https://github.com/volcano-sh/volcano/tree/master/example).
|
||||
|
||||
## Example
|
||||
|
||||
Before going through the example, remove any running Ray Clusters to ensure a successful run through of the example below.
|
||||
```shell
|
||||
kubectl delete raycluster --all
|
||||
```
|
||||
|
||||
### Gang scheduling
|
||||
|
||||
This example walks through how gang scheduling works with Volcano and KubeRay.
|
||||
|
||||
First, create a queue with a capacity of 4 CPUs and 6Gi of RAM:
|
||||
|
||||
```shell
|
||||
kubectl create -f - <<EOF
|
||||
apiVersion: scheduling.volcano.sh/v1beta1
|
||||
kind: Queue
|
||||
metadata:
|
||||
name: kuberay-test-queue
|
||||
spec:
|
||||
weight: 1
|
||||
capability:
|
||||
cpu: 4
|
||||
memory: 6Gi
|
||||
EOF
|
||||
```
|
||||
|
||||
The **weight** in the definition above indicates the relative weight of a queue in a cluster resource division. Use this parameter in cases where the total **capability** of all the queues in your cluster exceeds the total available resources, forcing the queues to share among themselves. Queues with higher weight are allocated a proportionally larger share of the total resources.
|
||||
|
||||
The **capability** is a hard constraint on the maximum resources the queue supports at any given time. You can update it as needed to allow more or fewer workloads to run at a time.
|
||||
|
||||
Next, create a RayCluster with a head node (1 CPU + 2Gi of RAM) and two workers (1 CPU + 1Gi of RAM each), for a total of 3 CPU and 4Gi of RAM:
|
||||
|
||||
```shell
|
||||
# Path: kuberay/ray-operator/config/samples
|
||||
# Includes the `volcano.sh/queue-name: kuberay-test-queue` label in the metadata.labels
|
||||
curl -LO https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-cluster.volcano-scheduler-queue.yaml
|
||||
kubectl apply -f ray-cluster.volcano-scheduler-queue.yaml
|
||||
```
|
||||
|
||||
Because the queue has a capacity of 4 CPU and 6Gi of RAM, this resource should schedule successfully without any issues. You can verify this by checking the status of the cluster's Volcano PodGroup to see that the phase is `Running` and the last status is `Scheduled`:
|
||||
|
||||
```shell
|
||||
kubectl get podgroup ray-test-cluster-0-pg -o yaml
|
||||
|
||||
# apiVersion: scheduling.volcano.sh/v1beta1
|
||||
# kind: PodGroup
|
||||
# metadata:
|
||||
# creationTimestamp: "2022-12-01T04:43:30Z"
|
||||
# generation: 2
|
||||
# name: ray-test-cluster-0-pg
|
||||
# namespace: test
|
||||
# ownerReferences:
|
||||
# - apiVersion: ray.io/v1alpha1
|
||||
# blockOwnerDeletion: true
|
||||
# controller: true
|
||||
# kind: RayCluster
|
||||
# name: test-cluster-0
|
||||
# uid: 7979b169-f0b0-42b7-8031-daef522d25cf
|
||||
# resourceVersion: "4427347"
|
||||
# uid: 78902d3d-b490-47eb-ba12-d6f8b721a579
|
||||
# spec:
|
||||
# minMember: 3
|
||||
# minResources:
|
||||
# cpu: "3"
|
||||
# memory: 4Gi
|
||||
# queue: kuberay-test-queue
|
||||
# status:
|
||||
# conditions:
|
||||
# - lastTransitionTime: "2022-12-01T04:43:31Z"
|
||||
# reason: tasks in the gang are ready to be scheduled
|
||||
# status: "True"
|
||||
# transitionID: f89f3062-ebd7-486b-8763-18ccdba1d585
|
||||
# type: Scheduled
|
||||
# phase: Running
|
||||
```
|
||||
|
||||
Check the status of the queue to see allocated resources:
|
||||
|
||||
```shell
|
||||
kubectl get queue kuberay-test-queue -o yaml
|
||||
|
||||
# apiVersion: scheduling.volcano.sh/v1beta1
|
||||
# kind: Queue
|
||||
# metadata:
|
||||
# creationTimestamp: "2022-12-01T04:43:21Z"
|
||||
# generation: 1
|
||||
# name: kuberay-test-queue
|
||||
# resourceVersion: "4427348"
|
||||
# uid: a6c4f9df-d58c-4da8-8a58-e01c93eca45a
|
||||
# spec:
|
||||
# capability:
|
||||
# cpu: 4
|
||||
# memory: 6Gi
|
||||
# reclaimable: true
|
||||
# weight: 1
|
||||
# status:
|
||||
# allocated:
|
||||
# cpu: "3"
|
||||
# memory: 4Gi
|
||||
# pods: "3"
|
||||
# reservation: {}
|
||||
# state: Open
|
||||
```
|
||||
|
||||
Next, add an additional RayCluster with the same configuration of head and worker nodes, but with a different name:
|
||||
|
||||
```shell
|
||||
# Path: kuberay/ray-operator/config/samples
|
||||
# Includes the `volcano.sh/queue-name: kuberay-test-queue` label in the metadata.labels
|
||||
# Replaces the name to test-cluster-1
|
||||
sed 's/test-cluster-0/test-cluster-1/' ray-cluster.volcano-scheduler-queue.yaml | kubectl apply -f-
|
||||
```
|
||||
|
||||
Check the status of its PodGroup to see that its phase is `Pending` and the last status is `Unschedulable`:
|
||||
|
||||
```shell
|
||||
kubectl get podgroup ray-test-cluster-1-pg -o yaml
|
||||
|
||||
# apiVersion: scheduling.volcano.sh/v1beta1
|
||||
# kind: PodGroup
|
||||
# metadata:
|
||||
# creationTimestamp: "2022-12-01T04:48:18Z"
|
||||
# generation: 2
|
||||
# name: ray-test-cluster-1-pg
|
||||
# namespace: test
|
||||
# ownerReferences:
|
||||
# - apiVersion: ray.io/v1alpha1
|
||||
# blockOwnerDeletion: true
|
||||
# controller: true
|
||||
# kind: RayCluster
|
||||
# name: test-cluster-1
|
||||
# uid: b3cf83dc-ef3a-4bb1-9c42-7d2a39c53358
|
||||
# resourceVersion: "4427976"
|
||||
# uid: 9087dd08-8f48-4592-a62e-21e9345b0872
|
||||
# spec:
|
||||
# minMember: 3
|
||||
# minResources:
|
||||
# cpu: "3"
|
||||
# memory: 4Gi
|
||||
# queue: kuberay-test-queue
|
||||
# status:
|
||||
# conditions:
|
||||
# - lastTransitionTime: "2022-12-01T04:48:19Z"
|
||||
# message: '3/3 tasks in gang unschedulable: pod group is not ready, 3 Pending,
|
||||
# 3 minAvailable; Pending: 3 Undetermined'
|
||||
# reason: NotEnoughResources
|
||||
# status: "True"
|
||||
# transitionID: 3956b64f-fc52-4779-831e-d379648eecfc
|
||||
# type: Unschedulable
|
||||
# phase: Pending
|
||||
```
|
||||
|
||||
Because the new cluster requires more CPU and RAM than our queue allows, even though one of the pods would fit in the remaining 1 CPU and 2Gi of RAM, none of the cluster's pods are placed until there is enough room for all the pods. Without using Volcano for gang scheduling in this way, one of the pods would ordinarily be placed, leading to the cluster being partially allocated, and some jobs (like [Horovod](https://github.com/horovod/horovod) training) being stuck waiting for resources to become available.
|
||||
|
||||
See the effect this has on scheduling the pods for our new RayCluster, which are listed as `Pending`:
|
||||
|
||||
```shell
|
||||
kubectl get pods
|
||||
|
||||
# NAME READY STATUS RESTARTS AGE
|
||||
# test-cluster-0-worker-worker-ddfbz 1/1 Running 0 7m
|
||||
# test-cluster-0-head 1/1 Running 0 7m
|
||||
# test-cluster-0-worker-worker-57pc7 1/1 Running 0 6m59s
|
||||
# test-cluster-1-worker-worker-6tzf7 0/1 Pending 0 2m12s
|
||||
# test-cluster-1-head 0/1 Pending 0 2m12s
|
||||
# test-cluster-1-worker-worker-n5g8k 0/1 Pending 0 2m12s
|
||||
```
|
||||
|
||||
Look at the pod details to see that Volcano cannot schedule the gang:
|
||||
|
||||
```shell
|
||||
kubectl describe pod test-cluster-1-head-6668q | tail -n 3
|
||||
|
||||
# Type Reason Age From Message
|
||||
# ---- ------ ---- ---- -------
|
||||
# Warning FailedScheduling 4m5s volcano 3/3 tasks in gang unschedulable: pod group is not ready, 3 Pending, 3 minAvailable; Pending: 3 Undetermined
|
||||
```
|
||||
|
||||
Delete the first RayCluster to make space in the queue:
|
||||
|
||||
```shell
|
||||
kubectl delete raycluster test-cluster-0
|
||||
```
|
||||
|
||||
The PodGroup for the second cluster changed to the `Running` state, because enough resources are now available to schedule the entire set of pods:
|
||||
|
||||
```shell
|
||||
kubectl get podgroup ray-test-cluster-1-pg -o yaml
|
||||
|
||||
# apiVersion: scheduling.volcano.sh/v1beta1
|
||||
# kind: PodGroup
|
||||
# metadata:
|
||||
# creationTimestamp: "2022-12-01T04:48:18Z"
|
||||
# generation: 9
|
||||
# name: ray-test-cluster-1-pg
|
||||
# namespace: test
|
||||
# ownerReferences:
|
||||
# - apiVersion: ray.io/v1alpha1
|
||||
# blockOwnerDeletion: true
|
||||
# controller: true
|
||||
# kind: RayCluster
|
||||
# name: test-cluster-1
|
||||
# uid: b3cf83dc-ef3a-4bb1-9c42-7d2a39c53358
|
||||
# resourceVersion: "4428864"
|
||||
# uid: 9087dd08-8f48-4592-a62e-21e9345b0872
|
||||
# spec:
|
||||
# minMember: 3
|
||||
# minResources:
|
||||
# cpu: "3"
|
||||
# memory: 4Gi
|
||||
# queue: kuberay-test-queue
|
||||
# status:
|
||||
# conditions:
|
||||
# - lastTransitionTime: "2022-12-01T04:54:04Z"
|
||||
# message: '3/3 tasks in gang unschedulable: pod group is not ready, 3 Pending,
|
||||
# 3 minAvailable; Pending: 3 Undetermined'
|
||||
# reason: NotEnoughResources
|
||||
# status: "True"
|
||||
# transitionID: db90bbf0-6845-441b-8992-d0e85f78db77
|
||||
# type: Unschedulable
|
||||
# - lastTransitionTime: "2022-12-01T04:55:10Z"
|
||||
# reason: tasks in the gang are ready to be scheduled
|
||||
# status: "True"
|
||||
# transitionID: 72bbf1b3-d501-4528-a59d-479504f3eaf5
|
||||
# type: Scheduled
|
||||
# phase: Running
|
||||
# running: 3
|
||||
```
|
||||
|
||||
Check the pods again to see that the second cluster is now up and running:
|
||||
|
||||
```shell
|
||||
kubectl get pods
|
||||
|
||||
# NAME READY STATUS RESTARTS AGE
|
||||
# test-cluster-1-worker-worker-n5g8k 1/1 Running 0 9m4s
|
||||
# test-cluster-1-head 1/1 Running 0 9m4s
|
||||
# test-cluster-1-worker-worker-6tzf7 1/1 Running 0 9m4s
|
||||
```
|
||||
|
||||
Finally, clean up the remaining cluster and queue:
|
||||
|
||||
```shell
|
||||
kubectl delete raycluster test-cluster-1
|
||||
kubectl delete queue kuberay-test-queue
|
||||
```
|
||||
|
||||
### Use Volcano for RayJob gang scheduling
|
||||
|
||||
Starting with KubeRay 1.6.0, KubeRay supports gang scheduling for RayJob custom resources.
|
||||
|
||||
First, create a queue with a capacity of 4 CPUs and 6Gi of RAM and RayJob a with a head node (1 CPU + 2Gi of RAM), two workers (1 CPU + 1Gi of RAM each) and a submitter pod (0.5 CPU + 200Mi of RAM), for a total of 3500m CPU and 4296Mi of RAM
|
||||
|
||||
```shell
|
||||
curl -LO https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-job.volcano-scheduler-queue.yaml
|
||||
kubectl apply -f ray-job.volcano-scheduler-queue.yaml
|
||||
```
|
||||
|
||||
Wait until all pods are in the Running state.
|
||||
```shell
|
||||
kubectl get pod
|
||||
|
||||
# NAME READY STATUS RESTARTS AGE
|
||||
# rayjob-sample-0-k449j-head-rlgxj 1/1 Running 0 93s
|
||||
# rayjob-sample-0-k449j-small-group-worker-c6dt8 1/1 Running 0 93s
|
||||
# rayjob-sample-0-k449j-small-group-worker-cq6xn 1/1 Running 0 93s
|
||||
# rayjob-sample-0-qmm8s 0/1 Completed 0 32s
|
||||
```
|
||||
|
||||
Add an additional RayJob with the same configuration but with a different name
|
||||
```shell
|
||||
sed 's/rayjob-sample-0/rayjob-sample-1/' ray-job.volcano-scheduler-queue.yaml | kubectl apply -f-
|
||||
```
|
||||
|
||||
All the pods stuck on pending for new RayJob
|
||||
```shell
|
||||
# NAME READY STATUS RESTARTS AGE
|
||||
# rayjob-sample-0-k449j-head-rlgxj 1/1 Running 0 3m27s
|
||||
# rayjob-sample-0-k449j-small-group-worker-c6dt8 1/1 Running 0 3m27s
|
||||
# rayjob-sample-0-k449j-small-group-worker-cq6xn 1/1 Running 0 3m27s
|
||||
# rayjob-sample-0-qmm8s 0/1 Completed 0 2m26s
|
||||
# rayjob-sample-1-mvgqf-head-qb7wm 0/1 Pending 0 21s
|
||||
# rayjob-sample-1-mvgqf-small-group-worker-jfzt5 0/1 Pending 0 21s
|
||||
# rayjob-sample-1-mvgqf-small-group-worker-ng765 0/1 Pending 0 21s
|
||||
```
|
||||
|
||||
Check the status of its PodGroup to see that its phase is `Pending` and the last status is `Unschedulable`:
|
||||
```shell
|
||||
kubectl get podgroup ray-rayjob-sample-1-pg -o yaml
|
||||
|
||||
# apiVersion: scheduling.volcano.sh/v1beta1
|
||||
# kind: PodGroup
|
||||
# metadata:
|
||||
# creationTimestamp: "2025-10-30T17:10:18Z"
|
||||
# generation: 2
|
||||
# name: ray-rayjob-sample-1-pg
|
||||
# namespace: default
|
||||
# ownerReferences:
|
||||
# - apiVersion: ray.io/v1
|
||||
# blockOwnerDeletion: true
|
||||
# controller: true
|
||||
# kind: RayJob
|
||||
# name: rayjob-sample-1
|
||||
# uid: 5835c896-c75d-4692-b10a-2871a79f141a
|
||||
# resourceVersion: "3226"
|
||||
# uid: 9fd55cbd-ba69-456d-b305-f61ffd6d935d
|
||||
# spec:
|
||||
# minMember: 3
|
||||
# minResources:
|
||||
# cpu: 3500m
|
||||
# memory: 4296Mi
|
||||
# queue: kuberay-test-queue
|
||||
# status:
|
||||
# conditions:
|
||||
# - lastTransitionTime: "2025-10-30T17:10:18Z"
|
||||
# message: '3/3 tasks in gang unschedulable: pod group is not ready, 3 Pending,
|
||||
# 3 minAvailable; Pending: 3 Unschedulable'
|
||||
# reason: NotEnoughResources
|
||||
# status: "True"
|
||||
# transitionID: 7866f533-6590-4a4d-83cf-8f1db0214609
|
||||
# type: Unschedulable
|
||||
# phase: Pending
|
||||
```
|
||||
|
||||
Delete the first RayJob to make space in the queue.
|
||||
```shell
|
||||
kubectl delete rayjob rayjob-sample-0
|
||||
```
|
||||
|
||||
The PodGroup for the second cluster changed to the Running state, because enough resources are now available to schedule the entire set of pods.
|
||||
```shell
|
||||
kubectl get podgroup ray-rayjob-sample-1-pg -o yaml
|
||||
|
||||
# apiVersion: scheduling.volcano.sh/v1beta1
|
||||
# kind: PodGroup
|
||||
# metadata:
|
||||
# creationTimestamp: "2025-10-30T17:10:18Z"
|
||||
# generation: 7
|
||||
# name: ray-rayjob-sample-1-pg
|
||||
# namespace: default
|
||||
# ownerReferences:
|
||||
# - apiVersion: ray.io/v1
|
||||
# blockOwnerDeletion: true
|
||||
# controller: true
|
||||
# kind: RayJob
|
||||
# name: rayjob-sample-1
|
||||
# uid: 5835c896-c75d-4692-b10a-2871a79f141a
|
||||
# resourceVersion: "3724"
|
||||
# uid: 9fd55cbd-ba69-456d-b305-f61ffd6d935d
|
||||
# spec:
|
||||
# minMember: 3
|
||||
# minResources:
|
||||
# cpu: 3500m
|
||||
# memory: 4296Mi
|
||||
# queue: kuberay-test-queue
|
||||
# status:
|
||||
# conditions:
|
||||
# - lastTransitionTime: "2025-10-30T17:10:18Z"
|
||||
# message: '3/3 tasks in gang unschedulable: pod group is not ready, 3 Pending,
|
||||
# 3 minAvailable; Pending: 3 Unschedulable'
|
||||
# reason: NotEnoughResources
|
||||
# status: "True"
|
||||
# transitionID: 7866f533-6590-4a4d-83cf-8f1db0214609
|
||||
# type: Unschedulable
|
||||
# - lastTransitionTime: "2025-10-30T17:14:44Z"
|
||||
# reason: tasks in gang are ready to be scheduled
|
||||
# status: "True"
|
||||
# transitionID: 36e0222d-eee3-444a-9889-5b9c255f41af
|
||||
# type: Scheduled
|
||||
# phase: Running
|
||||
# running: 4
|
||||
```
|
||||
|
||||
Check the pods again to see that the second RayJob is now up and running:
|
||||
```shell
|
||||
kubectl get pod
|
||||
# NAME READY STATUS RESTARTS AGE
|
||||
# rayjob-sample-1-mvgqf-head-qb7wm 1/1 Running 0 5m47s
|
||||
# rayjob-sample-1-mvgqf-small-group-worker-jfzt5 1/1 Running 0 5m47s
|
||||
# rayjob-sample-1-mvgqf-small-group-worker-ng765 1/1 Running 0 5m47s
|
||||
# rayjob-sample-1-tcd4m 0/1 Completed 0 84s
|
||||
```
|
||||
|
||||
Finally, clean up the remaining rayjob, queue and configmap:
|
||||
```
|
||||
kubectl delete rayjob rayjob-sample-1
|
||||
kubectl delete queue kuberay-test-queue
|
||||
kubectl delete configmap ray-job-code-sample
|
||||
```
|
||||
@@ -0,0 +1,189 @@
|
||||
(kuberay-yunikorn)=
|
||||
|
||||
# KubeRay integration with Apache YuniKorn
|
||||
|
||||
[Apache YuniKorn](https://yunikorn.apache.org/) is a light-weight, universal resource scheduler for container orchestrator systems. It performs fine-grained resource sharing for various workloads efficiently on a large scale, multi-tenant, and cloud-native environment. YuniKorn brings a unified, cross-platform, scheduling experience for mixed workloads that consist of stateless batch workloads and stateful services.
|
||||
|
||||
KubeRay's Apache YuniKorn integration enables more efficient scheduling of Ray Pods in multi-tenant Kubernetes environments.
|
||||
|
||||
:::{note}
|
||||
|
||||
This feature requires KubeRay version 1.2.2 or newer, and it's in alpha testing.
|
||||
|
||||
:::
|
||||
|
||||
## Step 1: Create a Kubernetes cluster with Kind
|
||||
Run the following command in a terminal:
|
||||
|
||||
```shell
|
||||
kind create cluster
|
||||
```
|
||||
|
||||
## Step 2: Install Apache YuniKorn
|
||||
|
||||
You need to successfully install Apache YuniKorn on your Kubernetes cluster before enabling Apache YuniKorn integration with KubeRay. See [Get Started](https://yunikorn.apache.org/docs/) for Apache YuniKorn installation instructions.
|
||||
|
||||
## Step 3: Install the KubeRay operator with Apache YuniKorn support
|
||||
|
||||
When installing KubeRay operator using Helm, pass the `--set batchScheduler.name=yunikorn` flag at the command line:
|
||||
|
||||
```shell
|
||||
helm install kuberay-operator kuberay/kuberay-operator --version 1.6.0 --set batchScheduler.name=yunikorn
|
||||
```
|
||||
|
||||
## Step 4: Use Apache YuniKorn for gang scheduling
|
||||
|
||||
This example demonstrates gang scheduling of RayCluster custom resources with Apache YuniKorn and KubeRay. Starting with KubeRay 1.6.0, KubeRay also supports gang scheduling for RayJob custom resources.
|
||||
|
||||
First, create a queue with a capacity of 4 CPUs and 6Gi of RAM by editing the ConfigMap:
|
||||
|
||||
Run `kubectl edit configmap -n yunikorn yunikorn-defaults`
|
||||
|
||||
Helm creates this ConfigMap during the installation of the Apache YuniKorn Helm chart.
|
||||
|
||||
Add a `queues.yaml` config under the `data` key. The `ConfigMap` should look like the following:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
# Metadata for the ConfigMap, skip for brevity.
|
||||
data:
|
||||
queues.yaml: |
|
||||
partitions:
|
||||
- name: default
|
||||
queues:
|
||||
- name: root
|
||||
queues:
|
||||
- name: test
|
||||
submitacl: "*"
|
||||
parent: false
|
||||
resources:
|
||||
guaranteed:
|
||||
memory: 6G
|
||||
vcore: 4
|
||||
max:
|
||||
memory: 6G
|
||||
vcore: 4
|
||||
```
|
||||
|
||||
Save the changes and exit the editor. This configuration creates a queue named `root.test` with a capacity of 4 CPUs and 6Gi of RAM.
|
||||
|
||||
Next, create a RayCluster with a head node with 1 CPU and 2GiB of RAM, and two workers with 1 CPU and 1GiB of RAM each, for a total of 3 CPU and 4GiB of RAM:
|
||||
|
||||
```shell
|
||||
# Path: kuberay/ray-operator/config/samples
|
||||
# Configure the necessary labels on the RayCluster custom resource for Apache YuniKorn scheduler's gang scheduling:
|
||||
# - `ray.io/gang-scheduling-enabled`: Set to `true` to enable gang scheduling.
|
||||
# - `yunikorn.apache.org/app-id`: Set to a unique identifier for the application in Kubernetes, even across different namespaces.
|
||||
# - `yunikorn.apache.org/queue`: Set to the name of one of the queues in Apache YuniKorn.
|
||||
wget https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-cluster.yunikorn-scheduler.yaml
|
||||
kubectl apply -f ray-cluster.yunikorn-scheduler.yaml
|
||||
```
|
||||
|
||||
Check the RayCluster that the KubeRay operator created:
|
||||
|
||||
```shell
|
||||
$ kubectl describe raycluster test-yunikorn-0
|
||||
|
||||
Name: test-yunikorn-0
|
||||
Namespace: default
|
||||
Labels: ray.io/gang-scheduling-enabled=true
|
||||
yunikorn.apache.org/app-id=test-yunikorn-0
|
||||
yunikorn.apache.org/queue=root.test
|
||||
Annotations: <none>
|
||||
API Version: ray.io/v1
|
||||
Kind: RayCluster
|
||||
Metadata:
|
||||
Creation Timestamp: 2024-09-29T09:52:30Z
|
||||
Generation: 1
|
||||
Resource Version: 951
|
||||
UID: cae1dbc9-5a67-4b43-b0d9-be595f21ab85
|
||||
# Other fields are skipped for brevity
|
||||
````
|
||||
|
||||
Note the labels on the RayCluster: `ray.io/gang-scheduling-enabled=true`, `yunikorn.apache.org/app-id=test-yunikorn-0`, and `yunikorn.apache.org/queue=root.test`.
|
||||
|
||||
:::{note}
|
||||
|
||||
You only need the `ray.io/gang-scheduling-enabled` label when you require gang scheduling. If you don't set this label, YuniKorn schedules the Ray cluster without enforcing gang scheduling.
|
||||
|
||||
:::
|
||||
|
||||
Because the queue has a capacity of 4 CPU and 6GiB of RAM, this resource should schedule successfully without any issues.
|
||||
|
||||
```shell
|
||||
$ kubectl get pods
|
||||
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
test-yunikorn-0-head-98fmp 1/1 Running 0 67s
|
||||
test-yunikorn-0-worker-worker-42tgg 1/1 Running 0 67s
|
||||
test-yunikorn-0-worker-worker-467mn 1/1 Running 0 67s
|
||||
```
|
||||
|
||||
Verify the scheduling by checking the [Apache YuniKorn dashboard](https://yunikorn.apache.org/docs/#access-the-web-ui).
|
||||
|
||||
```shell
|
||||
kubectl port-forward svc/yunikorn-service 9889:9889 -n yunikorn
|
||||
```
|
||||
|
||||
Go to `http://localhost:9889/#/applications` to see the running apps.
|
||||
|
||||
|
||||
|
||||
Next, add an additional RayCluster with the same configuration of head and worker nodes, but with a different name:
|
||||
|
||||
```shell
|
||||
# Replace the name with `test-yunikorn-1`.
|
||||
sed 's/test-yunikorn-0/test-yunikorn-1/' ray-cluster.yunikorn-scheduler.yaml | kubectl apply -f-
|
||||
```
|
||||
|
||||
Now all the Pods for `test-yunikorn-1` are in the `Pending` state:
|
||||
|
||||
```shell
|
||||
$ kubectl get pods
|
||||
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
test-yunikorn-0-head-98fmp 1/1 Running 0 4m22s
|
||||
test-yunikorn-0-worker-worker-42tgg 1/1 Running 0 4m22s
|
||||
test-yunikorn-0-worker-worker-467mn 1/1 Running 0 4m22s
|
||||
test-yunikorn-1-head-xl2r5 0/1 Pending 0 71s
|
||||
test-yunikorn-1-worker-worker-l6ttz 0/1 Pending 0 71s
|
||||
test-yunikorn-1-worker-worker-vjsts 0/1 Pending 0 71s
|
||||
tg-test-yunikorn-1-headgroup-vgzvoot0dh 0/1 Pending 0 69s
|
||||
tg-test-yunikorn-1-worker-eyti2bn2jv 1/1 Running 0 69s
|
||||
tg-test-yunikorn-1-worker-k8it0x6s73 0/1 Pending 0 69s
|
||||
```
|
||||
|
||||
Apache YuniKorn creates the Pods with the `tg-` prefix for gang scheduling purpose.
|
||||
|
||||
Go to `http://localhost:9889/#/applications` and to see `test-yunikorn-1` in the `Accepted` state but not running yet:
|
||||
|
||||
|
||||
|
||||
Because the new cluster requires more CPU and RAM than the queue allows, even though one of the Pods would fit in the remaining 1 CPU and 2GiB of RAM, Apache YuniKorn doesn't place the cluster's Pods until there's enough room for all of the Pods. Without using Apache YuniKorn for gang scheduling in this way, KubeRay would place one of the Pods, and only partially allocating the cluster.
|
||||
|
||||
Delete the first RayCluster to free up resources in the queue:
|
||||
|
||||
```shell
|
||||
kubectl delete raycluster test-yunikorn-0
|
||||
```
|
||||
|
||||
Now all the Pods for the second cluster change to the `Running` state, because enough resources are now available to schedule the entire set of Pods:
|
||||
|
||||
Check the Pods again to see that the second cluster is now up and running:
|
||||
|
||||
```shell
|
||||
$ kubectl get pods
|
||||
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
test-yunikorn-1-head-xl2r5 1/1 Running 0 3m34s
|
||||
test-yunikorn-1-worker-worker-l6ttz 1/1 Running 0 3m34s
|
||||
test-yunikorn-1-worker-worker-vjsts 1/1 Running 0 3m34s
|
||||
```
|
||||
|
||||
Clean up the resources:
|
||||
|
||||
```shell
|
||||
kubectl delete raycluster test-yunikorn-1
|
||||
```
|
||||
@@ -0,0 +1,14 @@
|
||||
(kuberay-api-reference)=
|
||||
# API Reference
|
||||
|
||||
To learn about RayCluster configuration, we recommend taking a look at the {ref}`configuration guide <kuberay-config>`.
|
||||
|
||||
For comprehensive coverage of all supported RayCluster fields, refer to the [API reference][APIReference].
|
||||
|
||||
## KubeRay API compatibility and guarantees
|
||||
|
||||
v1 APIs in the KubeRay project are stable and suitable for production environments. Fields in the v1 APIs will never be removed to maintain compatibility. Future major versions of the API (i.e. v2) may have breaking changes and fields removed from v1.
|
||||
|
||||
However, KubeRay maintainers preserve the right to mark fields as deprecated and remove functionality associated with deprecated fields after a minimum of two minor releases. In addition, some definitions of the API may see small changes in behavior. For example, the definition of a "ready" or "unhealthy" RayCluster could change to better handle new failure scenarios.
|
||||
|
||||
[APIReference]: https://ray-project.github.io/kuberay/reference/api/
|
||||
@@ -0,0 +1,13 @@
|
||||
(kuberay-troubleshooting)=
|
||||
|
||||
# KubeRay Troubleshooting
|
||||
|
||||
```{toctree}
|
||||
:hidden:
|
||||
|
||||
troubleshooting/troubleshooting
|
||||
troubleshooting/rayservice-troubleshooting
|
||||
```
|
||||
|
||||
- {ref}`kuberay-troubleshooting-guides`
|
||||
- {ref}`kuberay-raysvc-troubleshoot`
|
||||
@@ -0,0 +1,304 @@
|
||||
(kuberay-raysvc-troubleshoot)=
|
||||
|
||||
# RayService troubleshooting
|
||||
|
||||
RayService is a Custom Resource Definition (CRD) designed for Ray Serve. In KubeRay, creating a RayService will first create a RayCluster and then create Ray Serve applications once the RayCluster is ready. If the issue pertains to the data plane, specifically your Ray Serve scripts or Ray Serve configurations (`serveConfigV2`), troubleshooting may be challenging. This section provides some tips to help you debug these issues.
|
||||
|
||||
## Observability
|
||||
|
||||
### Method 1: Check KubeRay operator's logs for errors
|
||||
|
||||
```bash
|
||||
kubectl logs $KUBERAY_OPERATOR_POD -n $YOUR_NAMESPACE | tee operator-log
|
||||
```
|
||||
|
||||
The above command will redirect the operator's logs to a file called `operator-log`. You can then search for errors in the file.
|
||||
|
||||
### Method 2: Check RayService CR status
|
||||
|
||||
```bash
|
||||
kubectl describe rayservice $RAYSERVICE_NAME -n $YOUR_NAMESPACE
|
||||
```
|
||||
|
||||
You can check the status and events of the RayService CR to see if there are any errors.
|
||||
|
||||
### Method 3: Check logs of Ray Pods
|
||||
|
||||
You can also check the Ray Serve logs directly by accessing the log files on the pods. These log files contain system level logs from the Serve controller and HTTP proxy as well as access logs and user-level logs. See [Ray Serve Logging](serve-logging) and [Ray Logging](configure-logging) for more details.
|
||||
|
||||
```bash
|
||||
kubectl exec -it $RAY_POD -n $YOUR_NAMESPACE -- bash
|
||||
# Check the logs under /tmp/ray/session_latest/logs/serve/
|
||||
```
|
||||
|
||||
### Method 4: Check Dashboard
|
||||
|
||||
```bash
|
||||
kubectl port-forward $RAY_POD -n $YOUR_NAMESPACE 8265:8265
|
||||
# Check $YOUR_IP:8265 in your browser
|
||||
```
|
||||
|
||||
For more details about Ray Serve observability on the dashboard, you can refer to [the documentation](dash-serve-view) and [the YouTube video](https://youtu.be/eqXfwM641a4).
|
||||
|
||||
### Method 5: Ray State CLI
|
||||
|
||||
You can use the [Ray State CLI](state-api-cli-ref) on the head Pod to check the status of Ray Serve applications.
|
||||
|
||||
```bash
|
||||
# Log into the head Pod
|
||||
export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)
|
||||
kubectl exec -it $HEAD_POD -- ray summary actors
|
||||
|
||||
# [Example output]:
|
||||
# ======== Actors Summary: 2023-07-11 17:58:24.625032 ========
|
||||
# Stats:
|
||||
# ------------------------------------
|
||||
# total_actors: 14
|
||||
|
||||
|
||||
# Table (group by class):
|
||||
# ------------------------------------
|
||||
# CLASS_NAME STATE_COUNTS
|
||||
# 0 ServeController ALIVE: 1
|
||||
# 1 ServeReplica:fruit_app_OrangeStand ALIVE: 1
|
||||
# 2 ProxyActor ALIVE: 3
|
||||
# 4 ServeReplica:math_app_Multiplier ALIVE: 1
|
||||
# 5 ServeReplica:math_app_create_order ALIVE: 1
|
||||
# 7 ServeReplica:fruit_app_FruitMarket ALIVE: 1
|
||||
# 8 ServeReplica:math_app_Adder ALIVE: 1
|
||||
# 9 ServeReplica:math_app_Router ALIVE: 1
|
||||
# 10 ServeReplica:fruit_app_MangoStand ALIVE: 1
|
||||
# 11 ServeReplica:fruit_app_PearStand ALIVE: 1
|
||||
```
|
||||
|
||||
## Common issues
|
||||
|
||||
* {ref}`kuberay-raysvc-issue1`
|
||||
* {ref}`kuberay-raysvc-issue2`
|
||||
* {ref}`kuberay-raysvc-issue3`
|
||||
* {ref}`kuberay-raysvc-issue4`
|
||||
* {ref}`kuberay-raysvc-issue5`
|
||||
* {ref}`kuberay-raysvc-issue6`
|
||||
* {ref}`kuberay-raysvc-issue7`
|
||||
* {ref}`kuberay-raysvc-issue8`
|
||||
* {ref}`kuberay-raysvc-issue9`
|
||||
* {ref}`kuberay-raysvc-issue10`
|
||||
* {ref}`kuberay-raysvc-issue11`
|
||||
|
||||
(kuberay-raysvc-issue1)=
|
||||
### Issue 1: Ray Serve script is incorrect
|
||||
|
||||
It's better to test Ray Serve script locally or in a RayCluster before deploying it to a RayService. See [Development Workflow](serve-dev-workflow) for more details.
|
||||
|
||||
(kuberay-raysvc-issue2)=
|
||||
### Issue 2: `serveConfigV2` is incorrect
|
||||
|
||||
The RayService CR sets `serveConfigV2` as a YAML multi-line string for flexibility. This implies that there is no strict type checking for the Ray Serve configurations in `serveConfigV2` field. Some tips to help you debug the `serveConfigV2` field:
|
||||
|
||||
* Check [the documentation](serve-api) for the schema about the Ray Serve Multi-application API `PUT "/api/serve/applications/"`.
|
||||
* Unlike `serveConfig`, `serveConfigV2` adheres to the snake case naming convention. For example, `numReplicas` is used in `serveConfig`, while `num_replicas` is used in `serveConfigV2`.
|
||||
|
||||
(kuberay-raysvc-issue3)=
|
||||
### Issue 3: The Ray image doesn't include the required dependencies
|
||||
|
||||
You have two options to resolve this issue:
|
||||
|
||||
* Build your own Ray image with the required dependencies.
|
||||
* Specify the required dependencies using `runtime_env` in the `serveConfigV2` field.
|
||||
* For example, the MobileNet example requires `python-multipart`, which isn't included in the Ray image `rayproject/ray:x.y.z`. Therefore, the YAML file includes `python-multipart` in the runtime environment. For more details, refer to [the MobileNet example](kuberay-mobilenet-rayservice-example).
|
||||
|
||||
(kuberay-raysvc-issue4)=
|
||||
### Issue 4: Incorrect `import_path`.
|
||||
|
||||
You can refer to [the documentation](https://docs.ray.io/en/latest/serve/api/doc/ray.serve.schema.ServeApplicationSchema.html#ray.serve.schema.ServeApplicationSchema.import_path) for more details about the format of `import_path`. Taking [the MobileNet YAML file](https://github.com/ray-project/kuberay/blob/v1.0.0/ray-operator/config/samples/ray-service.mobilenet.yaml) as an example, the `import_path` is `mobilenet.mobilenet:app`. The first `mobilenet` is the name of the directory in the `working_dir`, the second `mobilenet` is the name of the Python file in the directory `mobilenet/`, and `app` is the name of the variable representing Ray Serve application within the Python file.
|
||||
|
||||
```yaml
|
||||
serveConfigV2: |
|
||||
applications:
|
||||
- name: mobilenet
|
||||
import_path: mobilenet.mobilenet:app
|
||||
runtime_env:
|
||||
working_dir: "https://github.com/ray-project/serve_config_examples/archive/b393e77bbd6aba0881e3d94c05f968f05a387b96.zip"
|
||||
pip: ["python-multipart==0.0.6"]
|
||||
```
|
||||
|
||||
(kuberay-raysvc-issue5)=
|
||||
### Issue 5: Fail to create / update Serve applications.
|
||||
|
||||
You may encounter the following error messages when KubeRay tries to create / update Serve applications:
|
||||
|
||||
#### Error message 1: `connect: connection refused`
|
||||
|
||||
```
|
||||
Put "http://${HEAD_SVC_FQDN}:52365/api/serve/applications/": dial tcp $HEAD_IP:52365: connect: connection refused
|
||||
```
|
||||
|
||||
For RayService, the KubeRay operator submits a request to the RayCluster for creating Serve applications once the head Pod is ready. It's important to note that the Dashboard, Dashboard Agent and GCS may take a few seconds to start up after the head Pod is ready. As a result, the request may fail a few times initially before the necessary components are fully operational.
|
||||
|
||||
If you continue to encounter this issue after waiting for 1 minute, it's possible that the dashboard or dashboard agent may have failed to start. For more information, you can check the `dashboard.log` and `dashboard_agent.log` files located at `/tmp/ray/session_latest/logs/` on the head Pod.
|
||||
|
||||
#### Error message 2: `i/o timeout`
|
||||
|
||||
```
|
||||
Put "http://${HEAD_SVC_FQDN}:52365/api/serve/applications/": dial tcp $HEAD_IP:52365: i/o timeout"
|
||||
```
|
||||
|
||||
One possible cause of this issue could be a Kubernetes NetworkPolicy blocking the traffic between the Ray Pods and the dashboard agent's port (i.e., 52365).
|
||||
|
||||
(kuberay-raysvc-issue6)=
|
||||
### Issue 6: `runtime_env`
|
||||
|
||||
In `serveConfigV2`, you can specify the runtime environment for the Ray Serve applications using `runtime_env`. Some common issues related to `runtime_env`:
|
||||
|
||||
* The `working_dir` points to a private AWS S3 bucket, but the Ray Pods do not have the necessary permissions to access the bucket.
|
||||
|
||||
* The NetworkPolicy blocks the traffic between the Ray Pods and the external URLs specified in `runtime_env`.
|
||||
|
||||
(kuberay-raysvc-issue7)=
|
||||
### Issue 7: Failed to get Serve application statuses.
|
||||
|
||||
You may encounter the following error message when KubeRay tries to get Serve application statuses:
|
||||
|
||||
```
|
||||
Get "http://${HEAD_SVC_FQDN}:52365/api/serve/applications/": dial tcp $HEAD_IP:52365: connect: connection refused"
|
||||
```
|
||||
|
||||
As mentioned in [Issue 5](#kuberay-raysvc-issue5), the KubeRay operator submits a `Put` request to the RayCluster for creating Serve applications once the head Pod is ready. After the successful submission of the `Put` request to the dashboard agent, a `Get` request is sent to the dashboard agent port (i.e., 52365). The successful submission indicates that all the necessary components, including the dashboard agent, are fully operational. Therefore, unlike Issue 5, the failure of the `Get` request is not expected.
|
||||
|
||||
If you consistently encounter this issue, there are several possible causes:
|
||||
|
||||
* The dashboard agent process on the head Pod is not running. You can check the `dashboard_agent.log` file located at `/tmp/ray/session_latest/logs/` on the head Pod for more information. In addition, you can also perform an experiment to reproduce this cause by manually killing the dashboard agent process on the head Pod.
|
||||
```bash
|
||||
# Step 1: Log in to the head Pod
|
||||
kubectl exec -it $HEAD_POD -n $YOUR_NAMESPACE -- bash
|
||||
|
||||
# Step 2: Check the PID of the dashboard agent process
|
||||
ps aux
|
||||
# [Example output]
|
||||
# ray 156 ... 0:03 ray::DashboardAgent --
|
||||
|
||||
# Step 3: Kill the dashboard agent process
|
||||
kill 156
|
||||
|
||||
# Step 4: Check the logs
|
||||
cat /tmp/ray/session_latest/logs/dashboard_agent.log
|
||||
|
||||
# [Example output]
|
||||
# 2023-07-10 11:24:31,962 INFO web_log.py:206 -- 10.244.0.5 [10/Jul/2023:18:24:31 +0000] "GET /api/serve/applications/ HTTP/1.1" 200 13940 "-" "Go-http-client/1.1"
|
||||
# 2023-07-10 11:24:34,001 INFO web_log.py:206 -- 10.244.0.5 [10/Jul/2023:18:24:33 +0000] "GET /api/serve/applications/ HTTP/1.1" 200 13940 "-" "Go-http-client/1.1"
|
||||
# 2023-07-10 11:24:36,043 INFO web_log.py:206 -- 10.244.0.5 [10/Jul/2023:18:24:36 +0000] "GET /api/serve/applications/ HTTP/1.1" 200 13940 "-" "Go-http-client/1.1"
|
||||
# 2023-07-10 11:24:38,082 INFO web_log.py:206 -- 10.244.0.5 [10/Jul/2023:18:24:38 +0000] "GET /api/serve/applications/ HTTP/1.1" 200 13940 "-" "Go-http-client/1.1"
|
||||
# 2023-07-10 11:24:38,590 WARNING agent.py:531 -- Exiting with SIGTERM immediately...
|
||||
|
||||
# Step 5: Open a new terminal and check the logs of the KubeRay operator
|
||||
kubectl logs $KUBERAY_OPERATOR_POD -n $YOUR_NAMESPACE | tee operator-log
|
||||
|
||||
# [Example output]
|
||||
# Get \"http://rayservice-sample-raycluster-rqlsl-head-svc.default.svc.cluster.local:52365/api/serve/applications/\": dial tcp 10.96.7.154:52365: connect: connection refused
|
||||
```
|
||||
|
||||
(kuberay-raysvc-issue8)=
|
||||
### Issue 8: A loop of restarting the RayCluster occurs when the Kubernetes cluster runs out of resources. (KubeRay v0.6.1 or earlier)
|
||||
|
||||
> Note: Currently, the KubeRay operator does not have a clear plan to handle situations where the Kubernetes cluster runs out of resources.
|
||||
Therefore, we recommend ensuring that the Kubernetes cluster has sufficient resources to accommodate the serve application.
|
||||
|
||||
If the status of a serve application remains non-`RUNNING` for more than `serviceUnhealthySecondThreshold` seconds, the KubeRay operator will consider the RayCluster as unhealthy and initiate the preparation of a new RayCluster. A common cause of this issue is that the Kubernetes cluster does not have enough resources to accommodate the serve application. In such cases, the KubeRay operator may continue to restart the RayCluster, leading to a loop of restarts.
|
||||
|
||||
We can also perform an experiment to reproduce this situation:
|
||||
|
||||
* A Kubernetes cluster with an 8-CPUs node
|
||||
* [ray-service.insufficient-resources.yaml](https://gist.github.com/kevin85421/6a7779308aa45b197db8015aca0c1faf)
|
||||
* RayCluster:
|
||||
* The cluster has 1 head Pod with 4 physical CPUs, but `num-cpus` is set to 0 in `rayStartParams` to prevent any serve replicas from being scheduled on the head Pod.
|
||||
* The cluster also has 1 worker Pod with 1 CPU by default.
|
||||
* `serveConfigV2` specifies 5 serve deployments, each with 1 replica and a requirement of 1 CPU.
|
||||
|
||||
```bash
|
||||
# Step 1: Get the number of CPUs available on the node
|
||||
kubectl get nodes -o custom-columns=NODE:.metadata.name,ALLOCATABLE_CPU:.status.allocatable.cpu
|
||||
|
||||
# [Example output]
|
||||
# NODE ALLOCATABLE_CPU
|
||||
# kind-control-plane 8
|
||||
|
||||
# Step 2: Install a KubeRay operator.
|
||||
|
||||
# Step 3: Create a RayService with autoscaling enabled.
|
||||
kubectl apply -f ray-service.insufficient-resources.yaml
|
||||
|
||||
# Step 4: The Kubernetes cluster will not have enough resources to accommodate the serve application.
|
||||
kubectl describe rayservices.ray.io rayservice-sample -n $YOUR_NAMESPACE
|
||||
|
||||
# [Example output]
|
||||
# fruit_app_FruitMarket:
|
||||
# Health Last Update Time: 2023-07-11T02:10:02Z
|
||||
# Last Update Time: 2023-07-11T02:10:35Z
|
||||
# Message: Deployment "fruit_app_FruitMarket" has 1 replicas that have taken more than 30s to be scheduled. This may be caused by waiting for the cluster to auto-scale, or waiting for a runtime environment to install. Resources required for each replica: {"CPU": 1.0}, resources available: {}.
|
||||
# Status: UPDATING
|
||||
|
||||
# Step 5: A new RayCluster will be created after `serviceUnhealthySecondThreshold` (300s here) seconds.
|
||||
# Check the logs of the KubeRay operator to find the reason for restarting the RayCluster.
|
||||
kubectl logs $KUBERAY_OPERATOR_POD -n $YOUR_NAMESPACE | tee operator-log
|
||||
|
||||
# [Example output]
|
||||
# 2023-07-11T02:14:58.109Z INFO controllers.RayService Restart RayCluster {"appName": "fruit_app", "restart reason": "The status of the serve application fruit_app has not been RUNNING for more than 300.000000 seconds. Hence, KubeRay operator labels the RayCluster unhealthy and will prepare a new RayCluster."}
|
||||
# 2023-07-11T02:14:58.109Z INFO controllers.RayService Restart RayCluster {"deploymentName": "fruit_app_FruitMarket", "appName": "fruit_app", "restart reason": "The status of the serve deployment fruit_app_FruitMarket or the serve application fruit_app has not been HEALTHY/RUNNING for more than 300.000000 seconds. Hence, KubeRay operator labels the RayCluster unhealthy and will prepare a new RayCluster. The message of the serve deployment is: Deployment \"fruit_app_FruitMarket\" has 1 replicas that have taken more than 30s to be scheduled. This may be caused by waiting for the cluster to auto-scale, or waiting for a runtime environment to install. Resources required for each replica: {\"CPU\": 1.0}, resources available: {}."}
|
||||
# .
|
||||
# .
|
||||
# .
|
||||
# 2023-07-11T02:14:58.122Z INFO controllers.RayService Restart RayCluster {"ServiceName": "default/rayservice-sample", "AvailableWorkerReplicas": 1, "DesiredWorkerReplicas": 5, "restart reason": "The serve application is unhealthy, restarting the cluster. If the AvailableWorkerReplicas is not equal to DesiredWorkerReplicas, this may imply that the Autoscaler does not have enough resources to scale up the cluster. Hence, the serve application does not have enough resources to run. Please check https://github.com/ray-project/kuberay/blob/master/docs/guidance/rayservice-troubleshooting.md for more details.", "RayCluster": {"apiVersion": "ray.io/v1alpha1", "kind": "RayCluster", "namespace": "default", "name": "rayservice-sample-raycluster-hvd9f"}}
|
||||
```
|
||||
|
||||
(kuberay-raysvc-issue9)=
|
||||
### Issue 9: Upgrade from Ray Serve's single-application API to its multi-application API without downtime
|
||||
|
||||
KubeRay v0.6.0 has begun supporting Ray Serve API V2 (multi-application) by exposing `serveConfigV2` in the RayService CRD. However, Ray Serve does not support deploying both API V1 and API V2 in the cluster simultaneously. Hence, if users want to perform in-place upgrades by replacing `serveConfig` with `serveConfigV2`, they may encounter the following error message:
|
||||
|
||||
```
|
||||
ray.serve.exceptions.RayServeException: You are trying to deploy a multi-application config, however a single-application
|
||||
config has been deployed to the current Serve instance already. Mixing single-app and multi-app is not allowed. Please either
|
||||
redeploy using the single-application config format `ServeApplicationSchema`, or shutdown and restart Serve to submit a
|
||||
multi-app config of format `ServeDeploySchema`. If you are using the REST API, you can submit a multi-app config to the
|
||||
the multi-app API endpoint `/api/serve/applications/`.
|
||||
```
|
||||
|
||||
To resolve this issue, you can replace `serveConfig` with `serveConfigV2` and modify `rayVersion` which has no effect when the Ray version is 2.0.0 or later to 2.100.0. This will trigger a new RayCluster preparation instead of an in-place update.
|
||||
|
||||
If, after following the steps above, you still see the error message and GCS fault tolerance is enabled, it may be due to the `ray.io/external-storage-namespace` annotation being the same for both old and new RayClusters. You can remove the annotation and KubeRay will automatically generate a unique key for each RayCluster custom resource. See [kuberay#1297](https://github.com/ray-project/kuberay/issues/1297) for more details.
|
||||
|
||||
(kuberay-raysvc-issue10)=
|
||||
### Issue 10: Upgrade RayService with GCS fault tolerance enabled without downtime
|
||||
|
||||
KubeRay uses the value of the annotation [ray.io/external-storage-namespace](kuberay-external-storage-namespace) to assign the environment variable `RAY_external_storage_namespace` to all Ray Pods managed by the RayCluster. This value represents the storage namespace in Redis where the Ray cluster metadata resides. In the process of a head Pod recovery, the head Pod attempts to reconnect to the Redis server using the `RAY_external_storage_namespace` value to recover the cluster data.
|
||||
|
||||
However, specifying the `RAY_external_storage_namespace` value in RayService can potentially lead to downtime during zero-downtime upgrades. Specifically, the new RayCluster accesses the same Redis storage namespace as the old one for cluster metadata. This configuration can lead the KubeRay operator to assume that the Ray Serve applications are operational, as indicated by the existing metadata in Redis. Consequently, the operator might deem it safe to retire the old RayCluster and redirect traffic to the new one, even though the latter may still require time to initialize the Ray Serve applications.
|
||||
|
||||
The recommended solution is to remove the `ray.io/external-storage-namespace` annotation from the RayService CRD. If the annotation isn't set, KubeRay automatically uses each RayCluster custom resource's UID as the `RAY_external_storage_namespace` value. Hence, both the old and new RayClusters have different `RAY_external_storage_namespace` values, and the new RayCluster is unable to access the old cluster metadata. Another solution is to set the `RAY_external_storage_namespace` value manually to a unique value for each RayCluster custom resource. See [kuberay#1296](https://github.com/ray-project/kuberay/issues/1296) for more details.
|
||||
|
||||
(kuberay-raysvc-issue11)=
|
||||
### Issue 11: RayService stuck in Initializing — use the initializing timeout to fail fast
|
||||
|
||||
If one or more underlying Pods are scheduled but fail to start (for example, ImagePullBackOff, CrashLoopBackOff, or other container startup errors), a `RayService` can remain in the Initializing state indefinitely. This state consumes cluster resources and makes the root cause harder to diagnose.
|
||||
|
||||
#### What to do
|
||||
KubeRay exposes a configurable initializing timeout via the annotation `ray.io/initializing-timeout`. When the timeout expires, the operator marks the `RayService` as failed and starts cleanup of associated `RayCluster` resources. Enabling the timeout requires only adding the annotation to the `RayService` metadata — no other CRD changes are necessary.
|
||||
|
||||
#### Operator behavior after timeout
|
||||
- The `RayServiceReady` condition is set to `False` with reason `InitializingTimeout`.
|
||||
- The `RayService` is placed into a **terminal (failed)** state; updating the spec will not trigger a retry. Recovery requires deleting and recreating the `RayService`.
|
||||
- Cluster names on the `RayService` CR are cleared, which triggers cleanup of the underlying `RayCluster` resources. Deletions still respect `RayClusterDeletionDelaySeconds`.
|
||||
- A `Warning` event is emitted that documents the timeout and the failure reason.
|
||||
|
||||
#### Enable the timeout
|
||||
Add the annotation to your `RayService` metadata. The annotation accepts either Go duration strings (for example, `"30m"` or `"1h"`) or integer seconds (for example, `"1800"`):
|
||||
|
||||
```yaml
|
||||
metadata:
|
||||
annotations:
|
||||
ray.io/initializing-timeout: "30m"
|
||||
```
|
||||
|
||||
#### Guidance
|
||||
- Pick a timeout that balances expected startup work with failing fast to conserve cluster resources.
|
||||
- See the upstream discussion [kuberay#4138](https://github.com/ray-project/kuberay/issues/4138) for more implementation details.
|
||||
@@ -0,0 +1,121 @@
|
||||
(kuberay-troubleshooting-guides)=
|
||||
|
||||
# Troubleshooting guide
|
||||
|
||||
This document addresses common inquiries. If you don't find an answer to your question here, please don't hesitate to connect with us via our [community channels](https://github.com/ray-project/kuberay#getting-involved).
|
||||
|
||||
# Contents
|
||||
|
||||
- [Use the right version of Ray](#use-the-right-version-of-ray)
|
||||
- [Use ARM-based docker images for Apple M1 or M2 MacBooks](#docker-image-for-apple-macbooks)
|
||||
- [Upgrade KubeRay](#upgrade-kuberay)
|
||||
- [Worker init container](#worker-init-container)
|
||||
- [Cluster domain](#cluster-domain)
|
||||
- [RayService](#rayservice)
|
||||
- [Autoscaler](#autoscaler)
|
||||
- [Multi-node GPU clusters](#multi-node-gpu)
|
||||
- [Other questions](#other-questions)
|
||||
|
||||
(use-the-right-version-of-ray)=
|
||||
## Use the right version of Ray
|
||||
|
||||
See the [upgrade guide](#kuberay-upgrade-guide) for the compatibility matrix between KubeRay versions and Ray versions.
|
||||
|
||||
```{admonition} Don't use Ray versions between 2.11.0 and 2.37.0.
|
||||
The [commit](https://github.com/ray-project/ray/pull/44658) introduces a bug in Ray 2.11.0.
|
||||
When a Ray job is created, the Ray dashboard agent process on the head node gets stuck, causing the readiness and liveness probes, which send health check requests for the Raylet to the dashboard agent, to fail.
|
||||
```
|
||||
|
||||
(docker-image-for-apple-macbooks)=
|
||||
## Use ARM-based docker images for Apple M1 or M2 MacBooks
|
||||
Ray builds different images for different platforms. Until Ray moves to building multi-architecture images, [tracked by this GitHub issue](https://github.com/ray-project/ray/issues/39364), use platform-specific docker images in the head and worker group specs of the [RayCluster config](https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/config.html#image).
|
||||
|
||||
Use an image with the tag `aarch64`, for example, `image: rayproject/ray:2.41.0-aarch64`), if you are running KubeRay on a MacBook M1 or M2.
|
||||
|
||||
[Link to issue details and discussion](https://ray-distributed.slack.com/archives/C02GFQ82JPM/p1712267296145549).
|
||||
|
||||
(upgrade-kuberay)=
|
||||
## Upgrade KubeRay
|
||||
|
||||
If you have issues upgrading KubeRay, see the [upgrade guide](#kuberay-upgrade-guide). Most issues are about the CRD version.
|
||||
|
||||
(worker-init-container)=
|
||||
## Worker init container
|
||||
|
||||
The KubeRay operator injects a default [init container](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/) into every worker Pod. This init container is responsible for waiting until the Global Control Service (GCS) on the head Pod is ready before establishing a connection to the head. The init container will use `ray health-check` to check the GCS server status continuously.
|
||||
|
||||
The default worker init container may not work for all use cases, or users may want to customize the init container.
|
||||
|
||||
### 1. Init container troubleshooting
|
||||
|
||||
Some common causes for the worker init container to stuck in `Init:0/1` status are:
|
||||
|
||||
* The GCS server process has failed in the head Pod. Please inspect the log directory `/tmp/ray/session_latest/logs/` in the head Pod for errors related to the GCS server.
|
||||
* The `ray` executable is not included in the `$PATH` for the image, so the init container will fail to run `ray health-check`.
|
||||
* The `CLUSTER_DOMAIN` environment variable is not set correctly. See the section [cluster domain](#cluster-domain) for more details.
|
||||
* The worker init container shares the same ***ImagePullPolicy***, ***SecurityContext***, ***Env***, ***VolumeMounts***, and ***Resources*** as the worker Pod template. Sharing these settings is possible to cause a deadlock. See [#1130](https://github.com/ray-project/kuberay/issues/1130) for more details.
|
||||
|
||||
If the init container remains stuck in `Init:0/1` status for 2 minutes, Ray stops redirecting the output messages to `/dev/null` and instead prints them to the worker Pod logs. To troubleshoot further, you can inspect the logs using `kubectl logs`.
|
||||
|
||||
### 2. Disable the init container injection
|
||||
|
||||
If you want to customize the worker init container, you can disable the init container injection and add your own. To disable the injection, set the `ENABLE_INIT_CONTAINER_INJECTION` environment variable in the KubeRay operator to `false` (applicable from KubeRay v0.5.2). Please refer to [#1069](https://github.com/ray-project/kuberay/pull/1069) and the [KubeRay Helm chart](https://github.com/ray-project/kuberay/blob/ddb5e528c29c2e1fb80994f05b1bd162ecbaf9f2/helm-chart/kuberay-operator/values.yaml#L83-L87) for instructions on how to set the environment variable. Once disabled, you can add your custom init container to the worker Pod template.
|
||||
|
||||
(cluster-domain)=
|
||||
## Cluster domain
|
||||
|
||||
In KubeRay, we use Fully Qualified Domain Names (FQDNs) to establish connections between workers and the head. The FQDN of the head service is `${HEAD_SVC}.${NAMESPACE}.svc.${CLUSTER_DOMAIN}`. The default [cluster domain](https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/#introduction) is `cluster.local`, which works for most Kubernetes clusters. However, it's important to note that some clusters may have a different cluster domain. You can check the cluster domain of your Kubernetes cluster by checking `/etc/resolv.conf` in a Pod.
|
||||
|
||||
To set a custom cluster domain, adjust the `CLUSTER_DOMAIN` environment variable in the KubeRay operator. Helm chart users can make this modification [here](https://github.com/ray-project/kuberay/blob/ddb5e528c29c2e1fb80994f05b1bd162ecbaf9f2/helm-chart/kuberay-operator/values.yaml#L88-L91). For more information, please refer to [#951](https://github.com/ray-project/kuberay/pull/951) and [#938](https://github.com/ray-project/kuberay/pull/938) for more details.
|
||||
|
||||
(rayservice)=
|
||||
## RayService
|
||||
|
||||
RayService is a Custom Resource Definition (CRD) designed for Ray Serve. In KubeRay, creating a RayService will first create a RayCluster and then create Ray Serve applications once the RayCluster is ready. If the issue pertains to the data plane, specifically your Ray Serve scripts or Ray Serve configurations (`serveConfigV2`), troubleshooting may be challenging. See [rayservice-troubleshooting](kuberay-raysvc-troubleshoot) for more details.
|
||||
|
||||
(autoscaler)=
|
||||
## Ray Autoscaler
|
||||
|
||||
### Ray Autoscaler doesn't scale up, causing new Ray tasks or actors to remain pending
|
||||
|
||||
One common cause is that the Ray tasks or actors require an amount of resources that exceeds what any single Ray node can provide. Note that Ray tasks and actors represent the smallest scheduling units in Ray, and a task or actor should be on a single Ray node. Take [kuberay#846](https://github.com/ray-project/kuberay/issues/846) as an example. The user attempts to schedule a Ray task that requires 2 CPUs, but the Ray Pods available for these tasks have only 1 CPU each. Consequently, the Ray Autoscaler decides not to scale up the RayCluster.
|
||||
|
||||
(multi-node-gpu)=
|
||||
## Multi-node GPU Deployments
|
||||
|
||||
For comprehensive troubleshooting of multi-node GPU serving issues, refer to {ref}`Troubleshooting multi-node GPU serving on KubeRay <serve-multi-node-gpu-troubleshooting>`.
|
||||
|
||||
(other-questions)=
|
||||
## Other questions
|
||||
|
||||
### Why are changes to the RayCluster or RayJob CR not taking effect?
|
||||
|
||||
Currently, only modifications to the `replicas` field in `RayCluster/RayJob` CR are supported. Changes to other fields may not take effect or could lead to unexpected results.
|
||||
|
||||
### How to configure reconcile concurrency when there are large mount of CRs?
|
||||
|
||||
In this example, [kuberay#3909](https://github.com/ray-project/kuberay/issues/3909), the user encountered high latency when processing RayCluster CRs and found that the ReconcileConcurrency value was set to 1.
|
||||
|
||||
The KubeRay operator supports configuring the `ReconcileConcurrency` setting, which controls the number of concurrent workers processing Ray custom resources (CRs).
|
||||
|
||||
To configure the `ReconcileConcurrency` number, you can edit the deployment's container args:
|
||||
|
||||
```bash
|
||||
kubectl edit deployment kuberay-operator
|
||||
```
|
||||
|
||||
Specify the `ReconcileConcurrency` number in the container args:
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
containers:
|
||||
- args:
|
||||
- --reconcile-concurrency
|
||||
- "10"
|
||||
```
|
||||
|
||||
You can also use the following command for kuberay version >= 1.6.0:
|
||||
|
||||
```bash
|
||||
helm install kuberay-operator kuberay/kuberay-operator --version 1.6.0 --set reconcileConcurrency=10
|
||||
```
|
||||
@@ -0,0 +1,76 @@
|
||||
(kuberay-guides)=
|
||||
|
||||
# User Guides
|
||||
|
||||
```{toctree}
|
||||
:hidden:
|
||||
|
||||
Deploy Ray Serve Apps <user-guides/rayservice>
|
||||
user-guides/rayservice-no-ray-serve-replica
|
||||
user-guides/rayservice-high-availability
|
||||
user-guides/kuberay-serve-high-throughput
|
||||
user-guides/rayservice-incremental-upgrade
|
||||
user-guides/observability
|
||||
user-guides/upgrade-guide
|
||||
user-guides/k8s-cluster-setup
|
||||
user-guides/storage
|
||||
user-guides/config
|
||||
user-guides/configuring-autoscaling
|
||||
user-guides/configuring-ippr
|
||||
user-guides/label-based-scheduling
|
||||
user-guides/kuberay-gcs-ft
|
||||
user-guides/kuberay-gcs-persistent-ft
|
||||
user-guides/gke-gcs-bucket
|
||||
user-guides/persist-kuberay-custom-resource-logs
|
||||
user-guides/persist-kuberay-operator-logs
|
||||
user-guides/gpu
|
||||
user-guides/tpu
|
||||
user-guides/pod-command
|
||||
user-guides/helm-chart-rbac
|
||||
user-guides/tls
|
||||
user-guides/k8s-autoscaler
|
||||
user-guides/kubectl-plugin
|
||||
user-guides/kuberay-auth
|
||||
user-guides/kuberay-auth-rbac
|
||||
user-guides/reduce-image-pull-latency
|
||||
user-guides/uv
|
||||
user-guides/kuberay-dashboard
|
||||
user-guides/resource-isolation-with-writable-cgroups
|
||||
user-guides/kuberay-history-server
|
||||
```
|
||||
|
||||
|
||||
:::{note}
|
||||
To learn the basics of Ray on Kubernetes, we recommend taking a look at the {ref}`introductory guide <kuberay-quickstart>` first.
|
||||
:::
|
||||
|
||||
* {ref}`kuberay-rayservice`
|
||||
* {ref}`kuberay-rayservice-no-ray-serve-replica`
|
||||
* {ref}`kuberay-rayservice-ha`
|
||||
* {ref}`kuberay-rayservice-incremental-upgrade`
|
||||
* {ref}`kuberay-serve-high-throughput`
|
||||
* {ref}`kuberay-observability`
|
||||
* {ref}`kuberay-upgrade-guide`
|
||||
* {ref}`kuberay-k8s-setup`
|
||||
* {ref}`kuberay-storage`
|
||||
* {ref}`kuberay-config`
|
||||
* {ref}`kuberay-autoscaling`
|
||||
* {ref}`kuberay-gpu`
|
||||
* {ref}`kuberay-tpu`
|
||||
* {ref}`kuberay-gcs-ft`
|
||||
* {ref}`kuberay-gcs-persistent-ft`
|
||||
* {ref}`persist-kuberay-custom-resource-logs`
|
||||
* {ref}`persist-kuberay-operator-logs`
|
||||
* {ref}`kuberay-pod-command`
|
||||
* {ref}`kuberay-helm-chart-rbac`
|
||||
* {ref}`kuberay-tls`
|
||||
* {ref}`kuberay-gke-bucket`
|
||||
* {ref}`ray-k8s-autoscaler-comparison`
|
||||
* {ref}`kubectl-plugin`
|
||||
* {ref}`kuberay-auth`
|
||||
* {ref}`kuberay-auth-rbac`
|
||||
* {ref}`reduce-image-pull-latency`
|
||||
* {ref}`kuberay-uv`
|
||||
* {ref}`kuberay-dashboard`
|
||||
* {ref}`resource-isolation-with-writable-cgroups`
|
||||
* {ref}`kuberay-history-server`
|
||||
@@ -0,0 +1,26 @@
|
||||
(kuberay-ack-gpu-cluster-setup)=
|
||||
|
||||
# Start an Aliyun ACK cluster with GPUs for KubeRay
|
||||
|
||||
This guide provides step-by-step instructions for creating an ACK cluster with GPU nodes specifically configured for KubeRay. The configuration outlined here can be applied to most KubeRay examples found in the documentation.
|
||||
|
||||
## Step 1: Create a Kubernetes cluster on Aliyun ACK
|
||||
|
||||
See [Create a cluster](https://www.alibabacloud.com/help/en/ack/ack-managed-and-ack-dedicated/user-guide/create-an-ack-managed-cluster-2) to create a Aliyun ACK cluster and see [Connect to clusters](https://www.alibabacloud.com/help/en/ack/ack-managed-and-ack-dedicated/user-guide/access-clusters) to configure your computer to communicate with the cluster.
|
||||
|
||||
## Step 2: Create node pools for the Aliyun ACK cluster
|
||||
|
||||
See [Create a node pool](https://www.alibabacloud.com/help/en/ack/ack-managed-and-ack-dedicated/user-guide/create-a-node-pool) to create node pools.
|
||||
|
||||
### Manage node labels and taints
|
||||
|
||||
If you need to set taints for nodes, see [Create and manage node labels](https://www.alibabacloud.com/help/en/ack/ack-managed-and-ack-dedicated/user-guide/manage-taints-and-tolerations) and [Create and manage node taints](https://www.alibabacloud.com/help/en/ack/ack-managed-and-ack-dedicated/user-guide/manage-taints-and-tolerations). For example, you can add a taint to GPU node pools so that Ray won't schedule head pods on these nodes.
|
||||
|
||||
### Upgrade drivers on the nodes
|
||||
|
||||
If you need to upgrade the drivers on the nodes, see [Step 2: Create a node pool and specify an NVIDIA driver version](https://www.alibabacloud.com/help/en/ack/ack-managed-and-ack-dedicated/user-guide/customize-the-gpu-driver-version-of-the-node-by-specifying-the-version-number) to upgrade drivers.
|
||||
|
||||
## Step 3: Install KubeRay addon in the cluster
|
||||
|
||||
See [Step 2: Install KubeRay-Operator](https://www.alibabacloud.com/help/en/ack/cloud-native-ai-suite/use-cases/efficient-deployment-and-optimization-practice-of-ray-in-ack-cluster?) to deploy KubeRay in ACK.
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
(kuberay-eks-gpu-cluster-setup)=
|
||||
|
||||
# Start Amazon EKS Cluster with GPUs for KubeRay
|
||||
|
||||
This guide walks you through the steps to create an Amazon EKS cluster with GPU nodes specifically for KubeRay. The configuration outlined here can be applied to most KubeRay examples found in the documentation.
|
||||
|
||||
## Step 1: Create a Kubernetes cluster on Amazon EKS
|
||||
|
||||
Follow the first two steps in [this AWS documentation](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-console.html#) to: (1) create your Amazon EKS cluster and (2) configure your computer to communicate with your cluster.
|
||||
|
||||
## Step 2: Create node groups for the Amazon EKS cluster
|
||||
|
||||
Follow "Step 3: Create nodes" in [this AWS documentation](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-console.html#) to create node groups. The following section provides more detailed information.
|
||||
|
||||
### Create a CPU node group
|
||||
|
||||
Typically, avoid running GPU workloads on the Ray head. Create a CPU node group for all Pods except Ray GPU workers, such as the KubeRay operator, Ray head, and CoreDNS Pods.
|
||||
|
||||
Here's a common configuration that works for most KubeRay examples in the docs:
|
||||
* Instance type: [**m5.xlarge**](https://aws.amazon.com/ec2/instance-types/m5/) (4 vCPU; 16 GB RAM)
|
||||
* Disk size: 256 GB
|
||||
* Desired size: 1, Min size: 0, Max size: 1
|
||||
|
||||
### Create a GPU node group
|
||||
|
||||
Create a GPU node group for Ray GPU workers.
|
||||
|
||||
1. Here's a common configuration that works for most KubeRay examples in the docs:
|
||||
* AMI type: Bottlerocket NVIDIA (BOTTLEROCKET_x86_64_NVIDIA)
|
||||
* Instance type: [**g5.xlarge**](https://aws.amazon.com/ec2/instance-types/g5/) (1 GPU; 24 GB GPU Memory; 4 vCPUs; 16 GB RAM)
|
||||
* Disk size: 1024 GB
|
||||
* Desired size: 1, Min size: 0, Max size: 1
|
||||
|
||||
2. Please install the NVIDIA device plugin. (Note: You can skip this step if you used the `BOTTLEROCKET_x86_64_NVIDIA` AMI in the step above.)
|
||||
* Install the DaemonSet for NVIDIA device plugin to run GPU enabled containers in your Amazon EKS cluster. You can refer to the [Amazon EKS optimized accelerated Amazon Linux AMIs](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html#gpu-ami) or [NVIDIA/k8s-device-plugin](https://github.com/NVIDIA/k8s-device-plugin) repository for more details.
|
||||
* If the GPU nodes have taints, add `tolerations` to `nvidia-device-plugin.yml` to enable the DaemonSet to schedule Pods on the GPU nodes.
|
||||
|
||||
> **Note:** If you encounter permission issues with `kubectl`, follow "Step 2: Configure your computer to communicate with your cluster"
|
||||
in the [AWS documentation](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-console.html#).
|
||||
|
||||
```sh
|
||||
# Install the DaemonSet
|
||||
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.9.0/nvidia-device-plugin.yml
|
||||
|
||||
# Verify that your nodes have allocatable GPUs. If the GPU node fails to detect GPUs,
|
||||
# please verify whether the DaemonSet schedules the Pod on the GPU node.
|
||||
kubectl get nodes "-o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"
|
||||
|
||||
# Example output:
|
||||
# NAME GPU
|
||||
# ip-....us-west-2.compute.internal 4
|
||||
# ip-....us-west-2.compute.internal <none>
|
||||
```
|
||||
|
||||
3. Add a Kubernetes taint to prevent scheduling CPU Pods on this GPU node group. For KubeRay examples, add the following taint to the GPU nodes: `Key: ray.io/node-type, Value: worker, Effect: NoSchedule`, and include the corresponding `tolerations` for GPU Ray worker Pods.
|
||||
|
||||
> Warning: GPU nodes are extremely expensive. Please remember to delete the cluster if you no longer need it.
|
||||
|
||||
## Step 3: Verify the node groups
|
||||
|
||||
> **Note:** If you encounter permission issues with `eksctl`, navigate to your AWS account's webpage and copy the
|
||||
credential environment variables, including `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN`, from the "Command line or programmatic access" page.
|
||||
|
||||
```sh
|
||||
eksctl get nodegroup --cluster ${YOUR_EKS_NAME}
|
||||
|
||||
# CLUSTER NODEGROUP STATUS CREATED MIN SIZE MAX SIZE DESIRED CAPACITY INSTANCE TYPE IMAGE ID ASG NAME TYPE
|
||||
# ${YOUR_EKS_NAME} cpu-node-group ACTIVE 2023-06-05T21:31:49Z 0 1 1 m5.xlarge AL2_x86_64 eks-cpu-node-group-... managed
|
||||
# ${YOUR_EKS_NAME} gpu-node-group ACTIVE 2023-06-05T22:01:44Z 0 1 1 g5.12xlarge BOTTLEROCKET_x86_64_NVIDIA eks-gpu-node-group-... managed
|
||||
```
|
||||
@@ -0,0 +1,52 @@
|
||||
(kuberay-aks-gpu-cluster-setup)=
|
||||
|
||||
# Start Azure AKS Cluster with GPUs for KubeRay
|
||||
|
||||
This guide walks you through the steps to create an Azure AKS cluster with GPU nodes specifically for KubeRay. The configuration outlined here can be applied to most KubeRay examples found in the documentation.
|
||||
|
||||
You can find the landing page for AKS [here](https://azure.microsoft.com/en-us/services/kubernetes-service/). If you have an account set up, you can immediately start experimenting with Kubernetes clusters in the provider's console. Alternatively, check out the [documentation](https://docs.microsoft.com/en-us/azure/aks/) and [quickstart guides](https://docs.microsoft.com/en-us/azure/aks/learn/quick-kubernetes-deploy-portal?tabs=azure-cli). To successfully deploy Ray on Kubernetes, you will need to use node pools following the guidance [here](https://docs.microsoft.com/en-us/azure/aks/use-multiple-node-pools).
|
||||
|
||||
## Step 1: Create a Resource Group
|
||||
|
||||
To create a resource group in a particular region:
|
||||
|
||||
```
|
||||
az group create -l eastus -n kuberay-rg
|
||||
```
|
||||
|
||||
## Step 2: Create AKS Cluster
|
||||
|
||||
To create an AKS cluster with system nodepool:
|
||||
```
|
||||
az aks create \
|
||||
-g kuberay-rg \
|
||||
-n kuberay-gpu-cluster \
|
||||
--nodepool-name system \
|
||||
--node-vm-size Standard_D8s_v3 \
|
||||
--node-count 3
|
||||
```
|
||||
|
||||
## Step 3: Add a GPU node group
|
||||
|
||||
To add a GPU nodepool with autoscaling:
|
||||
```
|
||||
az aks nodepool add \
|
||||
-g kuberay-rg \
|
||||
--cluster-name kuberay-gpu-cluster \
|
||||
--nodepool-name gpupool \
|
||||
--node-vm-size Standard_NC6s_v3 \
|
||||
--node-taints nvidia.com/gpu=present:NoSchedule \
|
||||
--min-count 0 \
|
||||
--max-count 3 \
|
||||
--enable-cluster-autoscaler
|
||||
```
|
||||
To use NVIDIA GPU operator alternatively, follow instructions [here](https://learn.microsoft.com/en-us/azure/aks/gpu-cluster?tabs=add-ubuntu-gpu-node-pool#skip-gpu-driver-installation-preview)
|
||||
|
||||
## Step 4: Get kubeconfig
|
||||
|
||||
To get kubeconfig:
|
||||
```
|
||||
az aks get-credentials --resource-group kuberay-rg \
|
||||
--name kuberay-gpu-cluster \
|
||||
--overwrite-existing
|
||||
```
|
||||
@@ -0,0 +1,207 @@
|
||||
(kuberay-config)=
|
||||
|
||||
# RayCluster Configuration
|
||||
|
||||
This guide covers the key aspects of Ray cluster configuration on Kubernetes.
|
||||
|
||||
## Introduction
|
||||
|
||||
Deployments of Ray on Kubernetes follow the [operator pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/). The key players are
|
||||
- A [custom resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) called a `RayCluster` describing the desired state of a Ray cluster.
|
||||
- A [custom controller](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#custom-controllers), the KubeRay operator, which manages Ray pods in order to match the `RayCluster`'s spec.
|
||||
|
||||
To deploy a Ray cluster, one creates a `RayCluster` custom resource (CR):
|
||||
```shell
|
||||
kubectl apply -f raycluster.yaml
|
||||
```
|
||||
|
||||
This guide covers the salient features of `RayCluster` CR configuration.
|
||||
|
||||
For reference, here is a condensed example of a `RayCluster` CR in yaml format.
|
||||
```yaml
|
||||
apiVersion: ray.io/v1alpha1
|
||||
kind: RayCluster
|
||||
metadata:
|
||||
name: raycluster-complete
|
||||
spec:
|
||||
rayVersion: "2.3.0"
|
||||
enableInTreeAutoscaling: true
|
||||
autoscalerOptions:
|
||||
...
|
||||
headGroupSpec:
|
||||
serviceType: ClusterIP # Options are ClusterIP, NodePort, and LoadBalancer
|
||||
rayStartParams:
|
||||
dashboard-host: "0.0.0.0"
|
||||
...
|
||||
template: # Pod template
|
||||
metadata: # Pod metadata
|
||||
spec: # Pod spec
|
||||
containers:
|
||||
- name: ray-head
|
||||
image: rayproject/ray-ml:2.3.0
|
||||
resources:
|
||||
limits:
|
||||
cpu: 14
|
||||
memory: 54Gi
|
||||
requests:
|
||||
cpu: 14
|
||||
memory: 54Gi
|
||||
ports: # Optional service port overrides
|
||||
- containerPort: 6379
|
||||
name: gcs
|
||||
- containerPort: 8265
|
||||
name: dashboard
|
||||
- containerPort: 10001
|
||||
name: client
|
||||
- containerPort: 8000
|
||||
name: serve
|
||||
...
|
||||
workerGroupSpecs:
|
||||
- groupName: small-group
|
||||
replicas: 1
|
||||
minReplicas: 1
|
||||
maxReplicas: 5
|
||||
rayStartParams:
|
||||
...
|
||||
template: # Pod template
|
||||
spec:
|
||||
...
|
||||
# Another workerGroup
|
||||
- groupName: medium-group
|
||||
...
|
||||
# Yet another workerGroup, with access to special hardware perhaps.
|
||||
- groupName: gpu-group
|
||||
...
|
||||
```
|
||||
|
||||
The rest of this guide will discuss the `RayCluster` CR's config fields. See also the [guide](kuberay-autoscaling-config) on configuring Ray autoscaling with KubeRay.
|
||||
|
||||
(kuberay-config-ray-version)=
|
||||
## The Ray Version
|
||||
The field `rayVersion` specifies the version of Ray used in the Ray cluster. The `rayVersion` is used to fill default values for certain config fields. The Ray container images specified in the RayCluster CR should carry the same Ray version as the CR's `rayVersion`. If you are using a nightly or development Ray image, it is fine to set `rayVersion` to the latest release version of Ray.
|
||||
|
||||
## Pod configuration: headGroupSpec and workerGroupSpecs
|
||||
|
||||
At a high level, a RayCluster is a collection of Kubernetes pods, similar to a Kubernetes Deployment or StatefulSet. Just as with the Kubernetes built-ins, the key pieces of configuration are
|
||||
* Pod specification
|
||||
* Scale information (how many pods are desired)
|
||||
|
||||
The key difference between a Deployment and a `RayCluster` is that a `RayCluster` is specialized for running Ray applications. A Ray cluster consists of
|
||||
|
||||
* One **head pod** which hosts global control processes for the Ray cluster. The head pod can also run Ray tasks and actors.
|
||||
* Any number of **worker pods**, which run Ray tasks and actors. Workers come in **worker groups** of identically configured pods. For each worker group, we must specify **replicas**, the number of pods we want of that group.
|
||||
|
||||
The head pod’s configuration is specified under `headGroupSpec`, while configuration for worker pods is specified under `workerGroupSpecs`. There may be multiple worker groups, each group with its own configuration. The `replicas` field of a `workerGroupSpec` specifies the number of worker pods of that group to keep in the cluster. Each `workerGroupSpec` also has optional `minReplicas` and `maxReplicas` fields; these fields are important if you wish to enable {ref}`autoscaling <kuberay-autoscaling-config>`.
|
||||
|
||||
### Pod templates
|
||||
The bulk of the configuration for a `headGroupSpec` or `workerGroupSpec` goes in the `template` field. The `template` is a Kubernetes Pod template which determines the configuration for the pods in the group. Here are some of the subfields of the pod `template` to pay attention to:
|
||||
|
||||
#### containers
|
||||
A Ray pod template specifies at minimum one container, namely the container that runs the Ray processes. A Ray pod template may also specify additional sidecar containers, for purposes such as {ref}`log processing <persist-kuberay-custom-resource-logs>`. However, the KubeRay operator assumes that the first container in the containers list is the main Ray container. Therefore, make sure to specify any sidecar containers **after** the main Ray container. In other words, the Ray container should be the **first** in the `containers` list.
|
||||
|
||||
#### resources
|
||||
|
||||
It's important to specify container CPU and memory resources for each group spec. Since CPU is a [compressible resource], you may want to set only CPU requests and not limits to guarantee your workloads a minimum amount of CPU but [allow them to take advantage of unused CPU and not get throttled][1] if they use more than their requested CPU.
|
||||
|
||||
For GPU workloads, you may also wish to specify GPU limits. For example, set `nvidia.com/gpu: 2` if using an NVIDIA GPU device plugin and you wish to specify a pod with access to 2 GPUs. See {ref}`this guide <kuberay-gpu>` for more details on GPU support.
|
||||
|
||||
KubeRay automatically configures Ray to use the CPU, memory, and GPU **limits** in the Ray container config. These values are the logical resource capacities of Ray pods in the head or worker group. As of KubeRay 1.3.0, KubeRay uses the CPU request if the limit is absent. KubeRay rounds up CPU quantities to the nearest integer. You can override these resource capacities with {ref}`rayStartParams`. KubeRay ignores memory and GPU **requests**. So **set memory and GPU resource requests equal to their limits** when possible
|
||||
|
||||
It's ideal to size each Ray pod to take up the entire Kubernetes node. In other words, it's best to run one large Ray pod per Kubernetes node. In general, it's more efficient to use a few large Ray pods than many small ones. The pattern of fewer large Ray pods has the following advantages:
|
||||
|
||||
- more efficient use of each Ray pod's shared memory object store
|
||||
- reduced communication overhead between Ray pods
|
||||
- reduced redundancy of per-pod Ray control structures such as Raylets
|
||||
|
||||
#### nodeSelector and tolerations
|
||||
You can control the scheduling of worker groups' Ray pods by setting the `nodeSelector` and `tolerations` fields of the pod spec. Specifically, these fields determine on which Kubernetes nodes the pods may be scheduled. See the [Kubernetes docs](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) for more about Pod-to-Node assignment.
|
||||
|
||||
#### image
|
||||
The Ray container images specified in the `RayCluster` CR should carry the same Ray version as the CR's `spec.rayVersion`. If you are using a nightly or development Ray image, you can specify Ray's latest release version under `spec.rayVersion`.
|
||||
|
||||
For Apple M1 or M2 MacBooks, see [Use ARM-based docker images for Apple M1 or M2 MacBooks](docker-image-for-apple-macbooks) to specify the correct image.
|
||||
|
||||
You must install code dependencies for a given Ray task or actor on each Ray node that might run the task or actor. The simplest way to achieve this configuration is to use the same Ray image for the Ray head and all worker groups. In any case, do make sure that all Ray images in your CR carry the same Ray version and Python version. To distribute custom code dependencies across your cluster, you can build a custom container image, using one of the [official Ray images](https://hub.docker.com/r/rayproject/ray) as the base. See {ref}`this guide <docker-images>` to learn more about the official Ray images. For dynamic dependency management geared towards iteration and development, you can also use {ref}`Runtime Environments <runtime-environments>`.
|
||||
|
||||
For `kuberay-operator` versions 1.1.0 and later, the Ray container image must have `wget` installed in it.
|
||||
|
||||
#### metadata.name and metadata.generateName
|
||||
The KubeRay operator will ignore the values of `metadata.name` and `metadata.generateName` set by users. The KubeRay operator will generate a `generateName` automatically to avoid name conflicts. See [KubeRay issue #587](https://github.com/ray-project/kuberay/pull/587) for more details.
|
||||
|
||||
(rayStartParams)=
|
||||
## Ray Start Parameters
|
||||
The ``rayStartParams`` field of each group spec is a string-string map of arguments to the Ray container’s `ray start` entrypoint. For the full list of arguments, refer to the documentation for {ref}`ray start <ray-start-doc>`. The RayCluster Kubernetes custom resource Custom Resource Definition (CRD) in KubeRay versions before 1.4.0 required this field to exist, but the value could be an empty map. As of KubeRay 1.4.0, ``rayStartParams`` is optional.
|
||||
|
||||
Note the following arguments:
|
||||
|
||||
### dashboard-host
|
||||
For most use-cases, this field should be set to "0.0.0.0" for the Ray head pod. This is required to expose the Ray dashboard outside the Ray cluster. (Future versions might set this parameter by default.)
|
||||
|
||||
(kuberay-num-cpus)=
|
||||
### num-cpus
|
||||
This optional field tells the Ray scheduler and autoscaler how many CPUs are available to the Ray pod. The CPU count can be autodetected from the Kubernetes resource limits specified in the group spec’s pod `template`. However, it is sometimes useful to override this autodetected value. For example, setting `num-cpus:"0"` for the Ray head pod will prevent Ray workloads with non-zero CPU requirements from being scheduled on the head. Note that the values of all Ray start parameters, including `num-cpus`, must be supplied as **strings**.
|
||||
|
||||
### num-gpus
|
||||
This field specifies the number of GPUs available to the Ray container. In future KubeRay versions, the number of GPUs will be auto-detected from Ray container resource limits. Note that the values of all Ray start parameters, including `num-gpus`, must be supplied as **strings**.
|
||||
|
||||
### memory
|
||||
The memory available to the Ray is detected automatically from the Kubernetes resource limits. If you wish, you may override this autodetected value by setting the desired memory value, in bytes, under `rayStartParams.memory`. Note that the values of all Ray start parameters, including `memory`, must be supplied as **strings**.
|
||||
|
||||
### resources
|
||||
This field can be used to specify custom resource capacities for the Ray pod. These resource capacities will be advertised to the Ray scheduler and Ray autoscaler. For example, the following annotation will mark a Ray pod as having 1 unit of `Custom1` capacity and 5 units of `Custom2` capacity.
|
||||
```yaml
|
||||
rayStartParams:
|
||||
resources: '"{\"Custom1\": 1, \"Custom2\": 5}"'
|
||||
```
|
||||
You can then annotate tasks and actors with annotations like `@ray.remote(resources={"Custom2": 1})`. The Ray scheduler and autoscaler will take appropriate action to schedule such tasks.
|
||||
|
||||
Note the format used to express the resources string. In particular, note that the backslashes are present as actual characters in the string. If you are specifying a `RayCluster` programmatically, you may have to [escape the backslashes](https://github.com/ray-project/ray/blob/cd9cabcadf1607bcda1512d647d382728055e688/python/ray/tests/kuberay/test_autoscaling_e2e.py#L92) to make sure they are processed as part of the string.
|
||||
|
||||
The field `rayStartParams.resources` should only be used for custom resources. The keys `CPU`, `GPU`, and `memory` are forbidden. If you need to specify overrides for those resource fields, use the Ray start parameters `num-cpus`, `num-gpus`, or `memory`.
|
||||
|
||||
(kuberay-networking)=
|
||||
## Services and Networking
|
||||
### The Ray head service.
|
||||
The KubeRay operator automatically configures a Kubernetes Service exposing the default ports for several services of the Ray head pod, including
|
||||
- Ray Client (default port 10001)
|
||||
- Ray Dashboard (default port 8265)
|
||||
- Ray GCS server (default port 6379)
|
||||
- Ray Serve (default port 8000)
|
||||
- Ray Prometheus metrics (default port 8080)
|
||||
|
||||
The name of the configured Kubernetes Service is the name, `metadata.name`, of the RayCluster followed by the suffix <nobr>`head-svc`</nobr>. For the example CR given on this page, the name of the head service will be
|
||||
<nobr>`raycluster-example-head-svc`</nobr>. Kubernetes networking (`kube-dns`) then allows us to address
|
||||
the Ray head's services using the name <nobr>`raycluster-example-head-svc`</nobr>. For example, the Ray Client server can be accessed from a pod in the same Kubernetes namespace using
|
||||
```python
|
||||
ray.init("ray://raycluster-example-head-svc:10001")
|
||||
```
|
||||
The Ray Client server can be accessed from a pod in another namespace using
|
||||
```python
|
||||
ray.init("ray://raycluster-example-head-svc.default.svc.cluster.local:10001")
|
||||
```
|
||||
(This assumes the Ray cluster was deployed into the default Kubernetes namespace. If the Ray cluster is deployed in a non-default namespace, use that namespace in place of `default`.)
|
||||
|
||||
### Specifying non-default ports.
|
||||
If you wish to override the ports exposed by the Ray head service, you may do so by specifying the Ray head container's `ports` list, under `headGroupSpec`. Here is an example of a list of non-default ports for the Ray head service.
|
||||
```yaml
|
||||
ports:
|
||||
- containerPort: 6380
|
||||
name: gcs
|
||||
- containerPort: 8266
|
||||
name: dashboard
|
||||
- containerPort: 10002
|
||||
name: client
|
||||
```
|
||||
If the head container's `ports` list is specified, the Ray head service will expose precisely the ports in the list. In the above example, the head service will expose just three ports; in particular there will be no port exposed for Ray Serve.
|
||||
|
||||
For the Ray head to actually use the non-default ports specified in the ports list, you must also specify the relevant `rayStartParams`. For the above example,
|
||||
```yaml
|
||||
rayStartParams:
|
||||
port: "6380"
|
||||
dashboard-port: "8266"
|
||||
ray-client-server-port: "10002"
|
||||
...
|
||||
```
|
||||
|
||||
[compressible resource]: https://kubernetes.io/blog/2021/11/26/qos-memory-resources/#:~:text=CPU%20is%20considered%20a%20%22compressible%22%20resource.%20If%20your%20app%20starts%20hitting%20your%20CPU%20limits%2C%20Kubernetes%20starts%20throttling%20your%20container%2C%20giving%20your%20app%20potentially%20worse%20performance.%20However%2C%20it%20won%E2%80%99t%20be%20terminated.%20That%20is%20what%20%22compressible%22%20means
|
||||
[1]: https://home.robusta.dev/blog/stop-using-cpu-limits
|
||||
@@ -0,0 +1,484 @@
|
||||
(kuberay-autoscaling)=
|
||||
|
||||
# KubeRay Autoscaling
|
||||
|
||||
This guide explains how to configure the Ray Autoscaler on Kubernetes. The Ray Autoscaler is a Ray cluster process that automatically scales a cluster up and down based on resource demand. The Autoscaler does this by adjusting the number of nodes (Ray Pods) in the cluster based on the resources required by tasks, actors, or placement groups.
|
||||
|
||||
The Autoscaler utilizes logical resource requests, indicated in `@ray.remote` and shown in `ray status`, not the physical machine utilization, to scale. If you launch an actor, task, or placement group and resources are insufficient, the Autoscaler queues the request. It adjusts the number of nodes to meet queue demands and removes idle nodes that have no tasks, actors, or objects over time.
|
||||
|
||||
```{admonition} When to use Autoscaling?
|
||||
Autoscaling can reduce workload costs, but adds node launch overheads and can be tricky to configure.
|
||||
We recommend starting with non-autoscaling clusters if you're new to Ray.
|
||||
```
|
||||
|
||||
```{admonition} Ray Autoscaling V2 alpha with KubeRay (@ray 2.10.0)
|
||||
With Ray 2.10, Ray Autoscaler V2 alpha is available with KubeRay. It has improvements on observability and stability. Please see the [section](kuberay-autoscaler-v2) for more details.
|
||||
```
|
||||
|
||||
## Overview
|
||||
|
||||
The following diagram illustrates the integration of the Ray Autoscaler with the KubeRay operator. Although depicted as a separate entity for clarity, the Ray Autoscaler is actually a sidecar container within the Ray head Pod in the actual implementation.
|
||||
|
||||
```{eval-rst}
|
||||
.. image:: ../images/AutoscalerOperator.svg
|
||||
:align: center
|
||||
..
|
||||
Find the source document here (https://docs.google.com/drawings/d/1LdOg9JQuN5AOII-vDpSaFBsTeg0JGWcsbyNNLP1yovg/edit)
|
||||
```
|
||||
|
||||
```{admonition} 3 levels of autoscaling in KubeRay
|
||||
* **Ray actor/task**: Some Ray libraries, like Ray Serve, can automatically adjust the number of Serve replicas (i.e., Ray actors) based on the incoming request volume.
|
||||
* **Ray node**: Ray Autoscaler automatically adjusts the number of Ray nodes (i.e., Ray Pods) based on the resource demand of Ray actors/tasks.
|
||||
* **Kubernetes node**: If the Kubernetes cluster lacks sufficient resources for the new Ray Pods that the Ray Autoscaler creates, the Kubernetes Autoscaler can provision a new Kubernetes node. ***You must configure the Kubernetes Autoscaler yourself.***
|
||||
```
|
||||
|
||||
* The Autoscaler scales up the cluster through the following sequence of events:
|
||||
1. A user submits a Ray workload.
|
||||
2. The Ray head container aggregates the workload resource requirements and communicates them to the Ray Autoscaler sidecar.
|
||||
3. The Autoscaler decides to add a Ray worker Pod to satisfy the workload's resource requirement.
|
||||
4. The Autoscaler requests an additional worker Pod by incrementing the RayCluster CR's `replicas` field.
|
||||
5. The KubeRay operator creates a Ray worker Pod to match the new `replicas` specification.
|
||||
6. The Ray scheduler places the user's workload on the new worker Pod.
|
||||
|
||||
* The Autoscaler also scales down the cluster by removing idle worker Pods. If it finds an idle worker Pod, it reduces the count in the RayCluster CR's `replicas` field and adds the identified Pods to the CR's `workersToDelete` field. Then, the KubeRay operator deletes the Pods in the `workersToDelete` field.
|
||||
|
||||
## Quickstart
|
||||
|
||||
### Step 1: Create a Kubernetes cluster with Kind
|
||||
|
||||
```bash
|
||||
kind create cluster --image=kindest/node:v1.26.0
|
||||
```
|
||||
|
||||
### Step 2: Install the KubeRay operator
|
||||
|
||||
Follow [this document](kuberay-operator-deploy) to install the latest stable KubeRay operator via Helm repository.
|
||||
|
||||
### Step 3: Create a RayCluster custom resource with autoscaling enabled
|
||||
|
||||
```bash
|
||||
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-cluster.autoscaler.yaml
|
||||
```
|
||||
|
||||
### Step 4: Verify the Kubernetes cluster status
|
||||
|
||||
```bash
|
||||
# Step 4.1: List all Ray Pods in the `default` namespace.
|
||||
kubectl get pods -l=ray.io/is-ray-node=yes
|
||||
|
||||
# [Example output]
|
||||
# NAME READY STATUS RESTARTS AGE
|
||||
# raycluster-autoscaler-head 2/2 Running 0 107s
|
||||
|
||||
# Step 4.2: Check the ConfigMap in the `default` namespace.
|
||||
kubectl get configmaps
|
||||
|
||||
# [Example output]
|
||||
# NAME DATA AGE
|
||||
# ray-example 2 21s
|
||||
# ...
|
||||
```
|
||||
|
||||
The RayCluster has one head Pod and zero worker Pods. The head Pod has two containers: a Ray head container and a Ray Autoscaler sidecar container. Additionally, the [ray-cluster.autoscaler.yaml](https://github.com/ray-project/kuberay/blob/v1.6.0/ray-operator/config/samples/ray-cluster.autoscaler.yaml) includes a ConfigMap named `ray-example` that contains two Python scripts: `detached_actor.py` and `terminate_detached_actor.py`.
|
||||
|
||||
* `detached_actor.py` is a Python script that creates a detached actor which requires 1 CPU.
|
||||
```py
|
||||
import ray
|
||||
import sys
|
||||
|
||||
@ray.remote(num_cpus=1)
|
||||
class Actor:
|
||||
pass
|
||||
|
||||
ray.init(namespace="default_namespace")
|
||||
Actor.options(name=sys.argv[1], lifetime="detached").remote()
|
||||
```
|
||||
|
||||
* `terminate_detached_actor.py` is a Python script that terminates a detached actor.
|
||||
```py
|
||||
import ray
|
||||
import sys
|
||||
|
||||
ray.init(namespace="default_namespace")
|
||||
detached_actor = ray.get_actor(sys.argv[1])
|
||||
ray.kill(detached_actor)
|
||||
```
|
||||
|
||||
### Step 5: Trigger RayCluster scale-up by creating detached actors
|
||||
|
||||
```bash
|
||||
# Step 5.1: Create a detached actor "actor1" which requires 1 CPU.
|
||||
export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)
|
||||
kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/detached_actor.py actor1
|
||||
|
||||
# Step 5.2: The Ray Autoscaler creates a new worker Pod.
|
||||
kubectl get pods -l=ray.io/is-ray-node=yes
|
||||
|
||||
# [Example output]
|
||||
# NAME READY STATUS RESTARTS AGE
|
||||
# raycluster-autoscaler-head 2/2 Running 0 xxm
|
||||
# raycluster-autoscaler-small-group-worker-yyyyy 1/1 Running 0 xxm
|
||||
|
||||
# Step 5.3: Create a detached actor which requires 1 CPU.
|
||||
kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/detached_actor.py actor2
|
||||
kubectl get pods -l=ray.io/is-ray-node=yes
|
||||
|
||||
# [Example output]
|
||||
# NAME READY STATUS RESTARTS AGE
|
||||
# raycluster-autoscaler-head 2/2 Running 0 xxm
|
||||
# raycluster-autoscaler-small-group-worker-yyyyy 1/1 Running 0 xxm
|
||||
# raycluster-autoscaler-small-group-worker-zzzzz 1/1 Running 0 xxm
|
||||
|
||||
# Step 5.4: List all actors in the Ray cluster.
|
||||
kubectl exec -it $HEAD_POD -- ray list actors
|
||||
|
||||
|
||||
# ======= List: 2023-09-06 13:26:49.228594 ========
|
||||
# Stats:
|
||||
# ------------------------------
|
||||
# Total: 2
|
||||
|
||||
# Table:
|
||||
# ------------------------------
|
||||
# ACTOR_ID CLASS_NAME STATE JOB_ID NAME ...
|
||||
# 0 xxxxxxxx Actor ALIVE 02000000 actor1 ...
|
||||
# 1 xxxxxxxx Actor ALIVE 03000000 actor2 ...
|
||||
```
|
||||
|
||||
The Ray Autoscaler generates a new worker Pod for each new detached actor. This is because the `rayStartParams` field in the Ray head specifies `num-cpus: "0"`, preventing the Ray scheduler from scheduling any Ray actors or tasks on the Ray head Pod. In addition, each Ray worker Pod has a capacity of 1 CPU, so the Autoscaler creates a new worker Pod to satisfy the resource requirement of the detached actor which requires 1 CPU.
|
||||
|
||||
* Using detached actors isn't necessary to trigger cluster scale-up. Normal actors and tasks can also initiate it. [Detached actors](actor-lifetimes) remain persistent even after the job's driver process exits, which is why the Autoscaler doesn't scale down the cluster automatically when the `detached_actor.py` process exits, making it more convenient for this tutorial.
|
||||
|
||||
* In this RayCluster custom resource, each Ray worker Pod possesses only 1 logical CPU from the perspective of the Ray Autoscaler. Therefore, if you create a detached actor with `@ray.remote(num_cpus=2)`, the Autoscaler doesn't initiate the creation of a new worker Pod because the capacity of the existing Pod is limited to 1 CPU.
|
||||
|
||||
* (Advanced) The Ray Autoscaler also offers a [Python SDK](ref-autoscaler-sdk), enabling advanced users, like Ray maintainers, to request resources directly from the Autoscaler. Generally, most users don't need to use the SDK.
|
||||
|
||||
### Step 6: Trigger RayCluster scale-down by terminating detached actors
|
||||
|
||||
```bash
|
||||
# Step 6.1: Terminate the detached actor "actor1".
|
||||
kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/terminate_detached_actor.py actor1
|
||||
|
||||
# Step 6.2: A worker Pod will be deleted after `idleTimeoutSeconds` (default 60s) seconds.
|
||||
kubectl get pods -l=ray.io/is-ray-node=yes
|
||||
|
||||
# [Example output]
|
||||
# NAME READY STATUS RESTARTS AGE
|
||||
# raycluster-autoscaler-head 2/2 Running 0 xxm
|
||||
# raycluster-autoscaler-small-group-worker-zzzzz 1/1 Running 0 xxm
|
||||
|
||||
# Step 6.3: Terminate the detached actor "actor2".
|
||||
kubectl exec -it $HEAD_POD -- python3 /home/ray/samples/terminate_detached_actor.py actor2
|
||||
|
||||
# Step 6.4: A worker Pod will be deleted after `idleTimeoutSeconds` (default 60s) seconds.
|
||||
kubectl get pods -l=ray.io/is-ray-node=yes
|
||||
|
||||
# [Example output]
|
||||
# NAME READY STATUS RESTARTS AGE
|
||||
# raycluster-autoscaler-head 2/2 Running 0 xxm
|
||||
```
|
||||
|
||||
### Step 7: Ray Autoscaler observability
|
||||
|
||||
```bash
|
||||
# Method 1: "ray status"
|
||||
kubectl exec $HEAD_POD -it -c ray-head -- ray status
|
||||
|
||||
# [Example output]:
|
||||
# ======== Autoscaler status: 2023-09-06 13:42:46.372683 ========
|
||||
# Node status
|
||||
# ---------------------------------------------------------------
|
||||
# Healthy:
|
||||
# 1 head-group
|
||||
# Pending:
|
||||
# (no pending nodes)
|
||||
# Recent failures:
|
||||
# (no failures)
|
||||
|
||||
# Resources
|
||||
# ---------------------------------------------------------------
|
||||
# Usage:
|
||||
# 0B/1.86GiB memory
|
||||
# 0B/514.69MiB object_store_memory
|
||||
|
||||
# Demands:
|
||||
# (no resource demands)
|
||||
|
||||
# Method 2: "kubectl logs"
|
||||
kubectl logs $HEAD_POD -c autoscaler | tail -n 20
|
||||
|
||||
# [Example output]:
|
||||
# 2023-09-06 13:43:22,029 INFO autoscaler.py:421 --
|
||||
# ======== Autoscaler status: 2023-09-06 13:43:22.028870 ========
|
||||
# Node status
|
||||
# ---------------------------------------------------------------
|
||||
# Healthy:
|
||||
# 1 head-group
|
||||
# Pending:
|
||||
# (no pending nodes)
|
||||
# Recent failures:
|
||||
# (no failures)
|
||||
|
||||
# Resources
|
||||
# ---------------------------------------------------------------
|
||||
# Usage:
|
||||
# 0B/1.86GiB memory
|
||||
# 0B/514.69MiB object_store_memory
|
||||
|
||||
# Demands:
|
||||
# (no resource demands)
|
||||
# 2023-09-06 13:43:22,029 INFO autoscaler.py:464 -- The autoscaler took 0.036 seconds to complete the update iteration.
|
||||
```
|
||||
|
||||
### Step 8: Clean up the Kubernetes cluster
|
||||
|
||||
```bash
|
||||
# Delete RayCluster and ConfigMap
|
||||
kubectl delete -f https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-cluster.autoscaler.yaml
|
||||
|
||||
# Uninstall the KubeRay operator
|
||||
helm uninstall kuberay-operator
|
||||
|
||||
# Delete the kind cluster
|
||||
kind delete cluster
|
||||
```
|
||||
|
||||
(kuberay-autoscaling-config)=
|
||||
## KubeRay Autoscaling Configurations
|
||||
|
||||
The [ray-cluster.autoscaler.yaml](https://github.com/ray-project/kuberay/blob/v1.6.0/ray-operator/config/samples/ray-cluster.autoscaler.yaml) used in the quickstart example contains detailed comments about the configuration options. ***It's recommended to read this section in conjunction with the YAML file.***
|
||||
|
||||
### 1. Enabling autoscaling
|
||||
|
||||
* **`enableInTreeAutoscaling`**: By setting `enableInTreeAutoscaling: true`, the KubeRay operator automatically configures an autoscaling sidecar container for the Ray head Pod.
|
||||
* **`minReplicas` / `maxReplicas` / `replicas`**: Set the `minReplicas` and `maxReplicas` fields to define the range for `replicas` in an autoscaling `workerGroup`. Typically, you would initialize both `replicas` and `minReplicas` with the same value during the deployment of an autoscaling cluster. Subsequently, the Ray Autoscaler adjusts the `replicas` field as it adds or removes Pods from the cluster.
|
||||
|
||||
### 2. Scale-up and scale-down speed
|
||||
|
||||
If necessary, you can regulate the pace of adding or removing nodes from the cluster. For applications with numerous short-lived tasks, considering a more conservative approach to adjusting the upscaling and downscaling speeds might be beneficial.
|
||||
|
||||
Utilize the `RayCluster` CR's `autoscalerOptions` field to accomplish this. This field encompasses the following sub-fields:
|
||||
|
||||
* **`upscalingMode`**: This controls the rate of scale-up process. The valid values are:
|
||||
- `Conservative`: Upscaling is rate-limited; the number of pending worker Pods is at most the number of worker pods connected to the Ray cluster.
|
||||
- `Default`: Upscaling isn't rate-limited.
|
||||
- `Aggressive`: An alias for Default; upscaling isn't rate-limited.
|
||||
|
||||
* **`idleTimeoutSeconds`** (default 60s): This denotes the waiting time in seconds before scaling down an idle worker pod. A worker node is idle when it has no active tasks, actors, or referenced objects, either stored in-memory or spilled to disk.
|
||||
|
||||
### 3. Autoscaler sidecar container
|
||||
|
||||
The `autoscalerOptions` field also provides options for configuring the Autoscaler container. Usually, it's not necessary to specify these options.
|
||||
|
||||
* **`resources`**: The `resources` sub-field of `autoscalerOptions` sets optional resource overrides for the Autoscaler sidecar container. These overrides should be specified in the standard [container resource spec format](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/Pod-v1/#resources). The default values are indicated below:
|
||||
```yaml
|
||||
resources:
|
||||
limits:
|
||||
cpu: "500m"
|
||||
memory: "512Mi"
|
||||
requests:
|
||||
cpu: "500m"
|
||||
memory: "512Mi"
|
||||
```
|
||||
|
||||
* **`image`**: This field overrides the Autoscaler container image. The container uses the same **image** as the Ray container by default.
|
||||
|
||||
* **`imagePullPolicy`**: This field overrides the Autoscaler container's image pull policy. The default is `IfNotPresent`.
|
||||
|
||||
* **`env`** and **`envFrom`**: These fields specify Autoscaler container environment variables. These fields should be formatted following the [Kubernetes API](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/Pod-v1/#environment-variables) for container environment variables.
|
||||
|
||||
* **`command`** and **`args`** (KubeRay >= v1.7.0):
|
||||
These fields independently override the autoscaler container's `command` and `args`.
|
||||
KubeRay also injects the environment variable `KUBERAY_GEN_AUTOSCALER_START_CMD` into the autoscaler container, so you can reference the generated command in your custom `args`:
|
||||
```sh
|
||||
# Example value of KUBERAY_GEN_AUTOSCALER_START_CMD inside the autoscaler container:
|
||||
ray kuberay-autoscaler --cluster-name $(RAY_CLUSTER_NAME) --cluster-namespace $(RAY_CLUSTER_NAMESPACE)
|
||||
```
|
||||
Note that `$(RAY_CLUSTER_NAME)` and `$(RAY_CLUSTER_NAMESPACE)` are resolved by the system before the container starts. When bash later expands `$KUBERAY_GEN_AUTOSCALER_START_CMD`, the command already contains the resolved values.
|
||||
|
||||
```yaml
|
||||
autoscalerOptions:
|
||||
version: v2
|
||||
# KubeRay uses `command` and `args` as-is when they are present,
|
||||
# instead of generating the autoscaler start command.
|
||||
command: ["/bin/bash", "-lc", "--"]
|
||||
# Reference the generated command via $KUBERAY_GEN_AUTOSCALER_START_CMD to wrap it with extra logic.
|
||||
args: ["echo 'Starting autoscaler...'; $KUBERAY_GEN_AUTOSCALER_START_CMD"]
|
||||
```
|
||||
|
||||
For a complete example, see [ray-cluster.autoscaler-v2-overwrite-cmd.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-cluster.autoscaler-v2-overwrite-cmd.yaml).
|
||||
|
||||
### 4. Set the `rayStartParams` and the resource limits for the Ray container
|
||||
|
||||
```{admonition} Resource limits are optional starting from Ray 2.41.0
|
||||
Starting from Ray 2.41.0, the Ray Autoscaler can read resource specifications from `rayStartParams`, resource limits, or resource requests of the Ray container. You must specify at least one of these fields.
|
||||
Earlier versions only support `rayStartParams` or resource limits, and don't recognize resource requests.
|
||||
```
|
||||
|
||||
```{admonition} rayStartParams is optional if you're using an autoscaler image with Ray 2.45.0 or later.
|
||||
`rayStartParams` is optional with RayCluster CRD from KubeRay 1.4.0 or later but required in earlier versions.
|
||||
If you omit `rayStartParams` and want to use autoscaling, the autoscaling image must have Ray 2.45.0 or later.
|
||||
```
|
||||
|
||||
The Ray Autoscaler reads the `rayStartParams` field or the Ray container's resource limits in the RayCluster custom resource specification to determine the Ray Pod's resource requirements. The information regarding the number of CPUs is essential for the Ray Autoscaler to scale the cluster. Therefore, without this information, the Ray Autoscaler reports an error and fails to start. Take [ray-cluster.autoscaler.yaml](https://github.com/ray-project/kuberay/blob/v1.6.0/ray-operator/config/samples/ray-cluster.autoscaler.yaml) as an example below:
|
||||
|
||||
* If users set `num-cpus` in `rayStartParams`, Ray Autoscaler would work regardless of the resource limits on the container.
|
||||
* If users don't set `rayStartParams`, the Ray container must have a specified CPU resource limit.
|
||||
|
||||
```yaml
|
||||
headGroupSpec:
|
||||
rayStartParams:
|
||||
num-cpus: "0"
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: ray-head
|
||||
resources:
|
||||
# The Ray Autoscaler still functions if you comment out the `limits` field for the
|
||||
# head container, as users have already specified `num-cpus` in `rayStartParams`.
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: "2G"
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: "2G"
|
||||
...
|
||||
workerGroupSpecs:
|
||||
- groupName: small-group
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: ray-worker
|
||||
resources:
|
||||
limits:
|
||||
# The Ray Autoscaler versions older than 2.41.0 will fail to start if the CPU resource limit for the worker
|
||||
# container is commented out because `rayStartParams` is empty.
|
||||
# The Ray Autoscaler starting from 2.41.0 will not fail but use the resource requests if the resource
|
||||
# limits are commented out and `rayStartParams` is empty.
|
||||
cpu: "1"
|
||||
memory: "1G"
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: "1G"
|
||||
```
|
||||
|
||||
### 5. Autoscaler environment configuration
|
||||
|
||||
You can configure the Ray autoscaler using environment variables specified in the `env` or `envFrom` fields under the `autoscalerOptions` section of your RayCluster custom resource. These variables provide fine-grained control over how the autoscaler behaves internally.
|
||||
|
||||
For example, `AUTOSCALER_UPDATE_INTERVAL_S` determines how frequently the autoscaler checks the cluster status and decides whether to scale up or down.
|
||||
|
||||
For complete examples, see [ray-cluster.autoscaler.yaml](https://github.com/ray-project/kuberay/blob/099bf616c012975031ea9e5bbf7843af03e5f05b/ray-operator/config/samples/ray-cluster.autoscaler.yaml#L28-L33) and [ray-cluster.autoscaler-v2.yaml](https://github.com/ray-project/kuberay/blob/099bf616c012975031ea9e5bbf7843af03e5f05b/ray-operator/config/samples/ray-cluster.autoscaler-v2.yaml#L16_L21).
|
||||
|
||||
```yaml
|
||||
autoscalerOptions:
|
||||
env:
|
||||
- name: AUTOSCALER_UPDATE_INTERVAL_S
|
||||
value: "5"
|
||||
```
|
||||
|
||||
## Next steps
|
||||
|
||||
See [(Advanced) Understanding the Ray Autoscaler in the Context of Kubernetes](ray-k8s-autoscaler-comparison) for more details about the relationship between the Ray Autoscaler and Kubernetes autoscalers.
|
||||
|
||||
(kuberay-autoscaler-v2)=
|
||||
### Autoscaler V2 with KubeRay
|
||||
|
||||
#### Prerequisites
|
||||
|
||||
* KubeRay v1.4.0 and the latest Ray version are the preferred setup for Autoscaler V2.
|
||||
|
||||
The release of Ray 2.10.0 introduces the alpha version of Ray Autoscaler V2 integrated with KubeRay, bringing enhancements in terms of observability and stability:
|
||||
|
||||
|
||||
1. **Observability**: The Autoscaler V2 provides instance level tracing for each Ray worker's lifecycle, making it easier to debug and understand the Autoscaler behavior. It also reports the idle information about each node, including details on why nodes are idle or active:
|
||||
|
||||
```bash
|
||||
|
||||
> ray status -v
|
||||
|
||||
======== Autoscaler status: 2024-03-08 21:06:21.023751 ========
|
||||
GCS request time: 0.003238s
|
||||
|
||||
Node status
|
||||
---------------------------------------------------------------
|
||||
Active:
|
||||
1 node_40f427230584b2d9c9f113d8db51d10eaf914aa9bf61f81dc7fabc64
|
||||
Idle:
|
||||
1 node_2d5fd3d4337ba5b5a8c3106c572492abb9a8de2dee9da7f6c24c1346
|
||||
Pending:
|
||||
(no pending nodes)
|
||||
Recent failures:
|
||||
(no failures)
|
||||
|
||||
Resources
|
||||
---------------------------------------------------------------
|
||||
Total Usage:
|
||||
1.0/64.0 CPU
|
||||
0B/72.63GiB memory
|
||||
0B/33.53GiB object_store_memory
|
||||
|
||||
Pending Demands:
|
||||
(no resource demands)
|
||||
|
||||
Node: 40f427230584b2d9c9f113d8db51d10eaf914aa9bf61f81dc7fabc64
|
||||
Usage:
|
||||
1.0/32.0 CPU
|
||||
0B/33.58GiB memory
|
||||
0B/16.79GiB object_store_memory
|
||||
# New in autoscaler V2: activity information
|
||||
Activity:
|
||||
Busy workers on node.
|
||||
Resource: CPU currently in use.
|
||||
|
||||
Node: 2d5fd3d4337ba5b5a8c3106c572492abb9a8de2dee9da7f6c24c1346
|
||||
# New in autoscaler V2: idle information
|
||||
Idle: 107356 ms
|
||||
Usage:
|
||||
0.0/32.0 CPU
|
||||
0B/39.05GiB memory
|
||||
0B/16.74GiB object_store_memory
|
||||
Activity:
|
||||
(no activity)
|
||||
```
|
||||
|
||||
2. **Stability** Autoscaler V2 makes significant improvements to idle node handling. The V1 autoscaler could stop nodes that became active during termination processing, potentially failing tasks or actors. V2 uses Ray's graceful draining mechanism, which safely stops idle nodes without disrupting ongoing work.
|
||||
|
||||
[ray-cluster.autoscaler-v2.yaml](https://github.com/ray-project/kuberay/blob/v1.6.0/ray-operator/config/samples/ray-cluster.autoscaler-v2.yaml) is an example YAML file of a RayCluster with Autoscaler V2 enabled that works with the latest KubeRay version.
|
||||
|
||||
If you're using KubeRay >= 1.4.0, enable V2 by setting `RayCluster.spec.autoscalerOptions.version: v2`.
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
enableInTreeAutoscaling: true
|
||||
# Set .spec.autoscalerOptions.version: v2
|
||||
autoscalerOptions:
|
||||
version: v2
|
||||
```
|
||||
|
||||
If you're using KubeRay < 1.4.0, enable V2 by setting the `RAY_enable_autoscaler_v2` environment variable in the head and using `restartPolicy: Never` on head and all worker groups.
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
enableInTreeAutoscaling: true
|
||||
headGroupSpec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: ray-head
|
||||
image: rayproject/ray:2.X.Y
|
||||
env:
|
||||
# Set this environment variable
|
||||
- name: RAY_enable_autoscaler_v2
|
||||
value: "1"
|
||||
restartPolicy: Never # Prevent container restart to maintain Ray health.
|
||||
|
||||
|
||||
# Prevent Kubernetes from restarting Ray worker pod containers, enabling correct instance management by Ray.
|
||||
workerGroupSpecs:
|
||||
- replicas: 1
|
||||
template:
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
...
|
||||
```
|
||||
@@ -0,0 +1,275 @@
|
||||
(kuberay-in-place-pod-resizing)=
|
||||
|
||||
# KubeRay In-Place Pod Resizing (IPPR)
|
||||
|
||||
This guide explains how to configure In-Place Pod Resizing (IPPR) for the Ray Autoscaler on Kubernetes 1.35+ with Ray 2.56 or later. IPPR allows the Ray Autoscaler to vertically resize running Pods (CPU and memory) to change the cluster capacity.
|
||||
|
||||
```{admonition} Alpha feature
|
||||
:class: warning
|
||||
|
||||
Ray Autoscaler integration with Kubernetes [In-Place Pod Resize](https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/) is still an alpha feature.
|
||||
APIs and behavior may change in future releases.
|
||||
```
|
||||
|
||||
## Overview
|
||||
|
||||
For background on the underlying Kubernetes feature, see [Resize CPU and Memory Resources assigned to Containers](https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/) in the Kubernetes documentation.
|
||||
|
||||
Without IPPR, the Ray Autoscaler scales horizontally only: when pending tasks, actors, or placement groups can't fit on existing Ray worker nodes, the Autoscaler launches new worker Pods. If the underlying Kubernetes cluster doesn't have capacity for those Pods, the [Kubernetes Cluster Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler) (which you configure separately) can in turn provision new Kubernetes nodes. See {ref}`the 3 levels of autoscaling in KubeRay for more details <kuberay-autoscaling>`.
|
||||
|
||||
With IPPR, the Autoscaler can first try to satisfy pending demand by *resizing* existing worker Pods without restarting them up to a per-group maximum, and only falls back to launching new Pods when in-place resizing isn't sufficient.
|
||||
|
||||
```{admonition} When to use IPPR?
|
||||
IPPR can reduce Pod-launch overhead and improve packing of long-lived workloads on a smaller number of larger Pods.
|
||||
It is most useful when:
|
||||
|
||||
* You have workloads with bursty resource demand that benefit from vertical scaling.
|
||||
* Worker Pod startup latency dominates your scale-up time.
|
||||
* The underlying Kubernetes nodes have headroom for larger worker Pods (or your Kubernetes cluster autoscaler can provide it).
|
||||
```
|
||||
|
||||
The Autoscaler's high-level behavior with IPPR enabled is:
|
||||
|
||||
1. After bin-packing pending tasks onto existing Ray worker nodes at their current capacity, the Autoscaler tries to bin-pack remaining demand onto Ray worker nodes that have no ongoing resize, this time using their *maximum* capacity. If a Ray worker node can absorb more demand by resizing, the Autoscaler issues a Kubernetes resize request for that node's Pod and records the status in the `ray.io/ippr-status` annotation on the Pod.
|
||||
2. If demand still remains, the Autoscaler falls back to horizontal scale-out, taking each worker group's maximum capacity into account.
|
||||
3. On the next reconciliation, the Autoscaler checks each in-flight resize:
|
||||
|
||||
* If the resize succeeded, the Autoscaler updates the Raylet's logical resources to match the new Pod size and updates `ray.io/ippr-status` accordingly.
|
||||
* If the resize timed out or errored, the Autoscaler queues a new Kubernetes resize request to adjust the Pod (for example, rolling it back to its previous size after a timeout, as shown in [Case 4](#case-4-resize-timeout-and-rollback)).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
* **Kubernetes 1.35 or later.** In-Place Pod Resize graduated to GA in Kubernetes 1.35 ([blog post](https://kubernetes.io/blog/2025/12/19/kubernetes-v1-35-in-place-pod-resize-ga/)).
|
||||
* **KubeRay v1.5.0 or later.**
|
||||
* **Ray Autoscaler V2** enabled on the RayCluster. See {ref}`kuberay-autoscaler-v2`.
|
||||
|
||||
## Configuration
|
||||
|
||||
Enable IPPR by setting the `ray.io/ippr` annotation on the RayCluster custom resource. The annotation value is a JSON document keyed by worker `groupName`:
|
||||
|
||||
```text
|
||||
{
|
||||
"groups": {
|
||||
"<groupName>": {
|
||||
"max-cpu": string|number,
|
||||
"max-memory": string|integer,
|
||||
"resize-timeout": integer
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`<groupName>` must match a `groupName` under `workerGroupSpecs` on the RayCluster. Each entry has the following required fields:
|
||||
|
||||
* **`max-cpu`**: The maximum CPU the Autoscaler may resize a Pod in this group to. Accepts any [Kubernetes CPU quantity](https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/quantity/), for example `"2"` or `"1500m"`.
|
||||
* **`max-memory`**: The maximum memory the Autoscaler may resize a Pod in this group to. Accepts any Kubernetes memory quantity, for example `"8Gi"` or `2147483648` (raw bytes).
|
||||
* **`resize-timeout`**: Number of seconds to wait for a Kubernetes Pod resize to complete before considering it timed out and rolling it back.
|
||||
|
||||
### Validation rules
|
||||
|
||||
In addition to the schema above, the Ray Autoscaler validates the following requirements when IPPR is enabled for a worker group. If validation fails, the Autoscaler raises an error during reconciliation (visible in the autoscaler logs in the head Pod) and will refuse to launch worker Pods for the misconfigured group until the RayCluster is updated accordingly.
|
||||
|
||||
1. **No CPU/memory in `rayStartParams`.** The corresponding worker group must not set `num-cpus` or `memory` in `rayStartParams`. Hard-coding logical resources there would cause Ray's view of the node's capacity to drift from the Pod's physical resources after a resize.
|
||||
2. **CPU and memory requests are required.** The Ray container in the worker group must specify both `cpu` and `memory` under `resources.requests`.
|
||||
3. **`resizePolicy` must use `restartPolicy: NotRequired`.** That's the Kubernetes default, so leaving `resizePolicy` unset is fine. Setting it to `RestartContainer`, which causes the container to be restarted, is not yet supported by Ray.
|
||||
|
||||
## Example
|
||||
|
||||
The following RayCluster excerpt enables IPPR for a worker group `small-group` whose Pods start at 1 CPU / 1Gi and may be resized up to 4 CPU / 4Gi.
|
||||
|
||||
```yaml
|
||||
apiVersion: ray.io/v1
|
||||
kind: RayCluster
|
||||
metadata:
|
||||
name: raycluster-ippr
|
||||
annotations:
|
||||
ray.io/ippr: |
|
||||
{
|
||||
"groups": {
|
||||
"small-group": {
|
||||
"max-cpu": "4",
|
||||
"max-memory": "4Gi",
|
||||
"resize-timeout": 60
|
||||
}
|
||||
}
|
||||
}
|
||||
spec:
|
||||
enableInTreeAutoscaling: true
|
||||
autoscalerOptions:
|
||||
# IPPR requires Autoscaler V2.
|
||||
version: v2
|
||||
env:
|
||||
- name: AUTOSCALER_UPDATE_INTERVAL_S
|
||||
value: "1"
|
||||
headGroupSpec:
|
||||
rayStartParams:
|
||||
num-cpus: "0"
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: ray-head
|
||||
image: rayproject/ray:2.56.0
|
||||
resources:
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: "4Gi"
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: "4Gi"
|
||||
workerGroupSpecs:
|
||||
- groupName: small-group
|
||||
replicas: 1
|
||||
minReplicas: 1
|
||||
maxReplicas: 10
|
||||
# Note: do NOT set num-cpus or memory in rayStartParams when using IPPR.
|
||||
rayStartParams: {}
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: ray-worker
|
||||
image: rayproject/ray:2.56.0
|
||||
# CPU and memory requests are required for IPPR.
|
||||
resources:
|
||||
requests:
|
||||
cpu: "1"
|
||||
memory: "1Gi"
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: "1Gi"
|
||||
# `resizePolicy` is omitted; IPPR requires the default
|
||||
# `restartPolicy: NotRequired` for cpu and memory.
|
||||
```
|
||||
|
||||
## Resize behavior
|
||||
|
||||
In the examples below, each entry of `WorkerGroup1=[...]` represents one Ray worker node (a worker Pod). `CPU` is the node's total CPU capacity (which IPPR can change in place) and `Available` is the unallocated portion. The cluster has one head node (`CPU=0`) and one worker group `WorkerGroup1` whose Pods start at `CPU=1` and may be resized up to `CPU=4`.
|
||||
|
||||
### Case 1: Resize an existing Ray worker node
|
||||
|
||||
A single idle worker can absorb a pending 2-CPU task by resizing in place.
|
||||
|
||||
```text
|
||||
T1: WorkerGroup1=[{CPU: 1, Available: 1}] Pending=[{CPU: 2}]
|
||||
-> Autoscaler resizes the worker to 4 CPUs:
|
||||
WorkerGroup1=[{CPU: 1 -> 4, Available: 1 -> 4}]
|
||||
|
||||
T2: WorkerGroup1=[{CPU: 4, Available: 2}] Pending=[]
|
||||
```
|
||||
|
||||
### Case 2: Resize an existing worker and scale out a new worker
|
||||
|
||||
A combination of resize and scale-out is used when demand exceeds what one resized Pod can absorb.
|
||||
|
||||
```text
|
||||
T1: WorkerGroup1=[{CPU: 1, Available: 1}] Pending=[{CPU: 2}, {CPU: 4}]
|
||||
-> Resize the existing worker to 4 CPUs and launch a new worker:
|
||||
WorkerGroup1=[{CPU: 1 -> 4, Available: 1 -> 4}, {CPU: 1, Available: 1}]
|
||||
|
||||
T2: WorkerGroup1=[{CPU: 4, Available: 0}, {CPU: 1, Available: 1}] Pending=[{CPU: 2}]
|
||||
-> Resize the new worker to 4 CPUs:
|
||||
WorkerGroup1=[{CPU: 4, Available: 0}, {CPU: 1 -> 4, Available: 1 -> 4}]
|
||||
|
||||
T3: WorkerGroup1=[{CPU: 4, Available: 0}, {CPU: 4, Available: 2}] Pending=[]
|
||||
```
|
||||
|
||||
### Case 3: Scale out with IPPR capacity in mind
|
||||
|
||||
When no worker exists yet, the Autoscaler still launches a new Pod. On a subsequent reconciliation, that Pod can be resized in place to absorb pending demand (as in Case 1).
|
||||
|
||||
```text
|
||||
T1: WorkerGroup1=[] Pending=[{CPU: 2}]
|
||||
-> Launch a new worker (IPPR capacity is considered for sizing decisions):
|
||||
WorkerGroup1=[{CPU: 1, Available: 1}]
|
||||
|
||||
T2: WorkerGroup1=[{CPU: 1, Available: 1}] Pending=[{CPU: 2}]
|
||||
-> Same as Case 1.
|
||||
```
|
||||
|
||||
### Case 4: Resize timeout and rollback
|
||||
|
||||
If a Kubernetes resize request doesn't complete within `resize-timeout` seconds, the Autoscaler rolls it back and falls back to horizontal scale-out.
|
||||
|
||||
```text
|
||||
T1: WorkerGroup1=[{CPU: 1, Available: 1}] Pending=[{CPU: 2}]
|
||||
-> Resize attempt:
|
||||
WorkerGroup1=[{CPU: 1 -> 4, Available: 1 -> 4}]
|
||||
|
||||
T2: (resize times out)
|
||||
-> Roll back the resize and scale out:
|
||||
WorkerGroup1=[{CPU: 4 -> 1, Available: 4 -> 1}, {CPU: 1, Available: 1}]
|
||||
|
||||
T3: WorkerGroup1=[{CPU: 1, Available: 1}, {CPU: 1, Available: 1}] Pending=[{CPU: 2}]
|
||||
-> Try resizing the second worker:
|
||||
WorkerGroup1=[{CPU: 1, Available: 1}, {CPU: 1 -> 4, Available: 1 -> 4}]
|
||||
|
||||
T4: WorkerGroup1=[{CPU: 1, Available: 1}, {CPU: 4, Available: 2}] Pending=[]
|
||||
```
|
||||
|
||||
## Observability
|
||||
|
||||
In-flight and recent resize state for each worker Pod is recorded in the `ray.io/ippr-status` annotation on the Pod. You can inspect it with `kubectl`:
|
||||
|
||||
```bash
|
||||
kubectl get pod <worker-pod> -o jsonpath='{.metadata.annotations.ray\.io/ippr-status}'
|
||||
```
|
||||
|
||||
For example, after a successful resize, `resizing-at: null` means no resize is in flight and `last-failed-at: null` means the most recent attempt succeeded:
|
||||
|
||||
```json
|
||||
{
|
||||
"resizing-at": null,
|
||||
"last-failed-at": null,
|
||||
"last-failed-reason": null
|
||||
}
|
||||
```
|
||||
|
||||
You can watch the Kubernetes-side progress of an in-flight resize via the Pod's [resize status conditions](https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/#pod-resize-status) (`PodResizePending` and `PodResizeInProgress`) and the resource events the Kubelet emits during the resize. The simplest way to view both at once is `kubectl describe pod <worker-pod>`.
|
||||
|
||||
After a successful resize, the `Conditions:` table contains only the standard Pod conditions (the transient `PodResizePending` / `PodResizeInProgress` fields are gone), the Pod's `Limits` / `Requests` reflect the new size, and the Kubelet emits a `ResizeCompleted` event:
|
||||
|
||||
```text
|
||||
Conditions:
|
||||
Type Status
|
||||
PodReadyToStartContainers True
|
||||
Initialized True
|
||||
Ready True
|
||||
ContainersReady True
|
||||
PodScheduled True
|
||||
Containers:
|
||||
ray-worker:
|
||||
...
|
||||
Limits:
|
||||
cpu: 4
|
||||
memory: 4Gi
|
||||
Requests:
|
||||
cpu: 4
|
||||
memory: 4Gi
|
||||
...
|
||||
Events:
|
||||
Type Reason Age From Message
|
||||
---- ------ ---- ---- -------
|
||||
Normal ResizeCompleted 2s kubelet Pod resize completed: {"containers":[{"name":"ray-worker","resources":{"limits":{"cpu":"4","memory":"4Gi"},"requests":{"cpu":"4","memory":"4Gi"}}}]}
|
||||
```
|
||||
|
||||
If a resize can't be granted, the Pod gains a `PodResizePending` condition:
|
||||
|
||||
```text
|
||||
Conditions:
|
||||
Type Status
|
||||
PodResizePending True
|
||||
PodReadyToStartContainers True
|
||||
Initialized True
|
||||
Ready True
|
||||
ContainersReady True
|
||||
PodScheduled True
|
||||
```
|
||||
|
||||
See [Pod resize status](https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/#pod-resize-status) and [Troubleshooting: Infeasible resize request](https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/#troubleshooting-infeasible-resize-request) in the Kubernetes documentation for the meaning of the condition's `reason` and `message` and how to inspect them.
|
||||
|
||||
## Limitations
|
||||
|
||||
* **Resize-up only.** The current implementation can only scale workers up to the configured maximum. Gradual resizing and downsizing will be added in future releases.
|
||||
* **CPU and memory only.** Other resources, including GPUs, can't be resized in place.
|
||||
* **Per-worker-group configuration.** IPPR is configured per worker `groupName`. Worker groups without an entry in `ray.io/ippr` aren't resized in place.
|
||||
* **Autoscaler V2 only.** IPPR is integrated only with Autoscaler V2.
|
||||
* **Upstream Kubernetes limitations apply.** Any [limitations of the underlying Kubernetes In-Place Pod Resize feature](https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/#limitations) (such as restrictions on QoS-class changes, swap, or specific runtimes) also apply to IPPR-managed Pods.
|
||||
@@ -0,0 +1,69 @@
|
||||
(kuberay-gke-gpu-cluster-setup)=
|
||||
|
||||
# Start Google Cloud GKE Cluster with GPUs for KubeRay
|
||||
|
||||
See <https://cloud.google.com/kubernetes-engine/docs/how-to/gpus> for full details, or continue reading for a quick start.
|
||||
|
||||
## Step 1: Create a Kubernetes cluster on GKE
|
||||
|
||||
Run this command and all following commands on your local machine or on the [Google Cloud Shell](https://cloud.google.com/shell). If running from your local machine, you need to install the [Google Cloud SDK](https://cloud.google.com/sdk/docs/install). The following command creates a Kubernetes cluster named `kuberay-gpu-cluster` with 1 CPU node in the `us-west1-b` zone. This example uses the `e2-standard-4` machine type, which has 4 vCPUs and 16 GB RAM.
|
||||
|
||||
```sh
|
||||
gcloud container clusters create kuberay-gpu-cluster \
|
||||
--num-nodes=1 --min-nodes 0 --max-nodes 1 --enable-autoscaling \
|
||||
--zone=us-west1-b --machine-type e2-standard-4
|
||||
```
|
||||
|
||||
```{admonition} Note
|
||||
You can also create a cluster from the [Google Cloud Console](https://console.cloud.google.com/kubernetes/list).
|
||||
```
|
||||
|
||||
## Step 2: Create a GPU node pool
|
||||
|
||||
Run the following command to create a GPU node pool for Ray GPU workers. You can also create it from the Google Cloud Console: <https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#console>
|
||||
|
||||
```sh
|
||||
gcloud container node-pools create gpu-node-pool \
|
||||
--accelerator type=nvidia-l4-vws,count=1 \
|
||||
--zone us-west1-b \
|
||||
--cluster kuberay-gpu-cluster \
|
||||
--num-nodes 1 \
|
||||
--min-nodes 0 \
|
||||
--max-nodes 1 \
|
||||
--enable-autoscaling \
|
||||
--machine-type g2-standard-4
|
||||
```
|
||||
|
||||
The `--accelerator` flag specifies the type and number of GPUs for each node in the node pool. This example uses the [NVIDIA L4](https://cloud.google.com/compute/docs/gpus#l4-gpus) GPU. The machine type `g2-standard-4` has 1 GPU, 24 GB GPU Memory, 4 vCPUs and 16 GB RAM.
|
||||
|
||||
```{admonition} Note
|
||||
GKE automatically configures taints and tolerations so that only GPU pods are scheduled on GPU nodes.
|
||||
For more details, see [GKE documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#create)
|
||||
```
|
||||
|
||||
## Step 3: Configure `kubectl` to connect to the cluster
|
||||
|
||||
Run the following command to download Google Cloud credentials and configure the Kubernetes CLI to use them.
|
||||
|
||||
```sh
|
||||
gcloud container clusters get-credentials kuberay-gpu-cluster --zone us-west1-b
|
||||
```
|
||||
|
||||
For more details, see [GKE documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl).
|
||||
|
||||
## Step 4: Install GPU drivers (optional)
|
||||
|
||||
If you encounter any issues with the GPU drivers installed by GKE, you can manually install the GPU drivers by following the instructions below.
|
||||
|
||||
```sh
|
||||
# Install NVIDIA GPU device driver
|
||||
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded-latest.yaml
|
||||
|
||||
# Verify that your nodes have allocatable GPUs
|
||||
kubectl get nodes "-o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"
|
||||
|
||||
# Verify that your nodes have allocatable GPUs
|
||||
# NAME GPU
|
||||
# ...... <none>
|
||||
# ...... 1
|
||||
```
|
||||
@@ -0,0 +1,104 @@
|
||||
(kuberay-gke-tpu-cluster-setup)=
|
||||
|
||||
# Start Google Cloud GKE Cluster with TPUs for KubeRay
|
||||
|
||||
See the [GKE documentation](<https://cloud.google.com/kubernetes-engine/docs/how-to/tpus>) for full details, or continue reading for a quick start.
|
||||
|
||||
## Step 1: Create a Kubernetes cluster on GKE
|
||||
|
||||
First, set the following environment variables to be used for GKE cluster creation:
|
||||
```sh
|
||||
export CLUSTER_NAME=CLUSTER_NAME
|
||||
export ZONE=ZONE
|
||||
export CLUSTER_VERSION=CLUSTER_VERSION
|
||||
```
|
||||
Replace the following:
|
||||
- CLUSTER_NAME: The name of the GKE cluster to be created.
|
||||
- ZONE: The zone with available TPU quota, for a list of TPU availability by zones, see the [GKE documentation](https://cloud.google.com/tpu/docs/regions-zones).
|
||||
- CLUSTER_VERSION: The GKE version to use. TPU v6e is supported in GKE versions 1.31.2-gke.1115000 or later. See the [GKE documentation](https://cloud.google.com/tpu/docs/tpus-in-gke#tpu-machine-types) for TPU generations and their minimum supported version.
|
||||
|
||||
Run the following commands on your local machine or on the [Google Cloud Shell](https://cloud.google.com/shell). If running from your local machine, install the [Google Cloud SDK](https://cloud.google.com/sdk/docs/install).
|
||||
|
||||
Create a Standard GKE cluster and enable the Ray Operator:
|
||||
|
||||
```sh
|
||||
gcloud container clusters create $CLUSTER_NAME \
|
||||
--addons=RayOperator \
|
||||
--machine-type=n1-standard-16 \
|
||||
--cluster-version=$CLUSTER_VERSION \
|
||||
--location=$ZONE
|
||||
```
|
||||
|
||||
Run the following command to add a TPU node pool to the cluster. You can also create it from the [Google Cloud Console](https://cloud.google.com/kubernetes-engine/docs/how-to/tpus#console):
|
||||
|
||||
Create a node pool with a single-host v4 TPU topology as follows:
|
||||
```sh
|
||||
gcloud container node-pools create v4-4 \
|
||||
--zone $ZONE \
|
||||
--cluster $CLUSTER_NAME \
|
||||
--num-nodes 1 \
|
||||
--min-nodes 0 \
|
||||
--max-nodes 10 \
|
||||
--enable-autoscaling \
|
||||
--machine-type ct4p-hightpu-4t \
|
||||
--tpu-topology 2x2x1
|
||||
```
|
||||
- For v4 TPUs, ZONE must be `us-central2-b`.
|
||||
|
||||
Alternatively, create a multi-host node pool as follows:
|
||||
|
||||
```sh
|
||||
gcloud container node-pools create v4-8 \
|
||||
--zone $ZONE \
|
||||
--cluster $CLUSTER_NAME \
|
||||
--num-nodes 2 \
|
||||
--min-nodes 0 \
|
||||
--max-nodes 10 \
|
||||
--enable-autoscaling \
|
||||
--machine-type ct4p-hightpu-4t \
|
||||
--tpu-topology 2x2x2
|
||||
```
|
||||
- For v4 TPUs, ZONE must be `us-central2-b`.
|
||||
- For other TPU types, see [TPU locations on Google Cloud documentation](https://docs.cloud.google.com/compute/docs/regions-zones/tpu-regions-zones).
|
||||
|
||||
The `--tpu-topology` flag specifies the physical topology of the TPU Pod slice. This example uses a v4 TPU slice with either a 2x2x1 or 2x2x2 topology. v4 TPUs have 4 chips per VM host, so a 2x2x2 v4 slice has 8 chips total and 2 TPU hosts, each scheduled on their own node. GKE treats multi-host TPU slices as atomic units, and scales them using node pools rather than singular nodes. Therefore, the number of TPU hosts should always equal the number of nodes in the TPU node pool. For more information about selecting a TPU topology and accelerator, see the [GKE documentation](https://cloud.google.com/kubernetes-engine/docs/concepts/tpus).
|
||||
|
||||
GKE uses Kubernetes node selectors to ensure TPU workloads run on the desired machine type and topology. For more details, see the [GKE documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/tpus#workload_preparation).
|
||||
|
||||
## Step 2: Connect to the GKE cluster
|
||||
|
||||
Run the following command to download Google Cloud credentials and configure the Kubernetes CLI to use them.
|
||||
|
||||
```sh
|
||||
gcloud container clusters get-credentials $CLUSTER_NAME --zone $ZONE
|
||||
```
|
||||
|
||||
The remote GKE cluster is now reachable through `kubectl`. For more details, see the [GKE documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl).
|
||||
|
||||
|
||||
### [Optional] Manually install KubeRay and the TPU webhook in a GKE cluster without the Ray Operator Addon:
|
||||
|
||||
In a cluster without the Ray Operator Addon enabled, KubeRay can be manually installed using [helm](https://ray-project.github.io/kuberay/deploy/helm/) with the following commands:
|
||||
|
||||
```sh
|
||||
helm repo add kuberay https://ray-project.github.io/kuberay-helm/
|
||||
|
||||
# Install both CRDs and KubeRay operator v1.6.0.
|
||||
helm install kuberay-operator kuberay/kuberay-operator --version 1.6.0
|
||||
```
|
||||
|
||||
GKE provides a [validating and mutating webhook](https://github.com/ai-on-gke/kuberay-tpu-webhook) to handle TPU Pod scheduling and bootstrap certain environment variables used for [JAX](https://github.com/google/jax) initialization. The Ray TPU webhook requires a KubeRay operator version of at least v1.1.0. GKE automatically installs the Ray TPU webhook through the [Ray Operator Addon](https://cloud.google.com/kubernetes-engine/docs/add-on/ray-on-gke/how-to/enable-ray-on-gke) with GKE versions 1.30.0-gke.1747000 or later.
|
||||
|
||||
When manually installing the webhook, [cert-manager](https://github.com/cert-manager/cert-manager) is required to handle TLS certificate injection. You can install cert-manager in both GKE Standard and Autopilot clusters using the following helm commands:
|
||||
|
||||
Install cert-manager:
|
||||
```
|
||||
helm repo add jetstack https://charts.jetstack.io
|
||||
helm repo update
|
||||
helm install --create-namespace --namespace cert-manager --set installCRDs=true --set global.leaderElection.namespace=cert-manager cert-manager jetstack/cert-manager
|
||||
```
|
||||
|
||||
Next, deploy the Ray TPU initialization webhook:
|
||||
1. `git clone https://github.com/GoogleCloudPlatform/ai-on-gke`
|
||||
2. `cd ray-on-gke/tpu/kuberay-tpu-webhook`
|
||||
3. `make deploy deploy-cert`
|
||||
@@ -0,0 +1,133 @@
|
||||
(kuberay-gke-bucket)=
|
||||
# Configuring KubeRay to use Google Cloud Storage Buckets in GKE
|
||||
|
||||
If you are already familiar with Workload Identity in GKE, you can skip this document. The gist is that you need to specify a service account in each of the Ray pods after linking your Kubernetes service account to your Google Cloud service account. Otherwise, read on.
|
||||
|
||||
This example is an abridged version of the documentation at <https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity>. The full documentation is worth reading if you are interested in the details.
|
||||
|
||||
## Create a Kubernetes cluster on GKE
|
||||
|
||||
This example creates a minimal KubeRay cluster using GKE.
|
||||
|
||||
Run this and all following commands on your local machine or on the [Google Cloud Shell](https://cloud.google.com/shell). If running from your local machine, install the [Google Cloud SDK](https://cloud.google.com/sdk/docs/install).
|
||||
|
||||
```bash
|
||||
PROJECT_ID=my-project-id # Replace my-project-id with your GCP project ID
|
||||
CLUSTER_NAME=cloud-bucket-cluster
|
||||
ZONE=us-west1-b
|
||||
|
||||
gcloud container clusters create $CLUSTER_NAME \
|
||||
--addons=RayOperator \
|
||||
--num-nodes=1 --min-nodes 0 --max-nodes 1 --enable-autoscaling \
|
||||
--zone=$ZONE --machine-type e2-standard-8 \
|
||||
--workload-pool=${PROJECT_ID}.svc.id.goog
|
||||
```
|
||||
|
||||
|
||||
This command creates a Kubernetes cluster named `cloud-bucket-cluster` with one node in the `us-west1-b` zone. This example uses the `e2-standard-8` machine type, which has 8 vCPUs and 32 GB RAM.
|
||||
|
||||
For more information on how to find your project ID, see <https://support.google.com/googleapi/answer/7014113?hl=en> or <https://cloud.google.com/resource-manager/docs/creating-managing-projects>.
|
||||
|
||||
Now get credentials for the cluster to use with `kubectl`:
|
||||
|
||||
```bash
|
||||
gcloud container clusters get-credentials $CLUSTER_NAME --zone $ZONE --project $PROJECT_ID
|
||||
```
|
||||
|
||||
## Create a Kubernetes Service Account
|
||||
|
||||
```bash
|
||||
NAMESPACE=default
|
||||
KSA=my-ksa
|
||||
kubectl create serviceaccount $KSA -n $NAMESPACE
|
||||
```
|
||||
|
||||
## Configure the GCS Bucket
|
||||
|
||||
Create a GCS bucket that Ray uses as the remote filesystem.
|
||||
```bash
|
||||
BUCKET=my-bucket
|
||||
gcloud storage buckets create gs://$BUCKET --uniform-bucket-level-access
|
||||
```
|
||||
|
||||
Bind the `roles/storage.objectUser` role to the Kubernetes service account and bucket IAM policy. See [Identifying projects](https://cloud.google.com/resource-manager/docs/creating-managing-projects#identifying_projects) to find your project ID and project number:
|
||||
```bash
|
||||
PROJECT_ID=<your project ID>
|
||||
PROJECT_NUMBER=<your project number>
|
||||
gcloud storage buckets add-iam-policy-binding gs://${BUCKET} --member "principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${PROJECT_ID}.svc.id.goog/subject/ns/${NAMESPACE}/sa/${KSA}" --role "roles/storage.objectUser"
|
||||
```
|
||||
|
||||
See [Authenticate to Google Cloud APIs from GKE workloads](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) for more details.
|
||||
|
||||
## Create a minimal RayCluster YAML manifest
|
||||
|
||||
You can download the RayCluster YAML manifest for this tutorial with `curl` as follows:
|
||||
|
||||
```bash
|
||||
curl -LO https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-cluster.gke-bucket.yaml
|
||||
```
|
||||
|
||||
The key parts are the following lines:
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
serviceAccountName: my-ksa
|
||||
nodeSelector:
|
||||
iam.gke.io/gke-metadata-server-enabled: "true"
|
||||
```
|
||||
|
||||
Include these lines in every pod spec of your Ray cluster. This example uses a single-node cluster (1 head node and 0 worker nodes) for simplicity.
|
||||
|
||||
## Create the RayCluster
|
||||
|
||||
```bash
|
||||
kubectl apply -f ray-cluster.gke-bucket.yaml
|
||||
```
|
||||
|
||||
## Test GCS bucket access from the RayCluster
|
||||
|
||||
Use `kubectl get pod` to get the name of the Ray head pod. Then run the following command to get a shell in the Ray head pod:
|
||||
|
||||
```bash
|
||||
kubectl exec -it raycluster-mini-head-xxxx -- /bin/bash
|
||||
```
|
||||
|
||||
In the shell, run `pip install google-cloud-storage` to install the Google Cloud Storage Python client library.
|
||||
|
||||
(For production use cases, you will need to make sure `google-cloud-storage` is installed on every node of your cluster, or use `ray.init(runtime_env={"pip": ["google-cloud-storage"]})` to have the package installed as needed at runtime -- see <https://docs.ray.io/en/latest/ray-core/handling-dependencies.html#runtime-environments> for more details.)
|
||||
|
||||
Then run the following Python code to test access to the bucket:
|
||||
|
||||
```python
|
||||
import ray
|
||||
import os
|
||||
from google.cloud import storage
|
||||
|
||||
GCP_GCS_BUCKET = "my-bucket"
|
||||
GCP_GCS_FILE = "test_file.txt"
|
||||
|
||||
ray.init(address="auto")
|
||||
|
||||
@ray.remote
|
||||
def check_gcs_read_write():
|
||||
client = storage.Client()
|
||||
bucket = client.bucket(GCP_GCS_BUCKET)
|
||||
blob = bucket.blob(GCP_GCS_FILE)
|
||||
|
||||
# Write to the bucket
|
||||
blob.upload_from_string("Hello, Ray on GKE!")
|
||||
|
||||
# Read from the bucket
|
||||
content = blob.download_as_text()
|
||||
|
||||
return content
|
||||
|
||||
result = ray.get(check_gcs_read_write.remote())
|
||||
print(result)
|
||||
```
|
||||
|
||||
You should see the following output:
|
||||
|
||||
```text
|
||||
Hello, Ray on GKE!
|
||||
```
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user