chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,42 @@
|
||||
(kuberay-operator-deploy)=
|
||||
|
||||
# KubeRay Operator Installation
|
||||
|
||||
## Step 1: Create a Kubernetes cluster
|
||||
|
||||
This step creates a local Kubernetes cluster using [Kind](https://kind.sigs.k8s.io/). If you already have a Kubernetes cluster, you can skip this step.
|
||||
|
||||
```sh
|
||||
kind create cluster --image=kindest/node:v1.26.0
|
||||
```
|
||||
|
||||
## Step 2: Install KubeRay operator
|
||||
|
||||
### Method 1: Helm (Recommended)
|
||||
|
||||
```sh
|
||||
helm repo add kuberay https://ray-project.github.io/kuberay-helm/
|
||||
helm repo update
|
||||
# Install both CRDs and KubeRay operator v1.6.0.
|
||||
helm install kuberay-operator kuberay/kuberay-operator --version 1.6.0
|
||||
```
|
||||
|
||||
### Method 2: Kustomize
|
||||
|
||||
```sh
|
||||
# Install CRD and KubeRay operator.
|
||||
kubectl create -k "github.com/ray-project/kuberay/ray-operator/config/default?ref=v1.6.0"
|
||||
```
|
||||
|
||||
## Step 3: Validate Installation
|
||||
|
||||
Confirm that the operator is running in the namespace `default`.
|
||||
|
||||
```sh
|
||||
kubectl get pods
|
||||
```
|
||||
|
||||
```text
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
kuberay-operator-6bc45dd644-gwtqv 1/1 Running 0 24s
|
||||
```
|
||||
@@ -0,0 +1,167 @@
|
||||
(kuberay-raycluster-quickstart)=
|
||||
|
||||
# RayCluster Quickstart
|
||||
|
||||
This guide shows you how to manage and interact with Ray clusters on Kubernetes.
|
||||
|
||||
## Preparation
|
||||
|
||||
* Install [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) (>= 1.23), [Helm](https://helm.sh/docs/intro/install/) (>= v3.4) if needed, [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation), and [Docker](https://docs.docker.com/engine/install/).
|
||||
* Make sure your Kubernetes cluster has at least 4 CPU and 4 GB RAM.
|
||||
|
||||
## Step 1: Create a Kubernetes cluster
|
||||
|
||||
This step creates a local Kubernetes cluster using [Kind](https://kind.sigs.k8s.io/). If you already have a Kubernetes cluster, you can skip this step.
|
||||
|
||||
```sh
|
||||
kind create cluster --image=kindest/node:v1.26.0
|
||||
```
|
||||
|
||||
## Step 2: Deploy a KubeRay operator
|
||||
|
||||
Follow [this document](kuberay-operator-deploy) to install the latest stable KubeRay operator from the Helm repository.
|
||||
|
||||
(raycluster-deploy)=
|
||||
## Step 3: Deploy a RayCluster custom resource
|
||||
|
||||
Once the KubeRay operator is running, you're ready to deploy a RayCluster. Create a RayCluster Custom Resource (CR) in the `default` namespace.
|
||||
|
||||
```sh
|
||||
# Deploy a sample RayCluster CR from the KubeRay Helm chart repo:
|
||||
helm install raycluster kuberay/ray-cluster --version 1.6.0
|
||||
```
|
||||
|
||||
|
||||
Once the RayCluster CR has been created, you can view it by running:
|
||||
|
||||
```sh
|
||||
# Once the RayCluster CR has been created, you can view it by running:
|
||||
kubectl get rayclusters
|
||||
```
|
||||
|
||||
```sh
|
||||
NAME DESIRED WORKERS AVAILABLE WORKERS CPUS MEMORY GPUS STATUS AGE
|
||||
raycluster-kuberay 1 1 2 3G 0 ready 55s
|
||||
```
|
||||
|
||||
The KubeRay operator detects the RayCluster object and starts your Ray cluster by creating head and worker pods. To view Ray cluster's pods, run the following command:
|
||||
|
||||
```sh
|
||||
# View the pods in the RayCluster named "raycluster-kuberay"
|
||||
kubectl get pods --selector=ray.io/cluster=raycluster-kuberay
|
||||
```
|
||||
|
||||
```sh
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
raycluster-kuberay-head 1/1 Running 0 XXs
|
||||
raycluster-kuberay-worker-workergroup-xvfkr 1/1 Running 0 XXs
|
||||
```
|
||||
|
||||
Wait for the pods to reach `Running` state. This may take a few minutes, downloading the Ray images takes most of this time. If your pods stick in the `Pending` state, you can check for errors using `kubectl describe pod raycluster-kuberay-xxxx-xxxxx` and ensure your Docker resource limits meet the requirements.
|
||||
|
||||
## Step 4: Run an application on a RayCluster
|
||||
|
||||
Now, interact with the RayCluster deployed.
|
||||
|
||||
### Method 1: Execute a Ray job in the head Pod
|
||||
|
||||
The most straightforward way to experiment with your RayCluster is to exec directly into the head pod. First, identify your RayCluster's head pod:
|
||||
|
||||
```sh
|
||||
export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)
|
||||
echo $HEAD_POD
|
||||
```
|
||||
|
||||
```sh
|
||||
raycluster-kuberay-head
|
||||
```
|
||||
|
||||
```sh
|
||||
# Print the cluster resources.
|
||||
kubectl exec -it $HEAD_POD -- python -c "import ray; ray.init(); print(ray.cluster_resources())"
|
||||
```
|
||||
|
||||
```sh
|
||||
2023-04-07 10:57:46,472 INFO worker.py:1243 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS
|
||||
2023-04-07 10:57:46,472 INFO worker.py:1364 -- Connecting to existing Ray cluster at address: 10.244.0.6:6379...
|
||||
2023-04-07 10:57:46,482 INFO worker.py:1550 -- Connected to Ray cluster. View the dashboard at http://10.244.0.6:8265
|
||||
{'CPU': 2.0,
|
||||
'memory': 3000000000.0,
|
||||
'node:10.244.0.6': 1.0,
|
||||
'node:10.244.0.7': 1.0,
|
||||
'node:__internal_head__': 1.0,
|
||||
'object_store_memory': 749467238.0}
|
||||
```
|
||||
|
||||
### Method 2: Submit a Ray job to the RayCluster using [ray job submission SDK](jobs-quickstart)
|
||||
|
||||
Unlike Method 1, this method doesn't require you to execute commands in the Ray head pod. Instead, you can use the [Ray job submission SDK](jobs-quickstart) to submit Ray jobs to the RayCluster through the Ray Dashboard port where Ray listens for Job requests. The KubeRay operator configures a [Kubernetes service](https://kubernetes.io/docs/concepts/services-networking/service/) targeting the Ray head Pod.
|
||||
|
||||
```sh
|
||||
kubectl get service raycluster-kuberay-head-svc
|
||||
```
|
||||
|
||||
```sh
|
||||
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
raycluster-kuberay-head-svc ClusterIP None <none> 10001/TCP,8265/TCP,6379/TCP,8080/TCP,8000/TCP 57s
|
||||
```
|
||||
|
||||
Now that the service name is available, use port-forwarding to access the Ray Dashboard port which is 8265 by default.
|
||||
|
||||
```sh
|
||||
# Execute this in a separate shell.
|
||||
kubectl port-forward service/raycluster-kuberay-head-svc 8265:8265 > /dev/null &
|
||||
```
|
||||
|
||||
Now that the Dashboard port is accessible, submit jobs to the RayCluster:
|
||||
|
||||
```sh
|
||||
# The following job's logs will show the Ray cluster's total resource capacity, including 2 CPUs.
|
||||
ray job submit --address http://localhost:8265 -- python -c "import ray; ray.init(); print(ray.cluster_resources())"
|
||||
```
|
||||
|
||||
```sh
|
||||
Job submission server address: http://localhost:8265
|
||||
|
||||
-------------------------------------------------------
|
||||
Job 'raysubmit_8vJ7dKqYrWKbd17i' submitted successfully
|
||||
-------------------------------------------------------
|
||||
|
||||
Next steps
|
||||
Query the logs of the job:
|
||||
ray job logs raysubmit_8vJ7dKqYrWKbd17i
|
||||
Query the status of the job:
|
||||
ray job status raysubmit_8vJ7dKqYrWKbd17i
|
||||
Request the job to be stopped:
|
||||
ray job stop raysubmit_8vJ7dKqYrWKbd17i
|
||||
|
||||
Tailing logs until the job exits (disable with --no-wait):
|
||||
2025-03-18 01:27:51,014 INFO job_manager.py:530 -- Runtime env is setting up.
|
||||
2025-03-18 01:27:51,744 INFO worker.py:1514 -- Using address 10.244.0.6:6379 set in the environment variable RAY_ADDRESS
|
||||
2025-03-18 01:27:51,744 INFO worker.py:1654 -- Connecting to existing Ray cluster at address: 10.244.0.6:6379...
|
||||
2025-03-18 01:27:51,750 INFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at 10.244.0.6:8265
|
||||
{'CPU': 2.0,
|
||||
'memory': 3000000000.0,
|
||||
'node:10.244.0.6': 1.0,
|
||||
'node:10.244.0.7': 1.0,
|
||||
'node:__internal_head__': 1.0,
|
||||
'object_store_memory': 749467238.0}
|
||||
|
||||
------------------------------------------
|
||||
Job 'raysubmit_8vJ7dKqYrWKbd17i' succeeded
|
||||
------------------------------------------
|
||||
```
|
||||
|
||||
## Step 5: Access the Ray Dashboard
|
||||
|
||||
Visit `${YOUR_IP}:8265` in your browser for the Dashboard. For example, `127.0.0.1:8265`. See the job you submitted in Step 4 in the **Recent jobs** pane as shown below.
|
||||
|
||||
|
||||
|
||||
## Step 6: Cleanup
|
||||
|
||||
```sh
|
||||
# Kill the `kubectl port-forward` background job in the earlier step
|
||||
killall kubectl
|
||||
kind delete cluster
|
||||
```
|
||||
@@ -0,0 +1,140 @@
|
||||
(kuberay-raycronjob-quickstart)=
|
||||
|
||||
# RayCronJob Quickstart
|
||||
|
||||
## Prerequisites
|
||||
|
||||
* This feature requires KubeRay version 1.6.0 or newer, and it's in alpha testing.
|
||||
* A running Kubernetes cluster.
|
||||
* The KubeRay operator installed and running in your cluster.
|
||||
|
||||
## What is a RayCronJob?
|
||||
|
||||
A `RayCronJob` is a Custom Resource (CR) that allows you to run `RayJob` workloads on a recurring, time-based schedule. It is heavily inspired by the native Kubernetes `CronJob` and brings automated, scheduled execution to your distributed Ray applications.
|
||||
|
||||
|
||||
This is particularly useful for recurring tasks such as scheduled model retraining, nightly batch inferences, or regular data processing pipelines.
|
||||
|
||||
|
||||
## RayCronJob Configuration
|
||||
|
||||
The `RayCronJob` CRD acts as an automated scheduler specifically designed to create and manage **RayJob** custom resources on a recurring basis. It does not execute workloads directly. Instead, it acts as a controller that creates a new `RayJob` each time the schedule triggers.
|
||||
|
||||
* `schedule` - The cron schedule string defining when a new Ray job should be created and run (e.g., `* * * * *` for every minute).
|
||||
* `jobTemplate` - Wraps a standard **RayJob** spec that the controller will use for each scheduled run. It supports the same fields as a RayJob spec. See the standard [RayJob Configuration](kuberay-rayjob-quickstart) documentation for the complete list of supported fields within the `jobTemplate`.
|
||||
* `suspend` (Optional): If `suspend` is true, the controller suspends the scheduling of future jobs. This does not apply to or interrupt any `RayJob`s that have already been created and are currently running.
|
||||
* `timeZone` (Optional): The time zone for the `schedule`, in [IANA Time Zone Database](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) format (e.g., `America/Los_Angeles`). If omitted, the schedule uses the local time zone of the KubeRay operator. Do not set `TZ` or `CRON_TZ` in the `schedule` string, use this field instead.
|
||||
|
||||
## How to Configure a RayCronJob
|
||||
|
||||
Configuring a `RayCronJob` requires wrapping a standard `RayJob` specification inside a `jobTemplate`, alongside your scheduling parameters.
|
||||
|
||||
The high-level structure looks like this:
|
||||
|
||||
```yaml
|
||||
apiVersion: ray.io/v1
|
||||
kind: RayCronJob
|
||||
metadata:
|
||||
name: example-raycronjob
|
||||
spec:
|
||||
schedule: "*/5 * * * *" # Run every 5 minutes
|
||||
timeZone: "America/Los_Angeles" # Optional, defaults to the operator's local time zone
|
||||
jobTemplate:
|
||||
# Everything below here is a standard RayJob spec
|
||||
entrypoint: python /home/ray/samples/sample_code.py
|
||||
# ... (RayCluster spec, runtimeEnv, etc.)
|
||||
```
|
||||
|
||||
## How to Run a Simple RayCronJob
|
||||
|
||||
Let's deploy a simple `RayCronJob` that executes a short Python script every minute.
|
||||
|
||||
### Step 1: Create a Kubernetes cluster with Kind
|
||||
|
||||
```sh
|
||||
kind create cluster --image=kindest/node:v1.26.0
|
||||
```
|
||||
|
||||
### Step 2: Install the KubeRay operator
|
||||
Install the KubeRay operator, following [these instructions](https://docs.ray.io/en/latest/cluster/kubernetes/getting-started/kuberay-operator-installation.html). The minimum version for this guide is v1.6.0. To use this feature, the `RayCronJob` feature gate must be enabled. To enable the feature gate when installing the kuberay operator, run the following command:
|
||||
|
||||
```sh
|
||||
helm repo add kuberay https://ray-project.github.io/kuberay-helm/
|
||||
helm repo update
|
||||
|
||||
# Install KubeRay operator v1.6.0 with the RayCronJob feature gate enabled
|
||||
helm install kuberay-operator kuberay/kuberay-operator \
|
||||
--version 1.6.0 \
|
||||
--set "featureGates[0].name=RayCronJob" \
|
||||
--set "featureGates[0].enabled=true"
|
||||
```
|
||||
|
||||
### Step 3: Install a RayCronJob
|
||||
|
||||
```sh
|
||||
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-cronjob.sample.yaml
|
||||
```
|
||||
|
||||
### Step 4: Monitor the RayCronJob
|
||||
|
||||
Check the status of your `RayCronJob`. The `SCHEDULE` field should be visible, while `LAST SCHEDULE` may be empty until the first run.
|
||||
|
||||
```sh
|
||||
kubectl get raycronjob raycronjob-sample
|
||||
|
||||
#You should see output listing the RayCronJob created
|
||||
# [Example output]
|
||||
|
||||
# NAME SCHEDULE LAST SCHEDULE AGE SUSPEND
|
||||
# raycronjob-sample * * * * * 10s
|
||||
```
|
||||
|
||||
Because our schedule is `* * * * *`, a new `RayJob` will be generated at the start of the next minute. You can watch the `RayJob` instances being created:
|
||||
|
||||
```shell
|
||||
kubectl get rayjob -w
|
||||
# [Example output]
|
||||
# NAME JOB STATUS DEPLOYMENT STATUS RAY CLUSTER NAME START TIME END TIME AGE
|
||||
# raycronjob-sample-l76h8 Initializing raycronjob-sample-l76h8-hjtrs 2026-04-03T05:57:00Z 2s
|
||||
# raycronjob-sample-l76h8 RUNNING Running raycronjob-sample-l76h8-hjtrs 2026-04-03T05:57:00Z 48s
|
||||
# raycronjob-sample-l76h8 SUCCEEDED Complete raycronjob-sample-l76h8-hjtrs 2026-04-03T05:57:00Z 2026-04-03T05:58:02Z 62s
|
||||
# raycronjob-sample-pct47 Initializing raycronjob-sample-pct47-bdspj 2026-04-03T05:58:00Z 0s
|
||||
# (Press Ctrl+C to stop watching once the job completes)
|
||||
```
|
||||
|
||||
### Step 5: Check the output of the RayCronJob
|
||||
|
||||
```shell
|
||||
# From the previous step, note the RayJob name label (e.g., raycronjob-sample-l76h8)
|
||||
# Use it to fetch the submitter pod logs directly:
|
||||
kubectl logs -l=job-name=<rayjob-name>
|
||||
# Example:
|
||||
# kubectl logs -l=job-name=raycronjob-sample-l76h8
|
||||
|
||||
# [Example output]
|
||||
# /home/ray/anaconda3/lib/python3.10/site-packages/ray/_private/worker.py:2062: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0
|
||||
# warnings.warn(
|
||||
# test_counter got 1
|
||||
# test_counter got 2
|
||||
# test_counter got 3
|
||||
# test_counter got 4
|
||||
# test_counter got 5
|
||||
# 2026-04-02 22:57:58,968 SUCC cli.py:65 -- ---------------------------------------------
|
||||
# 2026-04-02 22:57:58,968 SUCC cli.py:66 -- Job 'raycronjob-sample-l76h8-hmjz2' succeeded
|
||||
# 2026-04-02 22:57:58,968 SUCC cli.py:67 -- ---------------------------------------------
|
||||
```
|
||||
|
||||
### Step 6: Clean Up
|
||||
|
||||
To stop the recurring jobs and delete the resource, run:
|
||||
|
||||
```bash
|
||||
# Step 6.1: Delete the RayCronJob
|
||||
kubectl delete -f https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-cronjob.sample.yaml
|
||||
|
||||
# Step 6.2: Delete the KubeRay operator
|
||||
helm uninstall kuberay-operator
|
||||
|
||||
# Step 6.3: Delete the Kubernetes cluster
|
||||
kind delete cluster
|
||||
```
|
||||
@@ -0,0 +1,208 @@
|
||||
(kuberay-rayjob-quickstart)=
|
||||
|
||||
# RayJob Quickstart
|
||||
|
||||
## Prerequisites
|
||||
|
||||
* KubeRay v0.6.0 or higher
|
||||
* KubeRay v0.6.0 or v1.0.0: Ray 1.10 or higher.
|
||||
* KubeRay v1.1.1 or newer is highly recommended: Ray 2.8.0 or higher.
|
||||
|
||||
## What's a RayJob?
|
||||
|
||||
A RayJob manages two aspects:
|
||||
|
||||
* **RayCluster**: A RayCluster custom resource manages all Pods in a Ray cluster, including a head Pod and multiple worker Pods.
|
||||
* **Job**: A Kubernetes Job runs `ray job submit` to submit a Ray job to the RayCluster.
|
||||
|
||||
## What does the RayJob provide?
|
||||
|
||||
With RayJob, KubeRay automatically creates a RayCluster and submits a job when the cluster is ready. You can also configure RayJob to automatically delete the RayCluster once the Ray job finishes.
|
||||
|
||||
To understand the following content better, you should understand the difference between:
|
||||
* RayJob: A Kubernetes custom resource definition provided by KubeRay.
|
||||
* Ray job: A Ray job is a packaged Ray application that can run on a remote Ray cluster. See [this document](jobs-overview) for more details.
|
||||
* Submitter: The submitter is a Kubernetes Job that runs `ray job submit` to submit a Ray job to the RayCluster.
|
||||
|
||||
## RayJob Configuration
|
||||
|
||||
* RayCluster configuration
|
||||
* `rayClusterSpec` - Defines the **RayCluster** custom resource to run the Ray job on.
|
||||
* `clusterSelector` - Use existing **RayCluster** custom resources to run the Ray job instead of creating a new one. See [ray-job.use-existing-raycluster.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-job.use-existing-raycluster.yaml) for example configurations.
|
||||
* Ray job configuration
|
||||
* `entrypoint` - The submitter runs `ray job submit --address ... --submission-id ... -- $entrypoint` to submit a Ray job to the RayCluster.
|
||||
* `runtimeEnvYAML` (Optional): A runtime environment that describes the dependencies the Ray job needs to run, including files, packages, environment variables, and more. Provide the configuration as a multi-line YAML string. Example:
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
runtimeEnvYAML: |
|
||||
pip:
|
||||
- requests==2.26.0
|
||||
- pendulum==2.1.2
|
||||
env_vars:
|
||||
KEY: "VALUE"
|
||||
```
|
||||
|
||||
See {ref}`Runtime Environments <runtime-environments>` for more details. _(New in KubeRay version 1.0.0)_
|
||||
* `jobId` (Optional): Defines the submission ID for the Ray job. If not provided, KubeRay generates one automatically. See {ref}`Ray Jobs CLI API Reference <ray-job-submission-cli-ref>` for more details about the submission ID.
|
||||
* `metadata` (Optional): See {ref}`Ray Jobs CLI API Reference <ray-job-submission-cli-ref>` for more details about the `--metadata-json` option.
|
||||
* `entrypointNumCpus` / `entrypointNumGpus` / `entrypointResources` (Optional): See {ref}`Ray Jobs CLI API Reference <ray-job-submission-cli-ref>` for more details.
|
||||
* `backoffLimit` (Optional, added in version 1.2.0): Specifies the number of retries before marking this RayJob failed. Each retry creates a new RayCluster. The default value is 0.
|
||||
* Submission configuration
|
||||
* `submissionMode` (Optional): Specifies how RayJob submits the Ray job to the RayCluster. There are three possible values, with the default being `K8sJobMode`.
|
||||
* `K8sJobMode`: The KubeRay operator creates a submitter Kubernetes Job to submit the Ray job.
|
||||
* `HTTPMode`: The KubeRay operator sends a request to the RayCluster to create a Ray job.
|
||||
* `InteractiveMode`: The KubeRay operator waits for the user to submit a job to the RayCluster. This mode is currently in alpha and the [KubeRay kubectl plugin](kubectl-plugin) relies on it.
|
||||
* `SidecarMode`: The KubeRay operator injects a container into the Ray head Pod to submit the Ray job. This mode does not support `clusterSelector`, `submitterPodTemplate`, and `submitterConfig`, and requires the head Pod's restart policy to be `Never`.
|
||||
* `submitterPodTemplate` (Optional): Defines the Pod template for the submitter Kubernetes Job. This field is only effective when `submissionMode` is "K8sJobMode".
|
||||
* `RAY_DASHBOARD_ADDRESS` - The KubeRay operator injects this environment variable to the submitter Pod. The value is `$HEAD_SERVICE:$DASHBOARD_PORT`.
|
||||
* `RAY_JOB_SUBMISSION_ID` - The KubeRay operator injects this environment variable to the submitter Pod. The value is the `RayJob.Status.JobId` of the RayJob.
|
||||
* Example: `ray job submit --address=http://$RAY_DASHBOARD_ADDRESS --submission-id=$RAY_JOB_SUBMISSION_ID ...`
|
||||
* See [ray-job.sample.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-job.sample.yaml) for more details.
|
||||
* `submitterConfig` (Optional): Additional configurations for the submitter Kubernetes Job.
|
||||
* `backoffLimit` (Optional, added in version 1.2.0): The number of retries before marking the submitter Job as failed. The default value is 2.
|
||||
* Automatic resource cleanup
|
||||
* `preRunningDeadlineSeconds` (Optional): If the RayJob doesn't transition the `JobDeploymentStatus` to `Running` within `preRunningDeadlineSeconds` seconds, the KubeRay operator transitions the `JobDeploymentStatus` to `Failed` with reason `PreRunningDeadlineExceeded`. The default value is 0 (no pre-running deadline is enforced).
|
||||
* `shutdownAfterJobFinishes` (Optional): Determines whether to recycle the RayCluster after the Ray job finishes. The default value is false.
|
||||
* `ttlSecondsAfterFinished` (Optional): Only works if `shutdownAfterJobFinishes` is true. The KubeRay operator deletes the RayCluster and the submitter `ttlSecondsAfterFinished` seconds after the Ray job finishes. The default value is 0.
|
||||
* `activeDeadlineSeconds` (Optional): If the RayJob doesn't transition the `JobDeploymentStatus` to `Complete` or `Failed` within `activeDeadlineSeconds`, the KubeRay operator transitions the `JobDeploymentStatus` to `Failed`, citing `DeadlineExceeded` as the reason.
|
||||
* `DELETE_RAYJOB_CR_AFTER_JOB_FINISHES` (Optional, added in version 1.2.0): Set this environment variable for the KubeRay operator, not the RayJob resource. If you set this environment variable to true, the RayJob custom resource itself is deleted if you also set `shutdownAfterJobFinishes` to true. Note that KubeRay deletes all resources created by the RayJob, including the Kubernetes Job.
|
||||
* Others
|
||||
* `suspend` (Optional): If `suspend` is true, KubeRay deletes both the RayCluster and the submitter. Note that Kueue also implements scheduling strategies by mutating this field. Avoid manually updating this field if you use Kueue to schedule RayJob.
|
||||
* `deletionStrategy` (alpha in v1.5.1, beta in v1.6.0): Configures automated cleanup after the RayJob reaches a terminal state. This field requires the `RayJobDeletionPolicy` feature gate to be enabled. Two mutually exclusive styles are supported:
|
||||
* **Rules-based** (Recommended): Define `deletionRules` as a list of deletion actions triggered by specific conditions. Each rule specifies:
|
||||
* `policy`: The deletion action to perform — `DeleteCluster` (delete the entire RayCluster and its Pods), `DeleteWorkers` (delete only worker Pods), `DeleteSelf` (delete the RayJob and all associated resources), or `DeleteNone` (no deletion).
|
||||
* `condition`: When to trigger the deletion, based on `jobStatus` (`SUCCEEDED` or `FAILED`) and an optional `ttlSeconds` delay.
|
||||
* This approach enables flexible, multi-stage cleanup strategies (e.g., delete workers immediately on success, then delete the cluster after 300 seconds).
|
||||
* Rules-based mode is incompatible with `shutdownAfterJobFinishes` and the global `ttlSecondsAfterFinished`. Use per-rule `condition.ttlSeconds` instead.
|
||||
* See [ray-job.deletion-rules.yaml](https://github.com/ray-project/kuberay/blob/master/ray-operator/config/samples/ray-job.deletion-rules.yaml) for example configurations.
|
||||
* **Legacy** (Deprecated): Define both `onSuccess` and `onFailure` policies. This approach is deprecated and will be removed in v1.6.0. Migration to `deletionRules` is strongly encouraged.
|
||||
* Legacy mode can be combined with `shutdownAfterJobFinishes` and the global `ttlSecondsAfterFinished`.
|
||||
* For detailed API specifications, see the [KubeRay API Reference](https://ray-project.github.io/kuberay/reference/api/).
|
||||
|
||||
|
||||
## Example: Run a simple Ray job with RayJob
|
||||
|
||||
## Step 1: Create a Kubernetes cluster with Kind
|
||||
|
||||
```sh
|
||||
kind create cluster --image=kindest/node:v1.26.0
|
||||
```
|
||||
|
||||
## Step 2: Install the KubeRay operator
|
||||
|
||||
Follow the [KubeRay Operator Installation](kuberay-operator-deploy) to install the latest stable KubeRay operator by Helm repository.
|
||||
|
||||
## Step 3: Install a RayJob
|
||||
|
||||
```sh
|
||||
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-job.sample.yaml
|
||||
```
|
||||
|
||||
## Step 4: Verify the Kubernetes cluster status
|
||||
|
||||
```shell
|
||||
# Step 4.1: List all RayJob custom resources in the `default` namespace.
|
||||
kubectl get rayjob
|
||||
|
||||
# [Example output]
|
||||
# NAME JOB STATUS DEPLOYMENT STATUS RAY CLUSTER NAME START TIME END TIME AGE
|
||||
# rayjob-sample SUCCEEDED Complete rayjob-sample-qnftt 2025-06-25T16:21:21Z 2025-06-25T16:22:35Z 6m53s
|
||||
|
||||
# Step 4.2: List all RayCluster custom resources in the `default` namespace.
|
||||
kubectl get raycluster
|
||||
|
||||
# [Example output]
|
||||
# NAME DESIRED WORKERS AVAILABLE WORKERS CPUS MEMORY GPUS STATUS AGE
|
||||
# rayjob-sample-qnftt 1 1 400m 0 0 ready 7m48s
|
||||
|
||||
# Step 4.3: List all Pods in the `default` namespace.
|
||||
# The Pod created by the Kubernetes Job will be terminated after the Kubernetes Job finishes.
|
||||
kubectl get pods
|
||||
|
||||
# [Example output]
|
||||
# kuberay-operator-755f666c4b-wbcm4 1/1 Running 0 8m32s
|
||||
# rayjob-sample-n2vj5 0/1 Completed 0 7m18ss => Pod created by a Kubernetes Job
|
||||
# rayjob-sample-qnftt-head 1/1 Running 0 8m14s
|
||||
# rayjob-sample-qnftt-small-group-worker-4f5wz 1/1 Running 0 8m14s
|
||||
|
||||
# Step 4.4: Check the status of the RayJob.
|
||||
# The field `jobStatus` in the RayJob custom resource will be updated to `SUCCEEDED` and `jobDeploymentStatus`
|
||||
# should be `Complete` once the job finishes.
|
||||
kubectl get rayjobs.ray.io rayjob-sample -o jsonpath='{.status.jobStatus}'
|
||||
# [Expected output]: "SUCCEEDED"
|
||||
|
||||
kubectl get rayjobs.ray.io rayjob-sample -o jsonpath='{.status.jobDeploymentStatus}'
|
||||
# [Expected output]: "Complete"
|
||||
```
|
||||
|
||||
The KubeRay operator creates a RayCluster custom resource based on the `rayClusterSpec` and a submitter Kubernetes Job to submit a Ray job to the RayCluster. In this example, the `entrypoint` is `python /home/ray/samples/sample_code.py`, and `sample_code.py` is a Python script stored in a Kubernetes ConfigMap mounted to the head Pod of the RayCluster. Because the default value of `shutdownAfterJobFinishes` is false, the KubeRay operator doesn't delete the RayCluster or the submitter when the Ray job finishes.
|
||||
|
||||
## Step 5: Check the output of the Ray job
|
||||
|
||||
```sh
|
||||
kubectl logs -l=job-name=rayjob-sample
|
||||
|
||||
# [Example output]
|
||||
# 2025-06-25 09:22:27,963 INFO worker.py:1654 -- Connecting to existing Ray cluster at address: 10.244.0.6:6379...
|
||||
# 2025-06-25 09:22:27,977 INFO worker.py:1832 -- Connected to Ray cluster. View the dashboard at 10.244.0.6:8265
|
||||
# test_counter got 1
|
||||
# test_counter got 2
|
||||
# test_counter got 3
|
||||
# test_counter got 4
|
||||
# test_counter got 5
|
||||
# 2025-06-25 09:22:31,719 SUCC cli.py:63 -- -----------------------------------
|
||||
# 2025-06-25 09:22:31,719 SUCC cli.py:64 -- Job 'rayjob-sample-zdxm6' succeeded
|
||||
# 2025-06-25 09:22:31,719 SUCC cli.py:65 -- -----------------------------------
|
||||
```
|
||||
|
||||
The Python script `sample_code.py` used by `entrypoint` is a simple Ray script that executes a counter's increment function 5 times.
|
||||
|
||||
## Step 6: Delete the RayJob
|
||||
|
||||
```sh
|
||||
kubectl delete -f https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-job.sample.yaml
|
||||
```
|
||||
|
||||
## Step 7: Create a RayJob with `shutdownAfterJobFinishes` set to true
|
||||
|
||||
```sh
|
||||
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-job.shutdown.yaml
|
||||
```
|
||||
|
||||
The `ray-job.shutdown.yaml` defines a RayJob custom resource with `shutdownAfterJobFinishes: true` and `ttlSecondsAfterFinished: 10`. Hence, the KubeRay operator deletes the RayCluster 10 seconds after the Ray job finishes. Note that the submitter job isn't deleted because it contains the ray job logs and doesn't use any cluster resources once completed. In addition, the RayJob cleans up the submitter job when the RayJob is eventually deleted due to its owner reference back to the RayJob.
|
||||
|
||||
## Step 8: Check the RayJob status
|
||||
|
||||
```sh
|
||||
# Wait until `jobStatus` is `SUCCEEDED` and `jobDeploymentStatus` is `Complete`.
|
||||
kubectl get rayjobs.ray.io rayjob-sample-shutdown -o jsonpath='{.status.jobDeploymentStatus}'
|
||||
kubectl get rayjobs.ray.io rayjob-sample-shutdown -o jsonpath='{.status.jobStatus}'
|
||||
```
|
||||
|
||||
## Step 9: Check if the KubeRay operator deletes the RayCluster
|
||||
|
||||
```sh
|
||||
# List the RayCluster custom resources in the `default` namespace. The RayCluster
|
||||
# associated with the RayJob `rayjob-sample-shutdown` should be deleted.
|
||||
kubectl get raycluster
|
||||
```
|
||||
|
||||
## Step 10: Clean up
|
||||
|
||||
```sh
|
||||
# Step 10.1: Delete the RayJob
|
||||
kubectl delete -f https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-job.shutdown.yaml
|
||||
|
||||
# Step 10.2: Delete the KubeRay operator
|
||||
helm uninstall kuberay-operator
|
||||
|
||||
# Step 10.3: Delete the Kubernetes cluster
|
||||
kind delete cluster
|
||||
```
|
||||
|
||||
## Next steps
|
||||
|
||||
* [RayJob Batch Inference Example](kuberay-batch-inference-example)
|
||||
* [Priority Scheduling with RayJob and Kueue](kuberay-kueue-priority-scheduling-example)
|
||||
* [Gang Scheduling with RayJob and Kueue](kuberay-kueue-gang-scheduling-example)
|
||||
@@ -0,0 +1,142 @@
|
||||
(kuberay-rayservice-quickstart)=
|
||||
# RayService Quickstart
|
||||
|
||||
## Prerequisites
|
||||
|
||||
This guide mainly focuses on the behavior of KubeRay v1.6.0 and Ray 2.46.0.
|
||||
|
||||
## What's a RayService?
|
||||
|
||||
A RayService manages these components:
|
||||
|
||||
* **RayCluster**: Manages resources in a Kubernetes cluster.
|
||||
* **Ray Serve Applications**: Manages users' applications.
|
||||
|
||||
## What does the RayService provide?
|
||||
|
||||
* **Kubernetes-native support for Ray clusters and Ray Serve applications:** After using a Kubernetes configuration to define a Ray cluster and its Ray Serve applications, you can use `kubectl` to create the cluster and its applications.
|
||||
* **In-place updating for Ray Serve applications:** See [RayService](kuberay-rayservice) for more details.
|
||||
* **Zero downtime upgrading for Ray clusters:** See [RayService](kuberay-rayservice) for more details.
|
||||
* **High-availabilable services:** See [RayService high availability](kuberay-rayservice-ha) for more details.
|
||||
|
||||
## Example: Serve two simple Ray Serve applications using RayService
|
||||
|
||||
## Step 1: Create a Kubernetes cluster with Kind
|
||||
|
||||
```sh
|
||||
kind create cluster --image=kindest/node:v1.26.0
|
||||
```
|
||||
|
||||
## Step 2: Install the KubeRay operator
|
||||
|
||||
Follow [this document](kuberay-operator-deploy) to install the latest stable KubeRay operator from the Helm repository. Note that the YAML file in this example uses `serveConfigV2` to specify a multi-application Serve configuration, available starting from KubeRay v0.6.0.
|
||||
|
||||
## Step 3: Install a RayService
|
||||
|
||||
```sh
|
||||
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-service.sample.yaml
|
||||
```
|
||||
|
||||
## Step 4: Verify the Kubernetes cluster status
|
||||
|
||||
```sh
|
||||
# Step 4.1: List all RayService custom resources in the `default` namespace.
|
||||
kubectl get rayservice
|
||||
|
||||
# [Example output]
|
||||
# NAME SERVICE STATUS NUM SERVE ENDPOINTS
|
||||
# rayservice-sample Running 2
|
||||
|
||||
# Step 4.2: List all RayCluster custom resources in the `default` namespace.
|
||||
kubectl get raycluster
|
||||
|
||||
# [Example output]
|
||||
# NAME DESIRED WORKERS AVAILABLE WORKERS CPUS MEMORY GPUS STATUS AGE
|
||||
# rayservice-sample-cxm7t 1 1 2500m 4Gi 0 ready 79s
|
||||
|
||||
# Step 4.3: List all Ray Pods in the `default` namespace.
|
||||
kubectl get pods -l=ray.io/is-ray-node=yes
|
||||
|
||||
# [Example output]
|
||||
# NAME READY STATUS RESTARTS AGE
|
||||
# rayservice-sample-cxm7t-head 1/1 Running 0 3m5s
|
||||
# rayservice-sample-cxm7t-small-group-worker-8hrgg 1/1 Running 0 3m5s
|
||||
|
||||
# Step 4.4: Check the `Ready` condition of the RayService.
|
||||
# The RayService is ready to serve requests when the condition is `True`.
|
||||
kubectl describe rayservices.ray.io rayservice-sample
|
||||
|
||||
# [Example output]
|
||||
# Conditions:
|
||||
# Last Transition Time: 2025-06-26T13:23:06Z
|
||||
# Message: Number of serve endpoints is greater than 0
|
||||
# Observed Generation: 1
|
||||
# Reason: NonZeroServeEndpoints
|
||||
# Status: True
|
||||
# Type: Ready
|
||||
|
||||
# Step 4.5: List services in the `default` namespace.
|
||||
kubectl get services
|
||||
|
||||
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
# ...
|
||||
# rayservice-sample-cxm7t-head-svc ClusterIP None <none> 10001/TCP,8265/TCP,6379/TCP,8080/TCP,8000/TCP 71m
|
||||
# rayservice-sample-head-svc ClusterIP None <none> 10001/TCP,8265/TCP,6379/TCP,8080/TCP,8000/TCP 70m
|
||||
# rayservice-sample-serve-svc ClusterIP 10.96.125.107 <none> 8000/TCP 70m
|
||||
```
|
||||
|
||||
When the Ray Serve applications are healthy and ready, KubeRay creates a head service and a Ray Serve service for the RayService custom resource. For example, `rayservice-sample-head-svc` and `rayservice-sample-serve-svc` in Step 4.5.
|
||||
> **What do these services do?**
|
||||
|
||||
- **`rayservice-sample-head-svc`**
|
||||
This service points to the **head pod** of the active RayCluster and is typically used to view the **Ray Dashboard** (port `8265`).
|
||||
|
||||
- **`rayservice-sample-serve-svc`**
|
||||
This service exposes the **HTTP interface** of Ray Serve, typically on port `8000`.
|
||||
Use this service to send HTTP requests to your deployed Serve applications (e.g., REST API, ML inference, etc.).
|
||||
|
||||
|
||||
## Step 5: Verify the status of the Serve applications
|
||||
|
||||
```sh
|
||||
# (1) Forward the dashboard port to localhost.
|
||||
# (2) Check the Serve page in the Ray dashboard at http://localhost:8265/#/serve.
|
||||
kubectl port-forward svc/rayservice-sample-head-svc 8265:8265
|
||||
```
|
||||
|
||||
* Refer to [rayservice-troubleshooting.md](kuberay-raysvc-troubleshoot) for more details on RayService observability. Below is a screenshot example of the Serve page in the Ray dashboard. 
|
||||
|
||||
## Step 6: Send requests to the Serve applications by the Kubernetes serve service
|
||||
|
||||
```sh
|
||||
# Step 6.1: Run a curl Pod.
|
||||
# If you already have a curl Pod, you can use `kubectl exec -it <curl-pod> -- sh` to access the Pod.
|
||||
kubectl run curl --image=curlimages/curl:latest -i --tty -- sh
|
||||
|
||||
# Step 6.2: Send a request to the fruit stand app.
|
||||
curl -X POST -H 'Content-Type: application/json' rayservice-sample-serve-svc:8000/fruit/ -d '["MANGO", 2]'
|
||||
# [Expected output]: 6
|
||||
|
||||
# Step 6.3: Send a request to the calculator app.
|
||||
curl -X POST -H 'Content-Type: application/json' rayservice-sample-serve-svc:8000/calc/ -d '["MUL", 3]'
|
||||
# [Expected output]: "15 pizzas please!"
|
||||
```
|
||||
|
||||
## Step 7: Clean up the Kubernetes cluster
|
||||
|
||||
```sh
|
||||
# Delete the RayService.
|
||||
kubectl delete -f https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-service.sample.yaml
|
||||
|
||||
# Uninstall the KubeRay operator.
|
||||
helm uninstall kuberay-operator
|
||||
|
||||
# Delete the curl Pod.
|
||||
kubectl delete pod curl
|
||||
```
|
||||
|
||||
## Next steps
|
||||
|
||||
* See [RayService](kuberay-rayservice) document for the full list of RayService features, including in-place update, zero downtime upgrade, and high-availability.
|
||||
* See [RayService troubleshooting guide](kuberay-raysvc-troubleshoot) if you encounter any issues.
|
||||
* See [Examples](kuberay-examples) for more RayService examples. The [MobileNet example](kuberay-mobilenet-rayservice-example) is a good example to start with because it doesn't require GPUs and is easy to run on a local machine.
|
||||
Reference in New Issue
Block a user