wehub/ray-project--ray
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 13:17:40 +08:00
commit f1825c8ceb
10096 changed files with 2364182 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/cluster/kubernetes/examples/argocd.md
+532
View File
@@ -0,0 +1,532 @@
(deploying-on-argocd-example)=
# Deploying Ray Clusters via ArgoCD
This guide provides a step-by-step approach for deploying Ray clusters on Kubernetes using ArgoCD. ArgoCD is a declarative GitOps tool that enables you to manage Ray cluster configurations in Git repositories with automated synchronization, version control, and rollback capabilities. This approach is particularly valuable when managing multiple Ray clusters across different environments, implementing audit trails and approval workflows, or maintaining infrastructure-as-code practices. For simpler use cases like single-cluster development or quick experimentation, direct kubectl or Helm deployments may be sufficient. You can read more about the benefits of ArgoCD [here](https://argo-cd.readthedocs.io/en/stable/#why-argo-cd).
This example demonstrates how to deploy the KubeRay operator and a RayCluster with three different worker groups, leveraging ArgoCD's GitOps capabilities for automated cluster management.
## Prerequisites
Before proceeding with this guide, ensure you have the following:
* A Kubernetes cluster with appropriate resources for running Ray workloads.
* `kubectl` configured to access your Kubernetes cluster.
* (Optional)[ArgoCD installed](https://argo-cd.readthedocs.io/en/stable/getting_started/) on your Kubernetes cluster.
* (Optional)[ArgoCD CLI](https://argo-cd.readthedocs.io/en/stable/cli_installation/) installed on your local machine (recommended for easier application management. It might need [port-forwarding and login](https://argo-cd.readthedocs.io/en/stable/getting_started/#port-forwarding) depending on your environment).
* (Optional)Access to the ArgoCD UI or API server.
## Step 1: Deploy KubeRay Operator CRDs
First, deploy the Custom Resource Definitions (CRDs) required by the KubeRay operator. Create a file named `ray-operator-crds.yaml` with the following content:
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: ray-operator-crds
namespace: argocd
spec:
project: default
destination:
server: https://kubernetes.default.svc
namespace: ray-cluster
source:
repoURL: https://github.com/ray-project/kuberay
targetRevision: v1.6.0 # update this as necessary
path: helm-chart/kuberay-operator/crds
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- Replace=true
```
Apply the ArgoCD Application:
```sh
kubectl apply -f ray-operator-crds.yaml
```
Wait for the CRDs Application to sync and become healthy. You can check the status using:
```sh
kubectl get application ray-operator-crds -n argocd
```
Which should eventually give something like:
```
NAME SYNC STATUS HEALTH STATUS
ray-operator-crds Synced Healthy
```
Alternatively, if you have the ArgoCD CLI installed, you can wait for the application:
```sh
argocd app wait ray-operator-crds
```
## Step 2: Deploy the KubeRay Operator
After the CRDs are installed, deploy the KubeRay operator itself. Create a file named `ray-operator.yaml` with the following content:
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: ray-operator
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/ray-project/kuberay
targetRevision: v1.6.0 # update this as necessary
path: helm-chart/kuberay-operator
helm:
skipCrds: true # CRDs are already installed in Step 1
destination:
server: https://kubernetes.default.svc
namespace: ray-cluster
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
```
Note the `skipCrds: true` setting in the Helm configuration. This is required because the CRDs were installed separately in Step 1.
Apply the ArgoCD Application:
```sh
kubectl apply -f ray-operator.yaml
```
Wait for the operator Application to sync and become healthy. You can check the status using:
```sh
kubectl get application ray-operator -n argocd
```
Which should give the following output eventually:
```
NAME SYNC STATUS HEALTH STATUS
ray-operator Synced Healthy
```
Alternatively, if you have the ArgoCD CLI installed:
```sh
argocd app wait ray-operator
```
Verify that the KubeRay operator pod is running:
```sh
kubectl get pods -n ray-cluster -l app.kubernetes.io/name=kuberay-operator
```
## Step 3: Deploy a RayCluster
Now deploy a RayCluster with autoscaling enabled and three different worker groups. Create a file named `raycluster.yaml` with the following content:
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: raycluster
namespace: argocd
spec:
project: default
destination:
server: https://kubernetes.default.svc
namespace: ray-cluster
ignoreDifferences:
- group: ray.io
kind: RayCluster
name: raycluster-kuberay
namespace: ray-cluster
jqPathExpressions:
- .spec.workerGroupSpecs[].replicas
source:
repoURL: https://ray-project.github.io/kuberay-helm/
chart: ray-cluster
targetRevision: "1.6.0"
helm:
releaseName: raycluster
valuesObject:
image:
repository: docker.io/rayproject/ray
tag: latest
pullPolicy: IfNotPresent
head:
rayStartParams:
num-cpus: "0"
enableInTreeAutoscaling: true
autoscalerOptions:
version: v2
upscalingMode: Default
idleTimeoutSeconds: 600 # 10 minutes
env:
- name: AUTOSCALER_MAX_CONCURRENT_LAUNCHES
value: "100"
worker:
groupName: standard-worker
replicas: 1
minReplicas: 1
maxReplicas: 200
rayStartParams:
resources: '"{\"standard-worker\": 1}"'
resources:
requests:
cpu: "1"
memory: "1G"
additionalWorkerGroups:
additional-worker-group1:
image:
repository: docker.io/rayproject/ray
tag: latest
pullPolicy: IfNotPresent
disabled: false
replicas: 1
minReplicas: 1
maxReplicas: 30
rayStartParams:
resources: '"{\"additional-worker-group1\": 1}"'
resources:
requests:
cpu: "1"
memory: "1G"
additional-worker-group2:
image:
repository: docker.io/rayproject/ray
tag: latest
pullPolicy: IfNotPresent
disabled: false
replicas: 1
minReplicas: 1
maxReplicas: 200
rayStartParams:
resources: '"{\"additional-worker-group2\": 1}"'
resources:
requests:
cpu: "1"
memory: "1G"
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
```
Apply the ArgoCD Application:
```sh
kubectl apply -f raycluster.yaml
```
Wait for the RayCluster Application to sync and become healthy. You can check the status using:
```sh
kubectl get application raycluster -n argocd
```
Alternatively, if you have the ArgoCD CLI installed:
```sh
argocd app wait raycluster
```
Verify that the RayCluster is running:
```sh
kubectl get raycluster -n ray-cluster
```
Which will give something like:
```
NAME DESIRED WORKERS AVAILABLE WORKERS CPUS MEMORY GPUS STATUS AGE
raycluster-kuberay 3 3 ... ... ... ready ...
```
You should see the head pod and worker pods:
```sh
kubectl get pods -n ray-cluster
```
Gives something like:
```
NAME READY STATUS RESTARTS AGE
kuberay-operator-6c485bc876-28dnl 1/1 Running 0 11d
raycluster-kuberay-additional-worker-group1-n45rc 1/1 Running 0 5d21h
raycluster-kuberay-additional-worker-group2-b2455 1/1 Running 0 2d18h
raycluster-kuberay-head 2/2 Running 0 5d21h
raycluster-kuberay-standard-worker-worker-bs8t8 1/1 Running 0 5d21h
```
## Understanding Ray Autoscaling with ArgoCD
### Determining Fields to Ignore
The `ignoreDifferences` section in the RayCluster Application configuration is critical for proper autoscaling. To determine which fields need to be ignored, you can inspect the RayCluster resource to identify fields that change dynamically during runtime.
First, describe the RayCluster resource to see its full specification:
```sh
kubectl describe raycluster raycluster-kuberay -n ray-cluster
```
Or, get the resource in YAML format to see the exact field paths:
```sh
kubectl get raycluster raycluster-kuberay -n ray-cluster -o yaml
```
Look for fields that are modified by controllers or autoscalers. In the case of Ray, the autoscaler modifies the `replicas` field under each worker group spec. You'll see output similar to:
```yaml
spec:
workerGroupSpecs:
- replicas: 5 # This value changes dynamically
minReplicas: 1
maxReplicas: 200
groupName: standard-worker
# ...
```
When ArgoCD detects differences between the desired state (in Git) and the actual state (in the cluster), it will show these in the UI or via CLI:
```sh
argocd app diff raycluster
```
If you see repeated differences in fields that should be managed by controllers (like autoscalers), those are candidates for `ignoreDifferences`.
### Configuring ignoreDifferences
The `ignoreDifferences` section in the RayCluster Application configuration tells ArgoCD which fields to ignore. Without this setting, ArgoCD and the Ray Autoscaler may conflict, resulting in unexpected behavior when requesting workers dynamically (for example, using `ray.autoscaler.sdk.request_resources`). Specifically, when requesting N workers, the Autoscaler might not spin up the expected number of workers because ArgoCD could revert the replica count back to the original value defined in the Application manifest.
The recommended approach is to use `jqPathExpressions`, which automatically handles any number of worker groups:
```yaml
ignoreDifferences:
- group: ray.io
kind: RayCluster
name: raycluster-kuberay
namespace: ray-cluster
jqPathExpressions:
- .spec.workerGroupSpecs[].replicas
```
This configuration tells ArgoCD to ignore differences in the `replicas` field for all worker groups. The `jqPathExpressions` field uses JQ syntax with array wildcards (`[]`), which means you don't need to update the configuration when adding or removing worker groups.
**Note**: The `name` and `namespace` must match your RayCluster resource name and namespace. Verify these values separately if you've customized them.
**Alternative: Using jsonPointers**
If you prefer explicit configuration, you can use `jsonPointers` instead:
```yaml
ignoreDifferences:
- group: ray.io
kind: RayCluster
name: raycluster-kuberay
namespace: ray-cluster
jsonPointers:
- /spec/workerGroupSpecs/0/replicas
- /spec/workerGroupSpecs/1/replicas
- /spec/workerGroupSpecs/2/replicas
```
With `jsonPointers`, you must explicitly list each worker group by index:
- `/spec/workerGroupSpecs/0/replicas` - First worker group (the default `worker` group)
- `/spec/workerGroupSpecs/1/replicas` - Second worker group (`additional-worker-group1`)
- `/spec/workerGroupSpecs/2/replicas` - Third worker group (`additional-worker-group2`)
If you add or remove worker groups, you **must** update this list accordingly. The index corresponds to the order of worker groups as they appear in the RayCluster spec, with the default `worker` group at index 0 and `additionalWorkerGroups` following in the order they are defined.
See the [ArgoCD diff customization documentation](https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/) for more details on both approaches.
By ignoring these differences, ArgoCD allows the Ray Autoscaler to dynamically manage worker replicas without interference.
## Step 4: Access the Ray Dashboard
To access the Ray Dashboard, port-forward the head service:
```sh
kubectl port-forward -n ray-cluster svc/raycluster-kuberay-head-svc 8265:8265
```
Navigate to `http://localhost:8265` in your browser to view the Ray Dashboard.
## Customizing the Configuration
You can customize the RayCluster configuration by modifying the `valuesObject` section in the `raycluster.yaml` file:
* **Image**: Change the `repository` and `tag` to use different Ray versions.
* **Worker Groups**: Add or remove worker groups by modifying the `additionalWorkerGroups` section.
* **Autoscaling**: Adjust `minReplicas`, `maxReplicas`, and `idleTimeoutSeconds` to control autoscaling behavior.
* **Resources**: Modify `rayStartParams` to allocate custom resources to worker groups.
After making changes, commit them to your Git repository. ArgoCD will automatically sync the changes to your cluster if automated sync is enabled.
## Alternative: Deploy Everything in One File
If you prefer to deploy all components at once, you can combine all three ArgoCD Applications into a single file. Create a file named `ray-argocd-all.yaml` with the following content:
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: ray-operator-crds
namespace: argocd
spec:
project: default
destination:
server: https://kubernetes.default.svc
namespace: ray-cluster
source:
repoURL: https://github.com/ray-project/kuberay
targetRevision: v1.6.0 # update this as necessary
path: helm-chart/kuberay-operator/crds
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- Replace=true
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: ray-operator
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/ray-project/kuberay
targetRevision: v1.6.0 # update this as necessary
path: helm-chart/kuberay-operator
helm:
skipCrds: true # CRDs are installed in the first Application
destination:
server: https://kubernetes.default.svc
namespace: ray-cluster
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: raycluster
namespace: argocd
spec:
project: default
destination:
server: https://kubernetes.default.svc
namespace: ray-cluster
ignoreDifferences:
- group: ray.io
kind: RayCluster
name: raycluster-kuberay # ensure this is aligned with the release name
namespace: ray-cluster # ensure this is aligned with the namespace
jqPathExpressions:
- .spec.workerGroupSpecs[].replicas
source:
repoURL: https://ray-project.github.io/kuberay-helm/
chart: ray-cluster
targetRevision: "1.4.1"
helm:
releaseName: raycluster # this affects the ignoreDifferences field
valuesObject:
image:
repository: docker.io/rayproject/ray
tag: latest
pullPolicy: IfNotPresent
head:
rayStartParams:
num-cpus: "0"
enableInTreeAutoscaling: true
autoscalerOptions:
version: v2
upscalingMode: Default
idleTimeoutSeconds: 600 # 10 minutes
env:
- name: AUTOSCALER_MAX_CONCURRENT_LAUNCHES
value: "100"
worker:
groupName: standard-worker
replicas: 1
minReplicas: 1
maxReplicas: 200
rayStartParams:
resources: '"{\"standard-worker\": 1}"'
resources:
requests:
cpu: "1"
memory: "1G"
additionalWorkerGroups:
additional-worker-group1:
image:
repository: docker.io/rayproject/ray
tag: latest
pullPolicy: IfNotPresent
disabled: false
replicas: 1
minReplicas: 1
maxReplicas: 30
rayStartParams:
resources: '"{\"additional-worker-group1\": 1}"'
resources:
requests:
cpu: "1"
memory: "1G"
additional-worker-group2:
image:
repository: docker.io/rayproject/ray
tag: latest
pullPolicy: IfNotPresent
disabled: false
replicas: 1
minReplicas: 1
maxReplicas: 200
rayStartParams:
resources: '"{\"additional-worker-group2\": 1}"'
resources:
requests:
cpu: "1"
memory: "1G"
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
```
Note in this example, the `jqPathExpressions` approach is used.
Apply all three Applications at once:
```sh
kubectl apply -f ray-argocd-all.yaml
```
This single-file approach is convenient for quick deployments, but the step-by-step approach in the earlier sections provides better visibility into the deployment process and makes it easier to troubleshoot issues.
doc/source/cluster/kubernetes/examples/distributed-checkpointing-with-gcsfuse.md
+289
View File
@@ -0,0 +1,289 @@
(kuberay-distributed-checkpointing-gcsfuse)=
# Distributed checkpointing with KubeRay and GCSFuse
This example orchestrates distributed checkpointing with KubeRay, using the GCSFuse CSI driver and Google Cloud Storage as the remote storage system. To illustrate the concepts, this guide uses the [Finetuning a Pytorch Image Classifier with Ray Train](https://docs.ray.io/en/latest/train/examples/pytorch/pytorch_resnet_finetune.html) example.
## Why distributed checkpointing with GCSFuse?
In large-scale, high-performance machine learning, distributed checkpointing is crucial for fault tolerance, ensuring that if a node fails during training, Ray can resume the process from the latest saved checkpoint instead of starting from scratch. While it's possible to directly reference remote storage paths (e.g., `gs://my-checkpoint-bucket`), using Google Cloud Storage FUSE (GCSFuse) has distinct advantages for distributed applications. GCSFuse allows you to mount Cloud Storage buckets like local file systems, making checkpoint management more intuitive for distributed applications that rely on these semantics. Furthermore, GCSFuse is designed for high-performance workloads, delivering the performance and scalability you need for distributed checkpointing of large models.
[Distributed checkpointing](https://docs.ray.io/en/latest/train/user-guides/checkpoints.html), in combination with [GCSFuse](https://cloud.google.com/storage/docs/gcs-fuse), allows for larger-scale model training with increased availability and efficiency.
## Create a Kubernetes cluster on GKE
Create a GKE cluster with the [GCSFuse CSI driver](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver) and [Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) enabled, as well as a GPU node pool with 4 L4 GPUs:
```
export PROJECT_ID=<your project id>
gcloud container clusters create kuberay-with-gcsfuse \
--addons GcsFuseCsiDriver \
--cluster-version=1.29.4 \
--location=us-east4-c \
--machine-type=g2-standard-8 \
--release-channel=rapid \
--num-nodes=4 \
--accelerator type=nvidia-l4,count=1,gpu-driver-version=latest \
--workload-pool=${PROJECT_ID}.svc.id.goog
```
Verify the successful creation of your cluster with 4 GPUs:
```
$ kubectl get nodes "-o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"
NAME GPU
gke-kuberay-with-gcsfuse-default-pool-xxxx-0000 1
gke-kuberay-with-gcsfuse-default-pool-xxxx-1111 1
gke-kuberay-with-gcsfuse-default-pool-xxxx-2222 1
gke-kuberay-with-gcsfuse-default-pool-xxxx-3333 1
```
## Install the KubeRay operator
Follow [Deploy a KubeRay operator](kuberay-operator-deploy) to install the latest stable KubeRay operator from the Helm repository. The KubeRay operator Pod must be on the CPU node if you set up the taint for the GPU node pool correctly.
## Configuring the GCS Bucket
Create a GCS bucket that Ray uses as the remote filesystem.
```
BUCKET=<your GCS bucket>
gcloud storage buckets create gs://$BUCKET --uniform-bucket-level-access
```
Create a Kubernetes ServiceAccount that grants the RayCluster access to mount the GCS bucket:
```
kubectl create serviceaccount pytorch-distributed-training
```
Bind the `roles/storage.objectUser` role to the Kubernetes service account and bucket IAM policy. See [Identifying projects](https://cloud.google.com/resource-manager/docs/creating-managing-projects#identifying_projects) to find your project ID and project number:
```
PROJECT_ID=<your project ID>
PROJECT_NUMBER=<your project number>
gcloud storage buckets add-iam-policy-binding gs://${BUCKET} --member "principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${PROJECT_ID}.svc.id.goog/subject/ns/default/sa/pytorch-distributed-training" --role "roles/storage.objectUser"
```
See [Access Cloud Storage buckets with the Cloud Storage FUSE CSI driver](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver) for more details.
## Deploy the RayJob
Download the RayJob that executes all the steps documented in [Finetuning a Pytorch Image Classifier with Ray Train](https://docs.ray.io/en/latest/train/examples/pytorch/pytorch_resnet_finetune.html). The [source code](https://github.com/ray-project/kuberay/tree/master/ray-operator/config/samples/pytorch-resnet-image-classifier) is also in the KubeRay repository.
```
curl -LO https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/pytorch-resnet-image-classifier/ray-job.pytorch-image-classifier.yaml
```
Modify the RayJob by replacing all instances of the `GCS_BUCKET` placeholder with the Google Cloud Storage bucket you created earlier. Alternatively you can use `sed`:
```
sed -i "s/GCS_BUCKET/$BUCKET/g" ray-job.pytorch-image-classifier.yaml
```
Deploy the RayJob:
```
kubectl create -f ray-job.pytorch-image-classifier.yaml
```
The deployed RayJob includes the following configuration to enable distributed checkpointing to a shared filesystem:
* 4 Ray workers, each with a single GPU.
* All Ray nodes use the `pytorch-distributed-training` ServiceAccount, which we created earlier.
* Includes volumes that are managed by the `gcsfuse.csi.storage.gke.io` CSI driver.
* Mounts a shared storage path `/mnt/cluster_storage`, backed by the GCS bucket you created earlier.
You can configure the Pod with annotations, which allows for finer grain control of the GCSFuse sidecar container. See [Specify Pod annotations](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver#pod-annotations) for more details.
```
annotations:
gke-gcsfuse/volumes: "true"
gke-gcsfuse/cpu-limit: "0"
gke-gcsfuse/memory-limit: 5Gi
gke-gcsfuse/ephemeral-storage-limit: 10Gi
```
You can also specify mount options when defining the GCSFuse container volume:
```
csi:
driver: gcsfuse.csi.storage.gke.io
volumeAttributes:
bucketName: GCS_BUCKET
mountOptions: "implicit-dirs,uid=1000,gid=100"
```
See [Mount options](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver#mount-options) to learn more about mount options.
Logs from the Ray job should indicate the use of the shared remote filesystem in `/mnt/cluster_storage` and the checkpointing directory. For example:
```
Training finished iteration 10 at 2024-04-29 10:22:08. Total running time: 1min 30s
╭─────────────────────────────────────────╮
│ Training result │
├─────────────────────────────────────────┤
│ checkpoint_dir_name checkpoint_000009 │
│ time_this_iter_s 6.47154 │
│ time_total_s 74.5547 │
│ training_iteration 10 │
│ acc 0.24183 │
│ loss 0.06882 │
╰─────────────────────────────────────────╯
Training saved a checkpoint for iteration 10 at: (local)/mnt/cluster_storage/finetune-resnet/TorchTrainer_cbb82_00000_0_2024-04-29_10-20-37/checkpoint_000009
```
## Inspect checkpointing data
Once the RayJob completes, you can inspect the contents of your bucket using a tool like [gsutil](https://cloud.google.com/storage/docs/gsutil).
```
gsutil ls gs://my-ray-bucket/**
gs://my-ray-bucket/finetune-resnet/
gs://my-ray-bucket/finetune-resnet/.validate_storage_marker
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000007/
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000007/checkpoint.pt
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000008/
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000008/checkpoint.pt
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000009/
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000009/checkpoint.pt
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/error.pkl
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/error.txt
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/events.out.tfevents.1714436502.orch-image-classifier-nc2sq-raycluster-tdrfx-head-xzcl8
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/events.out.tfevents.1714436809.orch-image-classifier-zz4sj-raycluster-vn7kz-head-lwx8k
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/params.json
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/params.pkl
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/progress.csv
gs://my-ray-bucket/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/result.json
gs://my-ray-bucket/finetune-resnet/basic-variant-state-2024-04-29_17-21-29.json
gs://my-ray-bucket/finetune-resnet/basic-variant-state-2024-04-29_17-26-35.json
gs://my-ray-bucket/finetune-resnet/experiment_state-2024-04-29_17-21-29.json
gs://my-ray-bucket/finetune-resnet/experiment_state-2024-04-29_17-26-35.json
gs://my-ray-bucket/finetune-resnet/trainer.pkl
gs://my-ray-bucket/finetune-resnet/tuner.pkl
```
## Resuming from checkpoint
In the event of a failed job, you can use the latest checkpoint to resume training of the model. This example configures `TorchTrainer` to automatically resume from the latest checkpoint:
```python
experiment_path = os.path.expanduser("/mnt/cluster_storage/finetune-resnet")
if TorchTrainer.can_restore(experiment_path):
trainer = TorchTrainer.restore(experiment_path,
train_loop_per_worker=train_loop_per_worker,
train_loop_config=train_loop_config,
scaling_config=scaling_config,
run_config=run_config,
)
else:
trainer = TorchTrainer(
train_loop_per_worker=train_loop_per_worker,
train_loop_config=train_loop_config,
scaling_config=scaling_config,
run_config=run_config,
)
```
You can verify automatic checkpoint recovery by redeploying the same RayJob:
```
kubectl create -f ray-job.pytorch-image-classifier.yaml
```
If the previous job succeeded, the training job should restore the checkpoint state from the `checkpoint_000009` directory and then immediately complete training with 0 iterations:
```
2024-04-29 15:51:32,528 INFO experiment_state.py:366 -- Trying to find and download experiment checkpoint at /mnt/cluster_storage/finetune-resnet
2024-04-29 15:51:32,651 INFO experiment_state.py:396 -- A remote experiment checkpoint was found and will be used to restore the previous experiment state.
2024-04-29 15:51:32,652 INFO tune_controller.py:404 -- Using the newest experiment state file found within the experiment directory: experiment_state-2024-04-29_15-43-40.json
View detailed results here: /mnt/cluster_storage/finetune-resnet
To visualize your results with TensorBoard, run: `tensorboard --logdir /home/ray/ray_results/finetune-resnet`
Result(
metrics={'loss': 0.070047477101968, 'acc': 0.23529411764705882},
path='/mnt/cluster_storage/finetune-resnet/TorchTrainer_ecc04_00000_0_2024-04-29_15-43-40',
filesystem='local',
checkpoint=Checkpoint(filesystem=local, path=/mnt/cluster_storage/finetune-resnet/TorchTrainer_ecc04_00000_0_2024-04-29_15-43-40/checkpoint_000009)
)
```
If the previous job failed at an earlier checkpoint, the job should resume from the last saved checkpoint and run until `max_epochs=10`. For example, if the last run failed at epoch 7, the training automatically resumes using `checkpoint_000006` and run 3 more iterations until epoch 10:
```
(TorchTrainer pid=611, ip=10.108.2.65) Restored on 10.108.2.65 from checkpoint: Checkpoint(filesystem=local, path=/mnt/cluster_storage/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000006)
(RayTrainWorker pid=671, ip=10.108.2.65) Setting up process group for: env:// [rank=0, world_size=4]
(TorchTrainer pid=611, ip=10.108.2.65) Started distributed worker processes:
(TorchTrainer pid=611, ip=10.108.2.65) - (ip=10.108.2.65, pid=671) world_rank=0, local_rank=0, node_rank=0
(TorchTrainer pid=611, ip=10.108.2.65) - (ip=10.108.1.83, pid=589) world_rank=1, local_rank=0, node_rank=1
(TorchTrainer pid=611, ip=10.108.2.65) - (ip=10.108.0.72, pid=590) world_rank=2, local_rank=0, node_rank=2
(TorchTrainer pid=611, ip=10.108.2.65) - (ip=10.108.3.76, pid=590) world_rank=3, local_rank=0, node_rank=3
(RayTrainWorker pid=589, ip=10.108.1.83) Downloading: "https://download.pytorch.org/models/resnet50-0676ba61.pth" to /home/ray/.cache/torch/hub/checkpoints/resnet50-0676ba61.pth
(RayTrainWorker pid=671, ip=10.108.2.65)
0%| | 0.00/97.8M [00:00<?, ?B/s]
(RayTrainWorker pid=671, ip=10.108.2.65)
22%|██▏ | 21.8M/97.8M [00:00<00:00, 229MB/s]
(RayTrainWorker pid=671, ip=10.108.2.65)
92%|█████████▏| 89.7M/97.8M [00:00<00:00, 327MB/s]
(RayTrainWorker pid=671, ip=10.108.2.65)
100%|██████████| 97.8M/97.8M [00:00<00:00, 316MB/s]
(RayTrainWorker pid=671, ip=10.108.2.65) Moving model to device: cuda:0
(RayTrainWorker pid=671, ip=10.108.2.65) Wrapping provided model in DistributedDataParallel.
(RayTrainWorker pid=671, ip=10.108.2.65) Downloading: "https://download.pytorch.org/models/resnet50-0676ba61.pth" to /home/ray/.cache/torch/hub/checkpoints/resnet50-0676ba61.pth [repeated 3x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/ray-logging.html#log-deduplication for more options.)
(RayTrainWorker pid=590, ip=10.108.3.76)
0%| | 0.00/97.8M [00:00<?, ?B/s] [repeated 3x across cluster]
(RayTrainWorker pid=590, ip=10.108.0.72)
85%|████████▍ | 82.8M/97.8M [00:00<00:00, 256MB/s]
100%|██████████| 97.8M/97.8M [00:00<00:00, 231MB/s] [repeated 11x across cluster]
(RayTrainWorker pid=590, ip=10.108.3.76)
100%|██████████| 97.8M/97.8M [00:00<00:00, 238MB/s]
(RayTrainWorker pid=671, ip=10.108.2.65) Epoch 7-train Loss: 0.0903 Acc: 0.2418
(RayTrainWorker pid=671, ip=10.108.2.65) Epoch 7-val Loss: 0.0881 Acc: 0.2353
(RayTrainWorker pid=590, ip=10.108.0.72) Checkpoint successfully created at: Checkpoint(filesystem=local, path=/mnt/cluster_storage/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000007)
(RayTrainWorker pid=590, ip=10.108.0.72) Moving model to device: cuda:0 [repeated 3x across cluster]
(RayTrainWorker pid=590, ip=10.108.0.72) Wrapping provided model in DistributedDataParallel. [repeated 3x across cluster]
Training finished iteration 8 at 2024-04-29 17:27:29. Total running time: 54s
╭─────────────────────────────────────────╮
│ Training result │
├─────────────────────────────────────────┤
│ checkpoint_dir_name checkpoint_000007 │
│ time_this_iter_s 40.46113 │
│ time_total_s 95.00043 │
│ training_iteration 8 │
│ acc 0.23529 │
│ loss 0.08811 │
╰─────────────────────────────────────────╯
Training saved a checkpoint for iteration 8 at: (local)/mnt/cluster_storage/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000007
(RayTrainWorker pid=671, ip=10.108.2.65) Epoch 8-train Loss: 0.0893 Acc: 0.2459
(RayTrainWorker pid=671, ip=10.108.2.65) Epoch 8-val Loss: 0.0859 Acc: 0.2353
(RayTrainWorker pid=589, ip=10.108.1.83) Checkpoint successfully created at: Checkpoint(filesystem=local, path=/mnt/cluster_storage/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000008) [repeated 4x across cluster]
Training finished iteration 9 at 2024-04-29 17:27:36. Total running time: 1min 1s
╭─────────────────────────────────────────╮
│ Training result │
├─────────────────────────────────────────┤
│ checkpoint_dir_name checkpoint_000008 │
│ time_this_iter_s 5.99923 │
│ time_total_s 100.99965 │
│ training_iteration 9 │
│ acc 0.23529 │
│ loss 0.08592 │
╰─────────────────────────────────────────╯
Training saved a checkpoint for iteration 9 at: (local)/mnt/cluster_storage/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000008
2024-04-29 17:27:37,170 WARNING util.py:202 -- The `process_trial_save` operation took 0.540 s, which may be a performance bottleneck.
(RayTrainWorker pid=671, ip=10.108.2.65) Epoch 9-train Loss: 0.0866 Acc: 0.2377
(RayTrainWorker pid=671, ip=10.108.2.65) Epoch 9-val Loss: 0.0833 Acc: 0.2353
(RayTrainWorker pid=589, ip=10.108.1.83) Checkpoint successfully created at: Checkpoint(filesystem=local, path=/mnt/cluster_storage/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000009) [repeated 4x across cluster]
Training finished iteration 10 at 2024-04-29 17:27:43. Total running time: 1min 8s
╭─────────────────────────────────────────╮
│ Training result │
├─────────────────────────────────────────┤
│ checkpoint_dir_name checkpoint_000009 │
│ time_this_iter_s 6.71457 │
│ time_total_s 107.71422 │
│ training_iteration 10 │
│ acc 0.23529 │
│ loss 0.08333 │
╰─────────────────────────────────────────╯
Training saved a checkpoint for iteration 10 at: (local)/mnt/cluster_storage/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000009
Training completed after 10 iterations at 2024-04-29 17:27:45. Total running time: 1min 9s
2024-04-29 17:27:46,236 WARNING experiment_state.py:323 -- Experiment checkpoint syncing has been triggered multiple times in the last 30.0 seconds. A sync will be triggered whenever a trial has checkpointed more than `num_to_keep` times since last sync or if 300 seconds have passed since last sync. If you have set `num_to_keep` in your `CheckpointConfig`, consider increasing the checkpoint frequency or keeping more checkpoints. You can suppress this warning by changing the `TUNE_WARN_EXCESSIVE_EXPERIMENT_CHECKPOINT_SYNC_THRESHOLD_S` environment variable.
Result(
metrics={'loss': 0.08333033206416111, 'acc': 0.23529411764705882},
path='/mnt/cluster_storage/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29',
filesystem='local',
checkpoint=Checkpoint(filesystem=local, path=/mnt/cluster_storage/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000009)
)
(RayTrainWorker pid=590, ip=10.108.3.76) Checkpoint successfully created at: Checkpoint(filesystem=local, path=/mnt/cluster_storage/finetune-resnet/TorchTrainer_96923_00000_0_2024-04-29_17-21-29/checkpoint_000009) [repeated 3x across cluster]
```
doc/source/cluster/kubernetes/examples/mnist-training-example.md
+121
View File
@@ -0,0 +1,121 @@
(kuberay-mnist-training-example)=
# Train a PyTorch model on Fashion MNIST with CPUs on Kubernetes
This example runs distributed training of a PyTorch model on Fashion MNIST with Ray Train. See [Train a PyTorch model on Fashion MNIST](train-pytorch-fashion-mnist) for more details.
## Step 1: Create a Kubernetes cluster
This step creates a local Kubernetes cluster using [Kind](https://kind.sigs.k8s.io/). If you already have a Kubernetes cluster, you can skip this step.
```sh
kind create cluster --image=kindest/node:v1.26.0
```
## Step 2: Install KubeRay operator
Follow [this document](kuberay-operator-deploy) to install the latest stable KubeRay operator from the Helm repository.
## Step 3: Create a RayJob
A RayJob consists of a RayCluster custom resource and a job that can you can submit to the RayCluster. With RayJob, KubeRay creates a RayCluster and submits a job when the cluster is ready. The following is a CPU-only RayJob description YAML file for MNIST training on a PyTorch model.
```sh
# Download `ray-job.pytorch-mnist.yaml`
curl -LO https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/pytorch-mnist/ray-job.pytorch-mnist.yaml
```
You might need to adjust some fields in the RayJob description YAML file so that it can run in your environment:
* `replicas` under `workerGroupSpecs` in `rayClusterSpec`: This field specifies the number of worker Pods that KubeRay schedules to the Kubernetes cluster. Each worker Pod requires 3 CPUs, and the head Pod requires 1 CPU, as described in the `template` field. A RayJob submitter Pod requires 1 CPU. For example, if your machine has 8 CPUs, the maximum `replicas` value is 2 to allow all Pods to reach the `Running` status.
* `NUM_WORKERS` under `runtimeEnvYAML` in `spec`: This field indicates the number of Ray actors to launch (see `ScalingConfig` in this [Document](ray-train-configs-api) for more information). Each Ray actor must be served by a worker Pod in the Kubernetes cluster. Therefore, `NUM_WORKERS` must be less than or equal to `replicas`.
* `CPUS_PER_WORKER`: This must be set to less than or equal to `(CPU resource request per worker Pod) - 1`. For example, in the sample YAML file, the CPU resource request per worker Pod is 3 CPUs, so `CPUS_PER_WORKER` must be set to 2 or less.
```sh
# `replicas` and `NUM_WORKERS` set to 2.
# Create a RayJob.
kubectl apply -f ray-job.pytorch-mnist.yaml
# Check existing Pods: According to `replicas`, there should be 2 worker Pods.
# Make sure all the Pods are in the `Running` status.
kubectl get pods
# NAME READY STATUS RESTARTS AGE
# kuberay-operator-6dddd689fb-ksmcs 1/1 Running 0 6m8s
# rayjob-pytorch-mnist-raycluster-rkdmq-small-group-worker-c8bwx 1/1 Running 0 5m32s
# rayjob-pytorch-mnist-raycluster-rkdmq-small-group-worker-s7wvm 1/1 Running 0 5m32s
# rayjob-pytorch-mnist-nxmj2 1/1 Running 0 4m17s
# rayjob-pytorch-mnist-raycluster-rkdmq-head-m4dsl 1/1 Running 0 5m32s
```
Check that the RayJob is in the `RUNNING` status:
```sh
kubectl get rayjob
# NAME JOB STATUS DEPLOYMENT STATUS START TIME END TIME AGE
# rayjob-pytorch-mnist RUNNING Running 2024-06-17T04:08:25Z 11m
```
## Step 4: Wait until the RayJob completes and check the training results
Wait until the RayJob completes. It might take several minutes.
```sh
kubectl get rayjob
# NAME JOB STATUS DEPLOYMENT STATUS START TIME END TIME AGE
# rayjob-pytorch-mnist SUCCEEDED Complete 2024-06-17T04:08:25Z 2024-06-17T04:22:21Z 16m
```
After seeing `JOB_STATUS` marked as `SUCCEEDED`, you can check the training logs:
```sh
# Check Pods name.
kubectl get pods
# NAME READY STATUS RESTARTS AGE
# kuberay-operator-6dddd689fb-ksmcs 1/1 Running 0 113m
# rayjob-pytorch-mnist-raycluster-rkdmq-small-group-worker-c8bwx 1/1 Running 0 38m
# rayjob-pytorch-mnist-raycluster-rkdmq-small-group-worker-s7wvm 1/1 Running 0 38m
# rayjob-pytorch-mnist-nxmj2 0/1 Completed 0 38m
# rayjob-pytorch-mnist-raycluster-rkdmq-head-m4dsl 1/1 Running 0 38m
# Check training logs.
kubectl logs -f rayjob-pytorch-mnist-nxmj2
# 2024-06-16 22:23:01,047 INFO cli.py:36 -- Job submission server address: http://rayjob-pytorch-mnist-raycluster-rkdmq-head-svc.default.svc.cluster.local:8265
# 2024-06-16 22:23:01,844 SUCC cli.py:60 -- -------------------------------------------------------
# 2024-06-16 22:23:01,844 SUCC cli.py:61 -- Job 'rayjob-pytorch-mnist-l6ccc' submitted successfully
# 2024-06-16 22:23:01,844 SUCC cli.py:62 -- -------------------------------------------------------
# ...
# (RayTrainWorker pid=1138, ip=10.244.0.18)
# 0%| | 0/26421880 [00:00<?, ?it/s]
# (RayTrainWorker pid=1138, ip=10.244.0.18)
# 0%| | 32768/26421880 [00:00<01:27, 301113.97it/s]
# ...
# Training finished iteration 10 at 2024-06-16 22:33:05. Total running time: 7min 9s
# ╭───────────────────────────────╮
# │ Training result │
# ├───────────────────────────────┤
# │ checkpoint_dir_name │
# │ time_this_iter_s 28.2635 │
# │ time_total_s 423.388 │
# │ training_iteration 10 │
# │ accuracy 0.8748 │
# │ loss 0.35477 │
# ╰───────────────────────────────╯
# Training completed after 10 iterations at 2024-06-16 22:33:06. Total running time: 7min 10s
# Training result: Result(
# metrics={'loss': 0.35476621258825347, 'accuracy': 0.8748},
# path='/home/ray/ray_results/TorchTrainer_2024-06-16_22-25-55/TorchTrainer_122aa_00000_0_2024-06-16_22-25-55',
# filesystem='local',
# checkpoint=None
# )
# ...
```
## Clean up
Delete your RayJob with the following command:
```sh
kubectl delete -f ray-job.pytorch-mnist.yaml
```
doc/source/cluster/kubernetes/examples/mobilenet-rayservice.md
+52
View File
@@ -0,0 +1,52 @@
(kuberay-mobilenet-rayservice-example)=
# Serve a MobileNet image classifier on Kubernetes
> **Note:** The Python files for the Ray Serve application and its client are in the repository [ray-project/serve_config_examples](https://github.com/ray-project/serve_config_examples).
## Step 1: Create a Kubernetes cluster with Kind
```sh
kind create cluster --image=kindest/node:v1.26.0
```
## Step 2: Install KubeRay operator
Follow [this document](kuberay-operator-deploy) to install the latest stable KubeRay operator from the Helm repository. Note that the YAML file in this example uses `serveConfigV2`. You need KubeRay version v0.6.0 or later to use this feature.
## Step 3: Install a RayService
```sh
# Create a RayService
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-service.mobilenet.yaml
```
* The [mobilenet.py](https://github.com/ray-project/serve_config_examples/blob/master/mobilenet/mobilenet.py) file needs `tensorflow` as a dependency. Hence, the YAML file uses `rayproject/ray-ml` image instead of `rayproject/ray` image.
* The request parsing function `starlette.requests.form()` needs `python-multipart`, so the YAML file includes `python-multipart` in the runtime environment.
## Step 4: Forward the port for Ray Serve
```sh
# Wait for the RayService to be ready to serve requests
kubectl describe rayservice/rayservice-mobilenet
# Conditions:
# Last Transition Time: 2025-02-13T02:29:26Z
# Message: Number of serve endpoints is greater than 0
# Observed Generation: 1
# Reason: NonZeroServeEndpoints
# Status: True
# Type: Ready
# Forward the port for Ray Serve service
kubectl port-forward svc/rayservice-mobilenet-serve-svc 8000
```
## Step 5: Send a request to the ImageClassifier
* Step 5.1: Prepare an image file.
* Step 5.2: Update `image_path` in [mobilenet_req.py](https://github.com/ray-project/serve_config_examples/blob/master/mobilenet/mobilenet_req.py)
* Step 5.3: Send a request to the `ImageClassifier`.
```sh
python mobilenet_req.py
# sample output: {"prediction":["n02099601","golden_retriever",0.17944198846817017]}
```
doc/source/cluster/kubernetes/examples/rayjob-agent-sandbox.md
+170
View File
@@ -0,0 +1,170 @@
(kuberay-agent-sandbox)=
# Sandboxed Code Execution with Ray and Agent Sandbox
This example shows how to use the [Agent Sandbox](https://github.com/kubernetes-sigs/agent-sandbox) with Ray and KubeRay to orchestrate code execution in a secure sandboxed environment. This example uses GKE and gVisor but can be modified to work on other sandbox runtimes.
---
## What is Agent Sandbox?
[Agent Sandbox](https://github.com/kubernetes-sigs/agent-sandbox) is a Kubernetes project to streamline the management of sandboxes on Kubernetes. Agent sandbox provides declarative Kubernetes APIs that can be used with KubeRay to manage sandbox environments that can be invoked from a Ray cluster.
Agent sandbox is compatible with multiple runtimes that offer strong isolation guarantees such as [gVisor](https://github.com/google/gvisor) and [Kata containers](https://github.com/kata-containers/kata-containers). Consider using Ray and Agent Sandbox for agentic RL use-cases where you need to securely execute code generated from a model during its post-training phase.
Agent Sandbox provides a collection of declarative Kubernetes APIs to easily manage Sandbox runtimes. This example uses the following custom resources provided by Agent Sandbox:
- `Sandbox`: This is the foundational unit, it manages a single Pod with a stable hostname and network identity. Unlike standard Pods, a Sandbox can be configured with persistent storage via volumeClaimTemplates that survives restarts
- `SandboxClaim`: Allows users to create `Sandboxes` from a `SandboxTemplate`, abstracting away the details of the underlying Sandbox configuration.
- `SandboxTemplate`: Provides a way to define reusable templates for creating `Sandboxes`, making it easier to manage large numbers of similar `Sandboxes`.
- `SandboxWarmPool`: Manages a pool of pre-warmed `Sandboxes` that can be quickly (<200ms) allocated to users, reducing the time it takes to get a new Sandbox up and running.
The Agent Sandbox project also provides a [Python SDK](https://github.com/kubernetes-sigs/agent-sandbox/tree/main/clients/python/agentic-sandbox-client) which can be used from within Ray actors to invoke Sandbox creation and secure code execution on sandboxes.
## Deploying KubeRay with Agent Sandbox
The following example creates a KubeRay RayJob, which runs a Ray job that uses the Agent Sandbox SDK to invoke code execution in a secure sandbox. It is highly recommended to keep Pods used for sandboxing decoupled from the Ray cluster itself.
### Step 1: Create a GKE cluster and Node Pools
Run the following command to create a GKE cluster. In this example we will create two separate node pools, one for KubeRay provisioned Pods and one for Sandbox pods using the gVisor runtime:
```bash
gcloud container node-pools create ray-worker-pool \
--cluster=<YOUR_CLUSTER_NAME> \
--machine-type=e2-standard-4 \
--num-nodes=2
gcloud container node-pools create ray-gvisor-pool \
--cluster=<YOUR_CLUSTER_NAME> \
--sandbox type=gvisor \
--machine-type=e2-standard-4 \
--num-nodes=1
```
### Step 2: Install KubeRay operator
Follow the instructions in [KubeRay operator](kuberay-operator-deploy) to install the KubeRay operator.
### Step 3: Deploy Agent Sandbox
Install the Custom Resource Definitions (CRDs), controllers, and extensions from the official Agent Sandbox release.
```bash
export VERSION="v0.5.0"
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${VERSION}/manifest.yaml
kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/${VERSION}/extensions.yaml
```
### Step 4: Apply RBAC Permissions for the Ray Workers
The Agent Sandbox Python SDK running inside Ray Workers needs to talk to the Kubernetes API to claim and delete sandboxes. In this example we will use the default service account token in the default namespace to grant Ray workers the ability to spawn Sandboxes:
Create a file named `rbac.yaml` with the following content:
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: ray-sandbox-manager-role
namespace: default
rules:
- apiGroups: ["extensions.agents.x-k8s.io"]
resources: ["sandboxclaims"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["agents.x-k8s.io"]
resources: ["sandboxes"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: ray-sandbox-manager-binding
namespace: default
subjects:
- kind: ServiceAccount
name: default
namespace: default
roleRef:
kind: Role
name: ray-sandbox-manager-role
apiGroup: rbac.authorization.k8s.io
```
Apply the RBAC configurations:
```bash
kubectl apply -f rbac.yaml
```
### Step 5: Deploy Sandbox Infrastructure
Run the following command to create sandbox infrastructure using Agent Sandbox:
```bash
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/agent-sandbox/sandbox.yaml
```
The following resources are created:
- **`SandboxTemplate`**: defines the per-sandbox podSpec. The Pod is configured to use gVisor, sets `automountServiceAccountToken: false` so untrusted code inside the sandbox cannot read a Kubernetes ServiceAccount token, and sets `networkPolicyManagement: Unmanaged` because the NetworkPolicy below is stricter than the controller's Secure Default. The template also labels every sandbox pod with `app: python-runtime-pool` so other selectors (the NetworkPolicy podSelector, your own `kubectl get` queries) can target them by a stable, human-readable label.
- **`SandboxWarmPool`** (`python-runtime-pool`) — keeps 6 pre-booted sandbox pods ready so the Ray actors' claims complete in under 200ms.
- **`NetworkPolicy`** (`python-runtime-pool-restrict-egress`) — default-denies egress for every sandbox pod except DNS. This is what provides concrete containment, ensuring packets are dropped by the CNI at the node rather than relying on cluster-default policies.
Verify the warm pool pods are running:
```bash
kubectl get pods -l app=python-runtime-pool
```
Based on the configuration of the SandboxWarmpool, we expect 6 gVisor Pods to be running:
```bash
GVISOR_POD=$(kubectl get pod -l app=python-runtime-pool -o jsonpath='{.items[0].metadata.name}')
kubectl get pod "$GVISOR_POD" -o jsonpath='{.spec.automountServiceAccountToken}{"\n"}' # expect: false
kubectl exec "$GVISOR_POD" -- ls /var/run/secrets/kubernetes.io/serviceaccount/ 2>&1 # expect: No such file or directory
```
### Step 6: Create the RayJob
Run the following command to create a RayJob resource:
```sh
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/agent-sandbox/ray-cluster.yaml
```
The RayJob is configured to do the following:
1. Create a RayCluster
2. Submit a Ray job that runs `sandboxed_code_execution.py` on the Ray cluster
3. The driver script will run Ray actors that use the Agent Sandbox Python SDK to invoke Sandbox creation.
4. Once the Sandbox environments are created, the actor will execute some code in the sandboxed environment and verify its output.
### Step 7: Verify the output
Monitor the status and query the execution logs of the submitted RayJob:
```sh
# List running job pods
kubectl get pods -l job-name=agent-sandbox-code-execution-demo
# Stream the demo logs
kubectl logs -f -l job-name=agent-sandbox-code-execution-demo
```
Once the job starts, two `SandboxExecutor` Ray actors each claim one pod from `python-runtime-pool` (the SDK reports per-actor adoption latency — sub-200ms when the warm pool is healthy). Every Python snippet that follows runs **inside the sandbox pod, never on the Ray worker**: gVisor isolates the syscall surface, the `python-runtime-pool-restrict-egress` NetworkPolicy applied in Step 5 default-denies all egress except DNS, and `sandbox.commands.run(..., timeout=5)` bounds wall-clock blast radius per call.
Expected output (abridged):
```
Starting 2 SandboxExecutors...
Dispatching 2 code executors...
(SandboxExecutor pid=457, ip=10.72.5.24) [executor-1] claimed sandbox 'sandbox-claim-6d4504d8' in 0.257s
--- Execution Results ---
[compute_fib.py] (Exit Code: 0)
Stdout: fib(20) = 6765
[json_aggregation.py] (Exit Code: 0)
Stdout: {"mean": 11.0, "max": 25}
Cleaning up sandboxes...
(SandboxExecutor pid=342, ip=10.72.1.10) [executor-0] claimed sandbox 'sandbox-claim-3a93b626' in 0.212s
```
doc/source/cluster/kubernetes/examples/rayjob-batch-inference-example.md
+137
View File
@@ -0,0 +1,137 @@
(kuberay-batch-inference-example)=
# RayJob Batch Inference Example
This example demonstrates how to use the RayJob custom resource to run a batch inference job for an image classification workload on a Ray cluster. See [Image Classification Batch Inference with HuggingFace Vision Transformer](https://docs.ray.io/en/latest/data/examples/huggingface_vit_batch_prediction.html) for a full explanation of the code.
## Prerequisites
You must have a Kubernetes cluster running,`kubectl` configured to use it, and GPUs available. This example provides a brief tutorial for setting up the necessary GPUs on Google Kubernetes Engine (GKE), but you can use any Kubernetes cluster with GPUs.
## Step 0: Create a Kubernetes cluster on GKE (Optional)
If you already have a Kubernetes cluster with GPUs, you can skip this step.
Otherwise, follow [this tutorial](kuberay-gke-gpu-cluster-setup), but substitute the following GPU node pool creation command to create a Kubernetes cluster on GKE with four NVIDIA T4 GPUs:
```sh
gcloud container node-pools create gpu-node-pool \
--accelerator type=nvidia-tesla-t4,count=4,gpu-driver-version=default \
--zone us-west1-b \
--cluster kuberay-gpu-cluster \
--num-nodes 1 \
--min-nodes 0 \
--max-nodes 1 \
--enable-autoscaling \
--machine-type n1-standard-64
```
This example uses four [NVIDIA T4](https://cloud.google.com/compute/docs/gpus#nvidia_t4_gpus) GPUs. The machine type is `n1-standard-64`, which has [64 vCPUs and 240 GB RAM](https://cloud.google.com/compute/docs/general-purpose-machines#n1_machine_types).
## Step 1: Install the KubeRay Operator
Follow [this document](kuberay-operator-deploy) to install the latest stable KubeRay operator from the Helm repository. The KubeRay operator Pod must be on the CPU node if you have set up the taint for the GPU node pool correctly.
## Step 2: Submit the RayJob
Create the RayJob custom resource with [ray-job.batch-inference.yaml](https://github.com/ray-project/kuberay/blob/v1.6.0/ray-operator/config/samples/ray-job.batch-inference.yaml).
Download the file with `curl`:
```bash
curl -LO https://raw.githubusercontent.com/ray-project/kuberay/v1.6.0/ray-operator/config/samples/ray-job.batch-inference.yaml
```
Note that the `RayJob` spec contains a spec for the `RayCluster`. This tutorial uses a single-node cluster with 4 GPUs. For production use cases, use a multi-node cluster where the head node doesn't have GPUs, so that Ray can automatically schedule GPU workloads on worker nodes which won't interfere with critical Ray processes on the head node.
Note the following fields in the `RayJob` spec, which specify the Ray image and the GPU resources for the Ray node:
```yaml
spec:
containers:
- name: ray-head
image: rayproject/ray-ml:2.6.3-gpu
resources:
limits:
nvidia.com/gpu: "4"
cpu: "54"
memory: "54Gi"
requests:
nvidia.com/gpu: "4"
cpu: "54"
memory: "54Gi"
volumeMounts:
- mountPath: /home/ray/samples
name: code-sample
nodeSelector:
cloud.google.com/gke-accelerator: nvidia-tesla-t4 # This is the GPU type we used in the GPU node pool.
```
To submit the job, run the following command:
```bash
kubectl apply -f ray-job.batch-inference.yaml
```
Check the status with `kubectl describe rayjob rayjob-sample`.
Sample output:
```
[...]
Status:
Dashboard URL: rayjob-sample-raycluster-j6t8n-head-svc.default.svc.cluster.local:8265
End Time: ...
Job Deployment Status: Complete
Job Id: rayjob-sample-ft8lh
Job Status: SUCCEEDED
Message: Job finished successfully.
Observed Generation: 2
...
```
To view the logs, first find the name of the pod running the job with `kubectl get pods`.
Sample output:
```bash
NAME READY STATUS RESTARTS AGE
kuberay-operator-8b86754c-r4rc2 1/1 Running 0 25h
rayjob-sample-raycluster-j6t8n-head-kx2gz 1/1 Running 0 35m
rayjob-sample-w98c7 0/1 Completed 0 30m
```
The Ray cluster is still running because `shutdownAfterJobFinishes` isn't set in the `RayJob` spec. If you set `shutdownAfterJobFinishes` to `true`, the cluster is shut down after the job finishes.
Next, run:
```text
kubectl logs rayjob-sample-w98c7
```
to get the standard output of the `entrypoint` command for the `RayJob`. Sample output:
```text
[...]
Running: 62.0/64.0 CPU, 4.0/4.0 GPU, 955.57 MiB/12.83 GiB object_store_memory: 0%| | 0/200 [00:05<?, ?it/s]
Running: 61.0/64.0 CPU, 4.0/4.0 GPU, 999.41 MiB/12.83 GiB object_store_memory: 0%| | 0/200 [00:05<?, ?it/s]
Running: 61.0/64.0 CPU, 4.0/4.0 GPU, 999.41 MiB/12.83 GiB object_store_memory: 0%| | 1/200 [00:05<17:04, 5.15s/it]
Running: 61.0/64.0 CPU, 4.0/4.0 GPU, 1008.68 MiB/12.83 GiB object_store_memory: 0%| | 1/200 [00:05<17:04, 5.15s/it]
Running: 61.0/64.0 CPU, 4.0/4.0 GPU, 1008.68 MiB/12.83 GiB object_store_memory: 100%|██████████| 1/1 [00:05<00:00, 5.15s/it]
2023-08-22 15:48:33,905 WARNING actor_pool_map_operator.py:267 -- To ensure full parallelization across an actor pool of size 4, the specified batch size should be at most 5. Your configured batch size for this operator was 16.
<PIL.Image.Image image mode=RGB size=500x375 at 0x7B37546CF7F0>
Label: tench, Tinca tinca
<PIL.Image.Image image mode=RGB size=500x375 at 0x7B37546AE430>
Label: tench, Tinca tinca
<PIL.Image.Image image mode=RGB size=500x375 at 0x7B37546CF430>
Label: tench, Tinca tinca
<PIL.Image.Image image mode=RGB size=500x375 at 0x7B37546AE430>
Label: tench, Tinca tinca
<PIL.Image.Image image mode=RGB size=500x375 at 0x7B37546CF7F0>
Label: tench, Tinca tinca
2023-08-22 15:48:36,522 SUCC cli.py:33 -- -----------------------------------
2023-08-22 15:48:36,522 SUCC cli.py:34 -- Job 'rayjob-sample-ft8lh' succeeded
2023-08-22 15:48:36,522 SUCC cli.py:35 -- -----------------------------------
```
doc/source/cluster/kubernetes/examples/rayjob-kueue-gang-scheduling.md
+225
View File
@@ -0,0 +1,225 @@
(kuberay-kueue-gang-scheduling-example)=
# Gang Scheduling with RayJob and Kueue
This guide demonstrates how to use Kueue for gang scheduling RayJob resources, taking advantage of dynamic resource provisioning and queueing on Kubernetes. To illustrate the concepts, this guide uses the [Fine-tune a PyTorch Lightning Text Classifier with Ray Data](https://docs.ray.io/en/master/train/examples/lightning/lightning_cola_advanced.html) example.
## Gang scheduling
Gang scheduling in Kubernetes ensures that a group of related Pods, such as those in a Ray cluster, only start when all required resources are available. Having this requirement is crucial when working with expensive, limited resources like GPUs.
## Kueue
[Kueue](https://kueue.sigs.k8s.io/) is a Kubernetes-native system that manages quotas and how jobs consume them. Kueue decides when:
* To make a job wait.
* To admit a job to start, which triggers Kubernetes to create Pods.
* To preempt a job, which triggers Kubernetes to delete active Pods.
Kueue has native support for some KubeRay APIs. Specifically, you can use Kueue to manage resources that RayJob, RayCluster, and RayService consume. See the [Kueue documentation](https://kueue.sigs.k8s.io/docs/overview/) to learn more.
## Why use gang scheduling
Gang scheduling is essential when working with expensive, limited hardware accelerators like GPUs. It prevents RayJobs from partially provisioning Ray clusters and claiming but not using the GPUs. Kueue suspends a RayJob until the Kubernetes cluster and the underlying cloud provider can guarantee the capacity that the RayJob needs to execute. This approach greatly improves GPU utilization and cost, especially when GPU availability is limited.
## Create a Kubernetes cluster on GKE
Create a GKE cluster with the `enable-autoscaling` option:
```bash
gcloud container clusters create kuberay-gpu-cluster \
--num-nodes=1 --min-nodes 0 --max-nodes 1 --enable-autoscaling \
--zone=us-east4-c --machine-type e2-standard-4
```
Create a GPU node pool with the `enable-queued-provisioning` option enabled:
```bash
gcloud container node-pools create gpu-node-pool \
--accelerator type=nvidia-l4,count=1,gpu-driver-version=latest \
--enable-queued-provisioning \
--reservation-affinity=none \
--zone us-east4-c \
--cluster kuberay-gpu-cluster \
--num-nodes 0 \
--min-nodes 0 \
--max-nodes 10 \
--enable-autoscaling \
--machine-type g2-standard-4
```
This command creates a node pool, which initially has zero nodes. The `--enable-queued-provisioning` flag enables "queued provisioning" in the Kubernetes node autoscaler using the ProvisioningRequest API. More details are below. You need to use the `--reservation-affinity=none` flag because GKE doesn't support Node Reservations with ProvisioningRequest.
## Install the KubeRay operator
Follow [Deploy a KubeRay operator](kuberay-operator-deploy) to install the latest stable KubeRay operator from the Helm repository. The KubeRay operator Pod must be on the CPU node if you set up the taint for the GPU node pool correctly.
## Install Kueue
Install the latest released version of Kueue.
```
kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.13.4/manifests.yaml
```
See [Kueue Installation](https://kueue.sigs.k8s.io/docs/installation/#install-a-released-version) for more details on installing Kueue.
## Configure Kueue for gang scheduling
Next, configure Kueue for gang scheduling. Kueue leverages the ProvisioningRequest API for two key tasks:
1. Dynamically adding new nodes to the cluster when a job needs more resources.
2. Blocking the admission of new jobs that are waiting for sufficient resources to become available.
See [How ProvisioningRequest works](https://cloud.google.com/kubernetes-engine/docs/how-to/provisioningrequest#how-provisioningrequest-works) for more details.
### Create Kueue resources
This manifest creates the following resources:
* [ClusterQueue](https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/): Defines quotas and fair sharing rules
* [LocalQueue](https://kueue.sigs.k8s.io/docs/concepts/local_queue/): A namespaced queue, belonging to a tenant, that references a ClusterQueue
* [ResourceFlavor](https://kueue.sigs.k8s.io/docs/concepts/resource_flavor/): Defines what resources are available in the cluster, typically from Nodes
* [AdmissionCheck](https://kueue.sigs.k8s.io/docs/concepts/admission_check/): A mechanism allowing components to influence the timing of a workload admission
```yaml
# kueue-resources.yaml
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
name: "default-flavor"
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: AdmissionCheck
metadata:
name: rayjob-gpu
spec:
controllerName: kueue.x-k8s.io/provisioning-request
parameters:
apiGroup: kueue.x-k8s.io
kind: ProvisioningRequestConfig
name: rayjob-gpu-config
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ProvisioningRequestConfig
metadata:
name: rayjob-gpu-config
spec:
provisioningClassName: queued-provisioning.gke.io
managedResources:
- nvidia.com/gpu
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
name: "cluster-queue"
spec:
namespaceSelector: {} # match all
resourceGroups:
- coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
flavors:
- name: "default-flavor"
resources:
- name: "cpu"
nominalQuota: 10000 # infinite quotas
- name: "memory"
nominalQuota: 10000Gi # infinite quotas
- name: "nvidia.com/gpu"
nominalQuota: 10000 # infinite quotas
admissionChecks:
- rayjob-gpu
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
namespace: "default"
name: "user-queue"
spec:
clusterQueue: "cluster-queue"
```
Create the Kueue resources:
```bash
kubectl apply -f kueue-resources.yaml
```
:::{note}
This example configures Kueue to orchestrate the gang scheduling of GPUs. However, you can use other resources such as CPU and memory.
:::
## Deploy a RayJob
Download the RayJob that executes all the steps documented in [Fine-tune a PyTorch Lightning Text Classifier](https://docs.ray.io/en/master/train/examples/lightning/lightning_cola_advanced.html). The [source code](https://github.com/ray-project/kuberay/tree/master/ray-operator/config/samples/pytorch-text-classifier) is also in the KubeRay repository.
```bash
curl -LO https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/pytorch-text-classifier/ray-job.pytorch-distributed-training.yaml
```
Before creating the RayJob, modify the RayJob metadata with a label to assign the RayJob to the LocalQueue that you created earlier:
```yaml
metadata:
generateName: pytorch-text-classifier-
labels:
kueue.x-k8s.io/queue-name: user-queue
```
Deploy the RayJob:
```bash
$ kubectl create -f ray-job.pytorch-distributed-training.yaml
rayjob.ray.io/dev-pytorch-text-classifier-r6d4p created
```
## Gang scheduling with RayJob
Following is the expected behavior when you deploy a GPU-requiring RayJob to a cluster that initially lacks GPUs:
* Kueue suspends the RayJob due to insufficient GPU resources in the cluster.
* Kueue creates a ProvisioningRequest, specifying the GPU requirements for the RayJob.
* The Kubernetes node autoscaler monitors ProvisioningRequests and adds nodes with GPUs as needed.
* Once the required GPU nodes are available, the ProvisioningRequest is satisfied.
* Kueue admits the RayJob, allowing Kubernetes to schedule the Ray nodes on the newly provisioned nodes, and the RayJob execution begins.
If GPUs are unavailable, Kueue keeps suspending the RayJob. In addition, the node autoscaler avoids provisioning new nodes until it can fully satisfy the RayJob's GPU requirements.
Upon creating a RayJob, notice that the RayJob status is immediately `suspended` despite the ClusterQueue having GPU quotas available.
```bash
$ kubectl get rayjob pytorch-text-classifier-rj4sg -o yaml
apiVersion: ray.io/v1
kind: RayJob
metadata:
name: pytorch-text-classifier-rj4sg
labels:
kueue.x-k8s.io/queue-name: user-queue
...
...
...
status:
jobDeploymentStatus: Suspended # RayJob suspended
jobId: pytorch-text-classifier-rj4sg-pj9hx
jobStatus: PENDING
```
Kueue keeps suspending this RayJob until its corresponding ProvisioningRequest is satisfied. List ProvisioningRequest resources and their status with this command:
```bash
$ kubectl get provisioningrequest
NAME ACCEPTED PROVISIONED FAILED AGE
rayjob-pytorch-text-classifier-nv77q-e95ec-rayjob-gpu-1 True False False 22s
```
Note the two columns in the output: `ACCEPTED` and `PROVISIONED`. `ACCEPTED=True` means that Kueue and the Kubernetes node autoscaler have acknowledged the request. `PROVISIONED=True` means that the Kubernetes node autoscaler has completed provisioning nodes. Once both of these conditions are true, the ProvisioningRequest is satisfied.
```bash
$ kubectl get provisioningrequest
NAME ACCEPTED PROVISIONED FAILED AGE
rayjob-pytorch-text-classifier-nv77q-e95ec-rayjob-gpu-1 True True False 57s
```
Because the example RayJob requires 1 GPU for fine-tuning, the ProvisioningRequest is satisfied by the addition of a single GPU node in the `gpu-node-pool` Node Pool.
```bash
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-kuberay-gpu-cluster-default-pool-8d883840-fd6d Ready <none> 14m v1.29.0-gke.1381000
gke-kuberay-gpu-cluster-gpu-node-pool-b176212e-g3db Ready <none> 46s v1.29.0-gke.1381000 # new node with GPUs
```
Once the ProvisioningRequest is satisfied, Kueue admits the RayJob. The Kubernetes scheduler then immediately places the head and worker nodes onto the newly provisioned resources. The ProvisioningRequest ensures a seamless Ray cluster start up, with no scheduling delays for any Pods.
```bash
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
pytorch-text-classifier-nv77q-g6z57 1/1 Running 0 13s
torch-text-classifier-nv77q-raycluster-gstrk-head-phnfl 1/1 Running 0 6m43s
```
doc/source/cluster/kubernetes/examples/rayjob-kueue-priority-scheduling.md
+250
View File
@@ -0,0 +1,250 @@
(kuberay-kueue-priority-scheduling-example)=
# Priority Scheduling with RayJob and Kueue
This guide shows how to run [Fine-tune a PyTorch Lightning Text Classifier with Ray Data](https://docs.ray.io/en/master/train/examples/lightning/lightning_cola_advanced.html) example as a RayJob and leverage Kueue to orchestrate priority scheduling and quota management.
## What's Kueue?
[Kueue](https://kueue.sigs.k8s.io/) is a Kubernetes-native job queueing system that manages quotas and how jobs consume them. Kueue decides when:
* To make a job wait
* To admit a job to start, meaning that Kubernetes creates pods.
* To preempt a job, meaning that Kubernetes deletes active pods.
Kueue has native support for some KubeRay APIs. Specifically, you can use Kueue to manage resources consumed by RayJob and RayCluster. See the [Kueue documentation](https://kueue.sigs.k8s.io/docs/overview/) to learn more.
## Step 0: Create a Kubernetes cluster on GKE (Optional)
If you already have a Kubernetes cluster with GPUs, you can skip this step. Otherwise, follow [Start Google Cloud GKE Cluster with GPUs for KubeRay](kuberay-gke-gpu-cluster-setup) to set up a Kubernetes cluster on GKE.
## Step 1: Install the KubeRay operator
Follow [Deploy a KubeRay operator](kuberay-operator-deploy) to install the latest stable KubeRay operator from the Helm repository. The KubeRay operator Pod must be on the CPU node if you set up the taint for the GPU node pool correctly.
## Step 2: Install Kueue
```bash
VERSION=v0.13.4
kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/$VERSION/manifests.yaml
```
See [Kueue Installation](https://kueue.sigs.k8s.io/docs/installation/#install-a-released-version) for more details on installing Kueue.
## Step 3: Configure Kueue with priority scheduling
To understand this tutorial, it's important to understand the following Kueue concepts:
* [ResourceFlavor](https://kueue.sigs.k8s.io/docs/concepts/resource_flavor/)
* [ClusterQueue](https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/)
* [LocalQueue](https://kueue.sigs.k8s.io/docs/concepts/local_queue/)
* [WorkloadPriorityClass](https://kueue.sigs.k8s.io/docs/concepts/workload_priority_class/)
```yaml
# kueue-resources.yaml
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
name: "default-flavor"
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
name: "cluster-queue"
spec:
preemption:
withinClusterQueue: LowerPriority
namespaceSelector: {} # Match all namespaces.
resourceGroups:
- coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
flavors:
- name: "default-flavor"
resources:
- name: "cpu"
nominalQuota: 2
- name: "memory"
nominalQuota: 8G
- name: "nvidia.com/gpu" # ClusterQueue only has quota for a single GPU.
nominalQuota: 1
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
namespace: "default"
name: "user-queue"
spec:
clusterQueue: "cluster-queue"
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: WorkloadPriorityClass
metadata:
name: prod-priority
value: 1000
description: "Priority class for prod jobs"
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: WorkloadPriorityClass
metadata:
name: dev-priority
value: 100
description: "Priority class for development jobs"
```
The YAML manifest configures:
* **ResourceFlavor**
* The ResourceFlavor `default-flavor` is an empty ResourceFlavor because the compute resources in the Kubernetes cluster are homogeneous. In other words, users can request 1 GPU without considering whether it's an NVIDIA A100 or a T4 GPU.
* **ClusterQueue**
* The ClusterQueue `cluster-queue` only has 1 ResourceFlavor `default-flavor` with quotas for 2 CPUs, 8G memory, and 1 GPU. It exactly matches the resources requested by 1 RayJob custom resource. ***Hence, only 1 RayJob can run at a time.***
* The ClusterQueue `cluster-queue` has a preemption policy `withinClusterQueue: LowerPriority`. This policy allows the pending RayJob that doesn’t fit within the nominal quota for its ClusterQueue to preempt active RayJob custom resources in the ClusterQueue that have lower priority.
* **LocalQueue**
* The LocalQueue `user-queue` is a namespaced object in the `default` namespace which belongs to a ClusterQueue. A typical practice is to assign a namespace to a tenant, team or user, of an organization. Users submit jobs to a LocalQueue, instead of to a ClusterQueue directly.
* **WorkloadPriorityClass**
* The WorkloadPriorityClass `prod-priority` has a higher value than the WorkloadPriorityClass `dev-priority`. This means that RayJob custom resources with the `prod-priority` priority class take precedence over RayJob custom resources with the `dev-priority` priority class.
Create the Kueue resources:
```bash
kubectl apply -f kueue-resources.yaml
```
## Step 4: Deploy a RayJob
Download the RayJob that executes all the steps documented in [Fine-tune a PyTorch Lightning Text Classifier](https://docs.ray.io/en/master/train/examples/lightning/lightning_cola_advanced.html). The [source code](https://github.com/ray-project/kuberay/tree/master/ray-operator/config/samples/pytorch-text-classifier) is also in the KubeRay repository.
```bash
curl -LO https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/pytorch-text-classifier/ray-job.pytorch-distributed-training.yaml
```
Before creating the RayJob, modify the RayJob metadata with:
```yaml
metadata:
generateName: dev-pytorch-text-classifier-
labels:
kueue.x-k8s.io/queue-name: user-queue
kueue.x-k8s.io/priority-class: dev-priority
```
* `kueue.x-k8s.io/queue-name: user-queue`: As the previous step mentioned, users submit jobs to a LocalQueue instead of directly to a ClusterQueue.
* `kueue.x-k8s.io/priority-class: dev-priority`: Assign the RayJob with the `dev-priority` WorkloadPriorityClass.
* A modified name to indicate that this job is for development.
Also note the resources required for this RayJob by looking at the resources that the Ray head Pod requests:
```yaml
resources:
limits:
memory: "8G"
nvidia.com/gpu: "1"
requests:
cpu: "2"
memory: "8G"
nvidia.com/gpu: "1"
```
Now deploy the RayJob:
```bash
$ kubectl create -f ray-job.pytorch-distributed-training.yaml
rayjob.ray.io/dev-pytorch-text-classifier-r6d4p created
```
Verify that the RayCluster and the submitter Kubernetes Job are running:
```bash
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
dev-pytorch-text-classifier-r6d4p-4nczg 1/1 Running 0 4s # Submitter Kubernetes Job
torch-text-classifier-r6d4p-raycluster-br45j-head-8bbwt 1/1 Running 0 34s # Ray head Pod
```
Delete the RayJob after verifying that the job has completed successfully.
```bash
$ kubectl get rayjobs.ray.io dev-pytorch-text-classifier-r6d4p -o jsonpath='{.status.jobStatus}'
SUCCEEDED
$ kubectl get rayjobs.ray.io dev-pytorch-text-classifier-r6d4p -o jsonpath='{.status.jobDeploymentStatus}'
Complete
$ kubectl delete rayjob dev-pytorch-text-classifier-r6d4p
rayjob.ray.io "dev-pytorch-text-classifier-r6d4p" deleted
```
## Step 5: Queuing multiple RayJob resources
Create 3 RayJob custom resources to see how Kueue interacts with KubeRay to implement job queueing.
```bash
$ kubectl create -f ray-job.pytorch-distributed-training.yaml
rayjob.ray.io/dev-pytorch-text-classifier-8vg2c created
$ kubectl create -f ray-job.pytorch-distributed-training.yaml
rayjob.ray.io/dev-pytorch-text-classifier-n5k89 created
$ kubectl create -f ray-job.pytorch-distributed-training.yaml
rayjob.ray.io/dev-pytorch-text-classifier-ftcs9 created
```
Because each RayJob requests 1 GPU and the ClusterQueue has quotas for only 1 GPU, Kueue automatically suspends new RayJob resources until GPU quotas become available.
You can also inspect the `ClusterQueue` to see available and used quotas:
```bash
$ kubectl get clusterqueue
NAME COHORT PENDING WORKLOADS
cluster-queue 2
$ kubectl get clusterqueue cluster-queue -o yaml
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
...
...
...
status:
admittedWorkloads: 1 # Workloads admitted by queue.
flavorsReservation:
- name: default-flavor
resources:
- borrowed: "0"
name: cpu
total: "8"
- borrowed: "0"
name: memory
total: 19531250Ki
- borrowed: "0"
name: nvidia.com/gpu
total: "2"
flavorsUsage:
- name: default-flavor
resources:
- borrowed: "0"
name: cpu
total: "8"
- borrowed: "0"
name: memory
total: 19531250Ki
- borrowed: "0"
name: nvidia.com/gpu
total: "2"
pendingWorkloads: 2 # Queued workloads waiting for quotas.
reservingWorkloads: 1 # Running workloads that are using quotas.
```
## Step 6: Deploy a RayJob with higher priority
At this point there are multiple RayJob custom resources queued up but only enough quota to run a single RayJob. Now you can create a new RayJob with higher priority to preempt the already queued RayJob resources. Modify the RayJob with:
```yaml
metadata:
generateName: prod-pytorch-text-classifier-
labels:
kueue.x-k8s.io/queue-name: user-queue
kueue.x-k8s.io/priority-class: prod-priority
```
* `kueue.x-k8s.io/queue-name: user-queue`: As the previous step mentioned, users submit jobs to a LocalQueue instead of directly to a ClusterQueue.
* `kueue.x-k8s.io/priority-class: dev-priority`: Assign the RayJob with the `prod-priority` WorkloadPriorityClass.
* A modified name to indicate that this job is for production.
Create the new RayJob:
```sh
$ kubectl create -f ray-job.pytorch-distributed-training.yaml
rayjob.ray.io/prod-pytorch-text-classifier-gkp9b created
```
Note that higher priority jobs preempt lower priority jobs when there aren't enough quotas for both:
```bash
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
prod-pytorch-text-classifier-gkp9b-r9k5r 1/1 Running 0 5s
torch-text-classifier-gkp9b-raycluster-s2f65-head-hfvht 1/1 Running 0 35s
```
doc/source/cluster/kubernetes/examples/rayserve-deepseek-example.md
+198
View File
@@ -0,0 +1,198 @@
(kuberay-rayservice-deepseek-example)=
# Serve Deepseek R1 using Ray Serve LLM
This guide provides a step-by-step guide for deploying a Large Language Model (LLM) using Ray Serve LLM on Kubernetes. Leveraging KubeRay, Ray Serve, and vLLM, this guide deploys the `deepseek-ai/DeepSeek-R1` model from Hugging Face, enabling scalable, efficient, and OpenAI-compatible LLM serving within a Kubernetes environment. See [Serving LLMs](serving-llms) for information on Ray Serve LLM.
## Prerequisites
A DeepSeek model requires 2 nodes, each equipped with 8 H100 80 GB GPUs. It should be deployable on Kubernetes clusters that meet this requirement. This guide provides instructions for setting up a GKE cluster using [A3 High](https://cloud.google.com/compute/docs/gpus#a3-high) or [A3 Mega](https://cloud.google.com/compute/docs/gpus#a3-mega) machine types.
Before creating the cluster, ensure that your project has sufficient [quota](https://console.cloud.google.com/iam-admin/quotas) for the required accelerators.
## Step 1: Create a Kubernetes cluster on GKE
Run this command and all following commands on your local machine or on the [Google Cloud Shell](https://cloud.google.com/shell). If running from your local machine, you need to install the [Google Cloud SDK](https://cloud.google.com/sdk/docs/install). The following command creates a Kubernetes cluster named `kuberay-gpu-cluster` with 1 default CPU node in the `us-east5-a` zone. This example uses the `e2-standard-16` machine type, which has 16 vCPUs and 64 GB memory.
```sh
gcloud container clusters create kuberay-gpu-cluster \
--location=us-east5-a \
--machine-type=e2-standard-16 \
--num-nodes=1 \
--enable-image-streaming
```
Run the following command to create an on-demand GPU node pool for Ray GPU workers.
```sh
gcloud beta container node-pools create gpu-node-pool \
--cluster kuberay-gpu-cluster \
--machine-type a3-highgpu-8g \
--num-nodes 2 \
--accelerator "type=nvidia-h100-80gb,count=8" \
--zone us-east5-a \
--node-locations us-east5-a \
--host-maintenance-interval=PERIODIC
```
The `--accelerator` flag specifies the type and number of GPUs for each node in the node pool. This example uses the [A3 High](https://cloud.google.com/compute/docs/gpus#a3-high) GPU. The machine type `a3-highgpu-8g` has 8 GPU, 640 GB GPU Memory, 208 vCPUs, and 1872 GB RAM.
```{admonition} Note
:class: note
To create a node pool that uses reservations, you can specify the following parameters:
* `--reservation-affinity=specific`
* `--reservation=RESERVATION_NAME`
* `--placement-policy=PLACEMENT_POLICY_NAME` (Optional)
```
Run the following `gcloud` command to configure `kubectl` to communicate with your cluster:
```sh
gcloud container clusters get-credentials kuberay-gpu-cluster --zone us-east5-a
```
## Step 2: Install the KubeRay operator
Install the most recent stable KubeRay operator from the Helm repository by following [Deploy a KubeRay operator](kuberay-operator-deploy). The Kubernetes `NoSchedule` taint in the example config prevents the KubeRay operator Pod from running on a GPU node.
## Step 3: Deploy a RayService
Deploy DeepSeek-R1 as a RayService custom resource by running the following command:
```sh
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-service.deepseek.yaml
```
This step sets up a custom Ray Serve application to serve the `deepseek-ai/DeepSeek-R1` model on two worker nodes. You can inspect and modify the `serveConfigV2` section in the YAML file to learn more about the Serve application:
```yaml
serveConfigV2: |
applications:
- args:
llm_configs:
- model_loading_config:
model_id: "deepseek"
model_source: "deepseek-ai/DeepSeek-R1"
accelerator_type: "H100"
deployment_config:
autoscaling_config:
min_replicas: 1
max_replicas: 1
runtime_env:
env_vars:
VLLM_USE_V1: "1"
engine_kwargs:
tensor_parallel_size: 8
pipeline_parallel_size: 2
gpu_memory_utilization: 0.92
dtype: "auto"
max_num_seqs: 40
max_model_len: 16384
enable_chunked_prefill: true
enable_prefix_caching: true
import_path: ray.serve.llm:build_openai_app
name: llm_app
route_prefix: "/"
```
In particular, this configuration loads the model from `deepseek-ai/DeepSeek-R1` and sets its `model_id` to `deepseek`. The `LLMDeployment` initializes the underlying LLM engine using the `engine_kwargs` field, which includes key performance tuning parameters:
- `tensor_parallel_size: 8`
This setting enables tensor parallelism, splitting individual large layers of the model across 8 GPUs. Adjust this variable according to the number of GPUs used by cluster nodes.
- `pipeline_parallel_size: 2`
This setting enables pipeline parallelism, dividing the model's entire set of layers into 2 sequential stages. Adjust this variable according to cluster worker node numbers.
The `deployment_config` section sets the desired number of engine replicas. See [Serving LLMs](serving-llms) and the [Ray Serve config documentation](serve-in-production-config-file) for more information.
Wait for the RayService resource to become healthy. You can confirm its status by running the following command:
```sh
kubectl get rayservice deepseek-r1 -o yaml
```
After a few minutes, the result should be similar to the following:
```
status:
activeServiceStatus:
applicationStatuses:
llm_app:
serveDeploymentStatuses:
LLMDeployment:deepseek:
status: HEALTHY
LLMRouter:
status: HEALTHY
status: RUNNING
```
```{admonition} Note
:class: note
The model download and deployment will typically take 20-30 minutes. While this is in progress, use the Ray Dashboard (Step 4) Cluster tab to monitor the download progress as disk fills up.
```
## Step 4: View the Ray dashboard
```sh
# Forward the service port
kubectl port-forward svc/deepseek-r1-head-svc 8265:8265
```
Once forwarded, navigate to the Serve tab on the dashboard to review application status, deployments, routers, logs, and other relevant features. ![LLM Serve Application](../images/ray_dashboard_deepseek.png)
## Step 5: Send a request
To send requests to the Ray Serve deployment, port-forward port 8000 from the Serve app service:
```sh
kubectl port-forward svc/deepseek-r1-serve-svc 8000
```
Note that this Kubernetes service comes up only after Ray Serve apps are running and ready.
Test the service with the following command:
```sh
$ curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "deepseek",
"messages": [
{
"role": "user",
"content": "I have four boxes. I put the red box on the bottom and put the blue box on top. Then I put the yellow box on top the blue. Then I take the blue box out and put it on top. And finally I put the green box on the top. Give me the final order of the boxes from bottom to top. Show your reasoning but be brief"}
],
"temperature": 0.7
}'
```
The output should be in the following format:
```
{
"id": "deepseek-653881a7-18f3-493b-a43f-adc8501f01f8",
"object": "chat.completion",
"created": 1753345252,
"model": "deepseek",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"reasoning_content": null,
"content": "Okay, let's break this down step by step. The user has four boxes: red, blue, yellow, and green. The starting point is putting the red box on the bottom. Then blue is placed on top of red. Next, yellow goes on top of blue. At this point, the order is red (bottom), blue, yellow. \n\nThen the instruction says to take the blue box out and put it on top. Wait, when they take the blue box out from where? The current stack is red, blue, yellow. If we remove blue from between red and yellow, that leaves red and yellow. Then placing blue on top would make the stack red, yellow, blue. But the problem is, when you remove a box from the middle, the boxes above it should fall down, right? So after removing blue, yellow would be on top of red. Then putting blue on top of that stack would make it red, yellow, blue.\n\nThen the final step is putting the green box on top. So the final order would be red (bottom), yellow, blue, green. Let me verify again to make sure I didn't miss anything. Start with red at bottom. Blue on top of red: red, blue. Yellow on top of blue: red, blue, yellow. Remove blue from the middle, so yellow moves down to be on red, then put blue on top: red, yellow, blue. Finally, add green on top: red, yellow, blue, green. Yes, that seems right.\n</think>\n\nThe final order from bottom to top is: red, yellow, blue, green.\n\n1. Start with red at the bottom. \n2. Add blue on top: red → blue. \n3. Add yellow on top: red → blue → yellow. \n4. **Remove blue** from between red and yellow; yellow drops to second position. Now: red → yellow. \n5. Place blue back on top: red → yellow → blue. \n6. Add green on top: red → yellow → blue → green.",
"tool_calls": []
},
"logprobs": null,
"finish_reason": "stop",
"stop_reason": null
}
],
"usage": {
"prompt_tokens": 81,
"total_tokens": 505,
"completion_tokens": 424,
"prompt_tokens_details": null
},
"prompt_logprobs": null
}
```
doc/source/cluster/kubernetes/examples/rayserve-llm-example.md
+164
View File
@@ -0,0 +1,164 @@
(kuberay-rayservice-llm-example)=
# Serve a Large Language Model using Ray Serve LLM on Kubernetes
This guide provides a step-by-step guide for deploying a Large Language Model (LLM) using Ray Serve LLM on Kubernetes. Leveraging KubeRay, Ray Serve, and vLLM, this guide deploys the `Qwen/Qwen2.5-7B-Instruct` model from Hugging Face, enabling scalable, efficient, and OpenAI-compatible LLM serving within a Kubernetes environment. See [Serving LLMs](serving-llms) for information on Ray Serve LLM.
## Prerequisites
This example downloads model weights from the [`Qwen/Qwen2.5-7B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) Hugging Face repository. To completely finish this guide, you must fulfill the following requirements:
* A [Hugging Face account](https://huggingface.co/) and a Hugging Face [access token](https://huggingface.co/settings/tokens) with read access to gated repositories.
* In your RayService custom resource, set the `HUGGING_FACE_HUB_TOKEN` environment variable to the Hugging Face token to enable model downloads.
* A Kubernetes cluster with GPUs.
## Step 1: Create a Kubernetes cluster with GPUs
Refer to the Kubernetes cluster setup [instructions](../user-guides/k8s-cluster-setup.md) for guides on creating a Kubernetes cluster.
## Step 2: Install the KubeRay operator
Install the most recent stable KubeRay operator from the Helm repository by following [Deploy a KubeRay operator](../getting-started/kuberay-operator-installation.md). The Kubernetes `NoSchedule` taint in the example config prevents the KubeRay operator pod from running on a GPU node.
## Step 3: Create a Kubernetes Secret containing your Hugging Face access token
For additional security, instead of passing the HF access token directly as an environment variable, create a Kubernetes secret containing your Hugging Face access token. Download the Ray Serve LLM service config .yaml file using the following command:
```sh
curl -o ray-service.llm-serve.yaml https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-service.llm-serve.yaml
```
After downloading, update the value for `hf_token` to your private access token in the `Secret`.
```yaml
apiVersion: v1
kind: Secret
metadata:
name: hf-token
type: Opaque
stringData:
hf_token: <your-hf-access-token-value>
```
## Step 4: Deploy a RayService
After adding the Hugging Face access token, create a RayService custom resource using the config file:
```sh
kubectl apply -f ray-service.llm-serve.yaml
```
This step sets up a custom Ray Serve app to serve the `Qwen/Qwen2.5-7B-Instruct` model, creating an OpenAI-compatible server. You can inspect and modify the `serveConfigV2` section in the YAML file to learn more about the Serve app:
```yaml
serveConfigV2: |
applications:
- name: llms
import_path: ray.serve.llm:build_openai_app
route_prefix: "/"
args:
llm_configs:
- model_loading_config:
model_id: qwen2.5-7b-instruct
model_source: Qwen/Qwen2.5-7B-Instruct
engine_kwargs:
dtype: bfloat16
max_model_len: 1024
device: auto
gpu_memory_utilization: 0.75
deployment_config:
autoscaling_config:
min_replicas: 1
max_replicas: 4
target_ongoing_requests: 64
max_ongoing_requests: 128
```
In particular, this configuration loads the model from `Qwen/Qwen2.5-7B-Instruct` and sets its `model_id` to `qwen2.5-7b-instruct`. The `LLMDeployment` initializes the underlying LLM engine using the `engine_kwargs` field. The `deployment_config` section sets the desired number of engine replicas. By default, each replica requires one GPU. See [Serving LLMs](serving-llms) and the [Ray Serve config documentation](serve-in-production-config-file) for more information.
Wait for the RayService resource to become healthy. You can confirm its status by running the following command:
```sh
kubectl get rayservice ray-serve-llm -o yaml
```
After a few minutes, the result should be similar to the following:
```
status:
activeServiceStatus:
applicationStatuses:
llms:
serveDeploymentStatuses:
LLMDeployment:qwen2_5-7b-instruct:
status: HEALTHY
LLMRouter:
status: HEALTHY
status: RUNNING
```
## Step 5: Send a request
To send requests to the Ray Serve deployment, port-forward port 8000 from the Serve app service:
```sh
kubectl port-forward ray-serve-llm-serve-svc 8000
```
Note that this Kubernetes service comes up only after Ray Serve apps are running and ready.
Test the service with the following command:
```sh
curl --location 'http://localhost:8000/v1/chat/completions' --header 'Content-Type: application/json'
--data '{
"model": "qwen2.5-7b-instruct",
"messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "Provide steps to serve an LLM using Ray Serve."
}
]
}'
```
The output should be in the following format:
```
{
"id": "qwen2.5-7b-instruct-550d3fd491890a7e7bca74e544d3479e",
"object": "chat.completion",
"created": 1746595284,
"model": "qwen2.5-7b-instruct",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"reasoning_content": null,
"content": "Sure! Ray Serve is a library built on top of Ray...",
"tool_calls": []
},
"logprobs": null,
"finish_reason": "stop",
"stop_reason": null
}
],
"usage": {
"prompt_tokens": 30,
"total_tokens": 818,
"completion_tokens": 788,
"prompt_tokens_details": null
},
"prompt_logprobs": null
}
```
## Step 6: View the Ray dashboard
```sh
kubectl port-forward svc/ray-serve-llm-head-svc 8265
```
Once forwarded, navigate to the Serve tab on the dashboard to review application status, deployments, routers, logs, and other relevant features. ![LLM Serve Application](../images/ray_dashboard_llm_application.png)
doc/source/cluster/kubernetes/examples/stable-diffusion-rayservice.md
+75
View File
@@ -0,0 +1,75 @@
(kuberay-stable-diffusion-rayservice-example)=
# Serve a StableDiffusion text-to-image model on Kubernetes
> **Note:** The Python files for the Ray Serve application and its client are in the [ray-project/serve_config_examples](https://github.com/ray-project/serve_config_examples) repository
and [the Ray documentation](https://docs.ray.io/en/latest/serve/tutorials/stable-diffusion.html).
## Step 1: Create a Kubernetes cluster with GPUs
See [aws-eks-gpu-cluster.md](kuberay-eks-gpu-cluster-setup) or [gcp-gke-gpu-cluster.md](kuberay-gke-gpu-cluster-setup) or [ack-gpu-cluster.md](kuberay-ack-gpu-cluster-setup) to create a Kubernetes cluster with 1 CPU node and 1 GPU node.
## Step 2: Install KubeRay operator
Follow [this document](kuberay-operator-deploy) to install the latest stable KubeRay operator using the Helm repository. Note that the YAML file in this example uses `serveConfigV2`. This feature requires KubeRay v0.6.0 or later.
## Step 3: Install a RayService
```sh
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-service.stable-diffusion.yaml
```
This RayService configuration contains some important settings:
* In the RayService, the head Pod doesn't have any `tolerations`. Meanwhile, the worker Pods use the following `tolerations` so the scheduler won't assign the head Pod to the GPU node.
```yaml
# Please add the following taints to the GPU node.
tolerations:
- key: "ray.io/node-type"
operator: "Equal"
value: "worker"
effect: "NoSchedule"
```
* It includes `diffusers` in `runtime_env` since this package isn't included by default in the `ray-ml` image.
## Step 4: Forward the port of Serve
First get the service name from this command.
```sh
kubectl get services
```
Then, port forward to the serve.
```sh
# Wait until the RayService `Ready` condition is `True`. This means the RayService is ready to serve.
kubectl describe rayservices.ray.io stable-diffusion
# [Example output]
# Conditions:
# Last Transition Time: 2025-02-13T07:10:34Z
# Message: Number of serve endpoints is greater than 0
# Observed Generation: 1
# Reason: NonZeroServeEndpoints
# Status: True
# Type: Ready
# Forward the port of Serve
kubectl port-forward svc/stable-diffusion-serve-svc 8000
```
## Step 5: Send a request to the text-to-image model
```sh
# Step 5.1: Download `stable_diffusion_req.py`
curl -LO https://raw.githubusercontent.com/ray-project/serve_config_examples/master/stable_diffusion/stable_diffusion_req.py
# Step 5.2: Set your `prompt` in `stable_diffusion_req.py`.
# Step 5.3: Send a request to the Stable Diffusion model.
python stable_diffusion_req.py
# Check output.png
```
* You can refer to the document ["Serving a Stable Diffusion Model"](https://docs.ray.io/en/latest/serve/tutorials/stable-diffusion.html) for an example output image.
doc/source/cluster/kubernetes/examples/text-summarizer-rayservice.md
+69
View File
@@ -0,0 +1,69 @@
(kuberay-text-summarizer-rayservice-example)=
# Serve a text summarizer on Kubernetes
> **Note:** The Python files for the Ray Serve application and its client are in the [ray-project/serve_config_examples](https://github.com/ray-project/serve_config_examples) repository.
## Step 1: Create a Kubernetes cluster with GPUs
See [aws-eks-gpu-cluster.md](kuberay-eks-gpu-cluster-setup) or [gcp-gke-gpu-cluster.md](kuberay-gke-gpu-cluster-setup) or [ack-gpu-cluster.md](kuberay-ack-gpu-cluster-setup) to create a Kubernetes cluster with 1 CPU node and 1 GPU node.
## Step 2: Install KubeRay operator
Follow [this document](kuberay-operator-deploy) to install the latest stable KubeRay operator using the Helm repository.
## Step 3: Install a RayService
```sh
# Create a RayService
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-service.text-summarizer.yaml
```
* In the RayService, the head Pod doesn't have any `tolerations`. Meanwhile, the worker Pods use the following `tolerations` so the scheduler won't assign the head Pod to the GPU node.
```yaml
# Please add the following taints to the GPU node.
tolerations:
- key: "ray.io/node-type"
operator: "Equal"
value: "worker"
effect: "NoSchedule"
```
## Step 4: Forward the port of Serve
```sh
# Step 4.1: Wait until the RayService is ready to serve requests.
kubectl describe rayservices text-summarizer
# Step 4.2: Get the service name.
kubectl get services
# [Example output]
# text-summarizer-head-svc ClusterIP None <none> 10001/TCP,8265/TCP,6379/TCP,8080/TCP,8000/TCP 31s
# text-summarizer-raycluster-tb9zf-head-svc ClusterIP None <none> 10001/TCP,8265/TCP,6379/TCP,8080/TCP,8000/TCP 108s
# text-summarizer-serve-svc ClusterIP 34.118.226.139 <none> 8000/TCP 31s
# Step 4.3: Forward the port of Serve.
kubectl port-forward svc/text-summarizer-serve-svc 8000
```
## Step 5: Send a request to the text summarizer model
```sh
# Step 5.1: Download `text_summarizer_req.py`
curl -LO https://raw.githubusercontent.com/ray-project/serve_config_examples/master/text_summarizer/text_summarizer_req.py
# Step 5.2: Send a request to the Summarizer model.
python text_summarizer_req.py
# Check printed to console
```
## Step 6: Delete your service
```sh
kubectl delete -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-service.text-summarizer.yaml
```
## Step 7: Uninstall your KubeRay operator
Follow [this document](https://github.com/ray-project/kuberay/tree/master/helm-chart/kuberay-operator) to uninstall the latest stable KubeRay operator using the Helm repository.
doc/source/cluster/kubernetes/examples/tpu-serve-stable-diffusion.md
+77
View File
@@ -0,0 +1,77 @@
(kuberay-tpu-stable-diffusion-example)=
# Serve a Stable Diffusion model on GKE with TPUs
> **Note:** The Python files for the Ray Serve app and its client are in the [ray-project/serve_config_examples](https://github.com/ray-project/serve_config_examples). This guide adapts the [tensorflow/tpu](https://github.com/tensorflow/tpu/tree/master/tools/ray_tpu/src/serve) example.
## Step 1: Create a Kubernetes cluster with TPUs
Follow [Creating a GKE Cluster with TPUs for KubeRay](kuberay-gke-tpu-cluster-setup) to create a GKE cluster with 1 CPU node and 1 TPU node.
## Step 2: Install the KubeRay operator
Skip this step if the [Ray Operator Addon](https://cloud.google.com/kubernetes-engine/docs/add-on/ray-on-gke/concepts/overview) is enabled in your GKE cluster. Follow [Deploy a KubeRay operator](kuberay-operator-deploy) instructions to install the latest stable KubeRay operator from the Helm repository. Multi-host TPU support is available in KubeRay v1.1.0+. Note that the YAML file in this example uses `serveConfigV2`, which KubeRay supports starting from v0.6.0.
## Step 3: Install the RayService CR
```sh
# Creates a RayCluster with a single-host v4 TPU worker group of 2x2x1 topology.
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-service.tpu-single-host.yaml
```
KubeRay operator v1.1.0 adds a new `NumOfHosts` field to the RayCluster CR, supporting multi-host worker groups. This field specifies the number of workers to create per replica, with each replica representing a multi-host Pod slice. The value for `NumOfHosts` should match the number of TPU VM hosts that the given `cloud.google.com/gke-tpu-topology` node selector expects. For this example, the Stable Diffusion model is small enough to run on a single TPU host, so `numOfHosts` is set to 1 in the RayService manifest.
## Step 4: View the Serve deployment in the Ray Dashboard
Verify that you deployed the RayService CR and it's running:
```sh
kubectl get rayservice
# NAME SERVICE STATUS NUM SERVE ENDPOINTS
# stable-diffusion-tpu-serve-svc Running 2
```
Port-forward the Ray Dashboard from the Ray head service. To view the dashboard, open http://localhost:8265/ on your local machine.
```sh
kubectl port-forward svc/stable-diffusion-tpu-head-svc 8265:8265 &
```
Monitor the status of the RayService CR in the Ray Dashboard from the 'Serve' tab. The installed RayService CR should create a running app with the name 'stable_diffusion'. The app should have two deployments, the API ingress, which receives input prompts, and the Stable Diffusion model server.
![serve_dashboard](../images/serve_dashboard.png)
## Step 5: Send text-to-image prompts to the model server
Port forward the Ray Serve service:
```sh
kubectl port-forward svc/stable-diffusion-tpu-serve-svc 8000
```
In a separate terminal, download the Python prompt script:
```sh
curl -LO https://raw.githubusercontent.com/ray-project/serve_config_examples/master/stable_diffusion/stable_diffusion_tpu_req.py
```
Install the required dependencies to run the Python script locally:
```sh
# Create a Python virtual environment.
python3 -m venv myenv
source myenv/bin/activate
pip install numpy pillow requests tqdm
```
Submit a text-to-image prompt to the Stable Diffusion model server:
```sh
python stable_diffusion_tpu_req.py --save_pictures
```
* The Python prompt script saves the results of the Stable Diffusion inference to a file named diffusion_results.png.
![diffusion_results](../images/diffusion_results.png)
doc/source/cluster/kubernetes/examples/verl-post-training.md
+144
View File
@@ -0,0 +1,144 @@
(kuberay-verl)=
# Reinforcement Learning with Human Feedback (RLHF) for LLMs with verl on KubeRay
[verl](https://github.com/volcengine/verl) is an open-source framework that provides a flexible, efficient, and production-ready RL training library for large language models (LLMs). This guide demonstrates Proximal Policy Optimization (PPO) training on the GSM8K dataset with verl for `Qwen2.5-0.5B-Instruct` on KubeRay.
* To make it easier to follow, this guide launches a single-node RayCluster with 4 GPUs. You can easily use KubeRay to launch a multi-node RayCluster to train larger models.
* You can also use the [RayJob CRD](kuberay-rayjob-quickstart) for production use cases.
# Step 1: Create a Kubernetes cluster with GPUs
Follow the instructions in [Managed Kubernetes services](kuberay-k8s-setup) to create a Kubernetes cluster with GPUs.
This guide uses a Kubernetes cluster with 4 L4 GPUs.
For GKE, you can follow the instructions in [this tutorial](kuberay-gke-gpu-cluster-setup) and use the following command to create a GPU node pool with 4 L4 GPUs per Kubernetes node:
```bash
gcloud container node-pools create gpu-node-pool \
--accelerator type=nvidia-l4-vws,count=4 \
--zone us-west1-b \
--cluster kuberay-gpu-cluster \
--num-nodes 1 \
--min-nodes 0 \
--max-nodes 1 \
--enable-autoscaling \
--machine-type g2-standard-48
```
# Step 2: Install KubeRay operator
Follow the instructions in [KubeRay operator](kuberay-operator-deploy) to install the KubeRay operator.
# Step 3: Create a RayCluster
```sh
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-cluster.verl.yaml
```
# Step 4: Install verl in the head Pod
Log in to the head Pod and install verl. The verl community doesn't provide images with verl installed ([verl#2222](https://github.com/volcengine/verl/issues/2222)) at the moment.
```sh
# Log in to the head Pod.
export HEAD_POD=$(kubectl get pods --selector=ray.io/node-type=head -o custom-columns=POD:metadata.name --no-headers)
kubectl exec -it $HEAD_POD -- bash
# Follow the instructions in https://verl.readthedocs.io/en/latest/start/install.html#install-from-docker-image to install verl.
git clone https://github.com/volcengine/verl && cd verl
pip3 install -e .[vllm]
```
# Step 5: Prepare the dataset and download `Qwen2.5-0.5B-Instruct` model
Run the following commands in the head Pod's verl root directory to prepare the dataset and download the `Qwen2.5-0.5B-Instruct` model.
```sh
# Prepare the dataset.
python3 examples/data_preprocess/gsm8k.py --local_dir ~/data/gsm8k
# Download the `Qwen2.5-0.5B-Instruct` model.
python3 -c "import transformers; transformers.pipeline('text-generation', model='Qwen/Qwen2.5-0.5B-Instruct')"
```
# Step 6: Run a PPO training job
Run the following command to start a PPO training job. This differs slightly from the instructions in [verl's documentation](https://verl.readthedocs.io/en/latest/start/quickstart.html#step-3-perform-ppo-training-with-the-instruct-model). The main differences are the following:
* Set `n_gpus_per_node` to `4` because the head Pod has 4 GPUs.
* Set `save_freq` to `-1` to avoid disk pressure caused by checkpointing.
```sh
PYTHONUNBUFFERED=1 python3 -m verl.trainer.main_ppo \
data.train_files=$HOME/data/gsm8k/train.parquet \
data.val_files=$HOME/data/gsm8k/test.parquet \
data.train_batch_size=256 \
data.max_prompt_length=512 \
data.max_response_length=256 \
actor_rollout_ref.model.path=Qwen/Qwen2.5-0.5B-Instruct \
actor_rollout_ref.actor.optim.lr=1e-6 \
actor_rollout_ref.actor.ppo_mini_batch_size=64 \
actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=4 \
actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=8 \
actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
actor_rollout_ref.rollout.gpu_memory_utilization=0.4 \
actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=4 \
critic.optim.lr=1e-5 \
critic.model.path=Qwen/Qwen2.5-0.5B-Instruct \
critic.ppo_micro_batch_size_per_gpu=4 \
algorithm.kl_ctrl.kl_coef=0.001 \
trainer.logger=['console'] \
trainer.val_before_train=False \
trainer.default_hdfs_dir=null \
trainer.n_gpus_per_node=4 \
trainer.nnodes=1 \
trainer.save_freq=-1 \
trainer.test_freq=10 \
trainer.total_epochs=15 2>&1 | tee verl_demo.log
```
This job takes 5 hours to complete. While it's running, you can check the Ray dashboard to see more details about the PPO job and the Ray cluster. Additionally, you can follow the next step to check the PPO job logs to see how the model improves.
```sh
# Port forward the Ray dashboard to your local machine's port 8265.
kubectl port-forward $HEAD_POD 8265:8265
```
Open `127.0.0.1:8265` in your browser to view the Ray dashboard and check whether all GPUs are in use.
![Ray dashboard](../images/verl-ray-dashboard.png)
# Step 7: Check the PPO job logs
Check `verl_demo.log` in the head Pod to see the PPO job's logs. For every 10 steps, verl validates the model with a simple math problem.
* Math problem:
```
Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market? Let's think step by step and output the final answer after
```
* Answer: `(16 - 3 - 4) * 2 = 18`
You should be able to see the model becomes gradually better at this question after several steps.
In this example run, the model first got the correct answer after 130 steps, and the following is the log. Throughout the entire process, the validation ran 44 times and got the correct answer 20 times. It may vary depending on the random seed.
```
(TaskRunner pid=21297) [response] First, we calculate the number of eggs Janet's ducks lay in a day. Since there are 16 eggs per day and Janet lays these eggs every day, the number of eggs laid in a day is 16.
(TaskRunner pid=21297)
(TaskRunner pid=21297) Next, we calculate the number of eggs Janet eats in a day. She eats 3 eggs for breakfast and bakes 4 muffins, so the total number of eggs she eats in a day is 3 + 4 = 7.
(TaskRunner pid=21297)
(TaskRunner pid=21297) The number of eggs she sells in a day is the total number of eggs laid minus the number of eggs she eats, which is 16 - 7 = 9 eggs.
(TaskRunner pid=21297)
(TaskRunner pid=21297) She sells each egg for $2, so the total amount she makes every day is 9 * 2 = 18 dollars.
(TaskRunner pid=21297)
(TaskRunner pid=21297) #### 18
(TaskRunner pid=21297) #### 18 dollars
```
It's not necessary to wait for all steps to complete. You can stop the job if you observe the process of the model improving.
# Step 8: Clean up
```sh
kubectl delete -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-cluster.verl.yaml
```