wehub-resource-sync
533 lines
17 KiB
Markdown
533 lines
17 KiB
Markdown
(deploying-on-argocd-example)=
|
|
|
|
# Deploying Ray Clusters via ArgoCD
|
|
|
|
This guide provides a step-by-step approach for deploying Ray clusters on Kubernetes using ArgoCD. ArgoCD is a declarative GitOps tool that enables you to manage Ray cluster configurations in Git repositories with automated synchronization, version control, and rollback capabilities. This approach is particularly valuable when managing multiple Ray clusters across different environments, implementing audit trails and approval workflows, or maintaining infrastructure-as-code practices. For simpler use cases like single-cluster development or quick experimentation, direct kubectl or Helm deployments may be sufficient. You can read more about the benefits of ArgoCD [here](https://argo-cd.readthedocs.io/en/stable/#why-argo-cd).
|
|
|
|
This example demonstrates how to deploy the KubeRay operator and a RayCluster with three different worker groups, leveraging ArgoCD's GitOps capabilities for automated cluster management.
|
|
|
|
## Prerequisites
|
|
|
|
Before proceeding with this guide, ensure you have the following:
|
|
|
|
* A Kubernetes cluster with appropriate resources for running Ray workloads.
|
|
* `kubectl` configured to access your Kubernetes cluster.
|
|
* (Optional)[ArgoCD installed](https://argo-cd.readthedocs.io/en/stable/getting_started/) on your Kubernetes cluster.
|
|
* (Optional)[ArgoCD CLI](https://argo-cd.readthedocs.io/en/stable/cli_installation/) installed on your local machine (recommended for easier application management. It might need [port-forwarding and login](https://argo-cd.readthedocs.io/en/stable/getting_started/#port-forwarding) depending on your environment).
|
|
* (Optional)Access to the ArgoCD UI or API server.
|
|
|
|
## Step 1: Deploy KubeRay Operator CRDs
|
|
|
|
First, deploy the Custom Resource Definitions (CRDs) required by the KubeRay operator. Create a file named `ray-operator-crds.yaml` with the following content:
|
|
|
|
```yaml
|
|
apiVersion: argoproj.io/v1alpha1
|
|
kind: Application
|
|
metadata:
|
|
name: ray-operator-crds
|
|
namespace: argocd
|
|
spec:
|
|
project: default
|
|
destination:
|
|
server: https://kubernetes.default.svc
|
|
namespace: ray-cluster
|
|
source:
|
|
repoURL: https://github.com/ray-project/kuberay
|
|
targetRevision: v1.6.0 # update this as necessary
|
|
path: helm-chart/kuberay-operator/crds
|
|
syncPolicy:
|
|
automated:
|
|
prune: true
|
|
selfHeal: true
|
|
syncOptions:
|
|
- CreateNamespace=true
|
|
- Replace=true
|
|
```
|
|
|
|
Apply the ArgoCD Application:
|
|
|
|
```sh
|
|
kubectl apply -f ray-operator-crds.yaml
|
|
```
|
|
|
|
Wait for the CRDs Application to sync and become healthy. You can check the status using:
|
|
|
|
```sh
|
|
kubectl get application ray-operator-crds -n argocd
|
|
```
|
|
|
|
Which should eventually give something like:
|
|
```
|
|
NAME SYNC STATUS HEALTH STATUS
|
|
ray-operator-crds Synced Healthy
|
|
```
|
|
|
|
Alternatively, if you have the ArgoCD CLI installed, you can wait for the application:
|
|
|
|
```sh
|
|
argocd app wait ray-operator-crds
|
|
```
|
|
|
|
## Step 2: Deploy the KubeRay Operator
|
|
|
|
After the CRDs are installed, deploy the KubeRay operator itself. Create a file named `ray-operator.yaml` with the following content:
|
|
|
|
```yaml
|
|
apiVersion: argoproj.io/v1alpha1
|
|
kind: Application
|
|
metadata:
|
|
name: ray-operator
|
|
namespace: argocd
|
|
spec:
|
|
project: default
|
|
source:
|
|
repoURL: https://github.com/ray-project/kuberay
|
|
targetRevision: v1.6.0 # update this as necessary
|
|
path: helm-chart/kuberay-operator
|
|
helm:
|
|
skipCrds: true # CRDs are already installed in Step 1
|
|
destination:
|
|
server: https://kubernetes.default.svc
|
|
namespace: ray-cluster
|
|
syncPolicy:
|
|
automated:
|
|
prune: true
|
|
selfHeal: true
|
|
syncOptions:
|
|
- CreateNamespace=true
|
|
```
|
|
|
|
Note the `skipCrds: true` setting in the Helm configuration. This is required because the CRDs were installed separately in Step 1.
|
|
|
|
Apply the ArgoCD Application:
|
|
|
|
```sh
|
|
kubectl apply -f ray-operator.yaml
|
|
```
|
|
|
|
Wait for the operator Application to sync and become healthy. You can check the status using:
|
|
|
|
```sh
|
|
kubectl get application ray-operator -n argocd
|
|
```
|
|
|
|
Which should give the following output eventually:
|
|
```
|
|
NAME SYNC STATUS HEALTH STATUS
|
|
ray-operator Synced Healthy
|
|
```
|
|
|
|
Alternatively, if you have the ArgoCD CLI installed:
|
|
|
|
```sh
|
|
argocd app wait ray-operator
|
|
```
|
|
|
|
Verify that the KubeRay operator pod is running:
|
|
|
|
```sh
|
|
kubectl get pods -n ray-cluster -l app.kubernetes.io/name=kuberay-operator
|
|
```
|
|
|
|
|
|
|
|
## Step 3: Deploy a RayCluster
|
|
|
|
Now deploy a RayCluster with autoscaling enabled and three different worker groups. Create a file named `raycluster.yaml` with the following content:
|
|
|
|
```yaml
|
|
apiVersion: argoproj.io/v1alpha1
|
|
kind: Application
|
|
metadata:
|
|
name: raycluster
|
|
namespace: argocd
|
|
spec:
|
|
project: default
|
|
destination:
|
|
server: https://kubernetes.default.svc
|
|
namespace: ray-cluster
|
|
ignoreDifferences:
|
|
- group: ray.io
|
|
kind: RayCluster
|
|
name: raycluster-kuberay
|
|
namespace: ray-cluster
|
|
jqPathExpressions:
|
|
- .spec.workerGroupSpecs[].replicas
|
|
source:
|
|
repoURL: https://ray-project.github.io/kuberay-helm/
|
|
chart: ray-cluster
|
|
targetRevision: "1.6.0"
|
|
helm:
|
|
releaseName: raycluster
|
|
valuesObject:
|
|
image:
|
|
repository: docker.io/rayproject/ray
|
|
tag: latest
|
|
pullPolicy: IfNotPresent
|
|
head:
|
|
rayStartParams:
|
|
num-cpus: "0"
|
|
enableInTreeAutoscaling: true
|
|
autoscalerOptions:
|
|
version: v2
|
|
upscalingMode: Default
|
|
idleTimeoutSeconds: 600 # 10 minutes
|
|
env:
|
|
- name: AUTOSCALER_MAX_CONCURRENT_LAUNCHES
|
|
value: "100"
|
|
worker:
|
|
groupName: standard-worker
|
|
replicas: 1
|
|
minReplicas: 1
|
|
maxReplicas: 200
|
|
rayStartParams:
|
|
resources: '"{\"standard-worker\": 1}"'
|
|
resources:
|
|
requests:
|
|
cpu: "1"
|
|
memory: "1G"
|
|
additionalWorkerGroups:
|
|
additional-worker-group1:
|
|
image:
|
|
repository: docker.io/rayproject/ray
|
|
tag: latest
|
|
pullPolicy: IfNotPresent
|
|
disabled: false
|
|
replicas: 1
|
|
minReplicas: 1
|
|
maxReplicas: 30
|
|
rayStartParams:
|
|
resources: '"{\"additional-worker-group1\": 1}"'
|
|
resources:
|
|
requests:
|
|
cpu: "1"
|
|
memory: "1G"
|
|
additional-worker-group2:
|
|
image:
|
|
repository: docker.io/rayproject/ray
|
|
tag: latest
|
|
pullPolicy: IfNotPresent
|
|
disabled: false
|
|
replicas: 1
|
|
minReplicas: 1
|
|
maxReplicas: 200
|
|
rayStartParams:
|
|
resources: '"{\"additional-worker-group2\": 1}"'
|
|
resources:
|
|
requests:
|
|
cpu: "1"
|
|
memory: "1G"
|
|
syncPolicy:
|
|
automated:
|
|
prune: true
|
|
selfHeal: true
|
|
syncOptions:
|
|
- CreateNamespace=true
|
|
```
|
|
|
|
Apply the ArgoCD Application:
|
|
|
|
```sh
|
|
kubectl apply -f raycluster.yaml
|
|
```
|
|
|
|
Wait for the RayCluster Application to sync and become healthy. You can check the status using:
|
|
|
|
```sh
|
|
kubectl get application raycluster -n argocd
|
|
```
|
|
|
|
Alternatively, if you have the ArgoCD CLI installed:
|
|
|
|
```sh
|
|
argocd app wait raycluster
|
|
```
|
|
|
|
Verify that the RayCluster is running:
|
|
|
|
```sh
|
|
kubectl get raycluster -n ray-cluster
|
|
```
|
|
|
|
Which will give something like:
|
|
```
|
|
NAME DESIRED WORKERS AVAILABLE WORKERS CPUS MEMORY GPUS STATUS AGE
|
|
raycluster-kuberay 3 3 ... ... ... ready ...
|
|
```
|
|
|
|
|
|
You should see the head pod and worker pods:
|
|
|
|
```sh
|
|
kubectl get pods -n ray-cluster
|
|
```
|
|
|
|
Gives something like:
|
|
```
|
|
NAME READY STATUS RESTARTS AGE
|
|
kuberay-operator-6c485bc876-28dnl 1/1 Running 0 11d
|
|
raycluster-kuberay-additional-worker-group1-n45rc 1/1 Running 0 5d21h
|
|
raycluster-kuberay-additional-worker-group2-b2455 1/1 Running 0 2d18h
|
|
raycluster-kuberay-head 2/2 Running 0 5d21h
|
|
raycluster-kuberay-standard-worker-worker-bs8t8 1/1 Running 0 5d21h
|
|
```
|
|
|
|
## Understanding Ray Autoscaling with ArgoCD
|
|
|
|
### Determining Fields to Ignore
|
|
|
|
The `ignoreDifferences` section in the RayCluster Application configuration is critical for proper autoscaling. To determine which fields need to be ignored, you can inspect the RayCluster resource to identify fields that change dynamically during runtime.
|
|
|
|
First, describe the RayCluster resource to see its full specification:
|
|
|
|
```sh
|
|
kubectl describe raycluster raycluster-kuberay -n ray-cluster
|
|
```
|
|
|
|
Or, get the resource in YAML format to see the exact field paths:
|
|
|
|
```sh
|
|
kubectl get raycluster raycluster-kuberay -n ray-cluster -o yaml
|
|
```
|
|
|
|
Look for fields that are modified by controllers or autoscalers. In the case of Ray, the autoscaler modifies the `replicas` field under each worker group spec. You'll see output similar to:
|
|
|
|
```yaml
|
|
spec:
|
|
workerGroupSpecs:
|
|
- replicas: 5 # This value changes dynamically
|
|
minReplicas: 1
|
|
maxReplicas: 200
|
|
groupName: standard-worker
|
|
# ...
|
|
```
|
|
|
|
When ArgoCD detects differences between the desired state (in Git) and the actual state (in the cluster), it will show these in the UI or via CLI:
|
|
|
|
```sh
|
|
argocd app diff raycluster
|
|
```
|
|
|
|
If you see repeated differences in fields that should be managed by controllers (like autoscalers), those are candidates for `ignoreDifferences`.
|
|
|
|
### Configuring ignoreDifferences
|
|
|
|
The `ignoreDifferences` section in the RayCluster Application configuration tells ArgoCD which fields to ignore. Without this setting, ArgoCD and the Ray Autoscaler may conflict, resulting in unexpected behavior when requesting workers dynamically (for example, using `ray.autoscaler.sdk.request_resources`). Specifically, when requesting N workers, the Autoscaler might not spin up the expected number of workers because ArgoCD could revert the replica count back to the original value defined in the Application manifest.
|
|
|
|
The recommended approach is to use `jqPathExpressions`, which automatically handles any number of worker groups:
|
|
|
|
```yaml
|
|
ignoreDifferences:
|
|
- group: ray.io
|
|
kind: RayCluster
|
|
name: raycluster-kuberay
|
|
namespace: ray-cluster
|
|
jqPathExpressions:
|
|
- .spec.workerGroupSpecs[].replicas
|
|
```
|
|
|
|
This configuration tells ArgoCD to ignore differences in the `replicas` field for all worker groups. The `jqPathExpressions` field uses JQ syntax with array wildcards (`[]`), which means you don't need to update the configuration when adding or removing worker groups.
|
|
|
|
**Note**: The `name` and `namespace` must match your RayCluster resource name and namespace. Verify these values separately if you've customized them.
|
|
|
|
**Alternative: Using jsonPointers**
|
|
|
|
If you prefer explicit configuration, you can use `jsonPointers` instead:
|
|
|
|
```yaml
|
|
ignoreDifferences:
|
|
- group: ray.io
|
|
kind: RayCluster
|
|
name: raycluster-kuberay
|
|
namespace: ray-cluster
|
|
jsonPointers:
|
|
- /spec/workerGroupSpecs/0/replicas
|
|
- /spec/workerGroupSpecs/1/replicas
|
|
- /spec/workerGroupSpecs/2/replicas
|
|
```
|
|
|
|
With `jsonPointers`, you must explicitly list each worker group by index:
|
|
- `/spec/workerGroupSpecs/0/replicas` - First worker group (the default `worker` group)
|
|
- `/spec/workerGroupSpecs/1/replicas` - Second worker group (`additional-worker-group1`)
|
|
- `/spec/workerGroupSpecs/2/replicas` - Third worker group (`additional-worker-group2`)
|
|
|
|
If you add or remove worker groups, you **must** update this list accordingly. The index corresponds to the order of worker groups as they appear in the RayCluster spec, with the default `worker` group at index 0 and `additionalWorkerGroups` following in the order they are defined.
|
|
|
|
See the [ArgoCD diff customization documentation](https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/) for more details on both approaches.
|
|
|
|
By ignoring these differences, ArgoCD allows the Ray Autoscaler to dynamically manage worker replicas without interference.
|
|
|
|
## Step 4: Access the Ray Dashboard
|
|
|
|
To access the Ray Dashboard, port-forward the head service:
|
|
|
|
```sh
|
|
kubectl port-forward -n ray-cluster svc/raycluster-kuberay-head-svc 8265:8265
|
|
```
|
|
|
|
Navigate to `http://localhost:8265` in your browser to view the Ray Dashboard.
|
|
|
|
## Customizing the Configuration
|
|
|
|
You can customize the RayCluster configuration by modifying the `valuesObject` section in the `raycluster.yaml` file:
|
|
|
|
* **Image**: Change the `repository` and `tag` to use different Ray versions.
|
|
* **Worker Groups**: Add or remove worker groups by modifying the `additionalWorkerGroups` section.
|
|
* **Autoscaling**: Adjust `minReplicas`, `maxReplicas`, and `idleTimeoutSeconds` to control autoscaling behavior.
|
|
* **Resources**: Modify `rayStartParams` to allocate custom resources to worker groups.
|
|
|
|
After making changes, commit them to your Git repository. ArgoCD will automatically sync the changes to your cluster if automated sync is enabled.
|
|
|
|
## Alternative: Deploy Everything in One File
|
|
|
|
If you prefer to deploy all components at once, you can combine all three ArgoCD Applications into a single file. Create a file named `ray-argocd-all.yaml` with the following content:
|
|
|
|
```yaml
|
|
apiVersion: argoproj.io/v1alpha1
|
|
kind: Application
|
|
metadata:
|
|
name: ray-operator-crds
|
|
namespace: argocd
|
|
spec:
|
|
project: default
|
|
destination:
|
|
server: https://kubernetes.default.svc
|
|
namespace: ray-cluster
|
|
source:
|
|
repoURL: https://github.com/ray-project/kuberay
|
|
targetRevision: v1.6.0 # update this as necessary
|
|
path: helm-chart/kuberay-operator/crds
|
|
syncPolicy:
|
|
automated:
|
|
prune: true
|
|
selfHeal: true
|
|
syncOptions:
|
|
- CreateNamespace=true
|
|
- Replace=true
|
|
|
|
---
|
|
|
|
apiVersion: argoproj.io/v1alpha1
|
|
kind: Application
|
|
metadata:
|
|
name: ray-operator
|
|
namespace: argocd
|
|
spec:
|
|
project: default
|
|
source:
|
|
repoURL: https://github.com/ray-project/kuberay
|
|
targetRevision: v1.6.0 # update this as necessary
|
|
path: helm-chart/kuberay-operator
|
|
helm:
|
|
skipCrds: true # CRDs are installed in the first Application
|
|
destination:
|
|
server: https://kubernetes.default.svc
|
|
namespace: ray-cluster
|
|
syncPolicy:
|
|
automated:
|
|
prune: true
|
|
selfHeal: true
|
|
syncOptions:
|
|
- CreateNamespace=true
|
|
|
|
---
|
|
|
|
apiVersion: argoproj.io/v1alpha1
|
|
kind: Application
|
|
metadata:
|
|
name: raycluster
|
|
namespace: argocd
|
|
spec:
|
|
project: default
|
|
destination:
|
|
server: https://kubernetes.default.svc
|
|
namespace: ray-cluster
|
|
ignoreDifferences:
|
|
- group: ray.io
|
|
kind: RayCluster
|
|
name: raycluster-kuberay # ensure this is aligned with the release name
|
|
namespace: ray-cluster # ensure this is aligned with the namespace
|
|
jqPathExpressions:
|
|
- .spec.workerGroupSpecs[].replicas
|
|
source:
|
|
repoURL: https://ray-project.github.io/kuberay-helm/
|
|
chart: ray-cluster
|
|
targetRevision: "1.4.1"
|
|
helm:
|
|
releaseName: raycluster # this affects the ignoreDifferences field
|
|
valuesObject:
|
|
image:
|
|
repository: docker.io/rayproject/ray
|
|
tag: latest
|
|
pullPolicy: IfNotPresent
|
|
head:
|
|
rayStartParams:
|
|
num-cpus: "0"
|
|
enableInTreeAutoscaling: true
|
|
autoscalerOptions:
|
|
version: v2
|
|
upscalingMode: Default
|
|
idleTimeoutSeconds: 600 # 10 minutes
|
|
env:
|
|
- name: AUTOSCALER_MAX_CONCURRENT_LAUNCHES
|
|
value: "100"
|
|
worker:
|
|
groupName: standard-worker
|
|
replicas: 1
|
|
minReplicas: 1
|
|
maxReplicas: 200
|
|
rayStartParams:
|
|
resources: '"{\"standard-worker\": 1}"'
|
|
resources:
|
|
requests:
|
|
cpu: "1"
|
|
memory: "1G"
|
|
additionalWorkerGroups:
|
|
additional-worker-group1:
|
|
image:
|
|
repository: docker.io/rayproject/ray
|
|
tag: latest
|
|
pullPolicy: IfNotPresent
|
|
disabled: false
|
|
replicas: 1
|
|
minReplicas: 1
|
|
maxReplicas: 30
|
|
rayStartParams:
|
|
resources: '"{\"additional-worker-group1\": 1}"'
|
|
resources:
|
|
requests:
|
|
cpu: "1"
|
|
memory: "1G"
|
|
additional-worker-group2:
|
|
image:
|
|
repository: docker.io/rayproject/ray
|
|
tag: latest
|
|
pullPolicy: IfNotPresent
|
|
disabled: false
|
|
replicas: 1
|
|
minReplicas: 1
|
|
maxReplicas: 200
|
|
rayStartParams:
|
|
resources: '"{\"additional-worker-group2\": 1}"'
|
|
resources:
|
|
requests:
|
|
cpu: "1"
|
|
memory: "1G"
|
|
syncPolicy:
|
|
automated:
|
|
prune: true
|
|
selfHeal: true
|
|
syncOptions:
|
|
- CreateNamespace=true
|
|
```
|
|
|
|
Note in this example, the `jqPathExpressions` approach is used.
|
|
|
|
Apply all three Applications at once:
|
|
|
|
```sh
|
|
kubectl apply -f ray-argocd-all.yaml
|
|
```
|
|
|
|
This single-file approach is convenient for quick deployments, but the step-by-step approach in the earlier sections provides better visibility into the deployment process and makes it easier to troubleshoot issues.
|