chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,8 @@
|
||||
# Use the latest Ray master as base.
|
||||
FROM rayproject/ray:nightly-py310
|
||||
# Invalidate the cache so that fresh code is pulled in the next step.
|
||||
ARG BUILD_DATE
|
||||
# Retrieve your development code.
|
||||
ADD . ray
|
||||
# Install symlinks to your modified Python code.
|
||||
RUN python ray/python/ray/setup-dev.py -y
|
||||
@@ -0,0 +1,42 @@
|
||||
# How to run the KubeRay autoscaling test
|
||||
|
||||
This page provides suggestions on running the test `test_autoscaling_e2e` locally.
|
||||
You might want to do this if your PR is breaking this test in CI and you want to debug why.
|
||||
|
||||
Running the test must happen in stages:
|
||||
|
||||
1. Tear down any running `kind` cluster
|
||||
2. Remove the existing ray docker image that will be deployed to the cluster
|
||||
3. Build a new docker image containing the local ray repository
|
||||
4. Create a new `kind` cluster
|
||||
5. Load the docker image into the cluster
|
||||
6. Set up kuberay
|
||||
7. Run the test
|
||||
|
||||
To help with this, there is a `Dockerfile` and a `rune2e.sh` bash script which
|
||||
together run these things for you.
|
||||
|
||||
## Test requirements
|
||||
|
||||
1. Ensure `kind` and `kustomize` are both installed
|
||||
2. Run `ray/autoscaler/kuberay/init-config.sh` to clone `ray-project/kuberay`,
|
||||
which contains config files needed to set up kuberay.
|
||||
3. Finally, make sure that the `Dockerfile` is using the same python version as
|
||||
what you're using to run the test. By default, this dockerfile is built using
|
||||
the `rayproject/ray:nightly-py310` build.
|
||||
4. Modify `EXAMPLE_CLUSTER_PATH` in `test_autoscaling_e2e.py`.
|
||||
|
||||
Now you're ready to run the test.
|
||||
|
||||
## Running the test
|
||||
|
||||
Run `./rune2e.sh` to run the test.
|
||||
|
||||
The test itself does not tear down resources on failure; you can
|
||||
- examine a Ray cluster from a failed test (`kubectl get pods`, `kubectl get pod`, `kubectl get raycluster`)
|
||||
- view all logs (`kubectl logs <head pod name>`) or just logs associated with the autoscaler (`kubectl logs <head pod name> -c autoscaler`)
|
||||
- delete the Ray cluster (`kubectl delete raycluster -A`)
|
||||
- rerun the test without tearing the operator down (`RAY_IMAGE=<registry>/<repo>:<tag> python test_autoscaling_e2e.py`)
|
||||
- tear down the operator when you're done `python setup/teardown_kuberay.py`
|
||||
- copy files from a pod to your filesystem (`kubectl cp <pod>:/path/to/file /target/path/in/local/filesystem`)
|
||||
- access a bash prompt inside the pod (`kubectl exec -it <pod> bash`)
|
||||
Executable
+15
@@ -0,0 +1,15 @@
|
||||
#!/bin/bash
|
||||
|
||||
set -x
|
||||
RAY_IMAGE=rayproject/autoscaling_e2e_test_image
|
||||
kind delete cluster
|
||||
docker image rm $RAY_IMAGE
|
||||
|
||||
pushd ../../../..
|
||||
docker build --progress=plain --build-arg BUILD_DATE="$(date +%Y-%m-%d:%H:%M:%S)" -t $RAY_IMAGE -f ./python/ray/tests/kuberay/Dockerfile . || exit
|
||||
popd || exit
|
||||
|
||||
kind create cluster || exit
|
||||
kind load docker-image $RAY_IMAGE || exit
|
||||
python setup/setup_kuberay.py
|
||||
RAY_IMAGE=$RAY_IMAGE python test_autoscaling_e2e.py
|
||||
@@ -0,0 +1,17 @@
|
||||
import ray
|
||||
|
||||
|
||||
def main():
|
||||
"""This script runs in a container with 1 CPU limit and 1G memory limit.
|
||||
Validate that Ray reads the correct limits.
|
||||
"""
|
||||
cpu_limit = ray._private.utils.get_num_cpus()
|
||||
mem_limit_gb = round(ray._common.utils.get_system_memory() / 10**9, 2)
|
||||
assert cpu_limit == 1, cpu_limit
|
||||
assert mem_limit_gb == 2.00, mem_limit_gb
|
||||
print(f"Confirmed cpu limit {cpu_limit}.")
|
||||
print(f"Confirmed memory limit {mem_limit_gb} gigabyte.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,18 @@
|
||||
import ray
|
||||
|
||||
|
||||
def main():
|
||||
"""Requests placement of a GPU actor."""
|
||||
|
||||
@ray.remote(num_gpus=1, num_cpus=1)
|
||||
class GPUActor:
|
||||
def where_am_i(self):
|
||||
assert len(ray.get_gpu_ids()) == 1
|
||||
return "on-a-gpu-node"
|
||||
|
||||
GPUActor.options(name="gpu_actor", lifetime="detached").remote()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
ray.init("auto", namespace="gpu-test")
|
||||
main()
|
||||
@@ -0,0 +1,14 @@
|
||||
import ray
|
||||
|
||||
|
||||
def main():
|
||||
"""Confirms placement of a GPU actor."""
|
||||
gpu_actor = ray.get_actor("gpu_actor")
|
||||
actor_response = ray.get(gpu_actor.where_am_i.remote())
|
||||
return actor_response
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
ray.init("auto", namespace="gpu-test")
|
||||
out = main()
|
||||
print(out)
|
||||
@@ -0,0 +1,26 @@
|
||||
import ray
|
||||
from ray.autoscaler._private.kuberay.autoscaling_config import _generate_provider_config
|
||||
from ray.autoscaler._private.providers import _get_node_provider
|
||||
|
||||
|
||||
@ray.remote
|
||||
def count_non_terminated_nodes() -> int:
|
||||
"""Get the count of non terminated nodes for the Ray cluster raycluster-autoscaler
|
||||
in namespace default.
|
||||
"""
|
||||
provider_config = _generate_provider_config(ray_cluster_namespace="default")
|
||||
kuberay_node_provider = _get_node_provider(
|
||||
provider_config=provider_config, cluster_name="raycluster-autoscaler"
|
||||
)
|
||||
nodes = kuberay_node_provider.non_terminated_nodes({})
|
||||
return len(nodes)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
return ray.get(count_non_terminated_nodes.remote())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
ray.init("auto")
|
||||
out = main()
|
||||
print(out)
|
||||
@@ -0,0 +1,37 @@
|
||||
import ray
|
||||
from ray._common import test_utils
|
||||
|
||||
|
||||
def main():
|
||||
"""Removes CPU request, removes GPU actor.
|
||||
Waits for autoscaler scale-down events to get emitted to stdout.
|
||||
|
||||
The worker idle timeout is set to 10 seconds and the autoscaler's update interval is
|
||||
5 seconds, so it should be enough to wait 15 seconds.
|
||||
"""
|
||||
|
||||
# Before scale-down.
|
||||
cluster_resources = ray.cluster_resources()
|
||||
assert cluster_resources.get("CPU", 0) > 0, cluster_resources
|
||||
assert cluster_resources.get("GPU", 0) > 0, cluster_resources
|
||||
|
||||
# Remove resource demands
|
||||
ray.autoscaler.sdk.request_resources(num_cpus=0)
|
||||
gpu_actor = ray.get_actor("gpu_actor")
|
||||
ray.kill(gpu_actor)
|
||||
|
||||
# Wait for scale-down to happen.
|
||||
def verify():
|
||||
cluster_resources = ray.cluster_resources()
|
||||
# From head node
|
||||
assert cluster_resources.get("CPU", 0) == 1, cluster_resources
|
||||
assert cluster_resources.get("GPU", 0) == 0, cluster_resources
|
||||
|
||||
return True
|
||||
|
||||
test_utils.wait_for_condition(verify, timeout=60, retry_interval_ms=2000)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
ray.init("auto", namespace="gpu-test")
|
||||
main()
|
||||
@@ -0,0 +1,27 @@
|
||||
import ray
|
||||
from ray._common.test_utils import wait_for_condition
|
||||
|
||||
|
||||
def main():
|
||||
"""Submits CPU request"""
|
||||
ray.autoscaler.sdk.request_resources(num_cpus=2)
|
||||
from ray.autoscaler.v2.sdk import get_cluster_status
|
||||
from ray.autoscaler.v2.utils import ClusterStatusFormatter
|
||||
|
||||
gcs_address = ray.get_runtime_context().gcs_address
|
||||
|
||||
def verify():
|
||||
cluster_resources = ray.cluster_resources()
|
||||
|
||||
cluster_status = get_cluster_status(gcs_address)
|
||||
print(ClusterStatusFormatter.format(cluster_status, verbose=True))
|
||||
assert cluster_resources.get("CPU", 0) == 2, cluster_resources
|
||||
|
||||
return True
|
||||
|
||||
wait_for_condition(verify, timeout=60, retry_interval_ms=2000)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
ray.init("auto")
|
||||
main()
|
||||
@@ -0,0 +1,32 @@
|
||||
import time
|
||||
|
||||
import ray
|
||||
|
||||
|
||||
def main():
|
||||
"""Submits custom resource request.
|
||||
|
||||
Also, validates runtime env data submitted with the Ray Job that executes
|
||||
this script.
|
||||
"""
|
||||
|
||||
# Workers and head are annotated as having 5 "Custom2" capacity each,
|
||||
# so this should trigger upscaling of two workers.
|
||||
# (One of the bundles will be "placed" on the head.)
|
||||
ray.autoscaler.sdk.request_resources(
|
||||
bundles=[{"Custom2": 3}, {"Custom2": 3}, {"Custom2": 3}]
|
||||
)
|
||||
|
||||
while (
|
||||
ray.cluster_resources().get("Custom2", 0) < 3
|
||||
and ray.cluster_resources().get("Custom2", 0) < 6
|
||||
):
|
||||
time.sleep(0.1)
|
||||
|
||||
# Output something to validate the job logs.
|
||||
print("Submitted custom scale request!")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
ray.init("auto")
|
||||
main()
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
apiVersion: ray.io/v1
|
||||
kind: RayCluster
|
||||
metadata:
|
||||
name: raycluster-test
|
||||
spec:
|
||||
headGroupSpec:
|
||||
serviceType: ClusterIP
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: ray-test
|
||||
@@ -0,0 +1,8 @@
|
||||
from ray.tests.kuberay.utils import (
|
||||
setup_kuberay_operator,
|
||||
wait_for_raycluster_crd,
|
||||
)
|
||||
|
||||
if __name__ == "__main__":
|
||||
setup_kuberay_operator()
|
||||
wait_for_raycluster_crd()
|
||||
@@ -0,0 +1,4 @@
|
||||
from ray.tests.kuberay.utils import teardown_kuberay_operator
|
||||
|
||||
if __name__ == "__main__":
|
||||
teardown_kuberay_operator()
|
||||
@@ -0,0 +1,959 @@
|
||||
import copy
|
||||
import platform
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Optional, Type
|
||||
from unittest import mock
|
||||
|
||||
import pytest
|
||||
import requests
|
||||
import yaml
|
||||
|
||||
from ray.autoscaler._private.kuberay.autoscaling_config import (
|
||||
GKE_TPU_ACCELERATOR_LABEL,
|
||||
GKE_TPU_TOPOLOGY_LABEL,
|
||||
AutoscalingConfigProducer,
|
||||
_derive_autoscaling_config_from_ray_cr,
|
||||
_get_custom_resources,
|
||||
_get_num_tpus,
|
||||
_get_ray_resources_from_group_spec,
|
||||
_round_up_k8s_quantity,
|
||||
)
|
||||
from ray.autoscaler._private.kuberay.utils import tpu_node_selectors_to_type
|
||||
|
||||
AUTOSCALING_CONFIG_MODULE_PATH = "ray.autoscaler._private.kuberay.autoscaling_config"
|
||||
|
||||
|
||||
def get_basic_ray_cr() -> dict:
|
||||
"""Returns the example Ray CR included in the Ray documentation,
|
||||
modified to include a GPU worker group and a TPU worker group.
|
||||
"""
|
||||
cr_path = str(
|
||||
Path(__file__).resolve().parents[2]
|
||||
/ "autoscaler"
|
||||
/ "kuberay"
|
||||
/ "ray-cluster.complete.yaml"
|
||||
)
|
||||
config = yaml.safe_load(open(cr_path).read())
|
||||
gpu_group = copy.deepcopy(config["spec"]["workerGroupSpecs"][0])
|
||||
gpu_group["groupName"] = "gpu-group"
|
||||
gpu_group["template"]["spec"]["containers"][0]["resources"]["limits"].setdefault(
|
||||
"nvidia.com/gpu", 3
|
||||
)
|
||||
gpu_group["maxReplicas"] = 200
|
||||
config["spec"]["workerGroupSpecs"].append(gpu_group)
|
||||
tpu_group = copy.deepcopy(config["spec"]["workerGroupSpecs"][0])
|
||||
tpu_group["groupName"] = "tpu-group"
|
||||
tpu_group["template"]["spec"]["containers"][0]["resources"]["limits"].setdefault(
|
||||
"google.com/tpu", 4
|
||||
)
|
||||
tpu_group["template"]["spec"]["nodeSelector"] = {}
|
||||
tpu_group["template"]["spec"]["nodeSelector"][
|
||||
"cloud.google.com/gke-tpu-topology"
|
||||
] = "2x2x2"
|
||||
tpu_group["template"]["spec"]["nodeSelector"][
|
||||
"cloud.google.com/gke-tpu-accelerator"
|
||||
] = "tpu-v4-podslice"
|
||||
tpu_group["maxReplicas"] = 4
|
||||
tpu_group["numOfHosts"] = 2
|
||||
config["spec"]["workerGroupSpecs"].append(tpu_group)
|
||||
return config
|
||||
|
||||
|
||||
def _get_basic_autoscaling_config() -> dict:
|
||||
"""The expected autoscaling derived from the example Ray CR."""
|
||||
return {
|
||||
"cluster_name": "raycluster-complete",
|
||||
"provider": {
|
||||
"disable_node_updaters": True,
|
||||
"disable_launch_config_check": True,
|
||||
"foreground_node_launch": True,
|
||||
"worker_liveness_check": False,
|
||||
"namespace": "default",
|
||||
"type": "kuberay",
|
||||
},
|
||||
"available_node_types": {
|
||||
"headgroup": {
|
||||
"labels": {},
|
||||
"max_workers": 0,
|
||||
"min_workers": 0,
|
||||
"node_config": {},
|
||||
"resources": {
|
||||
"CPU": 1,
|
||||
"memory": 1000000000,
|
||||
"Custom1": 1,
|
||||
"Custom2": 5,
|
||||
},
|
||||
},
|
||||
"small-group": {
|
||||
"labels": {},
|
||||
"max_workers": 300,
|
||||
"min_workers": 0,
|
||||
"node_config": {},
|
||||
"resources": {
|
||||
"CPU": 1,
|
||||
"memory": 536870912,
|
||||
"Custom2": 5,
|
||||
"Custom3": 1,
|
||||
},
|
||||
},
|
||||
# Same as "small-group" with a GPU resource entry added
|
||||
# and modified max_workers.
|
||||
"gpu-group": {
|
||||
"labels": {},
|
||||
"max_workers": 200,
|
||||
"min_workers": 0,
|
||||
"node_config": {},
|
||||
"resources": {
|
||||
"CPU": 1,
|
||||
"memory": 536870912,
|
||||
"Custom2": 5,
|
||||
"Custom3": 1,
|
||||
"GPU": 3,
|
||||
},
|
||||
},
|
||||
# Same as "small-group" with a TPU resource entry added
|
||||
# and modified max_workers and node_config.
|
||||
"tpu-group": {
|
||||
"labels": {},
|
||||
"max_workers": 8,
|
||||
"min_workers": 0,
|
||||
"node_config": {},
|
||||
"resources": {
|
||||
"CPU": 1,
|
||||
"memory": 536870912,
|
||||
"Custom2": 5,
|
||||
"Custom3": 1,
|
||||
"TPU": 4,
|
||||
"TPU-v4-16-head": 1,
|
||||
},
|
||||
},
|
||||
},
|
||||
"auth": {},
|
||||
"cluster_synced_files": [],
|
||||
"file_mounts": {},
|
||||
"file_mounts_sync_continuously": False,
|
||||
"head_node_type": "headgroup",
|
||||
"head_setup_commands": [],
|
||||
"head_start_ray_commands": [],
|
||||
"idle_timeout_minutes": 1.0,
|
||||
"initialization_commands": [],
|
||||
"max_workers": 508,
|
||||
"setup_commands": [],
|
||||
"upscaling_speed": 1000,
|
||||
"worker_setup_commands": [],
|
||||
"worker_start_ray_commands": [],
|
||||
}
|
||||
|
||||
|
||||
def _get_ray_cr_no_cpu_error() -> dict:
|
||||
"""Incorrectly formatted Ray CR without num-cpus rayStartParam and without resource
|
||||
limits. Autoscaler should raise an error when reading this.
|
||||
"""
|
||||
cr = get_basic_ray_cr()
|
||||
# Verify that the num-cpus rayStartParam is not present for the worker type.
|
||||
assert "num-cpus" not in cr["spec"]["workerGroupSpecs"][0]["rayStartParams"]
|
||||
del cr["spec"]["workerGroupSpecs"][0]["template"]["spec"]["containers"][0][
|
||||
"resources"
|
||||
]["limits"]["cpu"]
|
||||
del cr["spec"]["workerGroupSpecs"][0]["template"]["spec"]["containers"][0][
|
||||
"resources"
|
||||
]["requests"]["cpu"]
|
||||
return cr
|
||||
|
||||
|
||||
def _get_no_cpu_error() -> str:
|
||||
return (
|
||||
"Autoscaler failed to detect `CPU` resources for group small-group."
|
||||
"\nSet the `--num-cpus` rayStartParam and/or "
|
||||
"the CPU resource limit for the Ray container."
|
||||
)
|
||||
|
||||
|
||||
def _get_ray_cr_with_overrides() -> dict:
|
||||
"""CR with memory, cpu, and gpu overrides from rayStartParams."""
|
||||
cr = get_basic_ray_cr()
|
||||
cr["spec"]["workerGroupSpecs"][0]["rayStartParams"]["memory"] = "300000000"
|
||||
# num-gpus rayStartParam with no gpus in container limits
|
||||
cr["spec"]["workerGroupSpecs"][0]["rayStartParams"]["num-gpus"] = "100"
|
||||
# num-gpus rayStartParam overriding gpus in container limits
|
||||
cr["spec"]["workerGroupSpecs"][1]["rayStartParams"]["num-gpus"] = "100"
|
||||
cr["spec"]["workerGroupSpecs"][0]["rayStartParams"]["num-cpus"] = "100"
|
||||
return cr
|
||||
|
||||
|
||||
def _get_autoscaling_config_with_overrides() -> dict:
|
||||
"""Autoscaling config with memory and gpu annotations."""
|
||||
config = _get_basic_autoscaling_config()
|
||||
config["available_node_types"]["small-group"]["resources"]["memory"] = 300000000
|
||||
config["available_node_types"]["small-group"]["resources"]["GPU"] = 100
|
||||
config["available_node_types"]["small-group"]["resources"]["CPU"] = 100
|
||||
config["available_node_types"]["gpu-group"]["resources"]["GPU"] = 100
|
||||
return config
|
||||
|
||||
|
||||
def _get_ray_cr_with_autoscaler_options() -> dict:
|
||||
cr = get_basic_ray_cr()
|
||||
cr["spec"]["autoscalerOptions"] = {
|
||||
"upscalingMode": "Conservative",
|
||||
"idleTimeoutSeconds": 300,
|
||||
}
|
||||
return cr
|
||||
|
||||
|
||||
def _get_ray_cr_with_tpu_custom_resource() -> dict:
|
||||
cr = get_basic_ray_cr()
|
||||
cr["spec"]["workerGroupSpecs"][2]["rayStartParams"][
|
||||
"resources"
|
||||
] = '"{"TPU": 4, "Custom2": 5, "Custom3": 1}"'
|
||||
# remove google.com/tpu k8s resource Pod limit
|
||||
del cr["spec"]["workerGroupSpecs"][2]["template"]["spec"]["containers"][0][
|
||||
"resources"
|
||||
]["limits"]["google.com/tpu"]
|
||||
|
||||
return cr
|
||||
|
||||
|
||||
def _get_ray_cr_with_tpu_k8s_resource_limit_and_custom_resource() -> dict:
|
||||
cr = get_basic_ray_cr()
|
||||
cr["spec"]["workerGroupSpecs"][2]["rayStartParams"][
|
||||
"resources"
|
||||
] = '"{"TPU": 4, "Custom2": 5, "Custom3": 1}"'
|
||||
|
||||
return cr
|
||||
|
||||
|
||||
def _get_ray_cr_with_top_level_labels() -> dict:
|
||||
"""CR with a top-level `labels` field."""
|
||||
cr = get_basic_ray_cr()
|
||||
# This top-level structured labels take priority.
|
||||
cr["spec"]["workerGroupSpecs"][0]["labels"] = {"instance-type": "mx5"}
|
||||
|
||||
# rayStartParams labels field should be ignored.
|
||||
cr["spec"]["workerGroupSpecs"][0]["rayStartParams"]["labels"] = "instance-type=n2"
|
||||
return cr
|
||||
|
||||
|
||||
def _get_autoscaling_config_with_top_level_labels() -> dict:
|
||||
config = _get_basic_autoscaling_config()
|
||||
config["available_node_types"]["small-group"]["labels"] = {"instance-type": "mx5"}
|
||||
return config
|
||||
|
||||
|
||||
def _get_ray_cr_with_invalid_top_level_labels() -> dict:
|
||||
"""CR with a syntactically invalid top-level `labels` field."""
|
||||
cr = get_basic_ray_cr()
|
||||
cr["spec"]["workerGroupSpecs"][0]["labels"] = {"!!invalid-key!!": "some-value"}
|
||||
return cr
|
||||
|
||||
|
||||
def _get_ray_cr_with_top_level_resources() -> dict:
|
||||
"""CR with a top-level `resources` field to test priority."""
|
||||
cr = get_basic_ray_cr()
|
||||
|
||||
# The top-level resources field should take priority.
|
||||
cr["spec"]["workerGroupSpecs"][1]["resources"] = {
|
||||
"CPU": "16",
|
||||
"GPU": "8",
|
||||
"memory": "2Gi",
|
||||
"CustomResource": "99",
|
||||
}
|
||||
# These rayStartParams should be ignored.
|
||||
cr["spec"]["workerGroupSpecs"][1]["rayStartParams"]["num-cpus"] = "1"
|
||||
cr["spec"]["workerGroupSpecs"][1]["rayStartParams"]["memory"] = "100000"
|
||||
cr["spec"]["workerGroupSpecs"][1]["rayStartParams"]["num-gpus"] = "2"
|
||||
cr["spec"]["workerGroupSpecs"][1]["rayStartParams"][
|
||||
"resources"
|
||||
] = '"{"Custom2": 1}"'
|
||||
return cr
|
||||
|
||||
|
||||
def _get_autoscaling_config_with_top_level_resources() -> dict:
|
||||
config = _get_basic_autoscaling_config()
|
||||
|
||||
config["available_node_types"]["gpu-group"]["resources"] = {
|
||||
"CPU": 16,
|
||||
"GPU": 8,
|
||||
"memory": 2147483648,
|
||||
"CustomResource": 99,
|
||||
}
|
||||
return config
|
||||
|
||||
|
||||
def _get_ray_cr_with_top_level_tpu_resource() -> dict:
|
||||
"""CR with a top-level `resources` field for the TPU custom resource."""
|
||||
cr = _get_ray_cr_with_tpu_k8s_resource_limit_and_custom_resource()
|
||||
|
||||
# The top-level field should take priority.
|
||||
cr["spec"]["workerGroupSpecs"][2]["resources"] = {"TPU": "8"}
|
||||
return cr
|
||||
|
||||
|
||||
def _get_ray_cr_with_no_tpus() -> dict:
|
||||
cr = get_basic_ray_cr()
|
||||
# remove TPU worker group
|
||||
cr["spec"]["workerGroupSpecs"].pop(2)
|
||||
|
||||
return cr
|
||||
|
||||
|
||||
def _get_ray_cr_with_only_requests() -> dict:
|
||||
"""CR contains only resource requests"""
|
||||
cr = get_basic_ray_cr()
|
||||
|
||||
for group in [cr["spec"]["headGroupSpec"]] + cr["spec"]["workerGroupSpecs"]:
|
||||
for container in group["template"]["spec"]["containers"]:
|
||||
container["resources"]["requests"] = container["resources"]["limits"]
|
||||
del container["resources"]["limits"]
|
||||
return cr
|
||||
|
||||
|
||||
def _get_ray_cr_with_labels() -> dict:
|
||||
"""CR with labels in rayStartParams of head and worker groups."""
|
||||
cr = get_basic_ray_cr()
|
||||
|
||||
# Pass invalid labels to the head group to test error handling.
|
||||
cr["spec"]["headGroupSpec"]["rayStartParams"]["labels"] = "!!ray.io/node-group=,"
|
||||
# Pass valid labels to each of the worker groups.
|
||||
cr["spec"]["workerGroupSpecs"][0]["rayStartParams"][
|
||||
"labels"
|
||||
] = "ray.io/availability-region=us-central2, ray.io/market-type=spot"
|
||||
cr["spec"]["workerGroupSpecs"][1]["rayStartParams"][
|
||||
"labels"
|
||||
] = "ray.io/accelerator-type=A100"
|
||||
cr["spec"]["workerGroupSpecs"][2]["rayStartParams"][
|
||||
"labels"
|
||||
] = "ray.io/accelerator-type=TPU-V4"
|
||||
return cr
|
||||
|
||||
|
||||
def _get_autoscaling_config_with_labels() -> dict:
|
||||
"""Autoscaling config with parsed labels for each group."""
|
||||
config = _get_basic_autoscaling_config()
|
||||
|
||||
# Since we passed invalid labels to the head group `rayStartParams`,
|
||||
# we expect an empty dictionary in the autoscaling config.
|
||||
config["available_node_types"]["headgroup"]["labels"] = {}
|
||||
config["available_node_types"]["small-group"]["labels"] = {
|
||||
"ray.io/availability-region": "us-central2",
|
||||
"ray.io/market-type": "spot",
|
||||
}
|
||||
config["available_node_types"]["gpu-group"]["labels"] = {
|
||||
"ray.io/accelerator-type": "A100"
|
||||
}
|
||||
config["available_node_types"]["tpu-group"]["labels"] = {
|
||||
"ray.io/accelerator-type": "TPU-V4"
|
||||
}
|
||||
return config
|
||||
|
||||
|
||||
def _get_autoscaling_config_with_options() -> dict:
|
||||
config = _get_basic_autoscaling_config()
|
||||
config["upscaling_speed"] = 1
|
||||
config["idle_timeout_minutes"] = 5.0
|
||||
return config
|
||||
|
||||
|
||||
def _get_tpu_group_with_no_node_selectors() -> dict[str, Any]:
|
||||
cr = get_basic_ray_cr()
|
||||
tpu_group = cr["spec"]["workerGroupSpecs"][2]
|
||||
tpu_group["template"]["spec"].pop("nodeSelector", None)
|
||||
return tpu_group
|
||||
|
||||
|
||||
def _get_tpu_group_without_accelerator_node_selector() -> dict[str, Any]:
|
||||
cr = get_basic_ray_cr()
|
||||
tpu_group = cr["spec"]["workerGroupSpecs"][2]
|
||||
tpu_group["template"]["spec"]["nodeSelector"].pop(GKE_TPU_ACCELERATOR_LABEL, None)
|
||||
return tpu_group
|
||||
|
||||
|
||||
def _get_tpu_group_without_topology_node_selector() -> dict[str, Any]:
|
||||
cr = get_basic_ray_cr()
|
||||
tpu_group = cr["spec"]["workerGroupSpecs"][2]
|
||||
tpu_group["template"]["spec"]["nodeSelector"].pop(GKE_TPU_TOPOLOGY_LABEL, None)
|
||||
return tpu_group
|
||||
|
||||
|
||||
def _get_tpu_group_with_v7x_node_selectors() -> dict[str, Any]:
|
||||
cr = get_basic_ray_cr()
|
||||
tpu_group = cr["spec"]["workerGroupSpecs"][2]
|
||||
tpu_group["template"]["spec"]["nodeSelector"][GKE_TPU_TOPOLOGY_LABEL] = "2x2x2"
|
||||
tpu_group["template"]["spec"]["nodeSelector"][GKE_TPU_ACCELERATOR_LABEL] = "tpu7x"
|
||||
return tpu_group
|
||||
|
||||
|
||||
def _get_ray_cr_with_tpu_v7x() -> dict[str, Any]:
|
||||
cr = get_basic_ray_cr()
|
||||
cr["spec"]["workerGroupSpecs"][2] = _get_tpu_group_with_v7x_node_selectors()
|
||||
return cr
|
||||
|
||||
|
||||
def _get_autoscaling_config_with_v7x() -> dict[str, Any]:
|
||||
config = _get_basic_autoscaling_config()
|
||||
config["available_node_types"]["tpu-group"]["resources"]["TPU-v7x-16-head"] = 1
|
||||
config["available_node_types"]["tpu-group"]["resources"].pop("TPU-v4-16-head", None)
|
||||
return config
|
||||
|
||||
|
||||
def _get_tpu_group_with_v5litepod_node_selectors() -> dict[str, Any]:
|
||||
cr = get_basic_ray_cr()
|
||||
tpu_group = cr["spec"]["workerGroupSpecs"][2]
|
||||
tpu_group["template"]["spec"]["nodeSelector"][GKE_TPU_TOPOLOGY_LABEL] = "2x4"
|
||||
tpu_group["template"]["spec"]["nodeSelector"][
|
||||
GKE_TPU_ACCELERATOR_LABEL
|
||||
] = "tpu-v5-lite-podslice"
|
||||
return tpu_group
|
||||
|
||||
|
||||
def _get_ray_cr_with_tpu_v5litepod() -> dict[str, Any]:
|
||||
cr = get_basic_ray_cr()
|
||||
cr["spec"]["workerGroupSpecs"][2] = _get_tpu_group_with_v5litepod_node_selectors()
|
||||
return cr
|
||||
|
||||
|
||||
def _get_autoscaling_config_with_v5litepod() -> dict[str, Any]:
|
||||
config = _get_basic_autoscaling_config()
|
||||
config["available_node_types"]["tpu-group"]["resources"]["TPU-v5litepod-8-head"] = 1
|
||||
config["available_node_types"]["tpu-group"]["resources"].pop("TPU-v4-16-head", None)
|
||||
return config
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"input,output",
|
||||
[
|
||||
# There's no particular discipline to these test cases.
|
||||
("100m", 1),
|
||||
("15001m", 16),
|
||||
("2", 2),
|
||||
("100Mi", 104857600),
|
||||
("1G", 1000000000),
|
||||
],
|
||||
)
|
||||
def test_resource_quantity(input: str, output: int):
|
||||
assert _round_up_k8s_quantity(input) == output, output
|
||||
|
||||
|
||||
PARAM_ARGS = ",".join(
|
||||
[
|
||||
"ray_cr_in",
|
||||
"expected_config_out",
|
||||
"expected_error",
|
||||
"expected_error_message",
|
||||
"expected_log_warning",
|
||||
]
|
||||
)
|
||||
|
||||
TEST_DATA = (
|
||||
[]
|
||||
if platform.system() == "Windows"
|
||||
else [
|
||||
pytest.param(
|
||||
get_basic_ray_cr(),
|
||||
_get_basic_autoscaling_config(),
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
id="basic",
|
||||
),
|
||||
pytest.param(
|
||||
_get_ray_cr_with_only_requests(),
|
||||
_get_basic_autoscaling_config(),
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
id="only-requests",
|
||||
),
|
||||
pytest.param(
|
||||
_get_ray_cr_no_cpu_error(),
|
||||
None,
|
||||
ValueError,
|
||||
_get_no_cpu_error(),
|
||||
None,
|
||||
id="no-cpu-error",
|
||||
),
|
||||
pytest.param(
|
||||
_get_ray_cr_with_overrides(),
|
||||
_get_autoscaling_config_with_overrides(),
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
id="overrides",
|
||||
),
|
||||
pytest.param(
|
||||
_get_ray_cr_with_autoscaler_options(),
|
||||
_get_autoscaling_config_with_options(),
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
id="autoscaler-options",
|
||||
),
|
||||
pytest.param(
|
||||
_get_ray_cr_with_tpu_custom_resource(),
|
||||
_get_basic_autoscaling_config(),
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
id="tpu-custom-resource",
|
||||
),
|
||||
pytest.param(
|
||||
get_basic_ray_cr(),
|
||||
_get_basic_autoscaling_config(),
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
id="tpu-k8s-resource-limit",
|
||||
),
|
||||
pytest.param(
|
||||
_get_ray_cr_with_tpu_k8s_resource_limit_and_custom_resource(),
|
||||
_get_basic_autoscaling_config(),
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
id="tpu-k8s-resource-limit-and-custom-resource",
|
||||
),
|
||||
pytest.param(
|
||||
_get_ray_cr_with_labels(),
|
||||
_get_basic_autoscaling_config(),
|
||||
None,
|
||||
None,
|
||||
"Ignoring labels: ray.io/accelerator-type=TPU-V4 set in rayStartParams for group 'tpu-group'. Group labels are supported in the top-level Labels field starting in KubeRay v1.5",
|
||||
id="groups-with-raystartparam-labels",
|
||||
),
|
||||
pytest.param(
|
||||
_get_ray_cr_with_top_level_labels(),
|
||||
_get_autoscaling_config_with_top_level_labels(),
|
||||
None,
|
||||
None,
|
||||
"Ignoring labels: instance-type=n2 set in rayStartParams for group 'small-group'. Group labels are supported in the top-level Labels field starting in KubeRay v1.5",
|
||||
id="groups-with-top-level-labels",
|
||||
),
|
||||
pytest.param(
|
||||
_get_ray_cr_with_invalid_top_level_labels(),
|
||||
_get_basic_autoscaling_config(),
|
||||
ValueError,
|
||||
None,
|
||||
None,
|
||||
id="invalid-top-level-labels",
|
||||
),
|
||||
pytest.param(
|
||||
_get_ray_cr_with_tpu_v7x(),
|
||||
_get_autoscaling_config_with_v7x(),
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
id="tpu-v7x",
|
||||
),
|
||||
pytest.param(
|
||||
_get_ray_cr_with_tpu_v5litepod(),
|
||||
_get_autoscaling_config_with_v5litepod(),
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
id="tpu-v5litepod",
|
||||
),
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.skipif(platform.system() == "Windows", reason="Not relevant.")
|
||||
@pytest.mark.parametrize(PARAM_ARGS, TEST_DATA)
|
||||
def test_autoscaling_config(
|
||||
ray_cr_in: Dict[str, Any],
|
||||
expected_config_out: Optional[Dict[str, Any]],
|
||||
expected_error: Optional[Type[Exception]],
|
||||
expected_error_message: Optional[str],
|
||||
expected_log_warning: Optional[str],
|
||||
):
|
||||
ray_cr_in["metadata"]["namespace"] = "default"
|
||||
# Reset log_once state to ensure each test case is independent.
|
||||
from ray.util.debug import _logged
|
||||
|
||||
_logged.clear()
|
||||
with mock.patch(f"{AUTOSCALING_CONFIG_MODULE_PATH}.logger") as mock_logger:
|
||||
if expected_error:
|
||||
with pytest.raises(expected_error, match=expected_error_message):
|
||||
_derive_autoscaling_config_from_ray_cr(ray_cr_in)
|
||||
else:
|
||||
assert (
|
||||
_derive_autoscaling_config_from_ray_cr(ray_cr_in) == expected_config_out
|
||||
)
|
||||
if expected_log_warning:
|
||||
mock_logger.warning.assert_called_with(expected_log_warning)
|
||||
else:
|
||||
mock_logger.warning.assert_not_called()
|
||||
|
||||
|
||||
@pytest.mark.skipif(platform.system() == "Windows", reason="Not relevant.")
|
||||
def test_cr_image_consistency():
|
||||
"""Verify that the example config uses the same Ray image for all Ray pods."""
|
||||
cr = get_basic_ray_cr()
|
||||
|
||||
group_specs = [cr["spec"]["headGroupSpec"]] + cr["spec"]["workerGroupSpecs"]
|
||||
# Head, CPU group, GPU group, TPU group.
|
||||
assert len(group_specs) == 4
|
||||
|
||||
ray_containers = [
|
||||
group_spec["template"]["spec"]["containers"][0] for group_spec in group_specs
|
||||
]
|
||||
|
||||
# All Ray containers in the example config have "ray-" in their name.
|
||||
assert all("ray-" in ray_container["name"] for ray_container in ray_containers)
|
||||
|
||||
# All Ray images are from the Ray repo.
|
||||
assert all(
|
||||
"rayproject/ray" in ray_container["image"] for ray_container in ray_containers
|
||||
)
|
||||
|
||||
# All Ray images are the same.
|
||||
assert len({ray_container["image"] for ray_container in ray_containers}) == 1
|
||||
|
||||
|
||||
@pytest.mark.parametrize("exception", [Exception, requests.HTTPError])
|
||||
@pytest.mark.parametrize("num_exceptions", range(6))
|
||||
def test_autoscaling_config_fetch_retries(exception, num_exceptions):
|
||||
"""Validates retry logic in
|
||||
AutoscalingConfigProducer._fetch_ray_cr_from_k8s_with_retries.
|
||||
"""
|
||||
|
||||
class MockKubernetesHttpApiClient:
|
||||
def __init__(self):
|
||||
self.exception_counter = 0
|
||||
|
||||
def get(self, *args, **kwargs):
|
||||
if self.exception_counter < num_exceptions:
|
||||
self.exception_counter += 1
|
||||
raise exception
|
||||
else:
|
||||
return {"ok-key": "ok-value"}
|
||||
|
||||
class MockAutoscalingConfigProducer(AutoscalingConfigProducer):
|
||||
def __init__(self, *args, **kwargs):
|
||||
self.kubernetes_api_client = MockKubernetesHttpApiClient()
|
||||
self._ray_cr_path = "rayclusters/mock"
|
||||
|
||||
config_producer = MockAutoscalingConfigProducer()
|
||||
# Patch retry backoff period.
|
||||
with mock.patch(
|
||||
"ray.autoscaler._private.kuberay.autoscaling_config.RAYCLUSTER_FETCH_RETRY_S",
|
||||
0,
|
||||
):
|
||||
# If you hit an exception and it's not HTTPError, expect to raise.
|
||||
# If you hit >= 5 exceptions, expect to raise.
|
||||
# Otherwise, don't expect to raise.
|
||||
if (
|
||||
num_exceptions > 0 and exception != requests.HTTPError
|
||||
) or num_exceptions >= 5:
|
||||
with pytest.raises(exception):
|
||||
config_producer._fetch_ray_cr_from_k8s_with_retries()
|
||||
else:
|
||||
out = config_producer._fetch_ray_cr_from_k8s_with_retries()
|
||||
assert out == {"ok-key": "ok-value"}
|
||||
|
||||
|
||||
TPU_TYPES_ARGS = ",".join(
|
||||
[
|
||||
"accelerator",
|
||||
"topology",
|
||||
"expected_tpu_type",
|
||||
]
|
||||
)
|
||||
TPU_TYPES_DATA = (
|
||||
[]
|
||||
if platform.system() == "Windows"
|
||||
else [
|
||||
pytest.param(
|
||||
"tpu-v4-podslice",
|
||||
None,
|
||||
None,
|
||||
id="tpu-none-topology",
|
||||
),
|
||||
pytest.param(
|
||||
None,
|
||||
"2x2x2",
|
||||
None,
|
||||
id="tpu-none-accelerator",
|
||||
),
|
||||
pytest.param(
|
||||
"tpu-v4-podslice",
|
||||
"2x2x2",
|
||||
"v4-16",
|
||||
id="tpu-v4-test",
|
||||
),
|
||||
pytest.param(
|
||||
"tpu-v5-lite-device",
|
||||
"2x2",
|
||||
"v5litepod-4",
|
||||
id="tpu-v5e-device-test",
|
||||
),
|
||||
pytest.param(
|
||||
"tpu-v5-lite-podslice",
|
||||
"2x4",
|
||||
"v5litepod-8",
|
||||
id="tpu-v5e-podslice-test",
|
||||
),
|
||||
pytest.param(
|
||||
"tpu-v5p-slice",
|
||||
"2x2x4",
|
||||
"v5p-32",
|
||||
id="tpu-v5p-test",
|
||||
),
|
||||
pytest.param(
|
||||
"tpu-v6e-slice",
|
||||
"16x16",
|
||||
"v6e-256",
|
||||
id="tpu-v6e-test",
|
||||
),
|
||||
pytest.param(
|
||||
"tpu7x",
|
||||
"2x2x2",
|
||||
"v7x-16",
|
||||
id="tpu-v7x-test",
|
||||
),
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.skipif(platform.system() == "Windows", reason="Not relevant.")
|
||||
@pytest.mark.parametrize(TPU_TYPES_ARGS, TPU_TYPES_DATA)
|
||||
def test_tpu_node_selectors_to_type(
|
||||
accelerator: str, topology: str, expected_tpu_type: str
|
||||
):
|
||||
"""Verify that tpu_node_selectors_to_type correctly returns TPU type from
|
||||
TPU nodeSelectors.
|
||||
"""
|
||||
tpu_type = tpu_node_selectors_to_type(topology, accelerator)
|
||||
assert expected_tpu_type == tpu_type
|
||||
|
||||
|
||||
TPU_PARAM_ARGS = ",".join(
|
||||
[
|
||||
"ray_cr_in",
|
||||
"expected_num_tpus",
|
||||
]
|
||||
)
|
||||
TPU_TEST_DATA = (
|
||||
[]
|
||||
if platform.system() == "Windows"
|
||||
else [
|
||||
pytest.param(
|
||||
get_basic_ray_cr(),
|
||||
4,
|
||||
id="tpu-k8s-resource-limits",
|
||||
),
|
||||
pytest.param(
|
||||
_get_ray_cr_with_tpu_custom_resource(),
|
||||
4,
|
||||
id="tpu-custom-resource",
|
||||
),
|
||||
pytest.param(
|
||||
_get_ray_cr_with_tpu_k8s_resource_limit_and_custom_resource(),
|
||||
4,
|
||||
id="tpu--k8s-resource-limits-and-custom-resource",
|
||||
),
|
||||
pytest.param(
|
||||
_get_ray_cr_with_no_tpus(),
|
||||
0,
|
||||
id="no-tpus-requested",
|
||||
),
|
||||
pytest.param(
|
||||
_get_ray_cr_with_top_level_tpu_resource(),
|
||||
8,
|
||||
id="tpu-top-level-resource",
|
||||
),
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.skipif(platform.system() == "Windows", reason="Not relevant.")
|
||||
@pytest.mark.parametrize(TPU_PARAM_ARGS, TPU_TEST_DATA)
|
||||
def test_get_num_tpus(ray_cr_in: Dict[str, Any], expected_num_tpus: int):
|
||||
"""Verify that _get_num_tpus correctly returns the number of requested TPUs."""
|
||||
for worker_group in ray_cr_in["spec"]["workerGroupSpecs"]:
|
||||
group_resources = worker_group.get("resources", {})
|
||||
ray_start_params = worker_group["rayStartParams"]
|
||||
custom_resources = _get_custom_resources(
|
||||
group_resources, ray_start_params, worker_group["groupName"]
|
||||
)
|
||||
k8s_resources = worker_group["template"]["spec"]["containers"][0]["resources"]
|
||||
|
||||
num_tpus = _get_num_tpus(group_resources, custom_resources, k8s_resources)
|
||||
|
||||
if worker_group["groupName"] == "tpu-group":
|
||||
assert num_tpus == expected_num_tpus
|
||||
else:
|
||||
assert num_tpus is None
|
||||
|
||||
|
||||
RAY_RESOURCES_PARAM_ARGS = ",".join(
|
||||
[
|
||||
"group_spec",
|
||||
"is_head",
|
||||
"expected_resources",
|
||||
]
|
||||
)
|
||||
RAY_RESOURCES_TEST_DATA = (
|
||||
[]
|
||||
if platform.system() == "Windows"
|
||||
else [
|
||||
pytest.param(
|
||||
get_basic_ray_cr()["spec"]["headGroupSpec"],
|
||||
True,
|
||||
{
|
||||
"CPU": 1,
|
||||
"memory": 1000000000,
|
||||
"Custom1": 1,
|
||||
"Custom2": 5,
|
||||
},
|
||||
id="head-group",
|
||||
),
|
||||
pytest.param(
|
||||
get_basic_ray_cr()["spec"]["workerGroupSpecs"][0],
|
||||
False,
|
||||
{
|
||||
"CPU": 1,
|
||||
"memory": 536870912,
|
||||
"Custom2": 5,
|
||||
"Custom3": 1,
|
||||
},
|
||||
id="cpu-group",
|
||||
),
|
||||
pytest.param(
|
||||
get_basic_ray_cr()["spec"]["workerGroupSpecs"][1],
|
||||
False,
|
||||
{
|
||||
"CPU": 1,
|
||||
"memory": 536870912,
|
||||
"Custom2": 5,
|
||||
"Custom3": 1,
|
||||
"GPU": 3,
|
||||
},
|
||||
id="gpu-group",
|
||||
),
|
||||
pytest.param(
|
||||
get_basic_ray_cr()["spec"]["workerGroupSpecs"][2],
|
||||
False,
|
||||
{
|
||||
"CPU": 1,
|
||||
"memory": 536870912,
|
||||
"Custom2": 5,
|
||||
"Custom3": 1,
|
||||
"TPU": 4,
|
||||
"TPU-v4-16-head": 1,
|
||||
},
|
||||
id="tpu-group",
|
||||
),
|
||||
pytest.param(
|
||||
_get_tpu_group_with_no_node_selectors(),
|
||||
False,
|
||||
{
|
||||
"CPU": 1,
|
||||
"memory": 536870912,
|
||||
"Custom2": 5,
|
||||
"Custom3": 1,
|
||||
"TPU": 4,
|
||||
},
|
||||
id="tpu-group-no-node-selectors",
|
||||
),
|
||||
pytest.param(
|
||||
_get_tpu_group_without_accelerator_node_selector(),
|
||||
False,
|
||||
{
|
||||
"CPU": 1,
|
||||
"memory": 536870912,
|
||||
"Custom2": 5,
|
||||
"Custom3": 1,
|
||||
"TPU": 4,
|
||||
},
|
||||
id="tpu-group-no-accelerator-node-selector",
|
||||
),
|
||||
pytest.param(
|
||||
_get_tpu_group_without_topology_node_selector(),
|
||||
False,
|
||||
{
|
||||
"CPU": 1,
|
||||
"memory": 536870912,
|
||||
"Custom2": 5,
|
||||
"Custom3": 1,
|
||||
"TPU": 4,
|
||||
},
|
||||
id="tpu-group-no-topology-node-selector",
|
||||
),
|
||||
pytest.param(
|
||||
_get_tpu_group_with_v7x_node_selectors(),
|
||||
False,
|
||||
{
|
||||
"CPU": 1,
|
||||
"memory": 536870912,
|
||||
"Custom2": 5,
|
||||
"Custom3": 1,
|
||||
"TPU": 4,
|
||||
"TPU-v7x-16-head": 1,
|
||||
},
|
||||
id="tpu-group-v7x",
|
||||
),
|
||||
pytest.param(
|
||||
_get_tpu_group_with_v5litepod_node_selectors(),
|
||||
False,
|
||||
{
|
||||
"CPU": 1,
|
||||
"memory": 536870912,
|
||||
"Custom2": 5,
|
||||
"Custom3": 1,
|
||||
"TPU": 4,
|
||||
"TPU-v5litepod-8-head": 1,
|
||||
},
|
||||
id="tpu-group-v5litepod",
|
||||
),
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.skipif(platform.system() == "Windows", reason="Not relevant.")
|
||||
@pytest.mark.parametrize(RAY_RESOURCES_PARAM_ARGS, RAY_RESOURCES_TEST_DATA)
|
||||
def test_get_ray_resources_from_group_spec(
|
||||
group_spec: Dict[str, Any],
|
||||
is_head: bool,
|
||||
expected_resources: Dict[str, Any],
|
||||
):
|
||||
assert _get_ray_resources_from_group_spec(group_spec, is_head) == expected_resources
|
||||
|
||||
|
||||
@pytest.mark.skipif(platform.system() == "Windows", reason="Not relevant.")
|
||||
def test_top_level_resources_override_warnings():
|
||||
"""
|
||||
Verify all override warnings are logged when a top-level `resources` field is used in
|
||||
addition to specifying those resources in the rayStartParams.
|
||||
"""
|
||||
ray_cr_in = _get_ray_cr_with_top_level_resources()
|
||||
ray_cr_in["metadata"]["namespace"] = "default"
|
||||
|
||||
with mock.patch(f"{AUTOSCALING_CONFIG_MODULE_PATH}.logger") as mock_logger:
|
||||
_derive_autoscaling_config_from_ray_cr(ray_cr_in)
|
||||
|
||||
expected_calls = [
|
||||
mock.call(
|
||||
"'CPU' specified in both the top-level 'resources' field and in 'rayStartParams'. "
|
||||
"Using the value from 'resources': 16."
|
||||
),
|
||||
mock.call(
|
||||
"'GPU' specified in both the top-level 'resources' field and in 'rayStartParams'. "
|
||||
"Using the value from 'resources': 8."
|
||||
),
|
||||
mock.call(
|
||||
"'memory' specified in both the top-level 'resources' field and in 'rayStartParams'. "
|
||||
"Using the value from 'resources': 2Gi."
|
||||
),
|
||||
mock.call(
|
||||
"custom resources specified in both the top-level 'resources' field and in 'rayStartParams'. "
|
||||
"Using the values from 'resources': {'CPU': '16', 'GPU': '8', 'memory': '2Gi', 'CustomResource': '99'}."
|
||||
),
|
||||
]
|
||||
|
||||
# Assert that all expected calls were made, in any order.
|
||||
mock_logger.warning.assert_has_calls(expected_calls, any_order=True)
|
||||
assert mock_logger.warning.call_count == 4
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(pytest.main(["-v", __file__]))
|
||||
@@ -0,0 +1,384 @@
|
||||
import base64
|
||||
import copy
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from typing import Any, Dict
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
from ray.tests.kuberay.utils import (
|
||||
get_pod,
|
||||
get_pod_names,
|
||||
get_raycluster,
|
||||
kubectl_delete,
|
||||
kubectl_exec_python_script,
|
||||
kubectl_logs,
|
||||
switch_to_ray_parent_dir,
|
||||
wait_for_pod_to_start,
|
||||
wait_for_pods,
|
||||
wait_for_ray_health,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# This image will be used for both the Ray nodes and the autoscaler.
|
||||
# The CI should pass an image built from the test branch.
|
||||
RAY_IMAGE = os.environ.get("RAY_IMAGE", "rayproject/ray:nightly-py38")
|
||||
# By default, use the same image for the autoscaler and Ray containers.
|
||||
AUTOSCALER_IMAGE = os.environ.get("AUTOSCALER_IMAGE", RAY_IMAGE)
|
||||
# Set to IfNotPresent in kind CI.
|
||||
PULL_POLICY = os.environ.get("PULL_POLICY", "IfNotPresent")
|
||||
# Set to enable autoscaler v2
|
||||
AUTOSCALER_V2 = os.environ.get("AUTOSCALER_V2", "False")
|
||||
logger.info(f"Using image `{RAY_IMAGE}` for Ray containers.")
|
||||
logger.info(f"Using image `{AUTOSCALER_IMAGE}` for Autoscaler containers.")
|
||||
logger.info(f"Using pull policy `{PULL_POLICY}` for all images.")
|
||||
logger.info(f"Using autoscaler v2: {AUTOSCALER_V2}")
|
||||
|
||||
# Path to example config inside the rayci container.
|
||||
EXAMPLE_CLUSTER_PATH = (
|
||||
"rayci/python/ray/tests/kuberay/test_files/ray-cluster.autoscaler-template.yaml"
|
||||
)
|
||||
EXAMPLE_CLUSTER_PATH_V2 = (
|
||||
"rayci/python/ray/tests/kuberay/test_files/ray-cluster.autoscaler-v2-template.yaml"
|
||||
)
|
||||
|
||||
HEAD_SERVICE = "raycluster-autoscaler-head-svc"
|
||||
HEAD_POD_PREFIX = "raycluster-autoscaler-head"
|
||||
CPU_WORKER_PREFIX = "raycluster-autoscaler-worker-small-group"
|
||||
RAY_CLUSTER_NAME = "raycluster-autoscaler"
|
||||
RAY_CLUSTER_NAMESPACE = "default"
|
||||
|
||||
# Test runs longer than the default timeout.
|
||||
pytestmark = pytest.mark.timeout(300)
|
||||
|
||||
|
||||
class KubeRayAutoscalingTest(unittest.TestCase):
|
||||
"""e2e verification of autoscaling following the steps in the Ray documentation.
|
||||
kubectl is used throughout, as that reflects the instructions in the docs.
|
||||
"""
|
||||
|
||||
def _get_ray_cr_config(
|
||||
self, min_replicas=0, cpu_replicas=0, gpu_replicas=0
|
||||
) -> Dict[str, Any]:
|
||||
"""Get Ray CR config yaml.
|
||||
|
||||
- Use configurable replica fields for a CPU workerGroup.
|
||||
|
||||
- Add a GPU-annotated group for testing GPU upscaling.
|
||||
|
||||
- Fill in Ray image, autoscaler image, and image pull policies from env
|
||||
variables.
|
||||
"""
|
||||
if AUTOSCALER_V2 == "True":
|
||||
with open(EXAMPLE_CLUSTER_PATH_V2) as ray_cr_config_file:
|
||||
ray_cr_config_str = ray_cr_config_file.read()
|
||||
else:
|
||||
with open(EXAMPLE_CLUSTER_PATH) as ray_cr_config_file:
|
||||
ray_cr_config_str = ray_cr_config_file.read()
|
||||
|
||||
for k8s_object in yaml.safe_load_all(ray_cr_config_str):
|
||||
if k8s_object["kind"] in ["RayCluster", "RayJob", "RayService"]:
|
||||
config = k8s_object
|
||||
break
|
||||
head_group = config["spec"]["headGroupSpec"]
|
||||
if "rayStartParams" not in head_group:
|
||||
head_group["rayStartParams"] = {}
|
||||
head_group["rayStartParams"][
|
||||
"resources"
|
||||
] = '"{\\"Custom1\\": 1, \\"Custom2\\": 5}"'
|
||||
|
||||
cpu_group = config["spec"]["workerGroupSpecs"][0]
|
||||
cpu_group["replicas"] = cpu_replicas
|
||||
cpu_group["minReplicas"] = min_replicas
|
||||
# Keep maxReplicas big throughout the test.
|
||||
cpu_group["maxReplicas"] = 300
|
||||
if "rayStartParams" not in cpu_group:
|
||||
cpu_group["rayStartParams"] = {}
|
||||
cpu_group["rayStartParams"][
|
||||
"resources"
|
||||
] = '"{\\"Custom1\\": 1, \\"Custom2\\": 5}"'
|
||||
|
||||
# Add a GPU-annotated group.
|
||||
# (We're not using real GPUs, just adding a GPU annotation for the autoscaler
|
||||
# and Ray scheduler.)
|
||||
gpu_group = copy.deepcopy(cpu_group)
|
||||
if "rayStartParams" not in gpu_group:
|
||||
gpu_group["rayStartParams"] = {}
|
||||
gpu_group["rayStartParams"]["num-gpus"] = "1"
|
||||
gpu_group["replicas"] = gpu_replicas
|
||||
gpu_group["minReplicas"] = 0
|
||||
gpu_group["maxReplicas"] = 1
|
||||
gpu_group["groupName"] = "fake-gpu-group"
|
||||
config["spec"]["workerGroupSpecs"].append(gpu_group)
|
||||
|
||||
# Substitute images.
|
||||
for group_spec in config["spec"]["workerGroupSpecs"] + [
|
||||
config["spec"]["headGroupSpec"]
|
||||
]:
|
||||
containers = group_spec["template"]["spec"]["containers"]
|
||||
|
||||
ray_container = containers[0]
|
||||
# Confirm the first container in the example config is the Ray container.
|
||||
assert ray_container["name"] in ["ray-head", "ray-worker"]
|
||||
# ("machine-learning" is the name of the worker Ray container)
|
||||
|
||||
ray_container["image"] = RAY_IMAGE
|
||||
|
||||
for container in containers:
|
||||
container["imagePullPolicy"] = PULL_POLICY
|
||||
|
||||
autoscaler_options = {
|
||||
"image": AUTOSCALER_IMAGE,
|
||||
"imagePullPolicy": PULL_POLICY,
|
||||
# Allow quick scale-down for test purposes.
|
||||
"idleTimeoutSeconds": 10,
|
||||
}
|
||||
config["spec"]["autoscalerOptions"] = autoscaler_options
|
||||
|
||||
return config
|
||||
|
||||
def _apply_ray_cr(
|
||||
self,
|
||||
min_replicas=0,
|
||||
cpu_replicas=0,
|
||||
gpu_replicas=0,
|
||||
validate_replicas: bool = False,
|
||||
) -> None:
|
||||
"""Apply Ray CR config yaml, with configurable replica fields for the cpu
|
||||
workerGroup.
|
||||
|
||||
If the CR does not yet exist, `replicas` can be set as desired.
|
||||
If the CR does already exist, the recommended usage is this:
|
||||
(1) Set `cpu_replicas` and `gpu_replicas` to what we currently expect them
|
||||
to be.
|
||||
(2) Set `validate_replicas` to True. We will then check that the replicas
|
||||
set on the CR coincides with `replicas`.
|
||||
"""
|
||||
if validate_replicas:
|
||||
raycluster = get_raycluster(
|
||||
RAY_CLUSTER_NAME, namespace=RAY_CLUSTER_NAMESPACE
|
||||
)
|
||||
assert raycluster["spec"]["workerGroupSpecs"][0]["replicas"] == cpu_replicas
|
||||
assert raycluster["spec"]["workerGroupSpecs"][1]["replicas"] == gpu_replicas
|
||||
logger.info(
|
||||
f"Validated that cpu and gpu worker replicas for "
|
||||
f"{RAY_CLUSTER_NAME} are currently {cpu_replicas} and"
|
||||
f" {gpu_replicas}, respectively."
|
||||
)
|
||||
cr_config = self._get_ray_cr_config(
|
||||
min_replicas=min_replicas,
|
||||
cpu_replicas=cpu_replicas,
|
||||
gpu_replicas=gpu_replicas,
|
||||
)
|
||||
|
||||
with tempfile.NamedTemporaryFile("w") as config_file:
|
||||
yaml.dump(cr_config, config_file)
|
||||
config_file.flush()
|
||||
|
||||
subprocess.check_call(
|
||||
["kubectl", "apply", "-f", config_file.name],
|
||||
stdout=sys.stdout,
|
||||
stderr=sys.stderr,
|
||||
)
|
||||
|
||||
def testAutoscaling(self):
|
||||
"""Test the following behaviors:
|
||||
|
||||
1. Spinning up a Ray cluster
|
||||
2. Scaling up Ray workers via autoscaler.sdk.request_resources()
|
||||
3. Scaling up by updating the CRD's minReplicas
|
||||
4. Scaling down by removing the resource request and reducing maxReplicas
|
||||
5. Autoscaler recognizes GPU annotations and Ray custom resources.
|
||||
6. Autoscaler and operator ignore pods marked for deletion.
|
||||
7. Autoscaler logs work. Autoscaler events are piped to the driver.
|
||||
8. Ray utils show correct resource limits in the head container.
|
||||
|
||||
TODO (Dmitri): Split up the test logic.
|
||||
Too much is stuffed into this one test case.
|
||||
|
||||
Resources requested by this test are safely within the bounds of an m5.xlarge
|
||||
instance.
|
||||
|
||||
The resource REQUESTS are:
|
||||
- One Ray head pod
|
||||
- Autoscaler: .25 CPU, .5 Gi memory
|
||||
- Ray node: .5 CPU, .5 Gi memeory
|
||||
- Three Worker pods
|
||||
- Ray node: .5 CPU, .5 Gi memory
|
||||
Total: 2.25 CPU, 2.5 Gi memory.
|
||||
|
||||
Including operator and system pods, the total CPU requested is around 3.
|
||||
|
||||
The cpu LIMIT of each Ray container is 1.
|
||||
The `num-cpus` arg to Ray start is 1 for each Ray container; thus Ray accounts
|
||||
1 CPU for each Ray node in the test.
|
||||
"""
|
||||
switch_to_ray_parent_dir()
|
||||
|
||||
# Cluster creation
|
||||
logger.info("Creating a RayCluster with no worker pods.")
|
||||
self._apply_ray_cr(min_replicas=0, cpu_replicas=0, gpu_replicas=0)
|
||||
|
||||
logger.info("Confirming presence of head.")
|
||||
wait_for_pods(goal_num_pods=1, namespace=RAY_CLUSTER_NAMESPACE)
|
||||
|
||||
logger.info("Waiting for head pod to start Running.")
|
||||
wait_for_pod_to_start(
|
||||
pod_name_filter=HEAD_POD_PREFIX, namespace=RAY_CLUSTER_NAMESPACE
|
||||
)
|
||||
logger.info("Confirming Ray is up on the head pod.")
|
||||
wait_for_ray_health(
|
||||
pod_name_filter=HEAD_POD_PREFIX, namespace=RAY_CLUSTER_NAMESPACE
|
||||
)
|
||||
|
||||
head_pod = get_pod(
|
||||
pod_name_filter=HEAD_POD_PREFIX, namespace=RAY_CLUSTER_NAMESPACE
|
||||
)
|
||||
assert head_pod, "Could not find the Ray head pod."
|
||||
|
||||
# Confirm head pod resource allocation.
|
||||
# (This is a misplaced test of Ray's resource detection in containers.
|
||||
# See the TODO in the docstring.)
|
||||
logger.info("Confirming head pod resource allocation.")
|
||||
out = kubectl_exec_python_script( # Interaction mode #1: `kubectl exec`
|
||||
script_name="check_cpu_and_memory.py",
|
||||
pod=head_pod,
|
||||
container="ray-head",
|
||||
namespace="default",
|
||||
)
|
||||
|
||||
# Scale-up
|
||||
logger.info("Scaling up to one worker via Ray resource request.")
|
||||
# The request for 2 cpus should give us a 1-cpu head (already present) and a
|
||||
# 1-cpu worker (will await scale-up).
|
||||
kubectl_exec_python_script( # Interaction mode #1: `kubectl exec`
|
||||
script_name="scale_up.py",
|
||||
pod=head_pod,
|
||||
container="ray-head",
|
||||
namespace="default",
|
||||
)
|
||||
# Check that stdout autoscaler logging is working.
|
||||
logs = kubectl_logs(head_pod, namespace="default", container="autoscaler")
|
||||
assert "Adding 1 node(s) of type small-group." in logs
|
||||
logger.info("Confirming number of workers.")
|
||||
wait_for_pods(goal_num_pods=2, namespace=RAY_CLUSTER_NAMESPACE)
|
||||
|
||||
# Ray CR updates.
|
||||
logger.info("Scaling up to two workers by editing minReplicas.")
|
||||
# replicas=1 reflects the current number of workers
|
||||
# (which is what we expect to be already present in the Ray CR)
|
||||
self._apply_ray_cr(
|
||||
min_replicas=2,
|
||||
cpu_replicas=1,
|
||||
gpu_replicas=0,
|
||||
# Confirm CPU, GPU replicas set on the Ray CR by the autoscaler are 1, 0:
|
||||
validate_replicas=True,
|
||||
)
|
||||
logger.info("Confirming number of workers.")
|
||||
wait_for_pods(goal_num_pods=3, namespace=RAY_CLUSTER_NAMESPACE)
|
||||
|
||||
# GPU upscaling.
|
||||
# 1. Check we haven't spuriously already started a fake GPU node.
|
||||
assert not any(
|
||||
"gpu" in pod_name
|
||||
for pod_name in get_pod_names(namespace=RAY_CLUSTER_NAMESPACE)
|
||||
)
|
||||
# 2. Trigger GPU upscaling by requesting placement of a GPU actor.
|
||||
logger.info("Scheduling an Actor with GPU demands.")
|
||||
kubectl_exec_python_script(
|
||||
script_name="gpu_actor_placement.py",
|
||||
pod=head_pod,
|
||||
container="ray-head",
|
||||
namespace="default",
|
||||
)
|
||||
|
||||
# 3. Confirm new pod number and presence of fake GPU worker.
|
||||
logger.info("Confirming fake GPU worker up-scaling.")
|
||||
wait_for_pods(goal_num_pods=4, namespace=RAY_CLUSTER_NAMESPACE)
|
||||
|
||||
gpu_workers = [
|
||||
pod_name
|
||||
for pod_name in get_pod_names(namespace=RAY_CLUSTER_NAMESPACE)
|
||||
if "gpu" in pod_name
|
||||
]
|
||||
assert len(gpu_workers) == 1
|
||||
# 4. Confirm that the GPU actor is up and that Ray believes
|
||||
# the node the actor is on has a GPU.
|
||||
logger.info("Confirming GPU actor placement.")
|
||||
out = kubectl_exec_python_script(
|
||||
script_name="gpu_actor_validation.py",
|
||||
pod=head_pod,
|
||||
container="ray-head",
|
||||
namespace="default",
|
||||
)
|
||||
|
||||
# Confirms the actor was placed on a GPU-annotated node.
|
||||
# (See gpu_actor_validation.py for details.)
|
||||
assert "on-a-gpu-node" in out
|
||||
|
||||
# Scale-down
|
||||
logger.info("Reducing min workers to 0.")
|
||||
# Max workers remains 300.
|
||||
self._apply_ray_cr(
|
||||
min_replicas=0,
|
||||
cpu_replicas=2,
|
||||
gpu_replicas=1,
|
||||
# Confirm CPU, GPU replicas set on the Ray CR by the autoscaler are 2, 1:
|
||||
validate_replicas=True,
|
||||
)
|
||||
logger.info("Removing resource demands.")
|
||||
kubectl_exec_python_script(
|
||||
script_name="scale_down.py",
|
||||
pod=head_pod,
|
||||
container="ray-head",
|
||||
namespace="default",
|
||||
)
|
||||
# Autoscaler should trigger scale-down after resource demands are removed.
|
||||
logger.info("Confirming workers are gone.")
|
||||
# Check that stdout autoscaler logging is working.
|
||||
logs = kubectl_logs(head_pod, namespace="default", container="autoscaler")
|
||||
assert "Removing 1 nodes of type fake-gpu-group (idle)." in logs
|
||||
wait_for_pods(goal_num_pods=1, namespace=RAY_CLUSTER_NAMESPACE, tries=120)
|
||||
|
||||
# Check custom resource upscaling.
|
||||
|
||||
# Submit two {"Custom2": 3} bundles to upscale two workers with 5
|
||||
# Custom2 capacity each.
|
||||
logger.info("Scaling up workers with request for custom resources.")
|
||||
out = kubectl_exec_python_script(
|
||||
script_name="scale_up_custom.py",
|
||||
pod=head_pod,
|
||||
container="ray-head",
|
||||
namespace="default",
|
||||
)
|
||||
assert "Submitted custom scale request!" in out, out
|
||||
|
||||
logger.info("Confirming two workers have scaled up.")
|
||||
wait_for_pods(goal_num_pods=3, namespace=RAY_CLUSTER_NAMESPACE)
|
||||
|
||||
# Cluster deletion
|
||||
logger.info("Deleting Ray cluster.")
|
||||
kubectl_delete(
|
||||
kind="raycluster", name=RAY_CLUSTER_NAME, namespace=RAY_CLUSTER_NAMESPACE
|
||||
)
|
||||
logger.info("Confirming Ray pods are gone.")
|
||||
wait_for_pods(goal_num_pods=0, namespace=RAY_CLUSTER_NAMESPACE)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
kubeconfig_base64 = os.environ.get("KUBECONFIG_BASE64")
|
||||
if kubeconfig_base64:
|
||||
kubeconfig_file = os.environ.get("KUBECONFIG")
|
||||
if not kubeconfig_file:
|
||||
raise ValueError("When KUBECONFIG_BASE64 is set, KUBECONFIG must be set.")
|
||||
|
||||
with open(kubeconfig_file, "wb") as f:
|
||||
f.write(base64.b64decode(kubeconfig_base64))
|
||||
|
||||
sys.exit(pytest.main(["-vv", __file__]))
|
||||
@@ -0,0 +1,607 @@
|
||||
apiVersion: v1
|
||||
items:
|
||||
- apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
annotations:
|
||||
ray.io/ft-enabled: "false"
|
||||
ray.io/health-state: ""
|
||||
creationTimestamp: "2022-11-14T23:10:15Z"
|
||||
generateName: raycluster-autoscaler-head-
|
||||
labels:
|
||||
app.kubernetes.io/created-by: kuberay-operator
|
||||
app.kubernetes.io/name: kuberay
|
||||
ray.io/cluster: raycluster-autoscaler
|
||||
ray.io/cluster-dashboard: raycluster-autoscaler-dashboard
|
||||
ray.io/group: headgroup
|
||||
ray.io/identifier: raycluster-autoscaler-head
|
||||
ray.io/is-ray-node: "yes"
|
||||
ray.io/node-type: head
|
||||
name: raycluster-autoscaler-head-8zsc8
|
||||
namespace: default
|
||||
ownerReferences:
|
||||
- apiVersion: ray.io/v1alpha1
|
||||
blockOwnerDeletion: true
|
||||
controller: true
|
||||
kind: RayCluster
|
||||
name: raycluster-autoscaler
|
||||
uid: ec79effb-0295-4f40-b08b-8633aa7f786a
|
||||
resourceVersion: "4519"
|
||||
uid: 539ea57c-8d51-4503-a395-08779efb3bf0
|
||||
spec:
|
||||
containers:
|
||||
- args:
|
||||
- 'ulimit -n 65536; ray start --head --resources="{\"Custom1\": 1, \"Custom2\":
|
||||
5}" --block --dashboard-host=0.0.0.0 --metrics-export-port=8080 --no-monitor --num-cpus=1 --memory=1000000000 '
|
||||
command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- --
|
||||
env:
|
||||
- name: RAY_IP
|
||||
value: 127.0.0.1
|
||||
- name: RAY_PORT
|
||||
value: "6379"
|
||||
- name: RAY_ADDRESS
|
||||
value: 127.0.0.1:6379
|
||||
- name: REDIS_PASSWORD
|
||||
image: gekho/ray
|
||||
imagePullPolicy: Always
|
||||
lifecycle:
|
||||
preStop:
|
||||
exec:
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- ray stop
|
||||
name: ray-head
|
||||
ports:
|
||||
- containerPort: 6379
|
||||
name: gcs
|
||||
protocol: TCP
|
||||
- containerPort: 8265
|
||||
name: dashboard
|
||||
protocol: TCP
|
||||
- containerPort: 10001
|
||||
name: client
|
||||
protocol: TCP
|
||||
- containerPort: 8080
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
resources:
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: 1G
|
||||
requests:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
terminationMessagePath: /dev/termination-log
|
||||
terminationMessagePolicy: File
|
||||
volumeMounts:
|
||||
- mountPath: /dev/shm
|
||||
name: shared-mem
|
||||
- mountPath: /tmp/ray
|
||||
name: ray-logs
|
||||
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
|
||||
name: kube-api-access-tmxvr
|
||||
readOnly: true
|
||||
- args:
|
||||
- kuberay-autoscaler
|
||||
- --cluster-name
|
||||
- $(RAY_CLUSTER_NAME)
|
||||
- --cluster-namespace
|
||||
- $(RAY_CLUSTER_NAMESPACE)
|
||||
command:
|
||||
- ray
|
||||
env:
|
||||
- name: RAY_CLUSTER_NAME
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
apiVersion: v1
|
||||
fieldPath: metadata.labels['ray.io/cluster']
|
||||
- name: RAY_CLUSTER_NAMESPACE
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
apiVersion: v1
|
||||
fieldPath: metadata.namespace
|
||||
image: gekho/ray
|
||||
imagePullPolicy: Always
|
||||
name: autoscaler
|
||||
resources:
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
requests:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
terminationMessagePath: /dev/termination-log
|
||||
terminationMessagePolicy: File
|
||||
volumeMounts:
|
||||
- mountPath: /tmp/ray
|
||||
name: ray-logs
|
||||
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
|
||||
name: kube-api-access-tmxvr
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirst
|
||||
enableServiceLinks: true
|
||||
nodeName: gke-cluster-1-default-pool-a5503908-181p
|
||||
preemptionPolicy: PreemptLowerPriority
|
||||
priority: 0
|
||||
restartPolicy: Always
|
||||
schedulerName: default-scheduler
|
||||
securityContext: {}
|
||||
serviceAccount: raycluster-autoscaler
|
||||
serviceAccountName: raycluster-autoscaler
|
||||
terminationGracePeriodSeconds: 30
|
||||
tolerations:
|
||||
- effect: NoExecute
|
||||
key: node.kubernetes.io/not-ready
|
||||
operator: Exists
|
||||
tolerationSeconds: 300
|
||||
- effect: NoExecute
|
||||
key: node.kubernetes.io/unreachable
|
||||
operator: Exists
|
||||
tolerationSeconds: 300
|
||||
volumes:
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 512Mi
|
||||
name: shared-mem
|
||||
- emptyDir: {}
|
||||
name: ray-logs
|
||||
- name: kube-api-access-tmxvr
|
||||
projected:
|
||||
defaultMode: 420
|
||||
sources:
|
||||
- serviceAccountToken:
|
||||
expirationSeconds: 3607
|
||||
path: token
|
||||
- configMap:
|
||||
items:
|
||||
- key: ca.crt
|
||||
path: ca.crt
|
||||
name: kube-root-ca.crt
|
||||
- downwardAPI:
|
||||
items:
|
||||
- fieldRef:
|
||||
apiVersion: v1
|
||||
fieldPath: metadata.namespace
|
||||
path: namespace
|
||||
status:
|
||||
conditions:
|
||||
- lastProbeTime: null
|
||||
lastTransitionTime: "2022-11-14T23:10:15Z"
|
||||
status: "True"
|
||||
type: Initialized
|
||||
- lastProbeTime: null
|
||||
lastTransitionTime: "2022-11-14T23:11:23Z"
|
||||
status: "True"
|
||||
type: Ready
|
||||
- lastProbeTime: null
|
||||
lastTransitionTime: "2022-11-14T23:11:23Z"
|
||||
status: "True"
|
||||
type: ContainersReady
|
||||
- lastProbeTime: null
|
||||
lastTransitionTime: "2022-11-14T23:10:15Z"
|
||||
status: "True"
|
||||
type: PodScheduled
|
||||
containerStatuses:
|
||||
- containerID: containerd://0b008432be839bec8dd97437d3f2be9ac8d7f017b91067a46ec45a487f141ebf
|
||||
image: docker.io/gekho/ray:latest
|
||||
imageID: docker.io/gekho/ray@sha256:7859a78d1a089bb88691864d5c4a2aad529f5353d7d9c82cc0274842fbda242b
|
||||
lastState: {}
|
||||
name: autoscaler
|
||||
ready: true
|
||||
restartCount: 0
|
||||
started: true
|
||||
state:
|
||||
running:
|
||||
startedAt: "2022-11-14T23:11:23Z"
|
||||
- containerID: containerd://b2aae80ed028cc41bad1e350bb70a0a4e8ea722df098b38781efabe54adbc5ec
|
||||
image: docker.io/gekho/ray:latest
|
||||
imageID: docker.io/gekho/ray@sha256:7859a78d1a089bb88691864d5c4a2aad529f5353d7d9c82cc0274842fbda242b
|
||||
lastState: {}
|
||||
name: ray-head
|
||||
ready: true
|
||||
restartCount: 0
|
||||
started: true
|
||||
state:
|
||||
running:
|
||||
startedAt: "2022-11-14T23:11:22Z"
|
||||
hostIP: 10.128.0.45
|
||||
phase: Running
|
||||
podIP: 10.4.2.6
|
||||
podIPs:
|
||||
- ip: 10.4.2.6
|
||||
qosClass: Burstable
|
||||
startTime: "2022-11-14T23:10:15Z"
|
||||
- apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
annotations:
|
||||
key: value
|
||||
ray.io/ft-enabled: "false"
|
||||
ray.io/health-state: ""
|
||||
creationTimestamp: "2022-11-14T23:11:45Z"
|
||||
deletionGracePeriodSeconds: 30
|
||||
deletionTimestamp: "2022-11-14T23:12:20Z"
|
||||
generateName: raycluster-autoscaler-worker-small-group-
|
||||
labels:
|
||||
app.kubernetes.io/created-by: kuberay-operator
|
||||
app.kubernetes.io/name: kuberay
|
||||
key: value
|
||||
ray.io/cluster: raycluster-autoscaler
|
||||
ray.io/cluster-dashboard: raycluster-autoscaler-dashboard
|
||||
ray.io/group: small-group
|
||||
ray.io/identifier: raycluster-autoscaler-worker
|
||||
ray.io/is-ray-node: "yes"
|
||||
ray.io/node-type: worker
|
||||
name: raycluster-autoscaler-worker-small-group-4wxfm
|
||||
namespace: default
|
||||
ownerReferences:
|
||||
- apiVersion: ray.io/v1alpha1
|
||||
blockOwnerDeletion: true
|
||||
controller: true
|
||||
kind: RayCluster
|
||||
name: raycluster-autoscaler
|
||||
uid: ec79effb-0295-4f40-b08b-8633aa7f786a
|
||||
resourceVersion: "4845"
|
||||
uid: 3698ed9b-7e06-4d47-b9f6-09e4bd08365a
|
||||
spec:
|
||||
containers:
|
||||
- args:
|
||||
- 'ulimit -n 65536; ray start --resources="{\"Custom1\": 1, \"Custom2\": 5}" --address=raycluster-autoscaler-head-svc:6379 --metrics-export-port=8080 --num-cpus=1 --memory=536870912 --block '
|
||||
command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- --
|
||||
env:
|
||||
- name: RAY_IP
|
||||
value: raycluster-autoscaler-head-svc
|
||||
- name: RAY_PORT
|
||||
value: "6379"
|
||||
- name: RAY_ADDRESS
|
||||
value: raycluster-autoscaler-head-svc:6379
|
||||
- name: REDIS_PASSWORD
|
||||
image: gekho/ray
|
||||
imagePullPolicy: Always
|
||||
lifecycle:
|
||||
preStop:
|
||||
exec:
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- ray stop
|
||||
name: machine-learning
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
resources:
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: 512Mi
|
||||
requests:
|
||||
cpu: 500m
|
||||
memory: 256Mi
|
||||
terminationMessagePath: /dev/termination-log
|
||||
terminationMessagePolicy: File
|
||||
volumeMounts:
|
||||
- mountPath: /dev/shm
|
||||
name: shared-mem
|
||||
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
|
||||
name: kube-api-access-wthw9
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirst
|
||||
enableServiceLinks: true
|
||||
initContainers:
|
||||
- command:
|
||||
- sh
|
||||
- -c
|
||||
- until nslookup $RAY_IP.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local;
|
||||
do echo waiting for myservice; sleep 2; done
|
||||
env:
|
||||
- name: RAY_IP
|
||||
value: raycluster-autoscaler-head-svc
|
||||
image: busybox:1.28
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: init-myservice
|
||||
resources: {}
|
||||
terminationMessagePath: /dev/termination-log
|
||||
terminationMessagePolicy: File
|
||||
volumeMounts:
|
||||
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
|
||||
name: kube-api-access-wthw9
|
||||
readOnly: true
|
||||
nodeName: gke-cluster-1-default-pool-a5503908-dpst
|
||||
preemptionPolicy: PreemptLowerPriority
|
||||
priority: 0
|
||||
restartPolicy: Always
|
||||
schedulerName: default-scheduler
|
||||
securityContext: {}
|
||||
serviceAccount: default
|
||||
serviceAccountName: default
|
||||
terminationGracePeriodSeconds: 30
|
||||
tolerations:
|
||||
- effect: NoExecute
|
||||
key: node.kubernetes.io/not-ready
|
||||
operator: Exists
|
||||
tolerationSeconds: 300
|
||||
- effect: NoExecute
|
||||
key: node.kubernetes.io/unreachable
|
||||
operator: Exists
|
||||
tolerationSeconds: 300
|
||||
volumes:
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 256Mi
|
||||
name: shared-mem
|
||||
- name: kube-api-access-wthw9
|
||||
projected:
|
||||
defaultMode: 420
|
||||
sources:
|
||||
- serviceAccountToken:
|
||||
expirationSeconds: 3607
|
||||
path: token
|
||||
- configMap:
|
||||
items:
|
||||
- key: ca.crt
|
||||
path: ca.crt
|
||||
name: kube-root-ca.crt
|
||||
- downwardAPI:
|
||||
items:
|
||||
- fieldRef:
|
||||
apiVersion: v1
|
||||
fieldPath: metadata.namespace
|
||||
path: namespace
|
||||
status:
|
||||
conditions:
|
||||
- lastProbeTime: null
|
||||
lastTransitionTime: "2022-11-14T23:11:47Z"
|
||||
status: "True"
|
||||
type: Initialized
|
||||
- lastProbeTime: null
|
||||
lastTransitionTime: "2022-11-14T23:11:45Z"
|
||||
message: 'containers with unready status: [machine-learning]'
|
||||
reason: ContainersNotReady
|
||||
status: "False"
|
||||
type: Ready
|
||||
- lastProbeTime: null
|
||||
lastTransitionTime: "2022-11-14T23:11:45Z"
|
||||
message: 'containers with unready status: [machine-learning]'
|
||||
reason: ContainersNotReady
|
||||
status: "False"
|
||||
type: ContainersReady
|
||||
- lastProbeTime: null
|
||||
lastTransitionTime: "2022-11-14T23:11:45Z"
|
||||
status: "True"
|
||||
type: PodScheduled
|
||||
containerStatuses:
|
||||
- image: gekho/ray
|
||||
imageID: ""
|
||||
lastState: {}
|
||||
name: machine-learning
|
||||
ready: false
|
||||
restartCount: 0
|
||||
started: false
|
||||
state:
|
||||
waiting:
|
||||
reason: PodInitializing
|
||||
hostIP: 10.128.0.31
|
||||
initContainerStatuses:
|
||||
- containerID: containerd://c7f5a0c3f63957213213ed1ebb6446cd205bd60346d010a879c5fa24e37f5145
|
||||
image: docker.io/library/busybox:1.28
|
||||
imageID: docker.io/library/busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47
|
||||
lastState: {}
|
||||
name: init-myservice
|
||||
ready: true
|
||||
restartCount: 0
|
||||
state:
|
||||
terminated:
|
||||
containerID: containerd://c7f5a0c3f63957213213ed1ebb6446cd205bd60346d010a879c5fa24e37f5145
|
||||
exitCode: 0
|
||||
finishedAt: "2022-11-14T23:11:47Z"
|
||||
reason: Completed
|
||||
startedAt: "2022-11-14T23:11:47Z"
|
||||
phase: Pending
|
||||
podIP: 10.4.0.4
|
||||
podIPs:
|
||||
- ip: 10.4.0.4
|
||||
qosClass: Burstable
|
||||
startTime: "2022-11-14T23:11:45Z"
|
||||
- apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
annotations:
|
||||
key: value
|
||||
ray.io/ft-enabled: "false"
|
||||
ray.io/health-state: ""
|
||||
creationTimestamp: "2022-11-14T23:11:50Z"
|
||||
generateName: raycluster-autoscaler-worker-small-group-
|
||||
labels:
|
||||
app.kubernetes.io/created-by: kuberay-operator
|
||||
app.kubernetes.io/name: kuberay
|
||||
key: value
|
||||
ray.io/cluster: raycluster-autoscaler
|
||||
ray.io/cluster-dashboard: raycluster-autoscaler-dashboard
|
||||
ray.io/group: small-group
|
||||
ray.io/identifier: raycluster-autoscaler-worker
|
||||
ray.io/is-ray-node: "yes"
|
||||
ray.io/node-type: worker
|
||||
name: raycluster-autoscaler-worker-small-group-dkz2r
|
||||
namespace: default
|
||||
ownerReferences:
|
||||
- apiVersion: ray.io/v1alpha1
|
||||
blockOwnerDeletion: true
|
||||
controller: true
|
||||
kind: RayCluster
|
||||
name: raycluster-autoscaler
|
||||
uid: ec79effb-0295-4f40-b08b-8633aa7f786a
|
||||
resourceVersion: "4776"
|
||||
uid: b4fb3233-6024-48a8-9f4f-a18f5e490629
|
||||
spec:
|
||||
containers:
|
||||
- args:
|
||||
- 'ulimit -n 65536; ray start --block --resources="{\"Custom1\": 1, \"Custom2\":
|
||||
5}" --address=raycluster-autoscaler-head-svc:6379 --metrics-export-port=8080 --num-cpus=1 --memory=536870912 '
|
||||
command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- --
|
||||
env:
|
||||
- name: RAY_IP
|
||||
value: raycluster-autoscaler-head-svc
|
||||
- name: RAY_PORT
|
||||
value: "6379"
|
||||
- name: RAY_ADDRESS
|
||||
value: raycluster-autoscaler-head-svc:6379
|
||||
- name: REDIS_PASSWORD
|
||||
image: gekho/ray
|
||||
imagePullPolicy: Always
|
||||
lifecycle:
|
||||
preStop:
|
||||
exec:
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- ray stop
|
||||
name: machine-learning
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
name: metrics
|
||||
protocol: TCP
|
||||
resources:
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: 512Mi
|
||||
requests:
|
||||
cpu: 500m
|
||||
memory: 256Mi
|
||||
terminationMessagePath: /dev/termination-log
|
||||
terminationMessagePolicy: File
|
||||
volumeMounts:
|
||||
- mountPath: /dev/shm
|
||||
name: shared-mem
|
||||
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
|
||||
name: kube-api-access-djtd9
|
||||
readOnly: true
|
||||
dnsPolicy: ClusterFirst
|
||||
enableServiceLinks: true
|
||||
initContainers:
|
||||
- command:
|
||||
- sh
|
||||
- -c
|
||||
- until nslookup $RAY_IP.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local;
|
||||
do echo waiting for myservice; sleep 2; done
|
||||
env:
|
||||
- name: RAY_IP
|
||||
value: raycluster-autoscaler-head-svc
|
||||
image: busybox:1.28
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: init-myservice
|
||||
resources: {}
|
||||
terminationMessagePath: /dev/termination-log
|
||||
terminationMessagePolicy: File
|
||||
volumeMounts:
|
||||
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
|
||||
name: kube-api-access-djtd9
|
||||
readOnly: true
|
||||
nodeName: gke-cluster-1-default-pool-a5503908-j51d
|
||||
preemptionPolicy: PreemptLowerPriority
|
||||
priority: 0
|
||||
restartPolicy: Always
|
||||
schedulerName: default-scheduler
|
||||
securityContext: {}
|
||||
serviceAccount: default
|
||||
serviceAccountName: default
|
||||
terminationGracePeriodSeconds: 30
|
||||
tolerations:
|
||||
- effect: NoExecute
|
||||
key: node.kubernetes.io/not-ready
|
||||
operator: Exists
|
||||
tolerationSeconds: 300
|
||||
- effect: NoExecute
|
||||
key: node.kubernetes.io/unreachable
|
||||
operator: Exists
|
||||
tolerationSeconds: 300
|
||||
volumes:
|
||||
- emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 256Mi
|
||||
name: shared-mem
|
||||
- name: kube-api-access-djtd9
|
||||
projected:
|
||||
defaultMode: 420
|
||||
sources:
|
||||
- serviceAccountToken:
|
||||
expirationSeconds: 3607
|
||||
path: token
|
||||
- configMap:
|
||||
items:
|
||||
- key: ca.crt
|
||||
path: ca.crt
|
||||
name: kube-root-ca.crt
|
||||
- downwardAPI:
|
||||
items:
|
||||
- fieldRef:
|
||||
apiVersion: v1
|
||||
fieldPath: metadata.namespace
|
||||
path: namespace
|
||||
status:
|
||||
conditions:
|
||||
- lastProbeTime: null
|
||||
lastTransitionTime: "2022-11-14T23:11:51Z"
|
||||
status: "True"
|
||||
type: Initialized
|
||||
- lastProbeTime: null
|
||||
lastTransitionTime: "2022-11-14T23:11:50Z"
|
||||
message: 'containers with unready status: [machine-learning]'
|
||||
reason: ContainersNotReady
|
||||
status: "False"
|
||||
type: Ready
|
||||
- lastProbeTime: null
|
||||
lastTransitionTime: "2022-11-14T23:11:50Z"
|
||||
message: 'containers with unready status: [machine-learning]'
|
||||
reason: ContainersNotReady
|
||||
status: "False"
|
||||
type: ContainersReady
|
||||
- lastProbeTime: null
|
||||
lastTransitionTime: "2022-11-14T23:11:50Z"
|
||||
status: "True"
|
||||
type: PodScheduled
|
||||
containerStatuses:
|
||||
- image: gekho/ray
|
||||
imageID: ""
|
||||
lastState: {}
|
||||
name: machine-learning
|
||||
ready: false
|
||||
restartCount: 0
|
||||
started: false
|
||||
state:
|
||||
waiting:
|
||||
reason: PodInitializing
|
||||
hostIP: 10.128.0.43
|
||||
initContainerStatuses:
|
||||
- containerID: containerd://672d9a5836e27a17f57a4e15e1d86431cfee6f2edef1210d60e864e3c510aac0
|
||||
image: docker.io/library/busybox:1.28
|
||||
imageID: docker.io/library/busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47
|
||||
lastState: {}
|
||||
name: init-myservice
|
||||
ready: true
|
||||
restartCount: 0
|
||||
state:
|
||||
terminated:
|
||||
containerID: containerd://672d9a5836e27a17f57a4e15e1d86431cfee6f2edef1210d60e864e3c510aac0
|
||||
exitCode: 0
|
||||
finishedAt: "2022-11-14T23:11:51Z"
|
||||
reason: Completed
|
||||
startedAt: "2022-11-14T23:11:51Z"
|
||||
phase: Pending
|
||||
podIP: 10.4.1.8
|
||||
podIPs:
|
||||
- ip: 10.4.1.8
|
||||
qosClass: Burstable
|
||||
startTime: "2022-11-14T23:11:50Z"
|
||||
kind: List
|
||||
metadata:
|
||||
resourceVersion: ""
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,73 @@
|
||||
# `ray-cluster.autoscaler-template.yaml` is a template for the RayCluster CR and
|
||||
# is used by the function `_get_ray_cr_config` in `test_autoscaling_e2e.py`.
|
||||
# [Note]
|
||||
# (1) The VM test runner only has 4 CPUs, so we lower the CPU requests.
|
||||
# (2) `test_autoscaling_e2e.py` assumes that each Ray Pod has 1 logical CPU.
|
||||
apiVersion: ray.io/v1
|
||||
kind: RayCluster
|
||||
metadata:
|
||||
name: raycluster-autoscaler
|
||||
spec:
|
||||
# The version of Ray you are using. Make sure all Ray containers are running this version of Ray.
|
||||
rayVersion: '2.46.0'
|
||||
# If `enableInTreeAutoscaling` is true, the Autoscaler sidecar will be added to the Ray head pod.
|
||||
# Ray Autoscaler integration is Beta with KubeRay >= 0.3.0 and Ray >= 2.0.0.
|
||||
enableInTreeAutoscaling: true
|
||||
autoscalerOptions:
|
||||
upscalingMode: Default
|
||||
idleTimeoutSeconds: 60
|
||||
imagePullPolicy: IfNotPresent
|
||||
resources:
|
||||
limits:
|
||||
cpu: "500m"
|
||||
memory: "512Mi"
|
||||
requests:
|
||||
cpu: "500m"
|
||||
memory: "512Mi"
|
||||
headGroupSpec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: ray-head
|
||||
image: rayproject/ray:2.46.0
|
||||
ports:
|
||||
- containerPort: 6379
|
||||
name: gcs
|
||||
- containerPort: 8265
|
||||
name: dashboard
|
||||
- containerPort: 10001
|
||||
name: client
|
||||
imagePullPolicy: IfNotPresent
|
||||
lifecycle:
|
||||
preStop:
|
||||
exec:
|
||||
command: ["/bin/sh","-c","ray stop"]
|
||||
resources:
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: "2G"
|
||||
requests:
|
||||
cpu: "500m"
|
||||
memory: "2G"
|
||||
workerGroupSpecs:
|
||||
- replicas: 1
|
||||
minReplicas: 1
|
||||
maxReplicas: 10
|
||||
groupName: small-group
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: ray-worker
|
||||
image: rayproject/ray:2.46.0
|
||||
lifecycle:
|
||||
preStop:
|
||||
exec:
|
||||
command: ["/bin/sh","-c","ray stop"]
|
||||
imagePullPolicy: IfNotPresent
|
||||
resources:
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: "1G"
|
||||
requests:
|
||||
cpu: "500m"
|
||||
memory: "1G"
|
||||
@@ -0,0 +1,76 @@
|
||||
# This is a copy from the `ray-cluster.autoscaler-template.yaml` with modifications needed
|
||||
# to make kuberary work with Ray's Autoscaler V2.
|
||||
# See more at `ray-cluster.autoscaler-template.yaml` for the non autoscaler-v2 definition.
|
||||
apiVersion: ray.io/v1
|
||||
kind: RayCluster
|
||||
metadata:
|
||||
name: raycluster-autoscaler
|
||||
spec:
|
||||
# The version of Ray you are using. Make sure all Ray containers are running this version of Ray.
|
||||
rayVersion: '2.46.0'
|
||||
# If `enableInTreeAutoscaling` is true, the Autoscaler sidecar will be added to the Ray head pod.
|
||||
# Ray Autoscaler integration is Beta with KubeRay >= 0.3.0 and Ray >= 2.0.0.
|
||||
enableInTreeAutoscaling: true
|
||||
autoscalerOptions:
|
||||
# Use version: v2 instead of env var RAY_enable_autoscaler_v2 and restartPolicy: Never below if
|
||||
# you're using KubeRay >= 1.4.0.
|
||||
version: v2
|
||||
upscalingMode: Default
|
||||
idleTimeoutSeconds: 60
|
||||
imagePullPolicy: IfNotPresent
|
||||
resources:
|
||||
limits:
|
||||
cpu: "500m"
|
||||
memory: "512Mi"
|
||||
requests:
|
||||
cpu: "500m"
|
||||
memory: "512Mi"
|
||||
headGroupSpec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: ray-head
|
||||
image: rayproject/ray:2.46.0
|
||||
ports:
|
||||
- containerPort: 6379
|
||||
name: gcs
|
||||
- containerPort: 8265
|
||||
name: dashboard
|
||||
- containerPort: 10001
|
||||
name: client
|
||||
imagePullPolicy: IfNotPresent
|
||||
lifecycle:
|
||||
preStop:
|
||||
exec:
|
||||
command: ["/bin/sh","-c","ray stop"]
|
||||
resources:
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: "2G"
|
||||
requests:
|
||||
cpu: "500m"
|
||||
memory: "2G"
|
||||
restartPolicy: Never # No restart to avoid reuse of pod for different ray nodes.
|
||||
workerGroupSpecs:
|
||||
- replicas: 1
|
||||
minReplicas: 1
|
||||
maxReplicas: 10
|
||||
groupName: small-group
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: ray-worker
|
||||
image: rayproject/ray:2.46.0
|
||||
lifecycle:
|
||||
preStop:
|
||||
exec:
|
||||
command: ["/bin/sh","-c","ray stop"]
|
||||
imagePullPolicy: IfNotPresent
|
||||
resources:
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: "1G"
|
||||
requests:
|
||||
cpu: "500m"
|
||||
memory: "1G"
|
||||
restartPolicy: Never # Never restart a pod to avoid pod reuse
|
||||
@@ -0,0 +1,371 @@
|
||||
import copy
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
from typing import List, Set
|
||||
from unittest import mock
|
||||
|
||||
import jsonpatch
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
from ray.autoscaler._private.kuberay.node_provider import (
|
||||
KubeRayNodeProvider,
|
||||
ScaleRequest,
|
||||
_worker_group_index,
|
||||
_worker_group_max_replicas,
|
||||
_worker_group_replicas,
|
||||
)
|
||||
from ray.autoscaler._private.util import NodeID
|
||||
from ray.autoscaler.batching_node_provider import NodeData
|
||||
from ray.tests.kuberay.test_autoscaling_config import get_basic_ray_cr
|
||||
|
||||
|
||||
def _get_basic_ray_cr_workers_to_delete(
|
||||
cpu_workers_to_delete: List[NodeID],
|
||||
gpu_workers_to_delete: List[NodeID],
|
||||
tpu_workers_to_delete: List[NodeID],
|
||||
):
|
||||
"""Generate a Ray cluster with non-empty workersToDelete field."""
|
||||
raycluster = get_basic_ray_cr()
|
||||
raycluster["spec"]["workerGroupSpecs"][0]["scaleStrategy"] = {
|
||||
"workersToDelete": cpu_workers_to_delete
|
||||
}
|
||||
raycluster["spec"]["workerGroupSpecs"][1]["scaleStrategy"] = {
|
||||
"workersToDelete": gpu_workers_to_delete
|
||||
}
|
||||
raycluster["spec"]["workerGroupSpecs"][2]["scaleStrategy"] = {
|
||||
"workersToDelete": tpu_workers_to_delete
|
||||
}
|
||||
return raycluster
|
||||
|
||||
|
||||
def _get_test_yaml(file_name):
|
||||
file_path = str(Path(__file__).resolve().parent / "test_files" / file_name)
|
||||
return yaml.safe_load(open(file_path).read())
|
||||
|
||||
|
||||
@pytest.mark.skipif(sys.platform.startswith("win"), reason="Not relevant on Windows.")
|
||||
@pytest.mark.parametrize(
|
||||
"group_name,expected_index", [("small-group", 0), ("gpu-group", 1)]
|
||||
)
|
||||
def test_worker_group_index(group_name, expected_index):
|
||||
"""Basic unit test for _worker_group_index.
|
||||
|
||||
Uses a RayCluster CR with worker groups "small-group" and "gpu-group",
|
||||
listed in that order.
|
||||
"""
|
||||
raycluster_cr = get_basic_ray_cr()
|
||||
assert _worker_group_index(raycluster_cr, group_name) == expected_index
|
||||
|
||||
|
||||
@pytest.mark.skipif(sys.platform.startswith("win"), reason="Not relevant on Windows.")
|
||||
@pytest.mark.parametrize(
|
||||
"group_index,expected_max_replicas,expected_replicas",
|
||||
[(0, 300, 1), (1, 200, 1), (2, 4, 1), (3, None, 0)],
|
||||
)
|
||||
def test_worker_group_replicas(group_index, expected_max_replicas, expected_replicas):
|
||||
"""Basic unit test for _worker_group_max_replicas and _worker_group_replicas
|
||||
|
||||
Uses a RayCluster CR with worker groups with 300 maxReplicas, 200 maxReplicas,
|
||||
and unspecified maxReplicas, in that order.
|
||||
"""
|
||||
raycluster = get_basic_ray_cr()
|
||||
|
||||
# Add a worker group without maxReplicas to confirm behavior
|
||||
# when maxReplicas is not specified.
|
||||
no_max_replicas_group = copy.deepcopy(raycluster["spec"]["workerGroupSpecs"][0])
|
||||
no_max_replicas_group["groupName"] = "no-max-replicas"
|
||||
del no_max_replicas_group["maxReplicas"]
|
||||
# Also, replicas field, just for the sake of testing.
|
||||
no_max_replicas_group["replicas"] = 0
|
||||
raycluster["spec"]["workerGroupSpecs"].append(no_max_replicas_group)
|
||||
|
||||
assert _worker_group_max_replicas(raycluster, group_index) == expected_max_replicas
|
||||
assert _worker_group_replicas(raycluster, group_index) == expected_replicas
|
||||
|
||||
|
||||
@pytest.mark.skipif(sys.platform.startswith("win"), reason="Not relevant on Windows.")
|
||||
@pytest.mark.parametrize(
|
||||
"attempted_target_replica_count,expected_target_replica_count",
|
||||
[(200, 200), (250, 250), (300, 300), (400, 300), (1000, 300)],
|
||||
)
|
||||
def test_create_node_cap_at_max(
|
||||
attempted_target_replica_count: int, expected_target_replica_count: int
|
||||
):
|
||||
"""Validates that KubeRayNodeProvider does not attempt to create more nodes than
|
||||
allowed by maxReplicas. For the config in this test, maxReplicas is fixed at 300.
|
||||
|
||||
Args:
|
||||
attempted_target_replica_count: The mocked desired replica count for a given
|
||||
worker group.
|
||||
expected_target_replica_count: The actual requested replicaCount. Should be
|
||||
capped at maxReplicas (300, for the config in this test.)
|
||||
"""
|
||||
raycluster = get_basic_ray_cr()
|
||||
with mock.patch.object(KubeRayNodeProvider, "__init__", return_value=None):
|
||||
kr_node_provider = KubeRayNodeProvider(provider_config={}, cluster_name="fake")
|
||||
scale_request = ScaleRequest(
|
||||
workers_to_delete=set(),
|
||||
desired_num_workers={"small-group": attempted_target_replica_count},
|
||||
)
|
||||
patch = kr_node_provider._scale_request_to_patch_payload(
|
||||
scale_request=scale_request, raycluster=raycluster
|
||||
)
|
||||
assert patch[0]["value"] == expected_target_replica_count
|
||||
|
||||
|
||||
@pytest.mark.skipif(sys.platform.startswith("win"), reason="Not relevant on Windows.")
|
||||
@pytest.mark.parametrize(
|
||||
"podlist_file,expected_node_data",
|
||||
[
|
||||
(
|
||||
# Pod list obtained by running kubectl get pod -o yaml at runtime.
|
||||
"podlist1.yaml",
|
||||
{
|
||||
"raycluster-autoscaler-head-8zsc8": NodeData(
|
||||
kind="head",
|
||||
type="headgroup",
|
||||
replica_index=None,
|
||||
ip="10.4.2.6",
|
||||
status="up-to-date",
|
||||
), # up-to-date status because the Ray container is in running status
|
||||
"raycluster-autoscaler-worker-small-group-dkz2r": NodeData(
|
||||
kind="worker",
|
||||
type="small-group",
|
||||
replica_index=None,
|
||||
ip="10.4.1.8",
|
||||
status="waiting",
|
||||
), # waiting status, because Ray container's state is "waiting".
|
||||
# The pod list includes a worker with non-null deletion timestamp.
|
||||
# It is excluded from the node data because it is considered
|
||||
# "terminated".
|
||||
},
|
||||
),
|
||||
(
|
||||
# Pod list obtained by running kubectl get pod -o yaml at runtime.
|
||||
"podlist2.yaml",
|
||||
{
|
||||
"raycluster-autoscaler-head-8zsc8": NodeData(
|
||||
kind="head",
|
||||
type="headgroup",
|
||||
replica_index=None,
|
||||
ip="10.4.2.6",
|
||||
status="up-to-date",
|
||||
),
|
||||
"raycluster-autoscaler-worker-fake-gpu-group-2qnhv": NodeData(
|
||||
kind="worker",
|
||||
type="fake-gpu-group",
|
||||
replica_index=None,
|
||||
ip="10.4.0.6",
|
||||
status="up-to-date",
|
||||
),
|
||||
"raycluster-autoscaler-worker-small-group-dkz2r": NodeData(
|
||||
kind="worker",
|
||||
type="small-group",
|
||||
replica_index=None,
|
||||
ip="10.4.1.8",
|
||||
status="up-to-date",
|
||||
),
|
||||
"raycluster-autoscaler-worker-small-group-lbfm4": NodeData(
|
||||
kind="worker",
|
||||
type="small-group",
|
||||
replica_index=None,
|
||||
ip="10.4.0.5",
|
||||
status="up-to-date",
|
||||
),
|
||||
"raycluster-autoscaler-tpu-group-worker-s8jhq": NodeData(
|
||||
kind="worker",
|
||||
type="tpu-group",
|
||||
replica_index="tpu-group-0",
|
||||
ip="10.24.9.4",
|
||||
status="up-to-date",
|
||||
),
|
||||
"raycluster-autoscaler-tpu-group-worker-jd69f": NodeData(
|
||||
kind="worker",
|
||||
type="tpu-group",
|
||||
replica_index="tpu-group-0",
|
||||
ip="10.24.8.4",
|
||||
status="up-to-date",
|
||||
),
|
||||
},
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_get_node_data(podlist_file: str, expected_node_data):
|
||||
"""Test translation of a K8s pod list into autoscaler node data."""
|
||||
pod_list = _get_test_yaml(podlist_file)
|
||||
|
||||
def mock_get(node_provider, path):
|
||||
if "pods" in path:
|
||||
return pod_list
|
||||
elif "raycluster" in path:
|
||||
return get_basic_ray_cr()
|
||||
else:
|
||||
raise ValueError("Invalid path.")
|
||||
|
||||
with mock.patch.object(
|
||||
KubeRayNodeProvider, "__init__", return_value=None
|
||||
), mock.patch.object(KubeRayNodeProvider, "_get", mock_get):
|
||||
kr_node_provider = KubeRayNodeProvider(provider_config={}, cluster_name="fake")
|
||||
kr_node_provider.cluster_name = "fake"
|
||||
kr_node_provider.replica_index_to_nodes = defaultdict(list[str])
|
||||
nodes = kr_node_provider.non_terminated_nodes({})
|
||||
assert kr_node_provider.node_data_dict == expected_node_data
|
||||
assert set(nodes) == set(expected_node_data.keys())
|
||||
|
||||
|
||||
@pytest.mark.skipif(sys.platform.startswith("win"), reason="Not relevant on Windows.")
|
||||
@pytest.mark.parametrize(
|
||||
"node_data_dict,scale_request,expected_patch_payload",
|
||||
[
|
||||
(
|
||||
{
|
||||
"raycluster-autoscaler-head-8zsc8": NodeData(
|
||||
kind="head",
|
||||
type="headgroup",
|
||||
replica_index=None,
|
||||
ip="10.4.2.6",
|
||||
status="up-to-date",
|
||||
),
|
||||
"raycluster-autoscaler-worker-fake-gpu-group-2qnhv": NodeData(
|
||||
kind="worker",
|
||||
type="fake-gpu-group",
|
||||
replica_index=None,
|
||||
ip="10.4.0.6",
|
||||
status="up-to-date",
|
||||
),
|
||||
"raycluster-autoscaler-worker-small-group-dkz2r": NodeData(
|
||||
kind="worker",
|
||||
type="small-group",
|
||||
replica_index=None,
|
||||
ip="10.4.1.8",
|
||||
status="up-to-date",
|
||||
),
|
||||
"raycluster-autoscaler-worker-small-group-lbfm4": NodeData(
|
||||
kind="worker",
|
||||
type="small-group",
|
||||
replica_index=None,
|
||||
ip="10.4.0.5",
|
||||
status="up-to-date",
|
||||
),
|
||||
},
|
||||
ScaleRequest(
|
||||
desired_num_workers={
|
||||
"small-group": 1, # Delete 1
|
||||
"gpu-group": 1, # Don't touch
|
||||
"blah-group": 5, # Create 5
|
||||
},
|
||||
workers_to_delete={
|
||||
"raycluster-autoscaler-worker-small-group-dkz2r",
|
||||
},
|
||||
),
|
||||
[
|
||||
{
|
||||
"op": "replace",
|
||||
"path": "/spec/workerGroupSpecs/3/replicas",
|
||||
"value": 5,
|
||||
},
|
||||
{
|
||||
"op": "replace",
|
||||
"path": "/spec/workerGroupSpecs/0/scaleStrategy",
|
||||
"value": {
|
||||
"workersToDelete": [
|
||||
"raycluster-autoscaler-worker-small-group-dkz2r"
|
||||
]
|
||||
},
|
||||
},
|
||||
],
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_submit_scale_request(node_data_dict, scale_request, expected_patch_payload):
|
||||
"""Test the KubeRayNodeProvider's RayCluster patch payload given a dict
|
||||
of current node counts and a scale request.
|
||||
"""
|
||||
raycluster = get_basic_ray_cr()
|
||||
# Add another worker group for the sake of this test.
|
||||
blah_group = copy.deepcopy(raycluster["spec"]["workerGroupSpecs"][1])
|
||||
blah_group["groupName"] = "blah-group"
|
||||
raycluster["spec"]["workerGroupSpecs"].append(blah_group)
|
||||
with mock.patch.object(KubeRayNodeProvider, "__init__", return_value=None):
|
||||
kr_node_provider = KubeRayNodeProvider(provider_config={}, cluster_name="fake")
|
||||
kr_node_provider.node_data_dict = node_data_dict
|
||||
patch_payload = kr_node_provider._scale_request_to_patch_payload(
|
||||
scale_request=scale_request, raycluster=raycluster
|
||||
)
|
||||
assert patch_payload == expected_patch_payload
|
||||
|
||||
|
||||
@pytest.mark.parametrize("node_set", [{"A", "B", "C", "D", "E"}])
|
||||
@pytest.mark.parametrize("cpu_workers_to_delete", ["A", "Z"])
|
||||
@pytest.mark.parametrize("gpu_workers_to_delete", ["B", "Y"])
|
||||
@pytest.mark.parametrize("tpu_workers_to_delete", ["C", "X"])
|
||||
@pytest.mark.skipif(sys.platform.startswith("win"), reason="Not relevant on Windows.")
|
||||
def test_safe_to_scale(
|
||||
node_set: Set[NodeID],
|
||||
cpu_workers_to_delete: List[NodeID],
|
||||
gpu_workers_to_delete: List[NodeID],
|
||||
tpu_workers_to_delete: List[NodeID],
|
||||
):
|
||||
# NodeData values unimportant for this test.
|
||||
mock_node_data = NodeData("-", "-", "-", "-", "-")
|
||||
node_data_dict = {node_id: mock_node_data for node_id in node_set}
|
||||
|
||||
raycluster = _get_basic_ray_cr_workers_to_delete(
|
||||
cpu_workers_to_delete, gpu_workers_to_delete, tpu_workers_to_delete
|
||||
)
|
||||
|
||||
def mock_patch(kuberay_provider, path, patch_payload):
|
||||
patch = jsonpatch.JsonPatch(patch_payload)
|
||||
kuberay_provider._patched_raycluster = patch.apply(kuberay_provider._raycluster)
|
||||
|
||||
with mock.patch.object(
|
||||
KubeRayNodeProvider, "__init__", return_value=None
|
||||
), mock.patch.object(KubeRayNodeProvider, "_patch", mock_patch):
|
||||
kr_node_provider = KubeRayNodeProvider(provider_config={}, cluster_name="fake")
|
||||
kr_node_provider.cluster_name = "fake"
|
||||
kr_node_provider._patched_raycluster = raycluster
|
||||
kr_node_provider._raycluster = raycluster
|
||||
kr_node_provider.node_data_dict = node_data_dict
|
||||
actual_safe = kr_node_provider.safe_to_scale()
|
||||
|
||||
expected_safe = (
|
||||
not any(
|
||||
cpu_worker_to_delete in node_set
|
||||
for cpu_worker_to_delete in cpu_workers_to_delete
|
||||
)
|
||||
and not any(
|
||||
gpu_worker_to_delete in node_set
|
||||
for gpu_worker_to_delete in gpu_workers_to_delete
|
||||
)
|
||||
and not any(
|
||||
tpu_worker_to_delete in node_set
|
||||
for tpu_worker_to_delete in tpu_workers_to_delete
|
||||
)
|
||||
)
|
||||
assert expected_safe is actual_safe
|
||||
patched_cpu_workers_to_delete = kr_node_provider._patched_raycluster["spec"][
|
||||
"workerGroupSpecs"
|
||||
][0]["scaleStrategy"]["workersToDelete"]
|
||||
patched_gpu_workers_to_delete = kr_node_provider._patched_raycluster["spec"][
|
||||
"workerGroupSpecs"
|
||||
][1]["scaleStrategy"]["workersToDelete"]
|
||||
patched_tpu_workers_to_delete = kr_node_provider._patched_raycluster["spec"][
|
||||
"workerGroupSpecs"
|
||||
][2]["scaleStrategy"]["workersToDelete"]
|
||||
|
||||
if expected_safe:
|
||||
# Cleaned up workers to delete
|
||||
assert patched_cpu_workers_to_delete == []
|
||||
assert patched_gpu_workers_to_delete == []
|
||||
assert patched_tpu_workers_to_delete == []
|
||||
else:
|
||||
# Did not clean up workers to delete
|
||||
assert patched_cpu_workers_to_delete == cpu_workers_to_delete
|
||||
assert patched_gpu_workers_to_delete == gpu_workers_to_delete
|
||||
assert patched_tpu_workers_to_delete == tpu_workers_to_delete
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(pytest.main(["-v", __file__]))
|
||||
@@ -0,0 +1,484 @@
|
||||
"""Utilities for e2e tests of KubeRay/Ray integration.
|
||||
For consistency, all K8s interactions use kubectl through subprocess calls.
|
||||
"""
|
||||
import atexit
|
||||
import contextlib
|
||||
import logging
|
||||
import os
|
||||
import pathlib
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
from typing import Any, Dict, Generator, List, Optional
|
||||
|
||||
import yaml
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
SCRIPTS_DIR = pathlib.Path(__file__).resolve().parent / "scripts"
|
||||
TEST_CR_PATH = (
|
||||
pathlib.Path(__file__).resolve().parent / "setup" / "raycluster_test.yaml"
|
||||
)
|
||||
TEST_CLUSTER_NAME = "raycluster-test"
|
||||
|
||||
# Parent directory of Ray repository
|
||||
RAY_PARENT = str(pathlib.Path(__file__).resolve().parents[5])
|
||||
|
||||
RAYCLUSTERS_QUALIFIED = "rayclusters.ray.io"
|
||||
|
||||
LOG_FORMAT = "[%(levelname)s %(asctime)s] %(filename)s: %(lineno)d %(message)s"
|
||||
|
||||
|
||||
def switch_to_ray_parent_dir():
|
||||
# Switch to parent of Ray repo, because that's what the doc examples do.
|
||||
logger.info("Switching to parent of Ray directory.")
|
||||
os.chdir(RAY_PARENT)
|
||||
|
||||
|
||||
def setup_kuberay_operator():
|
||||
"""Set up KubeRay operator and Ray autoscaler RBAC."""
|
||||
switch_to_ray_parent_dir()
|
||||
logger.info("Cloning KubeRay and setting up KubeRay configuration.")
|
||||
# For faster run-time when triggering the test locally, don't run the init
|
||||
# script if it has already been run.
|
||||
subprocess.check_call(
|
||||
[
|
||||
"bash",
|
||||
"-c",
|
||||
(
|
||||
"ls ray/python/ray/autoscaler/kuberay/config ||"
|
||||
" ./ray/python/ray/autoscaler/kuberay/init-config.sh"
|
||||
),
|
||||
]
|
||||
)
|
||||
logger.info("Creating KubeRay operator.")
|
||||
subprocess.check_call(
|
||||
[
|
||||
"kubectl",
|
||||
"create",
|
||||
"-k",
|
||||
"ray/python/ray/autoscaler/kuberay/config/default",
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def teardown_kuberay_operator():
|
||||
logger.info("Switching to parent of Ray directory.")
|
||||
os.chdir(RAY_PARENT)
|
||||
|
||||
logger.info("Deleting operator.")
|
||||
subprocess.check_call(
|
||||
[
|
||||
"kubectl",
|
||||
"delete",
|
||||
"--ignore-not-found",
|
||||
"-k",
|
||||
"ray/python/ray/autoscaler/kuberay/config/default",
|
||||
]
|
||||
)
|
||||
|
||||
logger.info("Double-checking no pods left over in namespace ray-system.")
|
||||
wait_for_pods(goal_num_pods=0, namespace="ray-system")
|
||||
|
||||
|
||||
def wait_for_raycluster_crd(tries=60, backoff_s=5):
|
||||
"""CRD creation can take a bit of time after the client request.
|
||||
This function waits until the crd with the provided name is registered.
|
||||
"""
|
||||
switch_to_ray_parent_dir()
|
||||
logger.info("Making sure RayCluster CRD has been registered.")
|
||||
for i in range(tries):
|
||||
get_crd_output = subprocess.check_output(["kubectl", "get", "crd"]).decode()
|
||||
if RAYCLUSTERS_QUALIFIED in get_crd_output:
|
||||
logger.info("Confirmed existence of RayCluster CRD.")
|
||||
break
|
||||
elif i < tries - 1:
|
||||
logger.info("Still waiting to register RayCluster CRD.")
|
||||
time.sleep(backoff_s)
|
||||
else:
|
||||
raise Exception("Failed to register RayCluster CRD.")
|
||||
|
||||
# Create a test RayCluster CR to make sure that the CRD is fully registered.
|
||||
for i in range(tries):
|
||||
try:
|
||||
subprocess.check_call(["kubectl", "apply", "-f", TEST_CR_PATH])
|
||||
break
|
||||
except subprocess.CalledProcessError as e:
|
||||
logger.info("Can't create RayCluster CR.")
|
||||
if i < tries - 1:
|
||||
logger.info("Retrying.")
|
||||
time.sleep(backoff_s)
|
||||
else:
|
||||
logger.info("Giving up.")
|
||||
raise e from None
|
||||
|
||||
# Confirm the test RayCluster exists.
|
||||
out = subprocess.check_output(["kubectl", "get", RAYCLUSTERS_QUALIFIED]).decode()
|
||||
assert TEST_CLUSTER_NAME in out, out
|
||||
|
||||
# Delete the test RayCluster.
|
||||
subprocess.check_call(["kubectl", "delete", "-f", TEST_CR_PATH])
|
||||
# Make sure the associated resources are gone before proceeding.
|
||||
wait_for_pods(goal_num_pods=0, namespace="default")
|
||||
|
||||
|
||||
def wait_for_pods(goal_num_pods: int, namespace: str, tries=60, backoff_s=5) -> None:
|
||||
"""Wait for the number of pods in the `namespace` to be exactly `num_pods`.
|
||||
|
||||
Raise an exception after exceeding `tries` attempts with `backoff_s` second waits.
|
||||
"""
|
||||
for i in range(tries):
|
||||
cur_num_pods = _get_num_pods(namespace)
|
||||
if cur_num_pods == goal_num_pods:
|
||||
logger.info(f"Confirmed {goal_num_pods} pod(s) in namespace {namespace}.")
|
||||
return
|
||||
elif i < tries - 1:
|
||||
logger.info(
|
||||
f"The number of pods in namespace {namespace} is {cur_num_pods}."
|
||||
f" Waiting until the number of pods is {goal_num_pods}."
|
||||
f"{get_pod_names(namespace)}"
|
||||
)
|
||||
time.sleep(backoff_s)
|
||||
else:
|
||||
raise Exception(
|
||||
f"Failed to scale to {goal_num_pods} pod(s) in namespace {namespace}."
|
||||
)
|
||||
|
||||
|
||||
def _get_num_pods(namespace: str) -> int:
|
||||
return len(get_pod_names(namespace))
|
||||
|
||||
|
||||
def get_pod_names(namespace: str) -> List[str]:
|
||||
"""Get the list of pod names in the namespace."""
|
||||
get_pods_output = (
|
||||
subprocess.check_output(
|
||||
[
|
||||
"kubectl",
|
||||
"-n",
|
||||
namespace,
|
||||
"get",
|
||||
"pods",
|
||||
"-o",
|
||||
"custom-columns=POD:metadata.name",
|
||||
"--no-headers",
|
||||
]
|
||||
)
|
||||
.decode()
|
||||
.strip()
|
||||
)
|
||||
|
||||
# If there aren't any pods, the output is any empty string.
|
||||
if not get_pods_output:
|
||||
return []
|
||||
else:
|
||||
return get_pods_output.split("\n")
|
||||
|
||||
|
||||
def wait_for_pod_to_start(
|
||||
pod_name_filter: str, namespace: str, tries=60, backoff_s=5
|
||||
) -> None:
|
||||
"""Waits for a pod to have Running status.phase.
|
||||
|
||||
More precisely, waits until there is a pod with name containing `pod_name_filter`
|
||||
and the pod has Running status.phase."""
|
||||
for i in range(tries):
|
||||
pod = get_pod(pod_name_filter=pod_name_filter, namespace=namespace)
|
||||
if not pod:
|
||||
# We didn't get a matching pod.
|
||||
continue
|
||||
pod_status = (
|
||||
subprocess.check_output(
|
||||
[
|
||||
"kubectl",
|
||||
"-n",
|
||||
namespace,
|
||||
"get",
|
||||
"pod",
|
||||
pod,
|
||||
"-o",
|
||||
"custom-columns=POD:status.phase",
|
||||
"--no-headers",
|
||||
]
|
||||
)
|
||||
.decode()
|
||||
.strip()
|
||||
)
|
||||
# "not found" is part of the kubectl output if the pod's not there.
|
||||
if "not found" in pod_status:
|
||||
raise Exception(f"Pod {pod} not found.")
|
||||
elif pod_status == "Running":
|
||||
logger.info(f"Confirmed pod {pod} is Running.")
|
||||
return
|
||||
elif i < tries - 1:
|
||||
logger.info(
|
||||
f"Pod {pod} has status {pod_status}. Waiting for the pod to enter "
|
||||
"Running status."
|
||||
)
|
||||
time.sleep(backoff_s)
|
||||
else:
|
||||
raise Exception(f"Timed out waiting for pod {pod} to enter Running status.")
|
||||
|
||||
|
||||
def wait_for_ray_health(
|
||||
pod_name_filter: str,
|
||||
namespace: str,
|
||||
tries=60,
|
||||
backoff_s=5,
|
||||
ray_container="ray-head",
|
||||
) -> None:
|
||||
"""Waits until a Ray pod passes `ray health-check`.
|
||||
|
||||
More precisely, waits until a Ray pod whose name includes the string
|
||||
`pod_name_filter` passes `ray health-check`.
|
||||
(Ensures Ray has completely started in the pod.)
|
||||
|
||||
Use case: Wait until there is a Ray head pod with Ray running on it.
|
||||
"""
|
||||
for i in range(tries):
|
||||
try:
|
||||
pod = get_pod(pod_name_filter=pod_name_filter, namespace="default")
|
||||
assert pod, f"Couldn't find a pod matching {pod_name_filter}."
|
||||
# `ray health-check` yields 0 exit status iff it succeeds
|
||||
kubectl_exec(
|
||||
["ray", "health-check"], pod, namespace, container=ray_container
|
||||
)
|
||||
logger.info(f"ray health check passes for pod {pod}")
|
||||
return
|
||||
except subprocess.CalledProcessError as e:
|
||||
logger.info(f"Failed ray health check for pod {pod}.")
|
||||
if i < tries - 1:
|
||||
logger.info("Trying again.")
|
||||
time.sleep(backoff_s)
|
||||
else:
|
||||
logger.info("Giving up.")
|
||||
raise e from None
|
||||
|
||||
|
||||
def get_pod(pod_name_filter: str, namespace: str) -> Optional[str]:
|
||||
"""Gets pods in the `namespace`.
|
||||
|
||||
Returns the first pod that has `pod_name_filter` as a
|
||||
substring of its name. Returns None if there are no matches.
|
||||
"""
|
||||
pod_names = get_pod_names(namespace)
|
||||
matches = [pod_name for pod_name in pod_names if pod_name_filter in pod_name]
|
||||
if not matches:
|
||||
logger.warning(f"No match for `{pod_name_filter}` in namespace `{namespace}`.")
|
||||
return None
|
||||
return matches[0]
|
||||
|
||||
|
||||
def kubectl_exec(
|
||||
command: List[str],
|
||||
pod: str,
|
||||
namespace: str,
|
||||
container: Optional[str] = None,
|
||||
) -> str:
|
||||
"""kubectl exec the `command` in the given `pod` in the given `namespace`.
|
||||
If a `container` is specified, will specify that container for kubectl.
|
||||
|
||||
Prints and return kubectl's output as a string.
|
||||
"""
|
||||
container_option = ["-c", container] if container else []
|
||||
kubectl_exec_command = (
|
||||
["kubectl", "exec", "-it", pod] + container_option + ["--"] + command
|
||||
)
|
||||
# Print for debugging convenience.
|
||||
try:
|
||||
out = subprocess.check_output(kubectl_exec_command).decode().strip()
|
||||
except subprocess.CalledProcessError as e:
|
||||
logger.error(f"Error running command {kubectl_exec_command}.")
|
||||
logger.error(f"Output: {e.output.decode()}")
|
||||
raise e from None
|
||||
print(out)
|
||||
return out
|
||||
|
||||
|
||||
def kubectl_logs(
|
||||
pod: str,
|
||||
namespace: str,
|
||||
container: Optional[str] = None,
|
||||
) -> str:
|
||||
"""Wrapper for kubectl logs.
|
||||
|
||||
Returns the logs as a string.
|
||||
"""
|
||||
container_option = ["-c", container] if container else []
|
||||
kubectl_logs_command = ["kubectl", "logs", pod] + container_option
|
||||
out = subprocess.check_output(kubectl_logs_command).decode().strip()
|
||||
return out
|
||||
|
||||
|
||||
def kubectl_exec_python_script(
|
||||
script_name: str,
|
||||
pod: str,
|
||||
namespace: str,
|
||||
container: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Runs a python script in a container via `kubectl exec`.
|
||||
Scripts live in `tests/kuberay/scripts`.
|
||||
|
||||
Prints and return kubectl's output as a string.
|
||||
"""
|
||||
script_path = SCRIPTS_DIR / script_name
|
||||
with open(script_path) as script_file:
|
||||
script_string = script_file.read()
|
||||
return kubectl_exec(["python", "-c", script_string], pod, namespace, container)
|
||||
|
||||
|
||||
def get_raycluster(raycluster: str, namespace: str) -> Dict[str, Any]:
|
||||
"""Gets the Ray CR with name `raycluster` in namespace `namespace`.
|
||||
|
||||
Returns the CR as a nested Dict.
|
||||
"""
|
||||
get_raycluster_output = (
|
||||
subprocess.check_output(
|
||||
["kubectl", "-n", namespace, "get", "raycluster", raycluster, "-o", "yaml"]
|
||||
)
|
||||
.decode()
|
||||
.strip()
|
||||
)
|
||||
return yaml.safe_load(get_raycluster_output)
|
||||
|
||||
|
||||
def _get_service_port(service: str, namespace: str, target_port: int) -> int:
|
||||
"""Given a K8s service and a port targetted by the service, returns the
|
||||
corresponding port exposed by the service.
|
||||
|
||||
Args:
|
||||
service: Name of a K8s service.
|
||||
namespace: Namespace to which the service belongs.
|
||||
target_port: Port targeted by the service.
|
||||
|
||||
Returns:
|
||||
service_port: The port exposed by the service.
|
||||
"""
|
||||
service_str = (
|
||||
subprocess.check_output(
|
||||
["kubectl", "-n", namespace, "get", "service", service, "-o", "yaml"]
|
||||
)
|
||||
.decode()
|
||||
.strip()
|
||||
)
|
||||
service_dict = yaml.safe_load(service_str)
|
||||
service_ports: List = service_dict["spec"]["ports"]
|
||||
matching_ports = [
|
||||
port for port in service_ports if port["targetPort"] == target_port
|
||||
]
|
||||
assert matching_ports
|
||||
service_port = matching_ports[0]["port"]
|
||||
return service_port
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def _kubectl_port_forward(
|
||||
service: str, namespace: str, target_port: int, local_port: Optional[int] = None
|
||||
) -> Generator[int, None, None]:
|
||||
"""Context manager which creates a kubectl port-forward process targeting a
|
||||
K8s service.
|
||||
|
||||
Terminates the port-forwarding process upon exit.
|
||||
|
||||
Args:
|
||||
service: Name of a K8s service.
|
||||
namespace: Namespace to which the service belongs.
|
||||
target_port: The port targeted by the service.
|
||||
local_port: Forward from this port. Optional. By default, uses the port exposed
|
||||
by the service.
|
||||
|
||||
Yields:
|
||||
int: The local port. The service can then be accessed at
|
||||
127.0.0.1:<local_port>.
|
||||
"""
|
||||
# First, figure out which port the service exposes for the given target port.
|
||||
service_port = _get_service_port(service, namespace, target_port)
|
||||
if not local_port:
|
||||
local_port = service_port
|
||||
|
||||
process = subprocess.Popen(
|
||||
[
|
||||
"kubectl",
|
||||
"-n",
|
||||
namespace,
|
||||
"port-forward",
|
||||
f"service/{service}",
|
||||
f"{local_port}:{service_port}",
|
||||
]
|
||||
)
|
||||
|
||||
def terminate_process():
|
||||
process.terminate()
|
||||
# Wait 10 seconds for the process to terminate.
|
||||
# This cleans up the zombie entry from the process table.
|
||||
# 10 seconds is a deliberately excessive amount of time to wait.
|
||||
process.wait(timeout=10)
|
||||
|
||||
# Ensure clean-up in case of interrupt.
|
||||
atexit.register(terminate_process)
|
||||
# terminate_process is ok to execute multiple times.
|
||||
|
||||
try:
|
||||
yield local_port
|
||||
finally:
|
||||
terminate_process()
|
||||
|
||||
|
||||
def kubectl_patch(
|
||||
kind: str,
|
||||
name: str,
|
||||
namespace: str,
|
||||
patch: Dict[str, Any],
|
||||
patch_type: str = "strategic",
|
||||
):
|
||||
"""Wrapper for kubectl patch.
|
||||
|
||||
Args:
|
||||
kind: Kind of the K8s resource (e.g. pod)
|
||||
name: Name of the K8s resource.
|
||||
namespace: Namespace of the K8s resource.
|
||||
patch: The patch to apply, as a dict.
|
||||
patch_type: json, merge, or strategic
|
||||
"""
|
||||
with tempfile.NamedTemporaryFile("w") as patch_file:
|
||||
yaml.dump(patch, patch_file)
|
||||
patch_file.flush()
|
||||
subprocess.check_call(
|
||||
[
|
||||
"kubectl",
|
||||
"-n",
|
||||
f"{namespace}",
|
||||
"patch",
|
||||
f"{kind}",
|
||||
f"{name}",
|
||||
"--patch-file",
|
||||
f"{patch_file.name}",
|
||||
"--type",
|
||||
f"{patch_type}",
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def kubectl_delete(kind: str, name: str, namespace: str, wait: bool = True):
|
||||
"""Wrapper for kubectl delete.
|
||||
|
||||
Args:
|
||||
kind: Kind of the K8s resource (e.g. pod)
|
||||
name: Name of the K8s resource.
|
||||
namespace: Namespace of the K8s resource.
|
||||
wait: Whether to pass ``--wait=true`` so ``kubectl`` blocks until the
|
||||
resource is fully removed.
|
||||
"""
|
||||
wait_str = "true" if wait else "false"
|
||||
subprocess.check_output(
|
||||
[
|
||||
"kubectl",
|
||||
"-n",
|
||||
f"{namespace}",
|
||||
"delete",
|
||||
f"{kind}",
|
||||
f"{name}",
|
||||
f"--wait={wait_str}",
|
||||
]
|
||||
)
|
||||
Reference in New Issue
Block a user