This page provides suggestions on running the test test_autoscaling_e2e locally. You might want to do this if your PR is breaking this test in CI and you want to debug why.

Running the test must happen in stages:

Tear down any running kind cluster
Remove the existing ray docker image that will be deployed to the cluster
Build a new docker image containing the local ray repository
Create a new kind cluster
Load the docker image into the cluster
Set up kuberay
Run the test

To help with this, there is a Dockerfile and a rune2e.sh bash script which together run these things for you.

Test requirements

Ensure kind and kustomize are both installed
Run ray/autoscaler/kuberay/init-config.sh to clone ray-project/kuberay, which contains config files needed to set up kuberay.
Finally, make sure that the Dockerfile is using the same python version as what you're using to run the test. By default, this dockerfile is built using the rayproject/ray:nightly-py310 build.
Modify EXAMPLE_CLUSTER_PATH in test_autoscaling_e2e.py.

Now you're ready to run the test.

Running the test

Run ./rune2e.sh to run the test.

The test itself does not tear down resources on failure; you can

examine a Ray cluster from a failed test (kubectl get pods, kubectl get pod, kubectl get raycluster)
view all logs (kubectl logs <head pod name>) or just logs associated with the autoscaler (kubectl logs <head pod name> -c autoscaler)
delete the Ray cluster (kubectl delete raycluster -A)
rerun the test without tearing the operator down (RAY_IMAGE=<registry>/<repo>:<tag> python test_autoscaling_e2e.py)
tear down the operator when you're done python setup/teardown_kuberay.py
copy files from a pod to your filesystem (kubectl cp <pod>:/path/to/file /target/path/in/local/filesystem)
access a bash prompt inside the pod (kubectl exec -it <pod> bash)