chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,48 @@
|
||||
# This is a Ray cluster configuration for exploration of the 100Gi Ray XGBoostTrainer benchmark.
|
||||
|
||||
# The configuration includes 1 Ray head node and 9 worker nodes.
|
||||
|
||||
cluster_name: ray-cluster-xgboost-benchmark
|
||||
|
||||
# The maximum number of worker nodes to launch in addition to the head
|
||||
# node.
|
||||
max_workers: 9
|
||||
|
||||
docker:
|
||||
image: "rayproject/ray-ml:2.0.0"
|
||||
container_name: "ray_container"
|
||||
|
||||
provider:
|
||||
type: aws
|
||||
region: us-west-2
|
||||
availability_zone: us-west-2a
|
||||
|
||||
auth:
|
||||
ssh_user: ubuntu
|
||||
|
||||
available_node_types:
|
||||
# Configurations for the head node.
|
||||
head:
|
||||
node_config:
|
||||
InstanceType: m5.4xlarge
|
||||
ImageId: latest_dlami
|
||||
BlockDeviceMappings:
|
||||
- DeviceName: /dev/sda1
|
||||
Ebs:
|
||||
VolumeSize: 1000
|
||||
|
||||
# Configurations for the worker nodes.
|
||||
worker:
|
||||
# To experiment with autoscaling, set min_workers to 0.
|
||||
# min_workers: 0
|
||||
min_workers: 9
|
||||
max_workers: 9
|
||||
node_config:
|
||||
InstanceType: m5.4xlarge
|
||||
ImageId: latest_dlami
|
||||
BlockDeviceMappings:
|
||||
- DeviceName: /dev/sda1
|
||||
Ebs:
|
||||
VolumeSize: 1000
|
||||
|
||||
head_node_type: head
|
||||
@@ -0,0 +1,18 @@
|
||||
(vm-cluster-examples)=
|
||||
|
||||
# Examples
|
||||
|
||||
```{toctree}
|
||||
:hidden:
|
||||
|
||||
ml-example
|
||||
```
|
||||
|
||||
:::{note}
|
||||
To learn the basics of Ray on Cloud VMs, we recommend taking a look at the {ref}`introductory guide <vm-cluster-quick-start>` first.
|
||||
:::
|
||||
|
||||
This section presents example Ray workloads to try out on your cloud cluster.
|
||||
|
||||
More examples will be added in the future. Running the distributed XGBoost example below is a great way to start experimenting with production Ray workloads in the cloud.
|
||||
- {ref}`clusters-vm-ml-example`
|
||||
@@ -0,0 +1,124 @@
|
||||
(clusters-vm-ml-example)=
|
||||
|
||||
# Ray Train XGBoostTrainer on VMs
|
||||
|
||||
:::{note}
|
||||
To learn the basics of Ray on VMs, we recommend taking a look at the {ref}`introductory guide <vm-cluster-quick-start>` first.
|
||||
:::
|
||||
|
||||
|
||||
In this guide, we show you how to run a sample Ray machine learning workload on AWS. The similar steps can be used to deploy on GCP or Azure as well.
|
||||
|
||||
We will run Ray's {ref}`XGBoost training benchmark <xgboost-benchmark>` with a 100 gigabyte training set. To learn more about using Ray's XGBoostTrainer, check out {ref}`the XGBoostTrainer documentation <train-xgboost>`.
|
||||
|
||||
## VM cluster setup
|
||||
|
||||
For the workload in this guide, it is recommended to use the following setup:
|
||||
- 10 nodes total
|
||||
- A capacity of 16 CPU and 64 Gi memory per node. For the major cloud providers, suitable instance types include
|
||||
* m5.4xlarge (Amazon Web Services)
|
||||
* Standard_D5_v2 (Azure)
|
||||
* e2-standard-16 (Google Cloud)
|
||||
- Each node should be configured with 1000 gigabytes of disk space (to store the training set).
|
||||
|
||||
The corresponding cluster configuration file is as follows:
|
||||
|
||||
```{literalinclude} ../configs/xgboost-benchmark.yaml
|
||||
:language: yaml
|
||||
```
|
||||
|
||||
```{admonition} Optional: Set up an autoscaling cluster
|
||||
**If you would like to try running the workload with autoscaling enabled**,
|
||||
change ``min_workers`` of worker nodes to 0.
|
||||
After the workload is submitted, 9 workers nodes will
|
||||
scale up to accommodate the workload. These nodes will scale back down after the workload is complete.
|
||||
```
|
||||
|
||||
## Deploy a Ray cluster
|
||||
|
||||
Now we're ready to deploy the Ray cluster with the configuration that's defined above. Before running the command, make sure your aws credentials are configured correctly.
|
||||
|
||||
```shell
|
||||
ray up -y cluster.yaml
|
||||
```
|
||||
|
||||
A Ray head node and 9 Ray worker nodes will be created.
|
||||
|
||||
## Run the workload
|
||||
|
||||
We will use {ref}`Ray Job Submission <jobs-overview>` to kick off the workload.
|
||||
|
||||
### Connect to the cluster
|
||||
|
||||
First, we connect to the Job server. Run the following blocking command in a separate shell.
|
||||
```shell
|
||||
ray dashboard cluster.yaml
|
||||
```
|
||||
This will forward remote port 8265 to port 8265 on localhost.
|
||||
|
||||
### Submit the workload
|
||||
|
||||
We'll use the {ref}`Ray Job Python SDK <ray-job-sdk>` to submit the XGBoost workload.
|
||||
|
||||
```{literalinclude} /cluster/doc_code/xgboost_submit.py
|
||||
:language: python
|
||||
```
|
||||
|
||||
To submit the workload, run the above Python script. The script is available [in the Ray repository][XGBSubmit].
|
||||
|
||||
```shell
|
||||
# Download the above script.
|
||||
curl https://raw.githubusercontent.com/ray-project/ray/releases/2.0.0/doc/source/cluster/doc_code/xgboost_submit.py -o xgboost_submit.py
|
||||
# Run the script.
|
||||
python xgboost_submit.py
|
||||
```
|
||||
|
||||
### Observe progress
|
||||
|
||||
The benchmark may take up to 30 minutes to run. Use the following tools to observe its progress.
|
||||
|
||||
#### Job logs
|
||||
|
||||
To follow the job's logs, use the command printed by the above submission script.
|
||||
```shell
|
||||
# Substitute the Ray Job's submission id.
|
||||
ray job logs 'raysubmit_xxxxxxxxxxxxxxxx' --address="http://localhost:8265" --follow
|
||||
```
|
||||
|
||||
#### Ray Dashboard
|
||||
|
||||
View `localhost:8265` in your browser to access the Ray Dashboard.
|
||||
|
||||
#### Ray Status
|
||||
|
||||
Observe autoscaling status and Ray resource usage with
|
||||
```shell
|
||||
ray exec cluster.yaml 'ray status'
|
||||
```
|
||||
|
||||
### Job completion
|
||||
|
||||
#### Benchmark results
|
||||
|
||||
Once the benchmark is complete, the job log will display the results:
|
||||
|
||||
```
|
||||
Results: {'training_time': 1338.488839321999, 'prediction_time': 403.36653568099973}
|
||||
```
|
||||
|
||||
The performance of the benchmark is sensitive to the underlying cloud infrastructure -- you might not match {ref}`the numbers quoted in the benchmark docs <xgboost-benchmark>`.
|
||||
|
||||
#### Model parameters
|
||||
The file `model.json` in the Ray head node contains the parameters for the trained model. Other result data will be available in the directory `ray_results` in the head node. Refer to the {ref}`XGBoostTrainer documentation <train-xgboost>` for details.
|
||||
|
||||
```{admonition} Scale-down
|
||||
If autoscaling is enabled, Ray worker nodes will scale down after the specified idle timeout.
|
||||
```
|
||||
|
||||
#### Clean-up
|
||||
Delete your Ray cluster with the following command:
|
||||
```shell
|
||||
ray down -y cluster.yaml
|
||||
```
|
||||
|
||||
[XGBSubmit]: https://github.com/ray-project/ray/blob/releases/2.0.0/doc/source/cluster/doc_code/xgboost_submit.py
|
||||
@@ -0,0 +1,329 @@
|
||||
.. _vm-cluster-quick-start:
|
||||
|
||||
Getting Started
|
||||
===============
|
||||
|
||||
This quick start demonstrates the capabilities of the Ray cluster. Using the Ray cluster, we'll take a sample application designed to run on a laptop and scale it up in the cloud. Ray will launch clusters and scale Python with just a few commands.
|
||||
|
||||
For launching a Ray cluster manually, you can refer to the :ref:`on-premise cluster setup <on-prem>` guide.
|
||||
|
||||
About the demo
|
||||
--------------
|
||||
|
||||
This demo will walk through an end-to-end flow:
|
||||
|
||||
1. Create a (basic) Python application.
|
||||
2. Launch a cluster on a cloud provider.
|
||||
3. Run the application in the cloud.
|
||||
|
||||
Requirements
|
||||
~~~~~~~~~~~~
|
||||
|
||||
To run this demo, you will need:
|
||||
|
||||
* Python installed on your development machine (typically your laptop), and
|
||||
* an account at your preferred cloud provider (AWS, GCP, Azure, Aliyun, or vSphere).
|
||||
|
||||
Setup
|
||||
~~~~~
|
||||
|
||||
Before we start, you will need to install some Python dependencies as follows:
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Ray Team Supported
|
||||
:sync: Ray Team Supported
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: AWS
|
||||
:sync: AWS
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ pip install -U "ray[default]" boto3
|
||||
|
||||
.. tab-item:: Azure
|
||||
:sync: Azure
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ pip install -U "ray[default]" azure-cli azure-core
|
||||
|
||||
.. tab-item:: GCP
|
||||
:sync: GCP
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ pip install -U "ray[default]" google-api-python-client
|
||||
|
||||
.. tab-item:: Community Supported
|
||||
:sync: Community Supported
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Aliyun
|
||||
:sync: Aliyun
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ pip install -U "ray[default]" aliyun-python-sdk-core aliyun-python-sdk-ecs
|
||||
|
||||
Aliyun Cluster Launcher Maintainers (GitHub handles): @zhuangzhuang131419, @chenk008
|
||||
|
||||
.. tab-item:: vSphere
|
||||
:sync: vSphere
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ pip install -U "ray[default]"
|
||||
|
||||
vSphere Cluster Launcher Maintainers (GitHub handles): @roshankathawate, @ankitasonawane30, @VamshikShetty
|
||||
|
||||
|
||||
Next, if you're not set up to use your cloud provider from the command line, you'll have to configure your credentials:
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Ray Team Supported
|
||||
:sync: Ray Team Supported
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: AWS
|
||||
:sync: AWS
|
||||
|
||||
Configure your credentials in ``~/.aws/credentials`` as described in `the AWS docs <https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html>`_.
|
||||
|
||||
.. tab-item:: Azure
|
||||
:sync: Azure
|
||||
|
||||
Log in using ``az login``, then configure your credentials with ``az account set -s <subscription_id>``.
|
||||
|
||||
.. tab-item:: GCP
|
||||
:sync: GCP
|
||||
|
||||
Set the ``GOOGLE_APPLICATION_CREDENTIALS`` environment variable as described in `the GCP docs <https://cloud.google.com/docs/authentication/getting-started>`_.
|
||||
|
||||
.. tab-item:: Community Supported
|
||||
:sync: Community Supported
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Aliyun
|
||||
:sync: Aliyun
|
||||
|
||||
Obtain and set the AccessKey pair of the Aliyun account as described in `the docs <https://www.alibabacloud.com/help/en/doc-detail/175967.htm>`__.
|
||||
|
||||
Make sure to grant the necessary permissions to the RAM user and set the AccessKey pair in your cluster config file.
|
||||
Refer to the provided `aliyun/example-full.yaml </ray/python/ray/autoscaler/aliyun/example-full.yaml>`__ for a sample cluster config.
|
||||
|
||||
.. tab-item:: vSphere
|
||||
:sync: vSphere
|
||||
|
||||
Make sure Ray supervisor service is up and running as per `the Ray-on-VCF docs <https://github-vcf.devops.broadcom.net/vcf/vmray>`
|
||||
|
||||
|
||||
Create a (basic) Python application
|
||||
-----------------------------------
|
||||
|
||||
We will write a simple Python application that tracks the IP addresses of the machines that its tasks are executed on:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from collections import Counter
|
||||
import socket
|
||||
import time
|
||||
|
||||
def f():
|
||||
time.sleep(0.001)
|
||||
# Return IP address.
|
||||
return socket.gethostbyname("localhost")
|
||||
|
||||
ip_addresses = [f() for _ in range(10000)]
|
||||
print(Counter(ip_addresses))
|
||||
|
||||
Save this application as ``script.py`` and execute it by running the command ``python script.py``. The application should take 10 seconds to run and output something similar to ``Counter({'127.0.0.1': 10000})``.
|
||||
|
||||
With some small changes, we can make this application run on Ray (for more information on how to do this, refer to :ref:`the Ray Core Walkthrough <core-walkthrough>`):
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from collections import Counter
|
||||
import socket
|
||||
import time
|
||||
|
||||
import ray
|
||||
|
||||
ray.init()
|
||||
|
||||
@ray.remote
|
||||
def f():
|
||||
time.sleep(0.001)
|
||||
# Return IP address.
|
||||
return socket.gethostbyname("localhost")
|
||||
|
||||
object_ids = [f.remote() for _ in range(10000)]
|
||||
ip_addresses = ray.get(object_ids)
|
||||
print(Counter(ip_addresses))
|
||||
|
||||
Finally, let's add some code to make the output more interesting:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from collections import Counter
|
||||
import socket
|
||||
import time
|
||||
|
||||
import ray
|
||||
|
||||
ray.init()
|
||||
|
||||
print('''This cluster consists of
|
||||
{} nodes in total
|
||||
{} CPU resources in total
|
||||
'''.format(len(ray.nodes()), ray.cluster_resources()['CPU']))
|
||||
|
||||
@ray.remote
|
||||
def f():
|
||||
time.sleep(0.001)
|
||||
# Return IP address.
|
||||
return socket.gethostbyname("localhost")
|
||||
|
||||
object_ids = [f.remote() for _ in range(10000)]
|
||||
ip_addresses = ray.get(object_ids)
|
||||
|
||||
print('Tasks executed')
|
||||
for ip_address, num_tasks in Counter(ip_addresses).items():
|
||||
print(' {} tasks on {}'.format(num_tasks, ip_address))
|
||||
|
||||
Running ``python script.py`` should now output something like:
|
||||
|
||||
.. parsed-literal::
|
||||
|
||||
This cluster consists of
|
||||
1 nodes in total
|
||||
4.0 CPU resources in total
|
||||
|
||||
Tasks executed
|
||||
10000 tasks on 127.0.0.1
|
||||
|
||||
Launch a cluster on a cloud provider
|
||||
------------------------------------
|
||||
|
||||
To start a Ray Cluster, first we need to define the cluster configuration. The cluster configuration is defined within a YAML file that will be used by the Cluster Launcher to launch the head node, and by the Autoscaler to launch worker nodes.
|
||||
|
||||
A minimal sample cluster configuration file looks as follows:
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Ray Team Supported
|
||||
:sync: Ray Team Supported
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: AWS
|
||||
:sync: AWS
|
||||
|
||||
.. literalinclude:: ../../../../python/ray/autoscaler/aws/example-minimal.yaml
|
||||
:language: yaml
|
||||
|
||||
.. tab-item:: Azure
|
||||
:sync: Azure
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
# An unique identifier for the head node and workers of this cluster.
|
||||
cluster_name: minimal
|
||||
|
||||
# Cloud-provider specific configuration.
|
||||
provider:
|
||||
type: azure
|
||||
location: westus2
|
||||
resource_group: ray-cluster
|
||||
|
||||
# How Ray will authenticate with newly launched nodes.
|
||||
auth:
|
||||
ssh_user: ubuntu
|
||||
# you must specify paths to matching private and public key pair files
|
||||
# use `ssh-keygen -t rsa -b 4096` to generate a new ssh key pair
|
||||
ssh_private_key: ~/.ssh/id_rsa
|
||||
# changes to this should match what is specified in file_mounts
|
||||
ssh_public_key: ~/.ssh/id_rsa.pub
|
||||
|
||||
.. tab-item:: GCP
|
||||
:sync: GCP
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
# A unique identifier for the head node and workers of this cluster.
|
||||
cluster_name: minimal
|
||||
|
||||
# Cloud-provider specific configuration.
|
||||
provider:
|
||||
type: gcp
|
||||
region: us-west1
|
||||
|
||||
.. tab-item:: Community Supported
|
||||
:sync: Community Supported
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Aliyun
|
||||
:sync: Aliyun
|
||||
|
||||
Please refer to `example-full.yaml </ray/python/ray/autoscaler/aliyun/example-full.yaml>`__.
|
||||
|
||||
Make sure your account balance is not less than 100 RMB, otherwise you will receive the error `InvalidAccountStatus.NotEnoughBalance`.
|
||||
|
||||
.. tab-item:: vSphere
|
||||
:sync: vSphere
|
||||
|
||||
.. literalinclude:: ../../../../python/ray/autoscaler/vsphere/example-minimal.yaml
|
||||
:language: yaml
|
||||
|
||||
|
||||
Save this configuration file as ``config.yaml``. You can specify a lot more details in the configuration file: instance types to use, minimum and maximum number of workers to start, autoscaling strategy, files to sync, and more. For a full reference on the available configuration properties, please refer to the :ref:`cluster YAML configuration options reference <cluster-config>`.
|
||||
|
||||
After defining our configuration, we will use the Ray cluster launcher to start a cluster on the cloud, creating a designated "head node" and worker nodes. To start the Ray cluster, we will use the :ref:`Ray CLI <ray-cluster-cli>`. Run the following command:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ ray up -y config.yaml
|
||||
|
||||
Running applications on a Ray Cluster
|
||||
-------------------------------------
|
||||
|
||||
We are now ready to execute an application on our Ray Cluster.
|
||||
``ray.init()`` will now automatically connect to the newly created cluster.
|
||||
|
||||
As a quick example, we execute a Python command on the Ray Cluster that connects to Ray and exits:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ ray exec config.yaml 'python -c "import ray; ray.init()"'
|
||||
2022-08-10 11:23:17,093 INFO worker.py:1312 -- Connecting to existing Ray cluster at address: <remote IP address>:6379...
|
||||
2022-08-10 11:23:17,097 INFO worker.py:1490 -- Connected to Ray cluster.
|
||||
|
||||
You can also optionally get a remote shell using ``ray attach`` and run commands directly on the cluster. This command will create an SSH connection to the head node of the Ray Cluster.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
# From a remote client:
|
||||
$ ray attach config.yaml
|
||||
|
||||
# Now on the head node...
|
||||
$ python -c "import ray; ray.init()"
|
||||
|
||||
For a full reference on the Ray Cluster CLI tools, please refer to :ref:`the cluster commands reference <cluster-commands>`.
|
||||
|
||||
While these tools are useful for ad-hoc execution on the Ray Cluster, the recommended way to execute an application on a Ray Cluster is to use :ref:`Ray Jobs <jobs-quickstart>`. Check out the :ref:`quickstart guide <jobs-quickstart>` to get started!
|
||||
|
||||
Deleting a Ray Cluster
|
||||
----------------------
|
||||
|
||||
To shut down your cluster, run the following command:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ ray down -y config.yaml
|
||||
@@ -0,0 +1,90 @@
|
||||
# Ray on Cloud VMs
|
||||
(cloud-vm-index)=
|
||||
|
||||
```{toctree}
|
||||
:hidden:
|
||||
|
||||
getting-started
|
||||
User Guides <user-guides/index>
|
||||
Examples <examples/index>
|
||||
references/index
|
||||
```
|
||||
|
||||
## Overview
|
||||
|
||||
In this section we cover how to launch Ray clusters on Cloud VMs. Ray ships with built-in support for launching AWS, GCP, and Azure clusters, and also has community-maintained integrations for Aliyun and vSphere. Each Ray cluster consists of a head node and a collection of worker nodes. Optional [autoscaling](vms-autoscaling) support allows the Ray cluster to be sized according to the requirements of your Ray workload, adding and removing worker nodes as needed. Ray supports clusters composed of multiple heterogeneous compute nodes (including GPU nodes).
|
||||
|
||||
Concretely, you will learn how to:
|
||||
|
||||
- Set up and configure Ray in public clouds
|
||||
- Deploy applications and monitor your cluster
|
||||
|
||||
## Learn More
|
||||
|
||||
The Ray docs present all the information you need to start running Ray workloads on VMs.
|
||||
|
||||
```{eval-rst}
|
||||
.. grid:: 1 2 2 2
|
||||
:gutter: 1
|
||||
:class-container: container pb-3
|
||||
|
||||
.. grid-item-card::
|
||||
|
||||
**Getting Started**
|
||||
^^^
|
||||
|
||||
Learn how to start a Ray cluster and deploy Ray applications in the cloud.
|
||||
|
||||
+++
|
||||
.. button-ref:: vm-cluster-quick-start
|
||||
:color: primary
|
||||
:outline:
|
||||
:expand:
|
||||
|
||||
Get Started with Ray on Cloud VMs
|
||||
|
||||
.. grid-item-card::
|
||||
|
||||
**Examples**
|
||||
^^^
|
||||
|
||||
Try example Ray workloads in the Cloud
|
||||
|
||||
+++
|
||||
.. button-ref:: vm-cluster-examples
|
||||
:color: primary
|
||||
:outline:
|
||||
:expand:
|
||||
|
||||
Try example workloads
|
||||
|
||||
.. grid-item-card::
|
||||
|
||||
**User Guides**
|
||||
^^^
|
||||
|
||||
Learn best practices for configuring cloud clusters
|
||||
|
||||
+++
|
||||
.. button-ref:: vm-cluster-guides
|
||||
:color: primary
|
||||
:outline:
|
||||
:expand:
|
||||
|
||||
Read the User Guides
|
||||
|
||||
.. grid-item-card::
|
||||
|
||||
**API Reference**
|
||||
^^^
|
||||
|
||||
Find API references for cloud clusters
|
||||
|
||||
+++
|
||||
.. button-ref:: vm-cluster-api-references
|
||||
:color: primary
|
||||
:outline:
|
||||
:expand:
|
||||
|
||||
Check API references
|
||||
```
|
||||
@@ -0,0 +1,14 @@
|
||||
(vm-cluster-api-references)=
|
||||
|
||||
# API References
|
||||
|
||||
The following pages provide reference documentation for using Ray Clusters on virtual machines.
|
||||
|
||||
```{toctree}
|
||||
:caption: "Reference documentation for Ray Clusters on VMs:"
|
||||
:maxdepth: '2'
|
||||
:name: ray-clusters-vms-reference
|
||||
|
||||
ray-cluster-cli
|
||||
ray-cluster-configuration
|
||||
```
|
||||
@@ -0,0 +1,232 @@
|
||||
.. _cluster-commands:
|
||||
|
||||
Cluster Launcher Commands
|
||||
=========================
|
||||
|
||||
This document overviews common commands for using the Ray cluster launcher.
|
||||
See the :ref:`Cluster Configuration <cluster-config>` docs on how to customize the configuration file.
|
||||
|
||||
Launching a cluster (``ray up``)
|
||||
--------------------------------
|
||||
|
||||
This will start up the machines in the cloud, install your dependencies and run
|
||||
any setup commands that you have, configure the Ray cluster automatically, and
|
||||
prepare you to scale your distributed system. See :ref:`the documentation
|
||||
<ray-up-doc>` for ``ray up``. The example config files can be accessed `here <https://github.com/ray-project/ray/tree/master/python/ray/autoscaler>`_.
|
||||
|
||||
.. tip:: The worker nodes will start only after the head node has finished
|
||||
starting. To monitor the progress of the cluster setup, you can run
|
||||
`ray monitor <cluster yaml>`.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
# Replace '<your_backend>' with one of: 'aws', 'gcp', 'kubernetes', or 'local'.
|
||||
$ BACKEND=<your_backend>
|
||||
|
||||
# Create or update the cluster.
|
||||
$ ray up ray/python/ray/autoscaler/$BACKEND/example-full.yaml
|
||||
|
||||
# Tear down the cluster.
|
||||
$ ray down ray/python/ray/autoscaler/$BACKEND/example-full.yaml
|
||||
|
||||
Updating an existing cluster (``ray up``)
|
||||
-----------------------------------------
|
||||
|
||||
If you want to update your cluster configuration (add more files, change dependencies), run ``ray up`` again on the existing cluster.
|
||||
|
||||
This command checks if the local configuration differs from the applied
|
||||
configuration of the cluster. This includes any changes to synced files
|
||||
specified in the ``file_mounts`` section of the config. If so, the new files
|
||||
and config will be uploaded to the cluster. Following that, Ray
|
||||
services/processes will be restarted.
|
||||
|
||||
.. tip:: Don't do this for the cloud provider specifications (e.g., change from
|
||||
AWS to GCP on a running cluster) or change the cluster name (as this
|
||||
will just start a new cluster and orphan the original one).
|
||||
|
||||
|
||||
You can also run ``ray up`` to restart a cluster if it seems to be in a bad
|
||||
state (this will restart all Ray services even if there are no config changes).
|
||||
|
||||
Running ``ray up`` on an existing cluster will do all the following:
|
||||
|
||||
* If the head node matches the cluster specification, the filemounts will be
|
||||
reapplied and the ``setup_commands`` and ``ray start`` commands will be run.
|
||||
There may be some caching behavior here to skip setup/file mounts.
|
||||
* If the head node is out of date from the specified YAML (e.g.,
|
||||
``head_node_type`` has changed on the YAML), then the out-of-date node will
|
||||
be terminated and a new node will be provisioned to replace it. Setup/file
|
||||
mounts/``ray start`` will be applied.
|
||||
* After the head node reaches a consistent state (after ``ray start`` commands
|
||||
are finished), the same above procedure will be applied to all the worker
|
||||
nodes. The ``ray start`` commands tend to run a ``ray stop`` + ``ray start``,
|
||||
so this will kill currently working jobs.
|
||||
|
||||
If you don't want the update to restart services (e.g. because the changes
|
||||
don't require a restart), pass ``--no-restart`` to the update call.
|
||||
|
||||
If you want to force re-generation of the config to pick up possible changes in
|
||||
the cloud environment, pass ``--no-config-cache`` to the update call.
|
||||
|
||||
If you want to skip the setup commands and only run ``ray stop``/``ray start``
|
||||
on all nodes, pass ``--restart-only`` to the update call.
|
||||
|
||||
See :ref:`the documentation <ray-up-doc>` for ``ray up``.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
# Reconfigure autoscaling behavior without interrupting running jobs.
|
||||
$ ray up ray/python/ray/autoscaler/$BACKEND/example-full.yaml \
|
||||
--max-workers=N --no-restart
|
||||
|
||||
Running shell commands on the cluster (``ray exec``)
|
||||
----------------------------------------------------
|
||||
|
||||
You can use ``ray exec`` to conveniently run commands on clusters. See :ref:`the documentation <ray-exec-doc>` for ``ray exec``.
|
||||
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
# Run a command on the cluster
|
||||
$ ray exec cluster.yaml 'echo "hello world"'
|
||||
|
||||
# Run a command on the cluster, starting it if needed
|
||||
$ ray exec cluster.yaml 'echo "hello world"' --start
|
||||
|
||||
# Run a command on the cluster, stopping the cluster after it finishes
|
||||
$ ray exec cluster.yaml 'echo "hello world"' --stop
|
||||
|
||||
# Run a command on a new cluster called 'experiment-1', stopping it after
|
||||
$ ray exec cluster.yaml 'echo "hello world"' \
|
||||
--start --stop --cluster-name experiment-1
|
||||
|
||||
# Run a command in a detached tmux session
|
||||
$ ray exec cluster.yaml 'echo "hello world"' --tmux
|
||||
|
||||
# Run a command in a screen (experimental)
|
||||
$ ray exec cluster.yaml 'echo "hello world"' --screen
|
||||
|
||||
If you want to run applications on the cluster that are accessible from a web
|
||||
browser (e.g., Jupyter notebook), you can use the ``--port-forward``. The local
|
||||
port opened is the same as the remote port.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ ray exec cluster.yaml --port-forward=8899 'source ~/anaconda3/bin/activate tensorflow_p36 && jupyter notebook --port=8899'
|
||||
|
||||
.. note:: For Kubernetes clusters, the ``port-forward`` option cannot be used
|
||||
while executing a command. To port forward and run a command you need
|
||||
to call ``ray exec`` twice separately.
|
||||
|
||||
Running Ray scripts on the cluster (``ray submit``)
|
||||
---------------------------------------------------
|
||||
|
||||
You can also use ``ray submit`` to execute Python scripts on clusters. This
|
||||
will ``rsync`` the designated file onto the head node cluster and execute it
|
||||
with the given arguments. See :ref:`the documentation <ray-submit-doc>` for
|
||||
``ray submit``.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
# Run a Python script in a detached tmux session
|
||||
$ ray submit cluster.yaml --tmux --start --stop tune_experiment.py
|
||||
|
||||
# Run a Python script with arguments.
|
||||
# This executes script.py on the head node of the cluster, using
|
||||
# the command: python ~/script.py --arg1 --arg2 --arg3
|
||||
$ ray submit cluster.yaml script.py -- --arg1 --arg2 --arg3
|
||||
|
||||
|
||||
Attaching to a running cluster (``ray attach``)
|
||||
-----------------------------------------------
|
||||
|
||||
You can use ``ray attach`` to attach to an interactive screen session on the
|
||||
cluster. See :ref:`the documentation <ray-attach-doc>` for ``ray attach`` or
|
||||
run ``ray attach --help``.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
# Open a screen on the cluster
|
||||
$ ray attach cluster.yaml
|
||||
|
||||
# Open a screen on a new cluster called 'session-1'
|
||||
$ ray attach cluster.yaml --start --cluster-name=session-1
|
||||
|
||||
# Attach to tmux session on cluster (creates a new one if none available)
|
||||
$ ray attach cluster.yaml --tmux
|
||||
|
||||
.. _ray-rsync:
|
||||
|
||||
Synchronizing files from the cluster (``ray rsync-up/down``)
|
||||
------------------------------------------------------------
|
||||
|
||||
To download or upload files to the cluster head node, use ``ray rsync_down`` or
|
||||
``ray rsync_up``:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ ray rsync_down cluster.yaml '/path/on/cluster' '/local/path'
|
||||
$ ray rsync_up cluster.yaml '/local/path' '/path/on/cluster'
|
||||
|
||||
.. _monitor-cluster:
|
||||
|
||||
Monitoring cluster status (``ray dashboard/status``)
|
||||
-----------------------------------------------------
|
||||
|
||||
The Ray also comes with an online dashboard. The dashboard is accessible via
|
||||
HTTP on the head node (by default it listens on ``localhost:8265``). You can
|
||||
also use the built-in ``ray dashboard`` to set up port forwarding
|
||||
automatically, making the remote dashboard viewable in your local browser at
|
||||
``localhost:8265``.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ ray dashboard cluster.yaml
|
||||
|
||||
You can monitor cluster usage and auto-scaling status by running (on the head node):
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ ray status
|
||||
|
||||
To see live updates to the status:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ watch -n 1 ray status
|
||||
|
||||
The Ray autoscaler also reports per-node status in the form of instance tags.
|
||||
In your cloud provider console, you can click on a Node, go to the "Tags" pane,
|
||||
and add the ``ray-node-status`` tag as a column. This lets you see per-node
|
||||
statuses at a glance:
|
||||
|
||||
.. image:: /images/autoscaler-status.png
|
||||
|
||||
Common Workflow: Syncing git branches
|
||||
-------------------------------------
|
||||
|
||||
A common use case is syncing a particular local git branch to all workers of
|
||||
the cluster. However, if you just put a `git checkout <branch>` in the setup
|
||||
commands, the autoscaler won't know when to rerun the command to pull in
|
||||
updates. There is a nice workaround for this by including the git SHA in the
|
||||
input (the hash of the file will change if the branch is updated):
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
file_mounts: {
|
||||
"/tmp/current_branch_sha": "/path/to/local/repo/.git/refs/heads/<YOUR_BRANCH_NAME>",
|
||||
}
|
||||
|
||||
setup_commands:
|
||||
- test -e <REPO_NAME> || git clone https://github.com/<REPO_ORG>/<REPO_NAME>.git
|
||||
- cd <REPO_NAME> && git fetch && git checkout `cat /tmp/current_branch_sha`
|
||||
|
||||
This tells ``ray up`` to sync the current git branch SHA from your personal
|
||||
computer to a temporary file on the cluster (assuming you've pushed the branch
|
||||
head already). Then, the setup commands read that file to figure out which SHA
|
||||
they should checkout on the nodes. Note that each command runs in its own
|
||||
session. The final workflow to update the cluster then becomes just this:
|
||||
|
||||
1. Make local changes to a git branch
|
||||
2. Commit the changes with ``git commit`` and ``git push``
|
||||
3. Update files on your Ray cluster with ``ray up``
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,42 @@
|
||||
.. _ref-cluster-setup:
|
||||
|
||||
Community Supported Cluster Managers
|
||||
====================================
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
|
||||
yarn
|
||||
slurm
|
||||
lsf
|
||||
|
||||
.. note::
|
||||
|
||||
If you're using AWS, Azure, GCP or vSphere you can use the :ref:`Ray cluster launcher <cluster-index>` to simplify the cluster setup process.
|
||||
|
||||
The following is a list of community supported cluster managers.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
yarn.rst
|
||||
slurm.rst
|
||||
lsf.rst
|
||||
spark.rst
|
||||
|
||||
.. _ref-additional-cloud-providers:
|
||||
|
||||
Using a custom cloud or cluster manager
|
||||
=======================================
|
||||
|
||||
The Ray cluster launcher currently supports AWS, Azure, GCP, Aliyun, vSphere and KubeRay out of the box. To use the Ray cluster launcher and Autoscaler on other cloud providers or cluster managers, you can implement the `node_provider.py <https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/node_provider.py>`_ interface (100 LOC).
|
||||
Once the node provider is implemented, you can register it in the `provider section <https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/local/example-full.yaml#L18>`_ of the cluster launcher config.
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
provider:
|
||||
type: "external"
|
||||
module: "my.module.MyCustomNodeProvider"
|
||||
|
||||
You can refer to `AWSNodeProvider <https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/_private/aws/node_provider.py#L95>`_, `KubeRayNodeProvider <https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/_private/kuberay/node_provider.py#L148>`_ and
|
||||
`LocalNodeProvider <https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/_private/local/node_provider.py#L166>`_ for more examples.
|
||||
@@ -0,0 +1,18 @@
|
||||
.. _ray-LSF-deploy:
|
||||
|
||||
Deploying on LSF
|
||||
================
|
||||
|
||||
This document describes a couple high-level steps to run Ray clusters on LSF.
|
||||
|
||||
1) Obtain desired nodes from LSF scheduler using bsub directives.
|
||||
2) Obtain free ports on the desired nodes to start ray services like dashboard, GCS etc.
|
||||
3) Start ray head node on one of the available nodes.
|
||||
4) Connect all the worker nodes to the head node.
|
||||
5) Perform port forwarding to access ray dashboard.
|
||||
|
||||
Steps 1-4 have been automated and can be easily run as a script, please refer to below github repo to access script and run sample workloads:
|
||||
|
||||
- `ray_LSF`_ Ray with LSF. Users can start up a Ray cluster on LSF, and run DL workloads through that either in a batch or interactive mode.
|
||||
|
||||
.. _`ray_LSF`: https://github.com/IBMSpectrumComputing/ray-integration
|
||||
@@ -0,0 +1,9 @@
|
||||
:orphan:
|
||||
|
||||
.. _slurm-basic:
|
||||
|
||||
slurm-basic.sh
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
.. literalinclude:: /cluster/doc_code/slurm-basic.sh
|
||||
:language: bash
|
||||
@@ -0,0 +1,8 @@
|
||||
:orphan:
|
||||
|
||||
.. _slurm-launch:
|
||||
|
||||
slurm-launch.py
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
.. literalinclude:: /cluster/doc_code/slurm-launch.py
|
||||
@@ -0,0 +1,9 @@
|
||||
:orphan:
|
||||
|
||||
.. _slurm-template:
|
||||
|
||||
slurm-template.sh
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. literalinclude:: /cluster/doc_code/slurm-template.sh
|
||||
:language: bash
|
||||
@@ -0,0 +1,281 @@
|
||||
.. _ray-slurm-deploy:
|
||||
|
||||
Deploying on Slurm
|
||||
==================
|
||||
|
||||
Slurm usage with Ray can be a little bit unintuitive.
|
||||
|
||||
* SLURM requires multiple copies of the same program are submitted multiple times to the same cluster to do cluster programming. This is particularly well-suited for MPI-based workloads.
|
||||
* Ray, on the other hand, expects a head-worker architecture with a single point of entry. That is, you'll need to start a Ray head node, multiple Ray worker nodes, and run your Ray script on the head node.
|
||||
|
||||
To bridge this gap, Ray 2.49 and above introduces ``ray symmetric-run`` command, which will start a Ray cluster on all nodes with given CPU and GPU resources and run your entrypoint script ONLY the head node.
|
||||
|
||||
Below, we provide a walkthrough using ``ray symmetric-run`` to run Ray on SLURM.
|
||||
|
||||
.. contents::
|
||||
:local:
|
||||
|
||||
|
||||
Walkthrough using Ray with SLURM
|
||||
--------------------------------
|
||||
|
||||
Many SLURM deployments require you to interact with slurm via ``sbatch``, which executes a batch script on SLURM.
|
||||
|
||||
To run a Ray job with ``sbatch``, you will want to start a Ray cluster in the sbatch job with multiple ``srun`` commands (tasks), and then execute your python script that uses Ray. Each task will run on a separate node and start/connect to a Ray runtime.
|
||||
|
||||
The below walkthrough will do the following:
|
||||
|
||||
1. Set the proper headers for the ``sbatch`` script.
|
||||
2. Load the proper environment/modules.
|
||||
3. Fetch a list of available computing nodes and their IP addresses.
|
||||
4. Launch a head ray process in one of the node (called the head node).
|
||||
5. Launch Ray processes in (n-1) worker nodes and connects them to the head node by providing the head node address.
|
||||
6. After the underlying ray cluster is ready, submit the user specified task.
|
||||
|
||||
See :ref:`slurm-basic.sh <slurm-basic>` for an end-to-end example.
|
||||
|
||||
.. _ray-slurm-headers:
|
||||
|
||||
sbatch directives
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
In your sbatch script, you'll want to add `directives to provide context <https://slurm.schedmd.com/sbatch.html>`__ for your job to SLURM.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
#!/bin/bash
|
||||
#SBATCH --job-name=my-workload
|
||||
|
||||
You'll need to tell SLURM to allocate nodes specifically for Ray. Ray will then find and manage all resources on each node.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
### Modify this according to your Ray workload.
|
||||
#SBATCH --nodes=4
|
||||
#SBATCH --exclusive
|
||||
|
||||
Important: To ensure that each Ray worker runtime will run on a separate node, set ``tasks-per-node``.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
#SBATCH --tasks-per-node=1
|
||||
|
||||
Since we've set `tasks-per-node = 1`, this will be used to guarantee that each Ray worker runtime will obtain the
|
||||
proper resources. In this example, we ask for at least 5 CPUs and 5 GB of memory per node.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
### Modify this according to your Ray workload.
|
||||
#SBATCH --cpus-per-task=5
|
||||
#SBATCH --mem-per-cpu=1GB
|
||||
### Similarly, you can also specify the number of GPUs per node.
|
||||
### Modify this according to your Ray workload. Sometimes this
|
||||
### should be 'gres' instead.
|
||||
#SBATCH --gpus-per-task=1
|
||||
|
||||
|
||||
You can also add other optional flags to your sbatch directives.
|
||||
|
||||
|
||||
Loading your environment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
First, you'll often want to Load modules or your own conda environment at the beginning of the script.
|
||||
|
||||
Note that this is an optional step, but it is often required for enabling the right set of dependencies.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Example: module load pytorch/v1.4.0-gpu
|
||||
# Example: conda activate my-env
|
||||
|
||||
conda activate my-env
|
||||
|
||||
Obtain the head IP address
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Next, we'll want to obtain a hostname and a node IP address for the head node. This way, when we start worker nodes, we'll be able to properly connect to the right head node.
|
||||
|
||||
.. literalinclude:: /cluster/doc_code/slurm-basic.sh
|
||||
:language: bash
|
||||
:start-after: __doc_head_address_start__
|
||||
:end-before: __doc_head_address_end__
|
||||
|
||||
.. note:: In Ray 2.49 and above, you can use IPv6 addresses/hostnames.
|
||||
|
||||
|
||||
Starting Ray and executing your script
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. note:: `ray symmetric-run` is available in Ray 2.49 and above. Check older versions of the documentation if you are using an older version of Ray.
|
||||
|
||||
Now, we'll use `ray symmetric-run` to start Ray on all nodes with given CPU and GPU resources and run your entrypoint script ONLY the head node.
|
||||
|
||||
Below, you'll see that we explicitly specify the number of CPUs (``num-cpus``)
|
||||
and number of GPUs (``num-gpus``) to Ray, as this will prevent Ray from using
|
||||
more resources than allocated. We also need to explicitly
|
||||
indicate the ``address`` parameter for the head node to identify itself and other nodes to connect to:
|
||||
|
||||
.. literalinclude:: /cluster/doc_code/slurm-basic.sh
|
||||
:language: bash
|
||||
:start-after: __doc_symmetric_run_start__
|
||||
:end-before: __doc_symmetric_run_end__
|
||||
|
||||
After the training job is completed, the Ray cluster will be stopped automatically.
|
||||
|
||||
.. note:: The -u argument tells python to print to stdout unbuffered, which is important with how slurm deals with rerouting output. If this argument is not included, you may get strange printing behavior such as printed statements not being logged by slurm until the program has terminated.
|
||||
|
||||
.. _slurm-network-ray:
|
||||
|
||||
SLURM networking caveats
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
There are two important networking aspects to keep in mind when working with
|
||||
SLURM and Ray:
|
||||
|
||||
1. Ports binding.
|
||||
2. IP binding.
|
||||
|
||||
One common use of a SLURM cluster is to have multiple users running concurrent
|
||||
jobs on the same infrastructure. This can easily conflict with Ray due to the
|
||||
way the head node communicates with its workers.
|
||||
|
||||
|
||||
Considering 2 users, if they both schedule a SLURM job using Ray
|
||||
at the same time, they are both creating a head node. In the backend, Ray will
|
||||
assign some internal ports to a few services. The issue is that as soon as the
|
||||
first head node is created, it will bind some ports and prevent them to be
|
||||
used by another head node. To prevent any conflicts, users have to manually
|
||||
specify non overlapping ranges of ports. The following ports are to be
|
||||
adjusted. For an explanation on ports, see :ref:`here <ray-ports>`::
|
||||
|
||||
# used for all ports
|
||||
--node-manager-port
|
||||
--object-manager-port
|
||||
--min-worker-port
|
||||
--max-worker-port
|
||||
# used for the head node
|
||||
--port
|
||||
--ray-client-server-port
|
||||
--redis-shard-ports
|
||||
|
||||
For instance, again with 2 users, they would run the following commands. Note that we don't use symmetric-run here
|
||||
because it does not currently work in multi-tenant environments:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# user 1
|
||||
...
|
||||
srun --nodes=1 --ntasks=1 -w "$head_node" \
|
||||
ray start --head --node-ip-address="$head_node_ip" \
|
||||
--port=6379 \
|
||||
--node-manager-port=6700 \
|
||||
--object-manager-port=6701 \
|
||||
--ray-client-server-port=10001 \
|
||||
--redis-shard-ports=6702 \
|
||||
--min-worker-port=10002 \
|
||||
--max-worker-port=19999 \
|
||||
--num-cpus "${SLURM_CPUS_PER_TASK}" --num-gpus "${SLURM_GPUS_PER_TASK}" --block &
|
||||
|
||||
python -u your_script.py
|
||||
|
||||
# user 2
|
||||
...
|
||||
srun --nodes=1 --ntasks=1 -w "$head_node" \
|
||||
ray start --head --node-ip-address="$head_node_ip" \
|
||||
--port=6380 \
|
||||
--node-manager-port=6800 \
|
||||
--object-manager-port=6801 \
|
||||
--ray-client-server-port=20001 \
|
||||
--redis-shard-ports=6802 \
|
||||
--min-worker-port=20002 \
|
||||
--max-worker-port=29999 \
|
||||
--num-cpus "${SLURM_CPUS_PER_TASK}" --num-gpus "${SLURM_GPUS_PER_TASK}" --block &
|
||||
|
||||
python -u your_script.py
|
||||
|
||||
As for the IP binding, on some cluster architecture the network interfaces
|
||||
do not allow to use external IPs between nodes. Instead, there are internal
|
||||
network interfaces (`eth0`, `eth1`, etc.). Currently, it's difficult to
|
||||
set an internal IP
|
||||
(see the open `issue <https://github.com/ray-project/ray/issues/22732>`_).
|
||||
|
||||
|
||||
Python-interface SLURM scripts
|
||||
------------------------------
|
||||
|
||||
[Contributed by @pengzhenghao] Below, we provide a helper utility (:ref:`slurm-launch.py <slurm-launch>`) to auto-generate SLURM scripts and launch.
|
||||
``slurm-launch.py`` uses an underlying template (:ref:`slurm-template.sh <slurm-template>`) and fills out placeholders given user input.
|
||||
|
||||
You can feel free to copy both files into your cluster for use. Feel free to also open any PRs for contributions to improve this script!
|
||||
|
||||
Usage example
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
If you want to utilize a multi-node cluster in slurm:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python slurm-launch.py --exp-name test --command "python your_file.py" --num-nodes 3
|
||||
|
||||
If you want to specify the computing node(s), just use the same node name(s) in the same format of the output of ``sinfo`` command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python slurm-launch.py --exp-name test --command "python your_file.py" --num-nodes 3 --node NODE_NAMES
|
||||
|
||||
|
||||
There are other options you can use when calling ``python slurm-launch.py``:
|
||||
|
||||
* ``--exp-name``: The experiment name. Will generate ``{exp-name}_{date}-{time}.sh`` and ``{exp-name}_{date}-{time}.log``.
|
||||
* ``--command``: The command you wish to run. For example: ``rllib train XXX`` or ``python XXX.py``.
|
||||
* ``--num-gpus``: The number of GPUs you wish to use in each computing node. Default: 0.
|
||||
* ``--node`` (``-w``): The specific nodes you wish to use, in the same form as the output of ``sinfo``. Nodes are automatically assigned if not specified.
|
||||
* ``--num-nodes`` (``-n``): The number of nodes you wish to use. Default: 1.
|
||||
* ``--partition`` (``-p``): The partition you wish to use. Default: "", will use user's default partition.
|
||||
* ``--load-env``: The command to setup your environment. For example: ``module load cuda/10.1``. Default: "".
|
||||
|
||||
Note that the :ref:`slurm-template.sh <slurm-template>` is compatible with both IPV4 and IPV6 ip address of the computing nodes.
|
||||
|
||||
Implementation
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
Concretely, the (:ref:`slurm-launch.py <slurm-launch>`) does the following things:
|
||||
|
||||
1. It automatically writes your requirements, e.g. number of CPUs, GPUs per node, the number of nodes and so on, to a sbatch script name ``{exp-name}_{date}-{time}.sh``. Your command (``--command``) to launch your own job is also written into the sbatch script.
|
||||
2. Then it will submit the sbatch script to slurm manager via a new process.
|
||||
3. Finally, the python process will terminate itself and leaves a log file named ``{exp-name}_{date}-{time}.log`` to record the progress of your submitted command. At the mean time, the ray cluster and your job is running in the slurm cluster.
|
||||
|
||||
|
||||
Examples and templates
|
||||
----------------------
|
||||
|
||||
Here are some community-contributed templates for using SLURM with Ray:
|
||||
|
||||
- `Ray sbatch submission scripts`_ used at `NERSC <https://www.nersc.gov/>`_, a US national lab.
|
||||
- `YASPI`_ (yet another slurm python interface) by @albanie. The goal of yaspi is to provide an interface to submitting slurm jobs, thereby obviating the joys of sbatch files. It does so through recipes - these are collections of templates and rules for generating sbatch scripts. Supports job submissions for Ray.
|
||||
|
||||
- `Convenient python interface`_ to launch ray cluster and submit task by @pengzhenghao
|
||||
|
||||
.. _`Ray sbatch submission scripts`: https://github.com/NERSC/slurm-ray-cluster
|
||||
|
||||
.. _`YASPI`: https://github.com/albanie/yaspi
|
||||
|
||||
.. _`Convenient python interface`: https://github.com/pengzhenghao/use-ray-with-slurm
|
||||
|
||||
|
||||
Troubleshooting
|
||||
---------------
|
||||
|
||||
.. _ray-slurm-docker-init:
|
||||
|
||||
Zombie processes in Docker containers
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Running Ray on Slurm inside a Docker container can produce zombie processes. See `this comment
|
||||
<https://github.com/ray-project/ray/pull/62591#issuecomment-4396615458>`_ for details.
|
||||
|
||||
Two recommended fixes:
|
||||
|
||||
1. Use ``docker run --init`` (`Docker CLI docs <https://docs.docker.com/reference/cli/docker/container/run/#init>`_).
|
||||
2. Set ``init: true`` in ``docker-compose.yaml`` (`Docker Compose docs <https://docs.docker.com/reference/compose-file/services/#init>`_).
|
||||
@@ -0,0 +1,105 @@
|
||||
.. _ray-Spark-deploy:
|
||||
|
||||
Deploying on Spark Standalone cluster
|
||||
=====================================
|
||||
|
||||
This document describes a couple high-level steps to run Ray clusters on `Spark Standalone cluster <https://spark.apache.org/docs/latest/spark-standalone.html>`_.
|
||||
|
||||
Running a basic example
|
||||
-----------------------
|
||||
|
||||
This is a spark application example code that starts Ray cluster on spark,
|
||||
and then execute ray application code, then shut down initiated ray cluster.
|
||||
|
||||
1) Create a python file that contains a spark application code,
|
||||
Assuming the python file name is 'ray-on-spark-example1.py'.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from pyspark.sql import SparkSession
|
||||
from ray.util.spark import setup_ray_cluster, shutdown_ray_cluster, MAX_NUM_WORKER_NODES
|
||||
if __name__ == "__main__":
|
||||
spark = SparkSession \
|
||||
.builder \
|
||||
.appName("Ray on spark example 1") \
|
||||
.config("spark.task.cpus", "4") \
|
||||
.getOrCreate()
|
||||
|
||||
# Set up a ray cluster on this spark application, it creates a background
|
||||
# spark job that each spark task launches one ray worker node.
|
||||
# ray head node is launched in spark application driver side.
|
||||
# Resources (CPU / GPU / memory) allocated to each ray worker node is equal
|
||||
# to resources allocated to the corresponding spark task.
|
||||
setup_ray_cluster(max_worker_nodes=MAX_NUM_WORKER_NODES)
|
||||
|
||||
# You can any ray application code here, the ray application will be executed
|
||||
# on the ray cluster setup above.
|
||||
# You don't need to set address for `ray.init`,
|
||||
# it will connect to the cluster created above automatically.
|
||||
ray.init()
|
||||
...
|
||||
|
||||
# Terminate ray cluster explicitly.
|
||||
# If you don't call it, when spark application is terminated, the ray cluster
|
||||
# will also be terminated.
|
||||
shutdown_ray_cluster()
|
||||
|
||||
2) Submit the spark application above to spark standalone cluster.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
#!/bin/bash
|
||||
spark-submit \
|
||||
--master spark://{spark_master_IP}:{spark_master_port} \
|
||||
path/to/ray-on-spark-example1.py
|
||||
|
||||
Creating a long running ray cluster on spark cluster
|
||||
----------------------------------------------------
|
||||
|
||||
This is a spark application example code that starts a long running Ray cluster on spark.
|
||||
The created ray cluster can be accessed by remote python processes.
|
||||
|
||||
1) Create a python file that contains a spark application code,
|
||||
Assuming the python file name is 'long-running-ray-cluster-on-spark.py'.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from pyspark.sql import SparkSession
|
||||
import time
|
||||
from ray.util.spark import setup_ray_cluster, MAX_NUM_WORKER_NODES
|
||||
|
||||
if __name__ == "__main__":
|
||||
spark = SparkSession \
|
||||
.builder \
|
||||
.appName("long running ray cluster on spark") \
|
||||
.config("spark.task.cpus", "4") \
|
||||
.getOrCreate()
|
||||
|
||||
cluster_address = setup_ray_cluster(
|
||||
max_worker_nodes=MAX_NUM_WORKER_NODES
|
||||
)
|
||||
print("Ray cluster is set up, you can connect to this ray cluster "
|
||||
f"via address ray://{cluster_address}")
|
||||
|
||||
# Sleep forever until the spark application being terminated,
|
||||
# at that time, the ray cluster will also be terminated.
|
||||
while True:
|
||||
time.sleep(10)
|
||||
|
||||
2) Submit the spark application above to spark standalone cluster.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
#!/bin/bash
|
||||
spark-submit \
|
||||
--master spark://{spark_master_IP}:{spark_master_port} \
|
||||
path/to/long-running-ray-cluster-on-spark.py
|
||||
|
||||
Ray on Spark APIs
|
||||
-----------------
|
||||
|
||||
.. autofunction:: ray.util.spark.setup_ray_cluster
|
||||
|
||||
.. autofunction:: ray.util.spark.shutdown_ray_cluster
|
||||
|
||||
.. autofunction:: ray.util.spark.setup_global_ray_cluster
|
||||
@@ -0,0 +1,206 @@
|
||||
.. _ray-yarn-deploy:
|
||||
|
||||
Deploying on YARN
|
||||
=================
|
||||
|
||||
.. warning::
|
||||
|
||||
Running Ray on YARN is still a work in progress. If you have a
|
||||
suggestion for how to improve this documentation or want to request
|
||||
a missing feature, please feel free to create a pull request or get in touch
|
||||
using one of the channels in the `Questions or Issues?`_ section below.
|
||||
|
||||
This document assumes that you have access to a YARN cluster and will walk
|
||||
you through using `Skein`_ to deploy a YARN job that starts a Ray cluster and
|
||||
runs an example script on it.
|
||||
|
||||
Skein uses a declarative specification (either written as a yaml file or using the Python API) and allows users to launch jobs and scale applications without the need to write Java code.
|
||||
|
||||
You will first need to install Skein: ``pip install skein``.
|
||||
|
||||
The Skein ``yaml`` file and example Ray program used here are provided in the
|
||||
`Ray repository`_ to get you started. Refer to the provided ``yaml``
|
||||
files to be sure that you maintain important configuration options for Ray to
|
||||
function properly.
|
||||
|
||||
.. _`Ray repository`: https://github.com/ray-project/ray/tree/master/doc/yarn
|
||||
|
||||
Skein Configuration
|
||||
-------------------
|
||||
|
||||
A Ray job is configured to run as two `Skein services`:
|
||||
|
||||
1. The ``ray-head`` service that starts the Ray head node and then runs the
|
||||
application.
|
||||
2. The ``ray-worker`` service that starts worker nodes that join the Ray cluster.
|
||||
You can change the number of instances in this configuration or at runtime
|
||||
using ``skein container scale`` to scale the cluster up/down.
|
||||
|
||||
The specification for each service consists of necessary files and commands that will be run to start the service.
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
services:
|
||||
ray-head:
|
||||
# There should only be one instance of the head node per cluster.
|
||||
instances: 1
|
||||
resources:
|
||||
# The resources for the worker node.
|
||||
vcores: 1
|
||||
memory: 2048
|
||||
files:
|
||||
...
|
||||
script:
|
||||
...
|
||||
ray-worker:
|
||||
# Number of ray worker nodes to start initially.
|
||||
# This can be scaled using 'skein container scale'.
|
||||
instances: 3
|
||||
resources:
|
||||
# The resources for the worker node.
|
||||
vcores: 1
|
||||
memory: 2048
|
||||
files:
|
||||
...
|
||||
script:
|
||||
...
|
||||
|
||||
Packaging Dependencies
|
||||
----------------------
|
||||
|
||||
Use the ``files`` option to specify files that will be copied into the YARN container for the application to use. See `the Skein file distribution page <https://jcrist.github.io/skein/distributing-files.html>`_ for more information.
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
services:
|
||||
ray-head:
|
||||
# There should only be one instance of the head node per cluster.
|
||||
instances: 1
|
||||
resources:
|
||||
# The resources for the head node.
|
||||
vcores: 1
|
||||
memory: 2048
|
||||
files:
|
||||
# ray/doc/yarn/example.py
|
||||
example.py: example.py
|
||||
# ray/doc/yarn/dashboard.py
|
||||
dashboard.py: dashboard.py
|
||||
# # A packaged python environment using `conda-pack`. Note that Skein
|
||||
# # doesn't require any specific way of distributing files, but this
|
||||
# # is a good one for python projects. This is optional.
|
||||
# # See https://jcrist.github.io/skein/distributing-files.html
|
||||
# environment: environment.tar.gz
|
||||
|
||||
Ray Setup in YARN
|
||||
-----------------
|
||||
|
||||
Below is a walkthrough of the bash commands used to start the ``ray-head`` and ``ray-worker`` services. Note that this configuration will launch a new Ray cluster for each application, not reuse the same cluster.
|
||||
|
||||
Head node commands
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Start by activating a pre-existing environment for dependency management.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
source environment/bin/activate
|
||||
|
||||
Register the Ray head address needed by the workers in the Skein key-value store.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
skein kv put --key=RAY_HEAD_ADDRESS --value=$(hostname -i) current
|
||||
|
||||
Start all the processes needed on the ray head node. By default, we set object store memory
|
||||
and heap memory to roughly 200 MB. This is conservative and should be set according to application needs.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ray start --head --port=6379 --object-store-memory=200000000 --memory 200000000 --num-cpus=1
|
||||
|
||||
Register the ray dashboard to Skein. This exposes the dashboard link on the Skein application page.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python dashboard.py "http://$(hostname -i):8265"
|
||||
|
||||
Execute the user script containing the Ray program.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
python example.py
|
||||
|
||||
Clean up all started processes even if the application fails or is killed.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ray stop
|
||||
skein application shutdown current
|
||||
|
||||
Putting things together, we have:
|
||||
|
||||
.. literalinclude:: /cluster/doc_code/yarn/ray-skein.yaml
|
||||
:language: yaml
|
||||
:start-after: # Head service
|
||||
:end-before: # Worker service
|
||||
|
||||
|
||||
Worker node commands
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Fetch the address of the head node from the Skein key-value store.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
RAY_HEAD_ADDRESS=$(skein kv get current --key=RAY_HEAD_ADDRESS)
|
||||
|
||||
Start all of the processes needed on a ray worker node, blocking until killed by Skein/YARN via SIGTERM. After receiving SIGTERM, all started processes should also die (ray stop).
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ray start --object-store-memory=200000000 --memory 200000000 --num-cpus=1 --address=$RAY_HEAD_ADDRESS:6379 --block; ray stop
|
||||
|
||||
Putting things together, we have:
|
||||
|
||||
.. literalinclude:: /cluster/doc_code/yarn/ray-skein.yaml
|
||||
:language: yaml
|
||||
:start-after: # Worker service
|
||||
|
||||
Running a Job
|
||||
-------------
|
||||
|
||||
Within your Ray script, use the following to connect to the started Ray cluster:
|
||||
|
||||
.. literalinclude:: /cluster/doc_code/yarn/example.py
|
||||
:language: python
|
||||
:start-after: if __name__ == "__main__"
|
||||
|
||||
You can use the following command to launch the application as specified by the Skein YAML file.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
skein application submit [TEST.YAML]
|
||||
|
||||
Once it has been submitted, you can see the job running on the YARN dashboard.
|
||||
|
||||
.. image:: /cluster/images/yarn-job.png
|
||||
|
||||
If you have registered the Ray dashboard address in the Skein as shown above, you can retrieve it on Skein's application page:
|
||||
|
||||
.. image:: /cluster/images/yarn-job-dashboard.png
|
||||
|
||||
Cleaning Up
|
||||
-----------
|
||||
|
||||
To clean up a running job, use the following (using the application ID):
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
skein application shutdown $appid
|
||||
|
||||
Questions or Issues?
|
||||
--------------------
|
||||
|
||||
.. include:: /_includes/_help.rst
|
||||
|
||||
.. _`Skein`: https://jcrist.github.io/skein/
|
||||
@@ -0,0 +1,56 @@
|
||||
.. _vms-autoscaling:
|
||||
|
||||
Configuring Autoscaling
|
||||
=======================
|
||||
|
||||
This guide explains how to configure the Ray autoscaler using the Ray cluster launcher.
|
||||
The Ray autoscaler is a Ray cluster process that automatically scales a cluster up and down based on resource demand.
|
||||
The autoscaler does this by adjusting the number of nodes in the cluster based on the resources required by tasks, actors or placement groups.
|
||||
|
||||
Note that the autoscaler only considers logical resource requests for scaling (i.e., those specified in ``@ray.remote`` and displayed in `ray status`), not physical machine utilization. If a user tries to launch an actor, task, or placement group but there are insufficient resources, the request will be queued. The autoscaler adds nodes to satisfy resource demands in this queue.
|
||||
The autoscaler also removes nodes after they become idle for some time.
|
||||
A node is considered idle if it has no active tasks, actors, or objects.
|
||||
|
||||
.. tip::
|
||||
**When to use Autoscaling?**
|
||||
|
||||
Autoscaling can reduce workload costs, but adds node launch overheads and can be tricky to configure.
|
||||
We recommend starting with non-autoscaling clusters if you're new to Ray.
|
||||
|
||||
Cluster Config Parameters
|
||||
-------------------------
|
||||
|
||||
The following options are available in your cluster config file.
|
||||
It is recommended that you set these before launching your cluster, but you can also modify them at run-time by updating the cluster config.
|
||||
|
||||
`max_workers[default_value=2, min_value=0]`: The max number of cluster worker nodes to launch. Note that this does not include the head node.
|
||||
|
||||
`min_workers[default_value=0, min_value=0]`: The min number of cluster worker nodes to launch, regardless of utilization. Note that this does not include the head node. This number must be less than the ``max_workers``.
|
||||
|
||||
.. note::
|
||||
|
||||
If `max_workers` is modified at runtime, the autoscaler will immediately remove nodes until this constraint
|
||||
is satisfied. This may disrupt running workloads.
|
||||
|
||||
If you are using more than one node type, you can also set min and max workers for each individual type:
|
||||
|
||||
`available_node_types.<node_type_name>.max_workers[default_value=cluster max_workers, min_value=0]`: The maximum number of worker nodes of a given type to launch. This number must be less than or equal to the `max_workers` for the cluster.
|
||||
|
||||
|
||||
`available_node_types.<node_type_name>.min_workers[default_value=0, min_value=0]`: The minimum number of worker nodes of a given type to launch, regardless of utilization. The sum of `min_workers` across all node types must be less than or equal to the `max_workers` for the cluster.
|
||||
|
||||
Upscaling and downscaling speed
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
If needed, you can also control the rate at which nodes should be added to or removed from the cluster. For applications with many short-lived tasks, you may wish to adjust the upscaling and downscaling speed to be more conservative.
|
||||
|
||||
`upscaling_speed[default_value=1.0, min_value=1.0]`: The number of nodes allowed to be pending as a multiple of the current number of nodes. The higher the value, the more aggressive upscaling will be. For example, if this is set to 1.0, the cluster can grow in size by at most 100% at any time, so if the cluster currently has 20 nodes, at most 20 pending
|
||||
launches are allowed. The minimum number of pending launches is 5 regardless of this setting.
|
||||
|
||||
`idle_timeout_minutes[default_value=5, min_value=0]`: The number of minutes that need to pass before an idle worker node is removed by the
|
||||
autoscaler. The smaller the value, the more aggressive downscaling will be. Worker nodes are considered idle when they hold no active tasks, actors, or referenced objects (either in-memory or spilled to disk). This parameter does not affect the head node.
|
||||
|
||||
Programmatic Scaling
|
||||
--------------------
|
||||
|
||||
For more information on programmatic access to the autoscaler, see the :ref:`Programmatic Cluster Scaling Guide <ref-autoscaler-sdk>`.
|
||||
@@ -0,0 +1,24 @@
|
||||
(vm-cluster-guides)=
|
||||
|
||||
# User Guides
|
||||
|
||||
```{toctree}
|
||||
:hidden:
|
||||
|
||||
launching-clusters/index
|
||||
large-cluster-best-practices
|
||||
configuring-autoscaling
|
||||
logging
|
||||
Community-supported Cluster Managers <community/index>
|
||||
```
|
||||
|
||||
:::{note}
|
||||
To learn the basics of Ray on Cloud VMs, we recommend taking a look at the {ref}`introductory guide <vm-cluster-quick-start>` first.
|
||||
:::
|
||||
|
||||
In these guides, we go into further depth on several topics related to deployments of Ray on cloud VMs or on-premises.
|
||||
* {ref}`launching-vm-clusters`
|
||||
* {ref}`vms-large-cluster`
|
||||
* {ref}`vms-autoscaling`
|
||||
* {ref}`ref-cluster-setup`
|
||||
* {ref}`cluster-FAQ`
|
||||
@@ -0,0 +1,142 @@
|
||||
.. _vms-large-cluster:
|
||||
|
||||
Best practices for deploying large clusters
|
||||
-------------------------------------------
|
||||
|
||||
This section aims to document best practices for deploying Ray clusters at
|
||||
large scale.
|
||||
|
||||
Networking configuration
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
End users should only need to directly interact with the head node of the
|
||||
cluster. In particular, there are 2 services which should be exposed to users:
|
||||
|
||||
1. The dashboard
|
||||
2. The Ray client server
|
||||
|
||||
.. note::
|
||||
|
||||
While users only need 2 ports to connect to a cluster, the nodes within a
|
||||
cluster require a much wider range of ports to communicate.
|
||||
|
||||
See :ref:`Ray port configuration <Ray-ports>` for a comprehensive list.
|
||||
|
||||
Applications (such as :ref:`Ray Serve <Rayserve>`) may also require
|
||||
additional ports to work properly.
|
||||
|
||||
System configuration
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
There are a few system level configurations that should be set when using Ray
|
||||
at a large scale.
|
||||
|
||||
* Make sure ``ulimit -n`` is set to at least 65535. Ray opens many direct
|
||||
connections between worker processes to avoid bottlenecks, so it can quickly
|
||||
use a large number of file descriptors.
|
||||
* Make sure ``/dev/shm`` is sufficiently large. Most ML/RL applications rely
|
||||
heavily on the plasma store. By default, Ray will try to use ``/dev/shm`` for
|
||||
the object store, but if it is not large enough (i.e. ``--object-store-memory``
|
||||
> size of ``/dev/shm``), Ray will write the plasma store to disk instead, which
|
||||
may cause significant performance problems.
|
||||
* Use NVMe SSDs (or other high performance storage) if possible. If
|
||||
:ref:`object spilling <object-spilling>` is enabled Ray will spill objects to
|
||||
disk if necessary. This is most commonly needed for data processing
|
||||
workloads.
|
||||
|
||||
.. _vms-large-cluster-configure-head-node:
|
||||
|
||||
Configuring the head node
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
In addition to the above changes, when deploying a large cluster, Ray's
|
||||
architecture means that the head node has extra stress due to
|
||||
additional system processes running on it like GCS.
|
||||
|
||||
* A good starting hardware specification for the head node is 8 CPUs and 32 GB memory.
|
||||
The actual hardware specification depends on the workload and the size of the cluster.
|
||||
Metrics that are useful for deciding the hardware specification are
|
||||
CPU usage, memory usage, and network bandwidth usage.
|
||||
* Make sure the head node has sufficient bandwidth. The most heavily stressed
|
||||
resource on the head node is outbound bandwidth. For large clusters (see the
|
||||
scalability envelope), we recommend using machines networking characteristics
|
||||
at least as good as an r5dn.16xlarge on AWS EC2.
|
||||
* Set ``resources: {"CPU": 0}`` on the head node.
|
||||
(For Ray clusters deployed using KubeRay,
|
||||
set ``rayStartParams: {"num-cpus": "0"}``.
|
||||
See the :ref:`configuration guide for KubeRay clusters <kuberay-num-cpus>`.)
|
||||
Due to the heavy networking load (and the GCS and dashboard processes), we
|
||||
recommend setting the quantity of logical CPU resources to 0 on the head node
|
||||
to avoid scheduling additional tasks on it.
|
||||
|
||||
For long-running clusters, head node memory usage can steadily increase over time.
|
||||
See :ref:`head-node-memory-management` for detailed information on causes and mitigation strategies.
|
||||
|
||||
Configuring the autoscaler
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
For large, long running clusters, there are a few parameters that can be tuned.
|
||||
|
||||
* Ensure your quotas for node types are set correctly.
|
||||
* For long running clusters, set the ``AUTOSCALER_MAX_NUM_FAILURES`` environment
|
||||
variable to a large number (or ``inf``) to avoid unexpected autoscaler
|
||||
crashes. The variable can be set by prepending \ ``export AUTOSCALER_MAX_NUM_FAILURES=inf;``
|
||||
to the head node's Ray start command.
|
||||
(Note: you may want a separate mechanism to detect if the autoscaler
|
||||
errors too often).
|
||||
* For large clusters, consider tuning ``upscaling_speed`` for faster
|
||||
autoscaling.
|
||||
|
||||
Picking nodes
|
||||
^^^^^^^^^^^^^
|
||||
|
||||
Here are some tips for how to set your ``available_node_types`` for a cluster,
|
||||
using AWS instance types as a concrete example.
|
||||
|
||||
General recommendations with AWS instance types:
|
||||
|
||||
**When to use GPUs**
|
||||
|
||||
* If you’re using some RL/ML framework
|
||||
* You’re doing something with tensorflow/pytorch/jax (some framework that can
|
||||
leverage GPUs well)
|
||||
|
||||
**What type of GPU?**
|
||||
|
||||
* The latest gen GPU is almost always the best bang for your buck (p3 > p2, g4
|
||||
> g3), for most well designed applications the performance outweighs the
|
||||
price. (The instance price may be higher, but you use the instance for less
|
||||
time.)
|
||||
* You may want to consider using older instances if you’re doing dev work and
|
||||
won’t actually fully utilize the GPUs though.
|
||||
* If you’re doing training (ML or RL), you should use a P instance. If you’re
|
||||
doing inference, you should use a G instance. The difference is
|
||||
processing:VRAM ratio (training requires more memory).
|
||||
|
||||
**What type of CPU?**
|
||||
|
||||
* Again stick to the latest generation, they’re typically cheaper and faster.
|
||||
* When in doubt use M instances, they have typically have the highest
|
||||
availability.
|
||||
* If you know your application is memory intensive (memory utilization is full,
|
||||
but cpu is not), go with an R instance
|
||||
* If you know your application is CPU intensive go with a C instance
|
||||
* If you have a big cluster, make the head node an instance with an n (r5dn or
|
||||
c5n)
|
||||
|
||||
**How many CPUs/GPUs?**
|
||||
|
||||
* Focus on your CPU:GPU ratio first and look at the utilization (Ray dashboard
|
||||
should help with this). If your CPU utilization is low add GPUs, or vice
|
||||
versa.
|
||||
* The exact ratio will be very dependent on your workload.
|
||||
* Once you find a good ratio, you should be able to scale up and keep the
|
||||
same ratio.
|
||||
* You can’t infinitely scale forever. Eventually, as you add more machines your
|
||||
performance improvements will become sub-linear/not worth it. There may not
|
||||
be a good one-size fits all strategy at this point.
|
||||
|
||||
.. note::
|
||||
|
||||
If you're using RLlib, check out :ref:`the RLlib scaling guide
|
||||
<rllib-scaling-guide>` for RLlib specific recommendations.
|
||||
@@ -0,0 +1,386 @@
|
||||
|
||||
# Launching Ray Clusters on AWS
|
||||
|
||||
This guide details the steps needed to start a Ray cluster on AWS.
|
||||
|
||||
To start an AWS Ray cluster, you should use the Ray cluster launcher with the AWS Python SDK.
|
||||
|
||||
## Install Ray cluster launcher
|
||||
|
||||
The Ray cluster launcher is part of the `ray` CLI. Use the CLI to start, stop and attach to a running ray cluster using commands such as `ray up`, `ray down` and `ray attach`. You can use pip to install the ray CLI with cluster launcher support. Follow [the Ray installation documentation](installation) for more detailed instructions.
|
||||
|
||||
```bash
|
||||
# install ray
|
||||
pip install -U ray[default]
|
||||
```
|
||||
|
||||
## Install and Configure AWS Python SDK (Boto3)
|
||||
|
||||
Next, install AWS SDK using `pip install -U boto3` and configure your AWS credentials following [the AWS guide](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html).
|
||||
|
||||
```bash
|
||||
# install AWS Python SDK (boto3)
|
||||
pip install -U boto3
|
||||
|
||||
# setup AWS credentials using environment variables
|
||||
export AWS_ACCESS_KEY_ID=foo
|
||||
export AWS_SECRET_ACCESS_KEY=bar
|
||||
export AWS_SESSION_TOKEN=baz
|
||||
|
||||
# alternatively, you can setup AWS credentials using ~/.aws/credentials file
|
||||
echo "[default]
|
||||
aws_access_key_id=foo
|
||||
aws_secret_access_key=bar
|
||||
aws_session_token=baz" >> ~/.aws/credentials
|
||||
```
|
||||
|
||||
## Start Ray with the Ray cluster launcher
|
||||
|
||||
Once Boto3 is configured to manage resources in your AWS account, you should be ready to launch your cluster using the cluster launcher. The provided [cluster config file](https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/aws/example-full.yaml) will create a small cluster with an m5.large head node (on-demand) configured to autoscale to up to two m5.large [spot-instance](https://aws.amazon.com/ec2/spot/) workers.
|
||||
|
||||
Test that it works by running the following commands from your local machine:
|
||||
|
||||
```bash
|
||||
# Download the example-full.yaml
|
||||
wget https://raw.githubusercontent.com/ray-project/ray/master/python/ray/autoscaler/aws/example-full.yaml
|
||||
|
||||
# Create or update the cluster. When the command finishes, it will print
|
||||
# out the command that can be used to SSH into the cluster head node.
|
||||
ray up example-full.yaml
|
||||
|
||||
# Get a remote shell on the head node.
|
||||
ray attach example-full.yaml
|
||||
|
||||
# Try running a Ray program.
|
||||
python -c 'import ray; ray.init()'
|
||||
exit
|
||||
|
||||
# Tear down the cluster.
|
||||
ray down example-full.yaml
|
||||
```
|
||||
|
||||
Congrats, you have started a Ray cluster on AWS!
|
||||
|
||||
|
||||
If you want to learn more about the Ray cluster launcher, see this blog post for a [step by step guide](https://medium.com/distributed-computing-with-ray/a-step-by-step-guide-to-scaling-your-first-python-application-in-the-cloud-8761fe331ef1).
|
||||
|
||||
|
||||
## AWS Configurations
|
||||
|
||||
(aws-cluster-efs)=
|
||||
|
||||
### Using Amazon EFS
|
||||
|
||||
To utilize Amazon EFS in the Ray cluster, you will need to install some additional utilities and mount the EFS in `setup_commands`. Note that these instructions only work if you are using the Ray cluster launcher on AWS.
|
||||
|
||||
```yaml
|
||||
# Note You need to replace the {{FileSystemId}} with your own EFS ID before using the config.
|
||||
# You may also need to modify the SecurityGroupIds for the head and worker nodes in the config file.
|
||||
|
||||
setup_commands:
|
||||
- sudo kill -9 `sudo lsof /var/lib/dpkg/lock-frontend | awk '{print $2}' | tail -n 1`;
|
||||
sudo pkill -9 apt-get;
|
||||
sudo pkill -9 dpkg;
|
||||
sudo dpkg --configure -a;
|
||||
sudo apt-get -y install binutils;
|
||||
cd $HOME;
|
||||
git clone https://github.com/aws/efs-utils;
|
||||
cd $HOME/efs-utils;
|
||||
./build-deb.sh;
|
||||
sudo apt-get -y install ./build/amazon-efs-utils*deb;
|
||||
cd $HOME;
|
||||
mkdir efs;
|
||||
sudo mount -t efs {{FileSystemId}}:/ efs;
|
||||
sudo chmod 777 efs;
|
||||
```
|
||||
|
||||
### Configuring IAM Role and EC2 Instance Profile
|
||||
|
||||
By default, Ray nodes in a Ray AWS cluster have full EC2 and S3 permissions (i.e. `arn:aws:iam::aws:policy/AmazonEC2FullAccess` and `arn:aws:iam::aws:policy/AmazonS3FullAccess`). This is a good default for trying out Ray clusters but you may want to change the permissions Ray nodes have for various reasons (e.g. to reduce the permissions for security reasons). You can do so by providing a custom `IamInstanceProfile` to the related `node_config`:
|
||||
|
||||
```yaml
|
||||
available_node_types:
|
||||
ray.worker.default:
|
||||
node_config:
|
||||
...
|
||||
IamInstanceProfile:
|
||||
Arn: arn:aws:iam::YOUR_AWS_ACCOUNT:YOUR_INSTANCE_PROFILE
|
||||
```
|
||||
|
||||
Please refer to this [discussion](https://github.com/ray-project/ray/issues/9327) for more details on configuring IAM role and EC2 instance profile.
|
||||
|
||||
(aws-cluster-s3)=
|
||||
|
||||
### Accessing S3
|
||||
|
||||
In various scenarios, worker nodes may need write access to an S3 bucket, e.g., Ray Tune has an option to write checkpoints to S3 instead of syncing them directly back to the driver.
|
||||
|
||||
If you see errors like “Unable to locate credentials”, make sure that the correct `IamInstanceProfile` is configured for worker nodes in your cluster config file. This may look like:
|
||||
|
||||
```yaml
|
||||
available_node_types:
|
||||
ray.worker.default:
|
||||
node_config:
|
||||
...
|
||||
IamInstanceProfile:
|
||||
Arn: arn:aws:iam::YOUR_AWS_ACCOUNT:YOUR_INSTANCE_PROFILE
|
||||
```
|
||||
|
||||
You can verify if the set up is correct by SSHing into a worker node and running
|
||||
|
||||
```bash
|
||||
aws configure list
|
||||
```
|
||||
|
||||
You should see something like
|
||||
|
||||
```bash
|
||||
Name Value Type Location
|
||||
---- ----- ---- --------
|
||||
profile <not set> None None
|
||||
access_key ****************XXXX iam-role
|
||||
secret_key ****************YYYY iam-role
|
||||
region <not set> None None
|
||||
```
|
||||
|
||||
Please refer to this [discussion](https://github.com/ray-project/ray/issues/9327) for more details on accessing S3.
|
||||
|
||||
## Monitor Ray using Amazon CloudWatch
|
||||
|
||||
```{eval-rst}
|
||||
Amazon CloudWatch is a monitoring and observability service that provides data and actionable insights to monitor your applications, respond to system-wide performance changes, and optimize resource utilization.
|
||||
CloudWatch integration with Ray requires an AMI (or Docker image) with the Unified CloudWatch Agent pre-installed.
|
||||
|
||||
AMIs with the Unified CloudWatch Agent pre-installed are provided by the Amazon Ray Team, and are currently available in the us-east-1, us-east-2, us-west-1, and us-west-2 regions.
|
||||
Please direct any questions, comments, or issues to the `Amazon Ray Team <https://github.com/amzn/amazon-ray/issues/new/choose>`_.
|
||||
|
||||
The table below lists AMIs with the Unified CloudWatch Agent pre-installed in each region, and you can also find AMIs at `DLAMI Release Notes <https://docs.aws.amazon.com/dlami/latest/devguide/appendix-ami-release-notes.html>`_. Each DLAMI (Deep Learning AMI) is pre-installed with the Unified CloudWatch Agent, and its corresponding release notes include AWS CLI commands to query the latest AMI ID.
|
||||
|
||||
.. list-table:: All available unified CloudWatch agent images
|
||||
|
||||
* - Base AMI
|
||||
- AMI ID
|
||||
- Region
|
||||
- Unified CloudWatch Agent Version
|
||||
* - AWS Deep Learning AMI (Ubuntu 24.04, 64-bit)
|
||||
- ami-087feac195f30e722
|
||||
- us-east-1
|
||||
- v1.300057.1b1167
|
||||
* - AWS Deep Learning AMI (Ubuntu 24.04, 64-bit)
|
||||
- ami-0ed6c422a7c93278a
|
||||
- us-east-2
|
||||
- v1.300057.1b1167
|
||||
* - AWS Deep Learning AMI (Ubuntu 24.04, 64-bit)
|
||||
- ami-0c5ddf2c101267018
|
||||
- us-west-1
|
||||
- v1.300057.1b1167
|
||||
* - AWS Deep Learning AMI (Ubuntu 24.04, 64-bit)
|
||||
- ami-0cfd95c6c87d00570
|
||||
- us-west-2
|
||||
- v1.300057.1b1167
|
||||
|
||||
.. note::
|
||||
|
||||
Using Amazon CloudWatch will incur charges, please refer to `CloudWatch pricing <https://aws.amazon.com/cloudwatch/pricing/>`_ for details.
|
||||
|
||||
Getting started
|
||||
---------------
|
||||
|
||||
1. Create a minimal cluster config YAML named ``cloudwatch-basic.yaml`` with the following contents:
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
provider:
|
||||
type: aws
|
||||
region: us-west-2
|
||||
availability_zone: us-west-2a
|
||||
# Start by defining a `cloudwatch` section to enable CloudWatch integration with your Ray cluster.
|
||||
cloudwatch:
|
||||
agent:
|
||||
# Path to Unified CloudWatch Agent config file
|
||||
config: "cloudwatch/example-cloudwatch-agent-config.json"
|
||||
dashboard:
|
||||
# CloudWatch Dashboard name
|
||||
name: "example-dashboard-name"
|
||||
# Path to the CloudWatch Dashboard config file
|
||||
config: "cloudwatch/example-cloudwatch-dashboard-config.json"
|
||||
|
||||
auth:
|
||||
ssh_user: ubuntu
|
||||
|
||||
available_node_types:
|
||||
ray.head.default:
|
||||
node_config:
|
||||
InstanceType: c5a.large
|
||||
ImageId: ami-0cfd95c6c87d00570 # Unified CloudWatch agent pre-installed AMI, us-west-2
|
||||
resources: {}
|
||||
ray.worker.default:
|
||||
node_config:
|
||||
InstanceType: c5a.large
|
||||
ImageId: ami-0cfd95c6c87d00570 # Unified CloudWatch agent pre-installed AMI, us-west-2
|
||||
IamInstanceProfile:
|
||||
Name: ray-autoscaler-cloudwatch-v1
|
||||
resources: {}
|
||||
min_workers: 0
|
||||
|
||||
2. Download CloudWatch Agent and Dashboard config.
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
First, create a ``cloudwatch`` directory in the same directory as ``cloudwatch-basic.yaml``.
|
||||
Then, download the example `CloudWatch Agent <https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/aws/cloudwatch/example-cloudwatch-agent-config.json>`_ and `CloudWatch Dashboard <https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/aws/cloudwatch/example-cloudwatch-dashboard-config.json>`_ config files to the ``cloudwatch`` directory.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ mkdir cloudwatch
|
||||
$ cd cloudwatch
|
||||
$ wget https://raw.githubusercontent.com/ray-project/ray/master/python/ray/autoscaler/aws/cloudwatch/example-cloudwatch-agent-config.json
|
||||
$ wget https://raw.githubusercontent.com/ray-project/ray/master/python/ray/autoscaler/aws/cloudwatch/example-cloudwatch-dashboard-config.json
|
||||
|
||||
3. Run ``ray up cloudwatch-basic.yaml`` to start your Ray Cluster.
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
This will launch your Ray cluster in ``us-west-2`` by default. When launching a cluster for a different region, you'll need to change your cluster config YAML file's ``region`` AND ``ImageId``.
|
||||
See the "Unified CloudWatch Agent Images" table above for available AMIs by region.
|
||||
|
||||
4. Check out your Ray cluster's logs, metrics, and dashboard in the `CloudWatch Console <https://console.aws.amazon.com/cloudwatch/>`_!
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
A tail can be acquired on all logs written to a CloudWatch log group by ensuring that you have the `AWS CLI V2+ installed <https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html>`_ and then running:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
aws logs tail $log_group_name --follow
|
||||
|
||||
Advanced Setup
|
||||
--------------
|
||||
|
||||
Refer to `example-cloudwatch.yaml <https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/aws/example-cloudwatch.yaml>`_ for a complete example.
|
||||
|
||||
1. Choose an AMI with the Unified CloudWatch Agent pre-installed.
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Ensure that you're launching your Ray EC2 cluster in the same region as the AMI,
|
||||
then specify the ``ImageId`` to use with your cluster's head and worker nodes in your cluster config YAML file.
|
||||
|
||||
The following CLI command returns the latest available Unified CloudWatch Agent Image for ``us-west-2``:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
aws ec2 describe-images --region us-west-2 --filters "Name=owner-id,Values=160082703681" "Name=name,Values=*cloudwatch*" --query 'Images[*].[ImageId,CreationDate]' --output text | sort -k2 -r | head -n1
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
available_node_types:
|
||||
ray.head.default:
|
||||
node_config:
|
||||
InstanceType: c5a.large
|
||||
ImageId: ami-0cfd95c6c87d00570
|
||||
ray.worker.default:
|
||||
node_config:
|
||||
InstanceType: c5a.large
|
||||
ImageId: ami-0cfd95c6c87d00570
|
||||
|
||||
To build your own AMI with the Unified CloudWatch Agent installed:
|
||||
|
||||
1. Follow the `CloudWatch Agent Installation <https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-on-EC2-Instance.html>`_ user guide to install the Unified CloudWatch Agent on an EC2 instance.
|
||||
2. Follow the `EC2 AMI Creation <https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html#creating-an-ami>`_ user guide to create an AMI from this EC2 instance.
|
||||
|
||||
2. Define your own CloudWatch Agent, Dashboard, and Alarm JSON config files.
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can start by using the example `CloudWatch Agent <https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/aws/cloudwatch/example-cloudwatch-agent-config.json>`_, `CloudWatch Dashboard <https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/aws/cloudwatch/example-cloudwatch-dashboard-config.json>`_ and `CloudWatch Alarm <https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/aws/cloudwatch/example-cloudwatch-alarm-config.json>`_ config files.
|
||||
|
||||
These example config files include the following features:
|
||||
|
||||
**Logs and Metrics**: Logs written to ``/tmp/ray/session_*/logs/**.out`` will be available in the ``{cluster_name}-ray_logs_out`` log group,
|
||||
and logs written to ``/tmp/ray/session_*/logs/**.err`` will be available in the ``{cluster_name}-ray_logs_err`` log group.
|
||||
Log streams are named after the EC2 instance ID that emitted their logs.
|
||||
Extended EC2 metrics including CPU/Disk/Memory usage and process statistics can be found in the ``{cluster_name}-ray-CWAgent`` metric namespace.
|
||||
|
||||
**Dashboard**: You will have a cluster-level dashboard showing total cluster CPUs and available object store memory.
|
||||
Process counts, disk usage, memory usage, and CPU utilization will be displayed as both cluster-level sums and single-node maximums/averages.
|
||||
|
||||
**Alarms**: Node-level alarms tracking prolonged high memory, disk, and CPU usage are configured. Alarm actions are NOT set,
|
||||
and must be manually provided in your alarm config file.
|
||||
|
||||
For more advanced options, see the `Agent <https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html>`_, `Dashboard <https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/CloudWatch-Dashboard-Body-Structure.html>`_ and `Alarm <https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_PutMetricAlarm.html>`_ config user guides.
|
||||
|
||||
CloudWatch Agent, Dashboard, and Alarm JSON config files support the following variables:
|
||||
|
||||
``{instance_id}``: Replaced with each EC2 instance ID in your Ray cluster.
|
||||
|
||||
``{region}``: Replaced with your Ray cluster's region.
|
||||
|
||||
``{cluster_name}``: Replaced with your Ray cluster name.
|
||||
|
||||
See CloudWatch Agent `Configuration File Details <https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html>`_ for additional variables supported natively by the Unified CloudWatch Agent.
|
||||
|
||||
.. note::
|
||||
Remember to replace the ``AlarmActions`` placeholder in your CloudWatch Alarm config file!
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
"AlarmActions":[
|
||||
"TODO: Add alarm actions! See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html"
|
||||
]
|
||||
|
||||
3. Reference your CloudWatch JSON config files in your cluster config YAML.
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Specify the file path to your CloudWatch JSON config files relative to the working directory that you will run ``ray up`` from:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
provider:
|
||||
cloudwatch:
|
||||
agent:
|
||||
config: "cloudwatch/example-cloudwatch-agent-config.json"
|
||||
|
||||
|
||||
4. Set your IAM Role and EC2 Instance Profile.
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
By default the ``ray-autoscaler-cloudwatch-v1`` IAM role and EC2 instance profile is created at Ray cluster launch time.
|
||||
This role contains all additional permissions required to integrate CloudWatch with Ray, namely the ``CloudWatchAgentAdminPolicy``, ``AmazonSSMManagedInstanceCore``, ``ssm:SendCommand``, ``ssm:ListCommandInvocations``, and ``iam:PassRole`` managed policies.
|
||||
|
||||
Ensure that all worker nodes are configured to use the ``ray-autoscaler-cloudwatch-v1`` EC2 instance profile in your cluster config YAML:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
ray.worker.default:
|
||||
node_config:
|
||||
InstanceType: c5a.large
|
||||
IamInstanceProfile:
|
||||
Name: ray-autoscaler-cloudwatch-v1
|
||||
|
||||
5. Export Ray system metrics to CloudWatch.
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
To export Ray's Prometheus system metrics to CloudWatch, first ensure that your cluster has the
|
||||
Ray Dashboard installed, then uncomment the ``head_setup_commands`` section in `example-cloudwatch.yaml file <https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/aws/example-cloudwatch.yaml>`_ file.
|
||||
You can find Ray Prometheus metrics in the ``{cluster_name}-ray-prometheus`` metric namespace.
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
head_setup_commands:
|
||||
# Make `ray_prometheus_waiter.sh` executable.
|
||||
- >-
|
||||
RAY_INSTALL_DIR=`pip show ray | grep -Po "(?<=Location:).*"`
|
||||
&& sudo chmod +x $RAY_INSTALL_DIR/ray/autoscaler/aws/cloudwatch/ray_prometheus_waiter.sh
|
||||
# Copy `prometheus.yml` to Unified CloudWatch Agent folder
|
||||
- >-
|
||||
RAY_INSTALL_DIR=`pip show ray | grep -Po "(?<=Location:).*"`
|
||||
&& sudo cp -f $RAY_INSTALL_DIR/ray/autoscaler/aws/cloudwatch/prometheus.yml /opt/aws/amazon-cloudwatch-agent/etc
|
||||
# First get current cluster name, then let the Unified CloudWatch Agent restart and use `AmazonCloudWatch-ray_agent_config_{cluster_name}` parameter at SSM Parameter Store.
|
||||
- >-
|
||||
nohup sudo sh -c "`pip show ray | grep -Po "(?<=Location:).*"`/ray/autoscaler/aws/cloudwatch/ray_prometheus_waiter.sh
|
||||
`cat ~/ray_bootstrap_config.yaml | jq '.cluster_name'`
|
||||
>> '/opt/aws/amazon-cloudwatch-agent/logs/ray_prometheus_waiter.out' 2>> '/opt/aws/amazon-cloudwatch-agent/logs/ray_prometheus_waiter.err'" &
|
||||
|
||||
6. Update CloudWatch Agent, Dashboard and Alarm config files.
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can apply changes to the CloudWatch Logs, Metrics, Dashboard, and Alarms for your cluster by simply modifying the CloudWatch config files referenced by your Ray cluster config YAML and re-running ``ray up example-cloudwatch.yaml``.
|
||||
The Unified CloudWatch Agent will be automatically restarted on all cluster nodes, and your config changes will be applied.
|
||||
```
|
||||
@@ -0,0 +1,132 @@
|
||||
|
||||
# Launching Ray Clusters on Azure
|
||||
|
||||
This guide details the steps needed to start a Ray cluster on Azure.
|
||||
|
||||
There are two ways to start an Azure Ray cluster.
|
||||
- Launch through Ray cluster launcher.
|
||||
- Deploy a cluster using Azure portal.
|
||||
|
||||
## Using Ray cluster launcher
|
||||
|
||||
|
||||
### Install Ray cluster launcher
|
||||
|
||||
The Ray cluster launcher is part of the `ray` CLI. Use the CLI to start, stop and attach to a running ray cluster using commands such as `ray up`, `ray down` and `ray attach`. You can use pip to install the ray CLI with cluster launcher support. Follow [the Ray installation documentation](installation) for more detailed instructions.
|
||||
|
||||
```bash
|
||||
# install ray
|
||||
pip install -U ray[default]
|
||||
```
|
||||
|
||||
### Install and Configure Azure CLI
|
||||
|
||||
Next, install the Azure CLI (`pip install -U azure-cli azure-identity`) and login using `az login`.
|
||||
|
||||
```bash
|
||||
# Install packages to use azure CLI.
|
||||
pip install azure-cli azure-identity
|
||||
|
||||
# Login to azure. This will redirect you to your web browser.
|
||||
az login
|
||||
```
|
||||
|
||||
### Install Azure SDK libraries
|
||||
|
||||
Now, install the Azure SDK libraries that enable the Ray cluster launcher to build Azure infrastructure.
|
||||
|
||||
```bash
|
||||
# Install azure SDK libraries.
|
||||
pip install azure-core azure-mgmt-network azure-mgmt-common azure-mgmt-resource azure-mgmt-compute msrestazure
|
||||
```
|
||||
|
||||
### Start Ray with the Ray cluster launcher
|
||||
|
||||
The provided [cluster config file](https://github.com/ray-project/ray/blob/eacc763c84d47c9c5b86b26a32fd62c685be84e6/python/ray/autoscaler/azure/example-full.yaml) will create a small cluster with a Standard DS2v3 on-demand head node that is configured to autoscale to up to two Standard DS2v3 [spot-instance](https://docs.microsoft.com/en-us/azure/virtual-machines/windows/spot-vms) worker nodes.
|
||||
|
||||
Note that you'll need to fill in your Azure [resource_group](https://github.com/ray-project/ray/blob/eacc763c84d47c9c5b86b26a32fd62c685be84e6/python/ray/autoscaler/azure/example-full.yaml#L42) and [location](https://github.com/ray-project/ray/blob/eacc763c84d47c9c5b86b26a32fd62c685be84e6/python/ray/autoscaler/azure/example-full.yaml#L41) in those templates. You also need set the subscription to use. You can do this from the command line with `az account set -s <subscription_id>` or by filling in the [subscription_id](https://github.com/ray-project/ray/blob/eacc763c84d47c9c5b86b26a32fd62c685be84e6/python/ray/autoscaler/azure/example-full.yaml#L44) in the cluster config file.
|
||||
|
||||
#### Download and configure the example configuration
|
||||
|
||||
Download the reference example locally:
|
||||
|
||||
```bash
|
||||
# Download the example-full.yaml
|
||||
wget https://raw.githubusercontent.com/ray-project/ray/master/python/ray/autoscaler/azure/example-full.yaml
|
||||
```
|
||||
|
||||
|
||||
|
||||
##### Automatic SSH Key Generation
|
||||
|
||||
To connect to the provisioned head node VM, Ray has automatic SSH Key Generation if none are specified in the config. This is the simplest approach and requires no manual key management.
|
||||
|
||||
The default configuration in `example-full.yaml` uses automatic key generation:
|
||||
|
||||
```yaml
|
||||
auth:
|
||||
ssh_user: ubuntu
|
||||
# SSH keys are auto-generated if not specified
|
||||
# Uncomment and specify custom paths if you want to use existing keys:
|
||||
# ssh_private_key: /path/to/your/key.pem
|
||||
# ssh_public_key: /path/to/your/key.pub
|
||||
```
|
||||
|
||||
##### (Optional) Manual SSH Key Configuration
|
||||
|
||||
If you prefer to use your own existing SSH keys, uncomment and specify both of the key paths in the `auth` section.
|
||||
|
||||
For example, to use an existing `ed25519` key pair:
|
||||
|
||||
```yaml
|
||||
auth:
|
||||
ssh_user: ubuntu
|
||||
ssh_private_key: ~/.ssh/id_ed25519
|
||||
ssh_public_key: ~/.ssh/id_ed25519.pub
|
||||
```
|
||||
|
||||
Or for RSA keys:
|
||||
|
||||
```yaml
|
||||
auth:
|
||||
ssh_user: ubuntu
|
||||
ssh_private_key: ~/.ssh/id_rsa
|
||||
ssh_public_key: ~/.ssh/id_rsa.pub
|
||||
```
|
||||
|
||||
Both methods inject the public key directly into the VM's `~/.ssh/authorized_keys` via Azure ARM templates.
|
||||
|
||||
#### Launch the Ray cluster on Azure
|
||||
|
||||
```bash
|
||||
# Create or update the cluster. When the command finishes, it will print
|
||||
# out the command that can be used to SSH into the cluster head node.
|
||||
ray up example-full.yaml
|
||||
|
||||
# Get a remote screen on the head node.
|
||||
ray attach example-full.yaml
|
||||
# Try running a Ray program.
|
||||
|
||||
# Tear down the cluster.
|
||||
ray down example-full.yaml
|
||||
```
|
||||
|
||||
Congratulations, you have started a Ray cluster on Azure!
|
||||
|
||||
## Using Azure portal
|
||||
|
||||
Alternatively, you can deploy a cluster using Azure portal directly. Please note that autoscaling is done using Azure VM Scale Sets and not through the Ray autoscaler. This will deploy [Azure Data Science VMs (DSVM)](https://azure.microsoft.com/en-us/services/virtual-machines/data-science-virtual-machines/) for both the head node and the auto-scalable cluster managed by [Azure Virtual Machine Scale Sets](https://azure.microsoft.com/en-us/services/virtual-machine-scale-sets/). The head node conveniently exposes both SSH as well as JupyterLab.
|
||||
|
||||
|
||||
|
||||
Once the template is successfully deployed the deployment Outputs page provides the ssh command to connect and the link to the JupyterHub on the head node (username/password as specified on the template input). Use the following code in a Jupyter notebook (using the conda environment specified in the template input, py38_tensorflow by default) to connect to the Ray cluster.
|
||||
|
||||
```python
|
||||
import ray; ray.init()
|
||||
```
|
||||
|
||||
Under the hood, the [azure-init.sh](https://github.com/ray-project/ray/blob/master/doc/azure/azure-init.sh) script is executed and performs the following actions:
|
||||
|
||||
1. Activates one of the conda environments available on DSVM
|
||||
2. Installs Ray and any other user-specified dependencies
|
||||
3. Sets up a systemd task (``/lib/systemd/system/ray.service``) to start Ray in head or worker mode
|
||||
@@ -0,0 +1,70 @@
|
||||
# Launching Ray Clusters on GCP
|
||||
|
||||
This guide details the steps needed to start a Ray cluster in GCP.
|
||||
|
||||
To start a GCP Ray cluster, you will use the Ray cluster launcher with the Google API client.
|
||||
|
||||
## Install Ray cluster launcher
|
||||
|
||||
The Ray cluster launcher is part of the `ray` CLI. Use the CLI to start, stop and attach to a running ray cluster using commands such as `ray up`, `ray down` and `ray attach`. You can use pip to install the ray CLI with cluster launcher support. Follow [the Ray installation documentation](installation) for more detailed instructions.
|
||||
|
||||
```bash
|
||||
# install ray
|
||||
pip install -U ray[default]
|
||||
```
|
||||
|
||||
## Install and Configure Google API Client
|
||||
|
||||
If you have never created a Google APIs Console project, read google Cloud's [Managing Projects page](https://cloud.google.com/resource-manager/docs/creating-managing-projects?visit_id=637952351450670909-433962807&rd=1) and create a project in the [Google API Console](https://console.developers.google.com/). Next, install the Google API Client using `pip install -U google-api-python-client`.
|
||||
|
||||
```bash
|
||||
# Install the Google API Client.
|
||||
pip install google-api-python-client
|
||||
```
|
||||
|
||||
## Start Ray with the Ray cluster launcher
|
||||
|
||||
Once the Google API client is configured to manage resources on your GCP account, you should be ready to launch your cluster. The provided [cluster config file](https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/gcp/example-full.yaml) will create a small cluster with an on-demand n1-standard-2 head node and is configured to autoscale to up to two n1-standard-2 [preemptible workers](https://cloud.google.com/preemptible-vms/). Note that you'll need to fill in your GCP [project_id](https://github.com/ray-project/ray/blob/eacc763c84d47c9c5b86b26a32fd62c685be84e6/python/ray/autoscaler/gcp/example-full.yaml#L42) in those templates.
|
||||
|
||||
Test that it works by running the following commands from your local machine:
|
||||
|
||||
```bash
|
||||
# Download the example-full.yaml
|
||||
wget https://raw.githubusercontent.com/ray-project/ray/master/python/ray/autoscaler/gcp/example-full.yaml
|
||||
|
||||
# Edit the example-full.yaml to update project_id.
|
||||
# vi example-full.yaml
|
||||
|
||||
# Create or update the cluster. When the command finishes, it will print
|
||||
# out the command that can be used to SSH into the cluster head node.
|
||||
ray up example-full.yaml
|
||||
|
||||
# Get a remote screen on the head node.
|
||||
ray attach example-full.yaml
|
||||
|
||||
# Try running a Ray program.
|
||||
python -c 'import ray; ray.init()'
|
||||
exit
|
||||
|
||||
# Tear down the cluster.
|
||||
ray down example-full.yaml
|
||||
```
|
||||
|
||||
Congrats, you have started a Ray cluster on GCP!
|
||||
|
||||
## GCP Configurations
|
||||
|
||||
### Running workers with Service Accounts
|
||||
|
||||
By default, only the head node runs with a Service Account (`ray-autoscaler-sa-v1@<project-id>.iam.gserviceaccount.com`). To enable workers to run with this same Service Account (to access Google Cloud Storage, or GCR), add the following configuration to the worker_node configuration:
|
||||
|
||||
```yaml
|
||||
available_node_types:
|
||||
ray.worker.default:
|
||||
node_config:
|
||||
...
|
||||
serviceAccounts:
|
||||
- email: ray-autoscaler-sa-v1@<YOUR_PROJECT_ID>.iam.gserviceaccount.com
|
||||
scopes:
|
||||
- https://www.googleapis.com/auth/cloud-platform
|
||||
```
|
||||
@@ -0,0 +1,18 @@
|
||||
.. _launching-vm-clusters:
|
||||
|
||||
Launching Ray Clusters on AWS, GCP, Azure, vSphere, On-Prem
|
||||
===========================================================
|
||||
|
||||
In this section, you can find guides for launching Ray clusters in various clouds or on-premises.
|
||||
|
||||
Table of Contents
|
||||
-----------------
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
aws.md
|
||||
gcp.md
|
||||
azure.md
|
||||
vsphere.md
|
||||
on-premises.md
|
||||
@@ -0,0 +1,117 @@
|
||||
(on-prem)=
|
||||
|
||||
# Launching an On-Premise Cluster
|
||||
|
||||
This document describes how to set up an on-premise Ray cluster, i.e., to run Ray on bare metal machines, or in a private cloud. We provide two ways to start an on-premise cluster.
|
||||
|
||||
* You can [manually set up](manual-setup-cluster) the Ray cluster by installing the Ray package and starting the Ray processes on each node.
|
||||
* Alternatively, if you know all the nodes in advance and have SSH access to them, you should start the Ray cluster using the [cluster-launcher](manual-cluster-launcher).
|
||||
|
||||
(manual-setup-cluster)=
|
||||
|
||||
## Manually Set up a Ray Cluster
|
||||
This section assumes that you have a list of machines and that the nodes in the cluster share the same network. It also assumes that Ray is installed on each machine. You can use pip to install the ray command line tool with cluster launcher support. Follow the [Ray installation instructions](installation) for more details.
|
||||
|
||||
```bash
|
||||
# install ray
|
||||
pip install -U "ray[default]"
|
||||
```
|
||||
|
||||
### Start the Head Node
|
||||
Choose any node to be the head node and run the following. If the `--port` argument is omitted, Ray will first choose port 6379, and then fall back to a random port if in 6379 is in use.
|
||||
|
||||
```bash
|
||||
ray start --head --port=6379
|
||||
```
|
||||
|
||||
The command will print out the Ray cluster address, which can be passed to `ray start` on other machines to start the worker nodes (see below). If you receive a ConnectionError, check your firewall settings and network configuration.
|
||||
|
||||
### Start Worker Nodes
|
||||
Then on each of the other nodes, run the following command to connect to the head node you just created.
|
||||
|
||||
```bash
|
||||
ray start --address=<head-node-address:port>
|
||||
```
|
||||
Make sure to replace `head-node-address:port` with the value printed by the command on the head node (it should look something like 123.45.67.89:6379).
|
||||
|
||||
Note that if your compute nodes are on their own subnetwork with Network Address Translation, the address printed by the head node will not work if connecting from a machine outside that subnetwork. You will need to use a head node address reachable from the remote machine. If the head node has a domain address like compute04.berkeley.edu, you can simply use that in place of an IP address and rely on DNS.
|
||||
|
||||
Ray auto-detects the resources (e.g., CPU) available on each node, but you can also manually override this by passing custom resources to the `ray start` command. For example, if you wish to specify that a machine has 10 CPUs and 1 GPU available for use by Ray, you can do this with the flags `--num-cpus=10` and `--num-gpus=1`. See the [Configuration page](configuring-ray) for more information.
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
If you see `Unable to connect to GCS at ...`, this means the head node is inaccessible at the given `--address`. Some possible causes include:
|
||||
|
||||
- the head node is not actually running;
|
||||
- a different version of Ray is running at the specified address;
|
||||
- the specified address is wrong;
|
||||
- or there are firewall settings preventing access.
|
||||
|
||||
If the connection fails, to check whether each port can be reached from a node, you can use a tool such as nmap or nc.
|
||||
|
||||
```bash
|
||||
$ nmap -sV --reason -p $PORT $HEAD_ADDRESS
|
||||
Nmap scan report for compute04.berkeley.edu (123.456.78.910)
|
||||
Host is up, received echo-reply ttl 60 (0.00087s latency).
|
||||
rDNS record for 123.456.78.910: compute04.berkeley.edu
|
||||
PORT STATE SERVICE REASON VERSION
|
||||
6379/tcp open redis? syn-ack
|
||||
Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
|
||||
$ nc -vv -z $HEAD_ADDRESS $PORT
|
||||
Connection to compute04.berkeley.edu 6379 port [tcp/*] succeeded!
|
||||
```
|
||||
|
||||
If the node cannot access that port at that IP address, you might see
|
||||
|
||||
```bash
|
||||
$ nmap -sV --reason -p $PORT $HEAD_ADDRESS
|
||||
Nmap scan report for compute04.berkeley.edu (123.456.78.910)
|
||||
Host is up (0.0011s latency).
|
||||
rDNS record for 123.456.78.910: compute04.berkeley.edu
|
||||
PORT STATE SERVICE REASON VERSION
|
||||
6379/tcp closed redis reset ttl 60
|
||||
Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
|
||||
$ nc -vv -z $HEAD_ADDRESS $PORT
|
||||
nc: connect to compute04.berkeley.edu port 6379 (tcp) failed: Connection refused
|
||||
```
|
||||
(manual-cluster-launcher)=
|
||||
|
||||
## Using Ray cluster launcher
|
||||
|
||||
The Ray cluster launcher is part of the `ray` command line tool. It allows you to start, stop and attach to a running ray cluster using commands such as `ray up`, `ray down` and `ray attach`. You can use pip to install it, or follow [install ray](installation) for more detailed instructions.
|
||||
|
||||
```bash
|
||||
# install ray
|
||||
pip install "ray[default]"
|
||||
```
|
||||
|
||||
### Start Ray with the Ray cluster launcher
|
||||
|
||||
The provided [example-full.yaml](https://github.com/ray-project/ray/blob/eacc763c84d47c9c5b86b26a32fd62c685be84e6/python/ray/autoscaler/local/example-full.yaml) cluster config file will create a Ray cluster given a list of nodes.
|
||||
|
||||
Note that you'll need to fill in your [head_ip](https://github.com/ray-project/ray/blob/eacc763c84d47c9c5b86b26a32fd62c685be84e6/python/ray/autoscaler/local/example-full.yaml#L20), a list of [worker_ips](https://github.com/ray-project/ray/blob/eacc763c84d47c9c5b86b26a32fd62c685be84e6/python/ray/autoscaler/local/example-full.yaml#L26), and the [ssh_user](https://github.com/ray-project/ray/blob/eacc763c84d47c9c5b86b26a32fd62c685be84e6/python/ray/autoscaler/local/example-full.yaml#L34) field in those templates
|
||||
|
||||
|
||||
|
||||
Test that it works by running the following commands from your local machine:
|
||||
|
||||
```bash
|
||||
# Download the example-full.yaml
|
||||
wget https://raw.githubusercontent.com/ray-project/ray/master/python/ray/autoscaler/local/example-full.yaml
|
||||
|
||||
# Update the example-full.yaml to update head_ip, worker_ips, and ssh_user.
|
||||
# vi example-full.yaml
|
||||
|
||||
# Create or update the cluster. When the command finishes, it will print
|
||||
# out the command that can be used to SSH into the cluster head node.
|
||||
ray up example-full.yaml
|
||||
|
||||
# Get a remote screen on the head node.
|
||||
ray attach example-full.yaml
|
||||
# Try running a Ray program.
|
||||
|
||||
# Tear down the cluster.
|
||||
ray down example-full.yaml
|
||||
```
|
||||
|
||||
Congrats, you have started a local Ray cluster!
|
||||
@@ -0,0 +1,64 @@
|
||||
# Launching Ray Clusters on vSphere
|
||||
|
||||
This guide details the steps needed to launch a Ray cluster in a vSphere environment.
|
||||
|
||||
To start a vSphere Ray cluster, you will use the Ray cluster launcher along with supervisor service (control plane) deployed on vSphere.
|
||||
|
||||
## Prepare the vSphere environment
|
||||
|
||||
If you don't already have a vSphere deployment, you can learn more about it by reading the [vSphere documentation](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere-supervisor/7-0/vsphere-with-tanzu-configuration-and-management-7-0/configuring-and-managing-a-supervisor-cluster/deploy-a-supervisor-with-nsx-networking.html). The vSphere Ray cluster launcher requires vSphere version 9.0 or later, along with the following prerequisites for creating Ray clusters.
|
||||
|
||||
* [A vSphere cluster with Workload Control Plane (WCP) enabled ](https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere-supervisor/7-0/vsphere-with-tanzu-configuration-and-management-7-0/configuring-and-managing-a-supervisor-cluster/deploy-a-supervisor-with-nsx-networking.html)
|
||||
|
||||
## Installing supervisor service for Ray on vSphere
|
||||
|
||||
Please refer [build and installation guide](https://github.com/vmware/ray-on-vcf) to install Ray control plane as a superviosr servise on vSphere. The vSphere Ray cluster launcher requires the vSphere environment to have a cotrol plane installed a s a supervisor service for deploying a Ray cluster. This service installs all the k8s CRDs used to rapidly create head and worker nodes. The details of the Ray cluster provisioning process using supervisor service can be found in this [Ray on vSphere architecture document](https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/_private/vsphere/ARCHITECTURE.md).
|
||||
|
||||
|
||||
## Install Ray cluster launcher
|
||||
|
||||
The Ray cluster launcher is part of the `ray` CLI. Use the CLI to start, stop and attach to a running ray cluster using commands such as `ray up`, `ray down` and `ray attach`. You can use pip to install the ray CLI with cluster launcher support. Follow [the Ray installation documentation](installation) for more detailed instructions.
|
||||
|
||||
```bash
|
||||
# install ray
|
||||
pip install -U ray[default]
|
||||
```
|
||||
|
||||
## Start Ray with the Ray cluster launcher
|
||||
|
||||
Once the Ray supervisor service is active, you should be ready to launch your cluster using the cluster launcher. The provided [cluster config file](https://raw.githubusercontent.com/ray-project/ray/master/python/ray/autoscaler/vsphere/example-full.yaml) will create a small cluster with a head node configured to autoscale to up to two workers.
|
||||
|
||||
Note that you need to configure your vSphere credentials and vCenter server address either via setting environment variables or adding them to the Ray cluster configuration YAML file.
|
||||
|
||||
Test that it works by running the following commands from your local machine:
|
||||
|
||||
```bash
|
||||
# Download the example-full.yaml
|
||||
wget https://raw.githubusercontent.com/ray-project/ray/master/python/ray/autoscaler/vsphere/example-full.yaml
|
||||
|
||||
# Create or update the cluster. When the command finishes, it will print
|
||||
# out the command that can be used to SSH into the cluster head node.
|
||||
ray up example-full.yaml
|
||||
|
||||
# Get a remote screen on the head node.
|
||||
ray attach example-full.yaml
|
||||
|
||||
# Try running a Ray program.
|
||||
python -c 'import ray; ray.init()'
|
||||
exit
|
||||
|
||||
# Tear down the cluster.
|
||||
ray down example-full.yaml
|
||||
```
|
||||
|
||||
Congrats, you have started a Ray cluster on vSphere!
|
||||
|
||||
## Configure vSAN File Service as persistent storage for Ray AI Libraries
|
||||
|
||||
Starting in Ray 2.7, Ray AI Libraries (Train and Tune) will require users to provide a cloud storage or NFS path when running distributed training or tuning jobs. In a vSphere environment with a vSAN datastore, you can utilize the vSAN File Service feature to employ vSAN as a shared persistent storage. You can refer to [this vSAN File Service document](https://techdocs.broadcom.com/us/en/vmware-cis/vsan/vsan/8-0/vsan-administration.html) to create and configure NFS file shares supported by vSAN. The general steps are as follows:
|
||||
|
||||
1. Enable vSAN File Service and configure it with domain information and IP address pools.
|
||||
2. Create a vSAN file share with NFS as the protocol.
|
||||
3. View the file share information to get NFS export path.
|
||||
|
||||
Once a file share is created, you can mount it into the head and worker node and use the mount path as the `storage_path` for the `RunConfig` parameter in Ray Train and Tune.
|
||||
@@ -0,0 +1,28 @@
|
||||
(vm-logging)=
|
||||
# Log Persistence
|
||||
|
||||
Logs are useful for troubleshooting Ray applications and Clusters. For example, you may want to access system logs if a node terminates unexpectedly.
|
||||
|
||||
Ray does not provide a native storage solution for log data. Users need to manage the lifecycle of the logs by themselves. The following sections provide instructions on how to collect logs from Ray Clusters running on VMs.
|
||||
|
||||
## Ray log directory
|
||||
By default, Ray writes logs to files in the directory `/tmp/ray/session_*/logs` on each Ray node's file system, including application logs and system logs. Learn more about the {ref}`log directory and log files <logging-directory>` and the {ref}`log rotation configuration <log-rotation>` before you start to collect logs.
|
||||
|
||||
|
||||
## Log processing tools
|
||||
|
||||
A number of open source log processing tools are available, such as [Vector][Vector], [FluentBit][FluentBit], [Fluentd][Fluentd], [Filebeat][Filebeat], and [Promtail][Promtail].
|
||||
|
||||
[Vector]: https://vector.dev/
|
||||
[FluentBit]: https://docs.fluentbit.io/manual
|
||||
[Filebeat]: https://www.elastic.co/guide/en/beats/filebeat/7.17/index.html
|
||||
[Fluentd]: https://docs.fluentd.org/
|
||||
[Promtail]: https://grafana.com/docs/loki/latest/clients/promtail/
|
||||
|
||||
## Log collection
|
||||
|
||||
After choosing a log processing tool based on your needs, you may need to perform the following steps:
|
||||
|
||||
1. Ingest log files on each node of your Ray Cluster as sources.
|
||||
2. Parse and transform the logs. You may want to use {ref}`Ray's structured logging <structured-logging>` to simplify this step.
|
||||
3. Ship the transformed logs to log storage or management systems.
|
||||
Reference in New Issue
Block a user