wehub-resource-sync
593b94c120
pytest / Unit Tests (push) Has been cancelled
pytest / Integration (integration_tests_a) (push) Has been cancelled
pytest / Integration (integration_tests_b) (push) Has been cancelled
pytest / Integration (integration_tests_c) (push) Has been cancelled
pytest / Integration (integration_tests_d) (push) Has been cancelled
pytest / Integration (integration_tests_e) (push) Has been cancelled
pytest / Integration (integration_tests_f) (push) Has been cancelled
pytest / Integration (integration_tests_g) (push) Has been cancelled
pytest / Integration (integration_tests_h) (push) Has been cancelled
pytest / Integration (integration_tests_i) (push) Has been cancelled
pytest / Integration (integration_tests_j) (push) Has been cancelled
pytest / Distributed (distributed_a) (push) Has been cancelled
pytest / Distributed (distributed_b) (push) Has been cancelled
pytest / Distributed (distributed_c) (push) Has been cancelled
pytest / Distributed (distributed_d) (push) Has been cancelled
pytest / Distributed (distributed_e) (push) Has been cancelled
pytest / Distributed (distributed_f) (push) Has been cancelled
pytest / Minimal Install (push) Has been cancelled
pytest / Event File (push) Has been cancelled
pytest (slow) / py-slow (push) Has been cancelled
Publish JSON Schema / publish-schema (push) Has been cancelled
Running on Kubernetes
Connect to k8s cluster with a Ray operator
You should now be pointing to your cluster with kubectl. Check the nodes to make sure you're connected correctly:
kubectl get nodes
We recommend using the Kuberay implementation of the Ray Operator to launch Ray clusters.
Configure the Ray cluster
First choose your preferred cluster template from clusters, for example:
export CLUSTER_NAME=ludwig-ray-cpu-cluster
Start the cluster
./utils/ray_up.sh $CLUSTER_NAME
Submit a script for execution
./utils/submit.sh $CLUSTER_NAME scripts/train.py
SSH into the head node
./utils/attach.sh $CLUSTER_NAME
Run the Ray Dashboard
./utils/dashboard.sh $CLUSTER_NAME
Navigate to http://localhost:8267
(For Ludwig Developers) Sync local Ludwig repo
./utils/rsync_up.sh $CLUSTER_NAME ~/repos/ludwig
Shutdown the cluster
./utils/ray_down.sh $CLUSTER_NAME
Connecting to remote filesystems (S3, GCS, etc.)
Build a custom Docker image deriving from ludwig-ray or ludwig-ray-gpu containing the library needed for your
data:
s3fsadlfsgcsfs
Set environment variables into the cluster YAML definition with your credentials. For example, you can connect to S3 using the environment variables described in the boto3 documentation.
You could also include the credentials directly into the Docker image if they don't need to be configured at runtime.