chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,296 @@
|
||||
.. _dask-on-ray:
|
||||
|
||||
Using Dask on Ray
|
||||
=================
|
||||
|
||||
|
||||
`Dask <https://dask.org/>`__ is a Python parallel computing library geared towards scaling analytics and
|
||||
scientific computing workloads. It provides `big data collections
|
||||
<https://docs.dask.org/en/latest/user-interfaces.html>`__ that mimic the APIs of
|
||||
the familiar `NumPy <https://numpy.org/>`__ and `Pandas <https://pandas.pydata.org/>`__ libraries,
|
||||
allowing those abstractions to represent
|
||||
larger-than-memory data and/or allowing operations on that data to be run on a multi-machine cluster,
|
||||
while also providing automatic data parallelism, smart scheduling,
|
||||
and optimized operations. Operations on these collections create a task graph, which is
|
||||
executed by a scheduler.
|
||||
|
||||
Ray provides a scheduler for Dask (`dask_on_ray`) which allows you to build data
|
||||
analyses using Dask's collections and execute
|
||||
the underlying tasks on a Ray cluster.
|
||||
|
||||
`dask_on_ray` uses Dask's scheduler API, which allows you to
|
||||
specify any callable as the scheduler that you would like Dask to use to execute your
|
||||
workload. Using the Dask-on-Ray scheduler, the entire Dask ecosystem can be executed on top of Ray.
|
||||
|
||||
.. note::
|
||||
We always ensure that the latest Dask versions are compatible with Ray nightly.
|
||||
The table below shows the latest Dask versions that are tested with Ray versions.
|
||||
|
||||
.. list-table:: Latest Dask versions for each Ray version.
|
||||
:header-rows: 1
|
||||
|
||||
* - Ray Version
|
||||
- Dask Version
|
||||
* - ``2.48.0`` or above
|
||||
- | ``2023.6.1 (Python version < 3.12)``
|
||||
| ``2025.5.0 (Python version >= 3.12)``
|
||||
* - ``2.40.0`` to ``2.47.1``
|
||||
- | ``2022.10.2 (Python version < 3.12)``
|
||||
| ``2024.6.0 (Python version >= 3.12)``
|
||||
* - ``2.34.0`` to ``2.39.0``
|
||||
- | ``2022.10.1 (Python version < 3.12)``
|
||||
| ``2024.6.0 (Python version >= 3.12)``
|
||||
* - ``2.8.0`` to ``2.33.x``
|
||||
- ``2022.10.1``
|
||||
* - ``2.5.0`` to ``2.7.x``
|
||||
- | ``2022.2.0 (Python version < 3.8)``
|
||||
| ``2022.10.1 (Python version >= 3.8)``
|
||||
* - ``2.4.0``
|
||||
- ``2022.10.1``
|
||||
* - ``2.3.0``
|
||||
- ``2022.10.1``
|
||||
* - ``2.2.0``
|
||||
- ``2022.10.1``
|
||||
* - ``2.1.0``
|
||||
- ``2022.2.0``
|
||||
* - ``2.0.0``
|
||||
- ``2022.2.0``
|
||||
* - ``1.13.0``
|
||||
- ``2022.2.0``
|
||||
* - ``1.12.0``
|
||||
- ``2022.2.0``
|
||||
* - ``1.11.0``
|
||||
- ``2022.1.0``
|
||||
* - ``1.10.0``
|
||||
- ``2021.12.0``
|
||||
* - ``1.9.2``
|
||||
- ``2021.11.0``
|
||||
* - ``1.9.1``
|
||||
- ``2021.11.0``
|
||||
* - ``1.9.0``
|
||||
- ``2021.11.0``
|
||||
* - ``1.8.0``
|
||||
- ``2021.9.1``
|
||||
* - ``1.7.0``
|
||||
- ``2021.9.1``
|
||||
* - ``1.6.0``
|
||||
- ``2021.8.1``
|
||||
* - ``1.5.0``
|
||||
- ``2021.7.0``
|
||||
* - ``1.4.1``
|
||||
- ``2021.6.1``
|
||||
* - ``1.4.0``
|
||||
- ``2021.5.0``
|
||||
|
||||
Scheduler
|
||||
---------
|
||||
|
||||
.. _dask-on-ray-scheduler:
|
||||
|
||||
The Dask-on-Ray scheduler can execute any valid Dask graph, and can be used with
|
||||
any Dask `.compute() <https://docs.dask.org/en/latest/api.html#dask.compute>`__
|
||||
call.
|
||||
Here's an example:
|
||||
|
||||
.. literalinclude:: doc_code/dask_on_ray_scheduler_example.py
|
||||
:language: python
|
||||
|
||||
.. note::
|
||||
For execution on a Ray cluster, you should *not* use the
|
||||
`Dask.distributed <https://distributed.dask.org/en/latest/quickstart.html>`__
|
||||
client; simply use plain Dask and its collections, and pass ``ray_dask_get``
|
||||
to ``.compute()`` calls, set the scheduler in one of the other ways detailed `here <https://docs.dask.org/en/latest/scheduling.html#configuration>`__, or use our ``enable_dask_on_ray`` configuration helper. Follow the instructions for
|
||||
:ref:`using Ray on a cluster <cluster-index>` to modify the
|
||||
``ray.init()`` call.
|
||||
|
||||
Why use Dask on Ray?
|
||||
|
||||
1. To take advantage of Ray-specific features such as the
|
||||
:ref:`launching cloud clusters <cluster-index>` and
|
||||
:ref:`shared-memory store <memory>`.
|
||||
2. If you'd like to use Dask and Ray libraries in the same application without having two different clusters.
|
||||
3. If you'd like to create data analyses using the familiar NumPy and Pandas APIs provided by Dask and execute them on a fast, fault-tolerant distributed task execution system geared towards production, like Ray.
|
||||
|
||||
Dask-on-Ray is an ongoing project and is not expected to achieve the same performance as using Ray directly. All `Dask abstractions <https://docs.dask.org/en/latest/user-interfaces.html>`__ should run seamlessly on top of Ray using this scheduler, so if you find that one of these abstractions doesn't run on Ray, please `open an issue <https://github.com/ray-project/ray/issues/new/choose>`__.
|
||||
|
||||
Best Practice for Large Scale workloads
|
||||
---------------------------------------
|
||||
For Ray 1.3, the default scheduling policy is to pack tasks to the same node as much as possible.
|
||||
It is more desirable to spread tasks if you run a large scale / memory intensive Dask on Ray workloads.
|
||||
|
||||
In this case, there are two recommended setups.
|
||||
- Reducing the config flag `scheduler_spread_threshold` to tell the scheduler to prefer spreading tasks across the cluster instead of packing.
|
||||
- Setting the head node's `num-cpus` to 0 so that tasks are not scheduled on a head node.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# Head node. Set `num_cpus=0` to avoid tasks being scheduled on a head node.
|
||||
RAY_scheduler_spread_threshold=0.0 ray start --head --num-cpus=0
|
||||
|
||||
# Worker node.
|
||||
RAY_scheduler_spread_threshold=0.0 ray start --address=[head-node-address]
|
||||
|
||||
Out-of-Core Data Processing
|
||||
---------------------------
|
||||
|
||||
.. _dask-on-ray-out-of-core:
|
||||
|
||||
Processing datasets larger than cluster memory is supported via Ray's :ref:`object spilling <object-spilling>`: if
|
||||
the in-memory object store is full, objects will be spilled to external storage (local disk by
|
||||
default). This feature is available but off by default in Ray 1.2, and is on by default
|
||||
in Ray 1.3+. Please see your Ray version's object spilling documentation for steps to enable and/or configure
|
||||
object spilling.
|
||||
|
||||
Persist
|
||||
-------
|
||||
|
||||
.. _dask-on-ray-persist:
|
||||
|
||||
Dask-on-Ray patches `dask.persist()
|
||||
<https://docs.dask.org/en/latest/api.html#dask.persist>`__ in order to match `Dask
|
||||
Distributed's persist semantics
|
||||
<https://distributed.dask.org/en/latest/manage-computation.html#client-persist>`__; namely, calling `dask.persist()` with a Dask-on-Ray
|
||||
scheduler will submit the tasks to the Ray cluster and return Ray futures inlined in the
|
||||
Dask collection. This is nice if you wish to compute some base collection (such as
|
||||
a Dask array), followed by multiple different downstream computations (such as
|
||||
aggregations): those downstream computations will be faster since that base collection
|
||||
computation was kicked off early and referenced by all downstream computations, often
|
||||
via shared memory.
|
||||
|
||||
.. literalinclude:: doc_code/dask_on_ray_persist_example.py
|
||||
:language: python
|
||||
|
||||
|
||||
Annotations, Resources, and Task Options
|
||||
----------------------------------------
|
||||
|
||||
.. _dask-on-ray-annotations:
|
||||
|
||||
|
||||
Dask-on-Ray supports specifying resources or any other Ray task option via `Dask's
|
||||
annotation API <https://docs.dask.org/en/stable/api.html#dask.annotate>`__. This
|
||||
annotation context manager can be used to attach resource requests (or any other Ray task
|
||||
option) to specific Dask operations, with the annotations funneling down to the
|
||||
underlying Ray tasks. Resource requests and other Ray task options can also be specified
|
||||
globally via the ``.compute(ray_remote_args={...})`` API, which will
|
||||
serve as a default for all Ray tasks launched via the Dask workload. Annotations on
|
||||
individual Dask operations will override this global default.
|
||||
|
||||
.. literalinclude:: doc_code/dask_on_ray_annotate_example.py
|
||||
:language: python
|
||||
|
||||
Note that you may need to disable graph optimizations since it can break annotations,
|
||||
see `this Dask issue <https://github.com/dask/dask/issues/7036>`__.
|
||||
|
||||
Custom optimization for Dask DataFrame shuffling
|
||||
------------------------------------------------
|
||||
|
||||
.. _dask-on-ray-shuffle-optimization:
|
||||
|
||||
Dask-on-Ray provides a Dask DataFrame optimizer that leverages Ray's ability to
|
||||
execute multiple-return tasks in order to speed up shuffling by as much as 4x on Ray.
|
||||
Simply set the `dataframe_optimize` configuration option to our optimizer function, similar to how you specify the Dask-on-Ray scheduler:
|
||||
|
||||
.. literalinclude:: doc_code/dask_on_ray_shuffle_optimization.py
|
||||
:language: python
|
||||
|
||||
Callbacks
|
||||
---------
|
||||
|
||||
.. _dask-on-ray-callbacks:
|
||||
|
||||
Dask's `custom callback abstraction <https://docs.dask.org/en/latest/diagnostics-local.html#custom-callbacks>`__
|
||||
is extended with Ray-specific callbacks, allowing the user to hook into the
|
||||
Ray task submission and execution lifecycles.
|
||||
With these hooks, implementing Dask-level scheduler and task introspection,
|
||||
such as progress reporting, diagnostics, caching, etc., is simple.
|
||||
|
||||
Here's an example that measures and logs the execution time of each task using
|
||||
the ``ray_pretask`` and ``ray_posttask`` hooks:
|
||||
|
||||
.. literalinclude:: doc_code/dask_on_ray_callbacks.py
|
||||
:language: python
|
||||
:start-after: __timer_callback_begin__
|
||||
:end-before: __timer_callback_end__
|
||||
|
||||
The following Ray-specific callbacks are provided:
|
||||
|
||||
1. :code:`ray_presubmit(task, key, deps)`: Run before submitting a Ray
|
||||
task. If this callback returns a non-`None` value, a Ray task will _not_
|
||||
be created and this value will be used as the would-be task's result
|
||||
value.
|
||||
2. :code:`ray_postsubmit(task, key, deps, object_ref)`: Run after submitting
|
||||
a Ray task.
|
||||
3. :code:`ray_pretask(key, object_refs)`: Run before executing a Dask task
|
||||
within a Ray task. This executes after the task has been submitted,
|
||||
within a Ray worker. The return value of this task will be passed to the
|
||||
ray_posttask callback, if provided.
|
||||
4. :code:`ray_posttask(key, result, pre_state)`: Run after executing a Dask
|
||||
task within a Ray task. This executes within a Ray worker. This callback
|
||||
receives the return value of the ray_pretask callback, if provided.
|
||||
5. :code:`ray_postsubmit_all(object_refs, dsk)`: Run after all Ray tasks
|
||||
have been submitted.
|
||||
6. :code:`ray_finish(result)`: Run after all Ray tasks have finished
|
||||
executing and the final result has been returned.
|
||||
|
||||
See the docstring for :class:`~ray.util.dask.RayDaskCallback`
|
||||
for further details about these callbacks, their arguments, and their return
|
||||
values.
|
||||
|
||||
When creating your own callbacks, you can use
|
||||
:class:`RayDaskCallback <ray.util.dask.callbacks.RayDaskCallback>`
|
||||
directly, passing the callback functions as constructor arguments:
|
||||
|
||||
.. literalinclude:: doc_code/dask_on_ray_callbacks.py
|
||||
:language: python
|
||||
:start-after: __ray_dask_callback_direct_begin__
|
||||
:end-before: __ray_dask_callback_direct_end__
|
||||
|
||||
or you can subclass it, implementing the callback methods that you need:
|
||||
|
||||
.. literalinclude:: doc_code/dask_on_ray_callbacks.py
|
||||
:language: python
|
||||
:start-after: __ray_dask_callback_subclass_begin__
|
||||
:end-before: __ray_dask_callback_subclass_end__
|
||||
|
||||
You can also specify multiple callbacks:
|
||||
|
||||
.. literalinclude:: doc_code/dask_on_ray_callbacks.py
|
||||
:language: python
|
||||
:start-after: __multiple_callbacks_begin__
|
||||
:end-before: __multiple_callbacks_end__
|
||||
|
||||
Combining Dask callbacks with an actor yields simple patterns for stateful data
|
||||
aggregation, such as capturing task execution statistics and caching results.
|
||||
Here is an example that does both, caching the result of a task if its
|
||||
execution time exceeds some user-defined threshold:
|
||||
|
||||
.. literalinclude:: doc_code/dask_on_ray_callbacks.py
|
||||
:language: python
|
||||
:start-after: __caching_actor_begin__
|
||||
:end-before: __caching_actor_end__
|
||||
|
||||
.. note::
|
||||
The existing Dask scheduler callbacks (``start``, ``start_state``,
|
||||
``pretask``, ``posttask``, ``finish``) are also available, which can be used to
|
||||
introspect the Dask task to Ray task conversion process, but note that the ``pretask``
|
||||
and ``posttask`` hooks are executed before and after the Ray task is *submitted*, not
|
||||
executed, and that ``finish`` is executed after all Ray tasks have been
|
||||
*submitted*, not executed.
|
||||
|
||||
This callback API is currently unstable and subject to change.
|
||||
|
||||
API
|
||||
---
|
||||
|
||||
.. autosummary::
|
||||
:nosignatures:
|
||||
:toctree: doc/
|
||||
|
||||
~ray.util.dask.RayDaskCallback
|
||||
~ray.util.dask.callbacks.RayDaskCallback._ray_presubmit
|
||||
~ray.util.dask.callbacks.RayDaskCallback._ray_postsubmit
|
||||
~ray.util.dask.callbacks.RayDaskCallback._ray_pretask
|
||||
~ray.util.dask.callbacks.RayDaskCallback._ray_posttask
|
||||
~ray.util.dask.callbacks.RayDaskCallback._ray_postsubmit_all
|
||||
~ray.util.dask.callbacks.RayDaskCallback._ray_finish
|
||||
@@ -0,0 +1,147 @@
|
||||
---
|
||||
orphan: true
|
||||
---
|
||||
|
||||
# Distributed Data Processing in Data-Juicer
|
||||
|
||||
Data-Juicer supports large-scale distributed data processing based on [Ray](https://github.com/ray-project/ray) and [Platform for AI](https://www.alibabacloud.com/en/product/machine-learning) of Alibaba Cloud.
|
||||
|
||||
With a dedicated design, you can seamlessly execute almost all operators that Data-Juicer implements in standalone mode, in Ray distributed mode. The Data-Juicer team continuously conducts engine-specific optimizations for large-scale scenarios, such as data subset splitting strategies that balance the number of files and workers, and streaming I/O patches for JSON files to Ray and Apache Arrow.
|
||||
|
||||
For reference, in experiments with 25 to 100 Alibaba Cloud nodes, Data-Juicer in Ray mode processes datasets containing 70 billion samples on 6400 CPU cores in 2 hours and 7 billion samples on 3200 CPU cores in 0.45 hours. Additionally, a MinHash-LSH-based deduplication operator in Ray mode can deduplicate terabyte-sized datasets on 8 nodes with 1280 CPU cores in 3 hours.
|
||||
|
||||
See the [Data-Juicer 2.0: Cloud-Scale Adaptive Data Processing for Foundation Models](http://dail-wlcb.oss-cn-wulanchabu.aliyuncs.com/data_juicer/DJ2.0_arXiv_preview.pdf) paper for more details.
|
||||
|
||||
<img src="https://img.alicdn.com/imgextra/i2/O1CN01EteoQ31taUweAW1UE_!!6000000005918-2-tps-4034-4146.png" align="center" width="600" />
|
||||
|
||||
## Implementation and optimizations
|
||||
|
||||
### Ray mode in Data-Juicer
|
||||
|
||||
- For most implementations of Data-Juicer [operators](https://github.com/modelscope/data-juicer/blob/main/docs/Operators.md), the core processing functions are engine-agnostic. Operator interoperability is managed primarily in [RayDataset](https://github.com/modelscope/data-juicer/blob/main/data_juicer/core/data/ray_dataset.py) and [RayExecutor](https://github.com/modelscope/data-juicer/blob/main/data_juicer/core/executor/ray_executor.py), which are subclasses of the base `DJDataset` and `BaseExecutor`, respectively, and support both Ray [Tasks](ray-remote-functions) and [Actors](actor-guide).
|
||||
- The exception is the deduplication operators, which are challenging to scale in standalone mode. The names of these operators follow the pattern of [`ray_xx_deduplicator`](https://github.com/modelscope/data-juicer/blob/main/data_juicer/ops/deduplicator/).
|
||||
|
||||
### Subset splitting
|
||||
|
||||
When a cluster has tens of thousands of nodes but only a few dataset files, Ray splits the dataset files according to available resources and distributes the blocks across all nodes, incurring high network communication costs and reducing CPU utilization. For more details, see [Ray's `_autodetect_parallelism` function](https://github.com/ray-project/ray/blob/2dbd08a46f7f08ea614d8dd20fd0bca5682a3078/python/ray/data/_internal/util.py#L201-L205) and [tuning output blocks for Ray](read_output_blocks).
|
||||
|
||||
This default execution plan can be quite inefficient especially for scenarios with a large number of nodes. To optimize performance for such cases, Data-Juicer automatically splits the original dataset into smaller files in advance, taking into consideration the features of Ray and Arrow. When you encounter such performance issues, you can use this feature or split the dataset according to your own preferences. In this auto-split strategy, the single file size is about 128 MB, and the result should ensure that the number of sub-files after splitting is at least twice the total number of CPU cores available in the cluster.
|
||||
|
||||
### Streaming reading of JSON files
|
||||
|
||||
Streaming reading of JSON files is a common requirement in data processing for foundation models, as many datasets are in JSONL format and large in size. However, the current implementation in Ray Datasets, which depends on the underlying Arrow library (up to Ray version 2.40 and Arrow version 18.1.0), doesn't support streaming reading of JSON files.
|
||||
|
||||
To address the lack of native support for streaming JSON data, the Data-Juicer team developed a streaming loading interface and contributed an in-house [patch](https://github.com/modelscope/data-juicer/pull/515) for Apache Arrow ([PR to the repository](https://github.com/apache/arrow/pull/45084)). This patch helps alleviate Out-of-Memory issues. With this patch, Data-Juicer in Ray mode, by default, uses the streaming loading interface to load JSON files. In addition, streaming-read support for CSV and Parquet files is already enabled.
|
||||
|
||||
|
||||
### Deduplication
|
||||
|
||||
Data-Juicer provides an optimized MinHash-LSH-based deduplication operator in Ray mode. It's a multiprocessing Union-Find set in Ray Actors and a load-balanced distributed algorithm, [BTS](https://ieeexplore.ieee.org/document/10598116), to complete equivalence class merging. This operator can deduplicate terabyte-sized datasets on 1280 CPU cores in 3 hours. The Data-Juicer team's ablation study shows 2x to 3x speedups with their dedicated optimizations for Ray mode compared to the vanilla version of this deduplication operator.
|
||||
|
||||
## Performance results
|
||||
|
||||
### Data Processing with Varied Scales
|
||||
|
||||
Data-Juicer team conducted experiments on datasets with billions of samples. They prepared a 560k-sample multimodal dataset and expanded it by different factors (1x to 125000x) to create datasets of varying sizes. The experimental results, shown in the figure below, demonstrate good scalability.
|
||||
|
||||
<img src="https://img.alicdn.com/imgextra/i3/O1CN01JV8wcC1oxn0G2xnBT_!!6000000005292-0-tps-1328-1742.jpg" align="center" width="600" />
|
||||
|
||||
### Distributed Deduplication on Large-Scale Datasets
|
||||
|
||||
Data-Juicer team tested the MinHash-based RayDeduplicator on datasets sized at 200 GB, 1 TB, and 5 TB, using CPU counts ranging from 640 to 1280 cores. As the table below shows, when the data size increases by 5x, the processing time increases by 4.02x to 5.62x. When the number of CPU cores doubles, the processing time decreases to 58.9% to 67.1% of the original time.
|
||||
|
||||
| # CPU | 200 GB Time | 1 TB Time | 5 TB Time |
|
||||
|---------|------------------|----------------|----------------|
|
||||
| 4 * 160 | 11.13 min | 50.83 min | 285.43 min |
|
||||
| 8 * 160 | 7.47 min | 30.08 min | 168.10 min |
|
||||
|
||||
## Quick Start
|
||||
|
||||
Before starting, you should install Data-Juicer and its `dist` requirements:
|
||||
|
||||
```shell
|
||||
pip install -v -e . # Install the minimal requirements of Data-Juicer
|
||||
pip install -v -e ".[dist]" # Include dependencies on Ray and other distributed libraries
|
||||
```
|
||||
|
||||
Then start a Ray cluster (ref to the [Ray doc](start-ray) for more details):
|
||||
|
||||
```shell
|
||||
# Start a cluster as the head node
|
||||
ray start --head
|
||||
|
||||
# (Optional) Connect to the cluster on other nodes/machines.
|
||||
ray start --address='{head_ip}:6379'
|
||||
```
|
||||
|
||||
Data-Juicer provides simple demos in the directory `demos/process_on_ray/`, which includes two config files and two test datasets.
|
||||
|
||||
```text
|
||||
demos/process_on_ray
|
||||
├── configs
|
||||
│ ├── demo.yaml
|
||||
│ └── dedup.yaml
|
||||
└── data
|
||||
├── demo-dataset.json
|
||||
└── demo-dataset.jsonl
|
||||
```
|
||||
|
||||
> **Important:**
|
||||
> If you run these demos on multiple nodes, you need to put the demo dataset to a shared disk (for example, Network-attached storage) and export the result dataset to it as well by modifying the `dataset_path` and `export_path` in the config files.
|
||||
|
||||
### Running Example of Ray Mode
|
||||
|
||||
In the `demo.yaml` config file, it sets the executor type to "ray" and specifies an automatic Ray address.
|
||||
|
||||
```yaml
|
||||
...
|
||||
dataset_path: './demos/process_on_ray/data/demo-dataset.jsonl'
|
||||
export_path: './outputs/demo/demo-processed'
|
||||
|
||||
executor_type: 'ray' # Set the executor type to "ray"
|
||||
ray_address: 'auto' # Set an automatic Ray address
|
||||
...
|
||||
```
|
||||
|
||||
Run the demo to process the dataset with 12 regular OPs:
|
||||
|
||||
```shell
|
||||
# Run the tool from source
|
||||
python tools/process_data.py --config demos/process_on_ray/configs/demo.yaml
|
||||
|
||||
# Use the command-line tool
|
||||
dj-process --config demos/process_on_ray/configs/demo.yaml
|
||||
```
|
||||
|
||||
Data-Juicer processes the demo dataset with the demo config file and exports the result datasets to the directory specified by the `export_path` argument in the config file.
|
||||
|
||||
### Running Example of Distributed Deduplication
|
||||
|
||||
In the `dedup.yaml` config file, it sets the executor type to "ray" and specifies an automatic Ray address. And it uses a dedicated distributed version of MinHash deduplication operator to deduplicate the dataset.
|
||||
|
||||
```yaml
|
||||
project_name: 'demo-dedup'
|
||||
dataset_path: './demos/process_on_ray/data/'
|
||||
export_path: './outputs/demo-dedup/demo-ray-bts-dedup-processed'
|
||||
|
||||
executor_type: 'ray' # Set the executor type to "ray"
|
||||
ray_address: 'auto' # Set an automatic Ray address
|
||||
|
||||
# process schedule
|
||||
# a list of several process operators with their arguments
|
||||
process:
|
||||
- ray_bts_minhash_deduplicator: # a distributed version of minhash deduplicator
|
||||
tokenization: 'character'
|
||||
```
|
||||
|
||||
Run the demo to deduplicate the dataset:
|
||||
|
||||
```shell
|
||||
# Run the tool from source
|
||||
python tools/process_data.py --config demos/process_on_ray/configs/dedup.yaml
|
||||
|
||||
# Use the command-line tool
|
||||
dj-process --config demos/process_on_ray/configs/dedup.yaml
|
||||
```
|
||||
|
||||
Data-Juicer deduplicates the demo dataset with the demo config file and exports the result datasets to the directory specified by the `export_path` argument in the config file.
|
||||
@@ -0,0 +1,52 @@
|
||||
import ray
|
||||
from ray.util.dask import enable_dask_on_ray
|
||||
import dask
|
||||
import dask.array as da
|
||||
|
||||
# Start Ray.
|
||||
# Tip: If connecting to an existing cluster, use ray.init(address="auto").
|
||||
ray.init(
|
||||
resources={
|
||||
"custom_resource": 1,
|
||||
"other_custom_resource": 1,
|
||||
"another_custom_resource": 1,
|
||||
}
|
||||
)
|
||||
|
||||
# Use our Dask config helper to set the scheduler to ray_dask_get globally,
|
||||
# without having to specify it on each compute call.
|
||||
enable_dask_on_ray()
|
||||
|
||||
# All Ray tasks that underly the Dask operations performed in an annotation
|
||||
# context will require the indicated resources: 2 CPUs and 0.01 of the custom
|
||||
# resource.
|
||||
with dask.annotate(
|
||||
ray_remote_args=dict(num_cpus=2, resources={"custom_resource": 0.01})
|
||||
):
|
||||
d_arr = da.ones(100)
|
||||
|
||||
# Operations on the same collection can have different annotations.
|
||||
with dask.annotate(ray_remote_args=dict(resources={"other_custom_resource": 0.01})):
|
||||
d_arr = 2 * d_arr
|
||||
|
||||
# This happens outside of the annotation context, so no resource constraints
|
||||
# will be attached to the underlying Ray tasks for the sum() operation.
|
||||
sum_ = d_arr.sum()
|
||||
|
||||
# Compute the result, passing in a default resource request that will be
|
||||
# applied to all operations that aren't already annotated with a resource
|
||||
# request. In this case, only the sum() operation will get this default
|
||||
# resource request.
|
||||
# We also give ray_remote_args, which will be given to every Ray task that
|
||||
# Dask-on-Ray submits; note that this can also be overridden for individual
|
||||
# Dask operations via the dask.annotate API.
|
||||
# NOTE: We disable graph optimization since it can break annotations,
|
||||
# see this issue: https://github.com/dask/dask/issues/7036.
|
||||
result = sum_.compute(
|
||||
ray_remote_args=dict(max_retries=5, resources={"another_custom_resource": 0.01}),
|
||||
optimize_graph=False,
|
||||
)
|
||||
print(result)
|
||||
# 200
|
||||
|
||||
ray.shutdown()
|
||||
@@ -0,0 +1,103 @@
|
||||
# flake8: noqa
|
||||
import ray
|
||||
import dask.array as da
|
||||
|
||||
z = da.ones(100)
|
||||
|
||||
# fmt: off
|
||||
# __timer_callback_begin__
|
||||
from ray.util.dask import RayDaskCallback, ray_dask_get
|
||||
from timeit import default_timer as timer
|
||||
|
||||
|
||||
class MyTimerCallback(RayDaskCallback):
|
||||
def _ray_pretask(self, key, object_refs):
|
||||
# Executed at the start of the Ray task.
|
||||
start_time = timer()
|
||||
return start_time
|
||||
|
||||
def _ray_posttask(self, key, result, pre_state):
|
||||
# Executed at the end of the Ray task.
|
||||
execution_time = timer() - pre_state
|
||||
print(f"Execution time for task {key}: {execution_time}s")
|
||||
|
||||
|
||||
with MyTimerCallback():
|
||||
# Any .compute() calls within this context will get MyTimerCallback()
|
||||
# as a Dask-Ray callback.
|
||||
z.compute(scheduler=ray_dask_get)
|
||||
# __timer_callback_end__
|
||||
# fmt: on
|
||||
|
||||
# fmt: off
|
||||
# __ray_dask_callback_direct_begin__
|
||||
def my_presubmit_cb(task, key, deps):
|
||||
print(f"About to submit task {key}!")
|
||||
|
||||
with RayDaskCallback(ray_presubmit=my_presubmit_cb):
|
||||
z.compute(scheduler=ray_dask_get)
|
||||
# __ray_dask_callback_direct_end__
|
||||
# fmt: on
|
||||
|
||||
# fmt: off
|
||||
# __ray_dask_callback_subclass_begin__
|
||||
class MyPresubmitCallback(RayDaskCallback):
|
||||
def _ray_presubmit(self, task, key, deps):
|
||||
print(f"About to submit task {key}!")
|
||||
|
||||
with MyPresubmitCallback():
|
||||
z.compute(scheduler=ray_dask_get)
|
||||
# __ray_dask_callback_subclass_end__
|
||||
# fmt: on
|
||||
|
||||
# fmt: off
|
||||
# __multiple_callbacks_begin__
|
||||
# The hooks for both MyTimerCallback and MyPresubmitCallback will be
|
||||
# called.
|
||||
with MyTimerCallback(), MyPresubmitCallback():
|
||||
z.compute(scheduler=ray_dask_get)
|
||||
# __multiple_callbacks_end__
|
||||
# fmt: on
|
||||
|
||||
# fmt: off
|
||||
# __caching_actor_begin__
|
||||
@ray.remote
|
||||
class SimpleCacheActor:
|
||||
def __init__(self):
|
||||
self.cache = {}
|
||||
|
||||
def get(self, key):
|
||||
# Raises KeyError if key isn't in cache.
|
||||
return self.cache[key]
|
||||
|
||||
def put(self, key, value):
|
||||
self.cache[key] = value
|
||||
|
||||
|
||||
class SimpleCacheCallback(RayDaskCallback):
|
||||
def __init__(self, cache_actor_handle, put_threshold=10):
|
||||
self.cache_actor = cache_actor_handle
|
||||
self.put_threshold = put_threshold
|
||||
|
||||
def _ray_presubmit(self, task, key, deps):
|
||||
try:
|
||||
return ray.get(self.cache_actor.get.remote(str(key)))
|
||||
except KeyError:
|
||||
return None
|
||||
|
||||
def _ray_pretask(self, key, object_refs):
|
||||
start_time = timer()
|
||||
return start_time
|
||||
|
||||
def _ray_posttask(self, key, result, pre_state):
|
||||
execution_time = timer() - pre_state
|
||||
if execution_time > self.put_threshold:
|
||||
self.cache_actor.put.remote(str(key), result)
|
||||
|
||||
|
||||
cache_actor = SimpleCacheActor.remote()
|
||||
cache_callback = SimpleCacheCallback(cache_actor, put_threshold=2)
|
||||
with cache_callback:
|
||||
z.compute(scheduler=ray_dask_get)
|
||||
# __caching_actor_end__
|
||||
# fmt: on
|
||||
@@ -0,0 +1,39 @@
|
||||
import ray
|
||||
from ray.util.dask import enable_dask_on_ray
|
||||
import dask
|
||||
import dask.array as da
|
||||
|
||||
# Start Ray.
|
||||
# Tip: If connecting to an existing cluster, use ray.init(address="auto").
|
||||
ray.init()
|
||||
|
||||
# Use our Dask config helper to set the scheduler to ray_dask_get globally,
|
||||
# without having to specify it on each compute call.
|
||||
enable_dask_on_ray()
|
||||
|
||||
d_arr = da.ones(100)
|
||||
|
||||
# Print the internal Dask graph. Replace this with `print(dask.base.collections_to_dsk([d_arr]))` when dask>=2024.11.0,<2025.4.0.
|
||||
print(dask.base.collections_to_expr([d_arr]).dask)
|
||||
# {('ones_like-5902a58f37d3b639948dee893f5c4f4a', 0):
|
||||
# <Task ('ones_like-5902a58f37d3b639948dee893f5c4f4a', 0)
|
||||
# ones_like(...)>}
|
||||
|
||||
# This submits all underlying Ray tasks to the cluster and returns
|
||||
# a Dask array with the Ray futures inlined.
|
||||
d_arr_p = d_arr.persist()
|
||||
|
||||
# Notice that the Ray ObjectRef is inlined. The dask.ones() task has
|
||||
# been submitted to and is running on the Ray cluster.
|
||||
# Replace this in a similar way when dask>=2024.11.0,<2025.4.0.
|
||||
print(dask.base.collections_to_expr([d_arr_p]).dask)
|
||||
# {('ones_like-5902a58f37d3b639948dee893f5c4f4a', 0):
|
||||
# DataNode(ObjectRef(2c329aa28fcae64affffffffffffffffffffffff2c00000001000000))}
|
||||
|
||||
# Future computations on this persisted Dask Array will be fast since we
|
||||
# already started computing d_arr_p in the background.
|
||||
d_arr_p.sum().compute()
|
||||
d_arr_p.min().compute()
|
||||
d_arr_p.max().compute()
|
||||
|
||||
ray.shutdown()
|
||||
@@ -0,0 +1,34 @@
|
||||
import ray
|
||||
from ray.util.dask import ray_dask_get, enable_dask_on_ray, disable_dask_on_ray
|
||||
import dask.array as da
|
||||
import dask.dataframe as dd
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
# Start Ray.
|
||||
# Tip: If connecting to an existing cluster, use ray.init(address="auto").
|
||||
ray.init()
|
||||
|
||||
d_arr = da.from_array(np.random.randint(0, 1000, size=(256, 256)))
|
||||
|
||||
# The Dask scheduler submits the underlying task graph to Ray.
|
||||
d_arr.mean().compute(scheduler=ray_dask_get)
|
||||
|
||||
# Use our Dask config helper to set the scheduler to ray_dask_get globally,
|
||||
# without having to specify it on each compute call.
|
||||
enable_dask_on_ray()
|
||||
|
||||
df = dd.from_pandas(
|
||||
pd.DataFrame(np.random.randint(0, 100, size=(1024, 2)), columns=["age", "grade"]),
|
||||
npartitions=2,
|
||||
)
|
||||
df.groupby(["age"]).mean().compute()
|
||||
|
||||
disable_dask_on_ray()
|
||||
|
||||
# The Dask config helper can be used as a context manager, limiting the scope
|
||||
# of the Dask-on-Ray scheduler to the context.
|
||||
with enable_dask_on_ray():
|
||||
d_arr.mean().compute()
|
||||
|
||||
ray.shutdown()
|
||||
@@ -0,0 +1,30 @@
|
||||
import ray
|
||||
from ray.util.dask import dataframe_optimize, ray_dask_get
|
||||
import dask
|
||||
import dask.dataframe as dd
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
# Start Ray.
|
||||
# Tip: If connecting to an existing cluster, use ray.init(address="auto").
|
||||
ray.init()
|
||||
|
||||
# Set the Dask DataFrame optimizer to
|
||||
# our custom optimization function, this time using the config setter as a
|
||||
# context manager.
|
||||
with dask.config.set(scheduler=ray_dask_get, dataframe_optimize=dataframe_optimize):
|
||||
npartitions = 100
|
||||
df = dd.from_pandas(
|
||||
pd.DataFrame(
|
||||
np.random.randint(0, 100, size=(10000, 2)), columns=["age", "grade"]
|
||||
),
|
||||
npartitions=npartitions,
|
||||
)
|
||||
# We set max_branch to infinity in order to ensure that the task-based
|
||||
# shuffle happens in a single stage, which is required in order for our
|
||||
# optimization to work.
|
||||
df.set_index(["age"], shuffle="tasks", max_branch=float("inf")).head(
|
||||
10, npartitions=-1
|
||||
)
|
||||
|
||||
ray.shutdown()
|
||||
@@ -0,0 +1,44 @@
|
||||
More Ray ML Libraries
|
||||
=====================
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
|
||||
joblib
|
||||
multiprocessing
|
||||
ray-collective
|
||||
dask-on-ray
|
||||
raydp
|
||||
mars-on-ray
|
||||
modin/index
|
||||
data_juicer_distributed_data_processing
|
||||
|
||||
|
||||
.. TODO: we added the three Ray Core examples below, since they don't really belong there.
|
||||
Going forward, make sure that all "Ray Lightning" and XGBoost topics are in one document or group,
|
||||
and not next to each other.
|
||||
|
||||
Ray has a variety of additional integrations with ecosystem libraries.
|
||||
|
||||
- :ref:`ray-joblib`
|
||||
- :ref:`ray-multiprocessing`
|
||||
- :ref:`ray-collective`
|
||||
- :ref:`dask-on-ray`
|
||||
- :ref:`spark-on-ray`
|
||||
- :ref:`mars-on-ray`
|
||||
- :ref:`modin-on-ray`
|
||||
- `daft <https://www.daft.ai>`_
|
||||
|
||||
.. _air-ecosystem-map:
|
||||
|
||||
Ecosystem Map
|
||||
-------------
|
||||
|
||||
The following map visualizes the landscape and maturity of Ray components and their integrations. Solid lines denote integrations between Ray components; dotted lines denote integrations with the broader ML ecosystem.
|
||||
|
||||
* **Stable**: This component is stable.
|
||||
* **Beta**: This component is under development and APIs may be subject to change.
|
||||
* **Alpha**: This component is in early development.
|
||||
* **Community-Maintained**: These integrations are community-maintained and may vary in quality.
|
||||
|
||||
.. image:: /images/air-ecosystem.svg
|
||||
@@ -0,0 +1,79 @@
|
||||
.. _ray-joblib:
|
||||
|
||||
Distributed Scikit-learn / Joblib
|
||||
=================================
|
||||
|
||||
.. _`issue on GitHub`: https://github.com/ray-project/ray/issues
|
||||
|
||||
Ray supports running distributed `scikit-learn`_ programs by
|
||||
implementing a Ray backend for `joblib`_ using `Ray Actors <actors.html>`__
|
||||
instead of local processes. This makes it easy to scale existing applications
|
||||
that use scikit-learn from a single node to a cluster.
|
||||
|
||||
.. note::
|
||||
|
||||
This API is new and may be revised in future Ray releases. If you encounter
|
||||
any bugs, please file an `issue on GitHub`_.
|
||||
|
||||
.. _`joblib`: https://joblib.readthedocs.io
|
||||
.. _`scikit-learn`: https://scikit-learn.org
|
||||
|
||||
Quickstart
|
||||
----------
|
||||
|
||||
To get started, first `install Ray <installation.html>`__, then use
|
||||
``from ray.util.joblib import register_ray`` and run ``register_ray()``.
|
||||
This will register Ray as a joblib backend for scikit-learn to use.
|
||||
Then run your original scikit-learn code inside
|
||||
``with joblib.parallel_backend('ray')``. This will start a local Ray cluster.
|
||||
See the `Run on a Cluster`_ section below for instructions to run on
|
||||
a multi-node Ray cluster instead.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import numpy as np
|
||||
from sklearn.datasets import load_digits
|
||||
from sklearn.model_selection import RandomizedSearchCV
|
||||
from sklearn.svm import SVC
|
||||
digits = load_digits()
|
||||
param_space = {
|
||||
'C': np.logspace(-6, 6, 30),
|
||||
'gamma': np.logspace(-8, 8, 30),
|
||||
'tol': np.logspace(-4, -1, 30),
|
||||
'class_weight': [None, 'balanced'],
|
||||
}
|
||||
model = SVC(kernel='rbf')
|
||||
search = RandomizedSearchCV(model, param_space, cv=5, n_iter=300, verbose=10)
|
||||
|
||||
import joblib
|
||||
from ray.util.joblib import register_ray
|
||||
register_ray()
|
||||
with joblib.parallel_backend('ray'):
|
||||
search.fit(digits.data, digits.target)
|
||||
|
||||
You can also set the ``ray_remote_args`` argument in ``parallel_backend`` to :func:`configure
|
||||
the Ray Actors <ray.remote>` making up the Pool. This can be used to e.g., :ref:`assign resources
|
||||
to Actors, such as GPUs <actor-resource-guide>`.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
# Allows to use GPU-enabled estimators, such as cuML
|
||||
with joblib.parallel_backend('ray', ray_remote_args=dict(num_gpus=1)):
|
||||
search.fit(digits.data, digits.target)
|
||||
|
||||
Run on a Cluster
|
||||
----------------
|
||||
|
||||
This section assumes that you have a running Ray cluster. To start a Ray cluster,
|
||||
see the :ref:`cluster setup <cluster-index>` instructions.
|
||||
|
||||
To connect scikit-learn to a running Ray cluster, you have to specify the address of the
|
||||
head node by setting the ``RAY_ADDRESS`` environment variable.
|
||||
|
||||
You can also start Ray manually by calling ``ray.init()`` (with any of its supported
|
||||
configuration options) before calling ``with joblib.parallel_backend('ray')``.
|
||||
|
||||
.. warning::
|
||||
|
||||
If you do not set the ``RAY_ADDRESS`` environment variable and do not provide
|
||||
``address`` in ``ray.init(address=<address>)`` then scikit-learn will run on a SINGLE node!
|
||||
@@ -0,0 +1,80 @@
|
||||
.. _mars-on-ray:
|
||||
|
||||
Using Mars on Ray
|
||||
=================
|
||||
|
||||
.. _`issue on GitHub`: https://github.com/mars-project/mars/issues
|
||||
|
||||
|
||||
`Mars`_ is a tensor-based unified framework for large-scale data computation which scales NumPy, Pandas and Scikit-learn.
|
||||
Mars on Ray makes it easy to scale your programs with a Ray cluster. Currently Mars on Ray supports both Ray actors
|
||||
and tasks as an execution backend. The task will be scheduled by Mars scheduler if Ray actors are used. This mode can reuse
|
||||
all Mars scheduler optimizations. If Ray tasks mode is used, all tasks will be scheduled by Ray, which can reuse failover and
|
||||
pipeline capabilities provided by Ray futures.
|
||||
|
||||
|
||||
.. _`Mars`: https://mars-project.readthedocs.io/en/latest/
|
||||
|
||||
|
||||
Installation
|
||||
-------------
|
||||
You can simply install Mars via pip:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pip install pymars>=0.8.3
|
||||
|
||||
|
||||
Getting started
|
||||
----------------
|
||||
|
||||
It's easy to run Mars jobs on a Ray cluster.
|
||||
|
||||
|
||||
Starting a new Mars on Ray runtime locally via:
|
||||
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import ray
|
||||
ray.init()
|
||||
import mars
|
||||
mars.new_ray_session()
|
||||
import mars.tensor as mt
|
||||
mt.random.RandomState(0).rand(1000_0000, 5).sum().execute()
|
||||
|
||||
|
||||
Or connecting to a Mars on Ray runtime which is already initialized:
|
||||
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import mars
|
||||
mars.new_ray_session('http://<web_ip>:<ui_port>')
|
||||
# perform computation
|
||||
|
||||
|
||||
Interact with Dataset:
|
||||
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import mars.tensor as mt
|
||||
import mars.dataframe as md
|
||||
df = md.DataFrame(
|
||||
mt.random.rand(1000_0000, 4),
|
||||
columns=list('abcd'))
|
||||
# Convert mars dataframe to ray dataset
|
||||
import ray
|
||||
# ds = md.to_ray_dataset(df)
|
||||
ds = ray.data.from_mars(df)
|
||||
print(ds.schema(), ds.count())
|
||||
ds.filter(lambda row: row["a"] > 0.5).show(5)
|
||||
# Convert ray dataset to mars dataframe
|
||||
# df2 = md.read_ray_dataset(ds)
|
||||
df2 = ds.to_mars()
|
||||
print(df2.head(5).execute())
|
||||
|
||||
Refer to `Mars on Ray`_ for more information.
|
||||
|
||||
.. _`Mars on Ray`: https://mars-project.readthedocs.io/en/latest/installation/ray.html#mars-ray
|
||||
@@ -0,0 +1,81 @@
|
||||
.. _modin-on-ray:
|
||||
|
||||
Using Pandas on Ray (Modin)
|
||||
===========================
|
||||
|
||||
Modin_, previously Pandas on Ray, is a dataframe manipulation library that
|
||||
allows users to speed up their pandas workloads by acting as a drop-in
|
||||
replacement. Modin also provides support for other APIs (e.g. spreadsheet)
|
||||
and libraries, like xgboost.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import modin.pandas as pd
|
||||
import ray
|
||||
|
||||
ray.init()
|
||||
df = pd.read_parquet("s3://my-bucket/big.parquet")
|
||||
|
||||
You can use Modin on Ray with your laptop or cluster. In this document,
|
||||
we show instructions for how to set up a Modin compatible Ray cluster
|
||||
and connect Modin to Ray.
|
||||
|
||||
.. note:: In previous versions of Modin, you had to initialize Ray before importing Modin. As of Modin 0.9.0, this is no longer the case.
|
||||
|
||||
Using Modin with Ray's autoscaler
|
||||
---------------------------------
|
||||
|
||||
In order to use Modin with :ref:`Ray's autoscaler <cluster-index>`, you need to ensure that the
|
||||
correct dependencies are installed at startup. Modin's repository has an
|
||||
example `yaml file and set of tutorial notebooks`_ to ensure that the Ray
|
||||
cluster has the correct dependencies. Once the cluster is up, connect Modin
|
||||
by simply importing.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import modin.pandas as pd
|
||||
import ray
|
||||
|
||||
ray.init(address="auto")
|
||||
df = pd.read_parquet("s3://my-bucket/big.parquet")
|
||||
|
||||
As long as Ray is initialized before any dataframes are created, Modin
|
||||
will be able to connect to and use the Ray cluster.
|
||||
|
||||
How Modin uses Ray
|
||||
------------------
|
||||
|
||||
Modin has a layered architecture, and the core abstraction for data manipulation
|
||||
is the Modin Dataframe, which implements a novel algebra that enables Modin to
|
||||
handle all of pandas (see Modin's documentation_ for more on the architecture).
|
||||
Modin's internal dataframe object has a scheduling layer that is able to partition
|
||||
and operate on data with Ray.
|
||||
|
||||
Dataframe operations
|
||||
''''''''''''''''''''
|
||||
|
||||
The Modin Dataframe uses Ray Tasks to perform data manipulations. Ray Tasks have
|
||||
a number of benefits over the actor model for data manipulation:
|
||||
|
||||
- Multiple tasks may be manipulating the same objects simultaneously
|
||||
- Objects in Ray's object store are immutable, making provenance and lineage easier
|
||||
to track
|
||||
- As new workers come online the shuffling of data will happen as tasks are
|
||||
scheduled on the new node
|
||||
- Identical partitions need not be replicated, especially beneficial for operations
|
||||
that selectively mutate the data (e.g., ``fillna``).
|
||||
- Finer grained parallelism with finer grained placement control
|
||||
|
||||
Machine Learning
|
||||
''''''''''''''''
|
||||
|
||||
Modin uses Ray Actors for the machine learning support it currently provides.
|
||||
Modin's implementation of XGBoost is able to spin up one actor for each node
|
||||
and aggregate all of the partitions on that node to the XGBoost actor. Modin
|
||||
is able to specify precisely the node IP for each actor on creation, giving
|
||||
fine-grained control over placement - a must for distributed training
|
||||
performance.
|
||||
|
||||
.. _Modin: https://github.com/modin-project/modin
|
||||
.. _documentation: https://modin.readthedocs.io/en/latest/development/architecture.html
|
||||
.. _yaml file and set of tutorial notebooks: https://github.com/modin-project/modin/tree/master/examples/tutorial/jupyter/execution/pandas_on_ray/cluster
|
||||
@@ -0,0 +1,71 @@
|
||||
.. _ray-multiprocessing:
|
||||
|
||||
Distributed multiprocessing.Pool
|
||||
================================
|
||||
|
||||
.. _`issue on GitHub`: https://github.com/ray-project/ray/issues
|
||||
|
||||
Ray supports running distributed Python programs with the `multiprocessing.Pool API`_
|
||||
using `Ray Actors <actors.html>`__ instead of local processes. This makes it easy
|
||||
to scale existing applications that use ``multiprocessing.Pool`` from a single node
|
||||
to a cluster.
|
||||
|
||||
.. _`multiprocessing.Pool API`: https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.pool
|
||||
|
||||
Quickstart
|
||||
----------
|
||||
|
||||
To get started, first `install Ray <installation.html>`__, then use
|
||||
``ray.util.multiprocessing.Pool`` in place of ``multiprocessing.Pool``.
|
||||
This will start a local Ray cluster the first time you create a ``Pool`` and
|
||||
distribute your tasks across it. See the `Run on a Cluster`_ section below for
|
||||
instructions to run on a multi-node Ray cluster instead.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from ray.util.multiprocessing import Pool
|
||||
|
||||
def f(index):
|
||||
return index
|
||||
|
||||
pool = Pool()
|
||||
for result in pool.map(f, range(100)):
|
||||
print(result)
|
||||
|
||||
The full ``multiprocessing.Pool`` API is currently supported. Please see the
|
||||
`multiprocessing documentation`_ for details.
|
||||
|
||||
.. warning::
|
||||
The ``context`` argument in the ``Pool`` constructor is ignored when using Ray.
|
||||
|
||||
.. _`multiprocessing documentation`: https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.pool
|
||||
|
||||
Run on a Cluster
|
||||
----------------
|
||||
|
||||
This section assumes that you have a running Ray cluster. To start a Ray cluster,
|
||||
see the :ref:`cluster setup <cluster-index>` instructions.
|
||||
|
||||
To connect a ``Pool`` to a running Ray cluster, you can specify the address of the
|
||||
head node in one of two ways:
|
||||
|
||||
- By setting the ``RAY_ADDRESS`` environment variable.
|
||||
- By passing the ``ray_address`` keyword argument to the ``Pool`` constructor.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from ray.util.multiprocessing import Pool
|
||||
|
||||
# Starts a new local Ray cluster.
|
||||
pool = Pool()
|
||||
|
||||
# Connects to a running Ray cluster, with the current node as the head node.
|
||||
# Alternatively, set the environment variable RAY_ADDRESS="auto".
|
||||
pool = Pool(ray_address="auto")
|
||||
|
||||
# Connects to a running Ray cluster, with a remote node as the head node.
|
||||
# Alternatively, set the environment variable RAY_ADDRESS="<ip_address>:<port>".
|
||||
pool = Pool(ray_address="<ip_address>:<port>")
|
||||
|
||||
You can also start Ray manually by calling ``ray.init()`` (with any of its supported
|
||||
configuration options) before creating a ``Pool``.
|
||||
@@ -0,0 +1,271 @@
|
||||
.. _ray-collective-custom-backend:
|
||||
|
||||
Custom Collective Backends
|
||||
==========================
|
||||
|
||||
This guide shows how to create and use custom collective backends with Ray.
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
Ray collective operations support custom backends through a registration API. You can implement your own backend by:
|
||||
|
||||
1. Creating a class that inherits from :py:class:`~ray.util.collective.collective_group.base_collective_group.BaseGroup`
|
||||
2. Implementing required collective operations
|
||||
3. Registering your backend with :py:func:`~ray.util.collective.backend_registry.register_collective_backend`
|
||||
|
||||
Creating a Custom Backend
|
||||
-------------------------
|
||||
|
||||
Step 1: Define Your Backend Class
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Your backend class must inherit from :py:class:`~ray.util.collective.collective_group.base_collective_group.BaseGroup` and implement required methods. See the :py:class:`~ray.util.collective.collective_group.base_collective_group.BaseGroup` API reference for the complete list of required methods.
|
||||
|
||||
Here's an example using the ``MockInternalKVGroup`` backend that uses Ray's internal KV store for communication:
|
||||
|
||||
.. testcode::
|
||||
:skipif: True
|
||||
|
||||
from ray.util.collective.examples.mock_internal_kv_example import MockInternalKVGroup
|
||||
|
||||
# MockInternalKVGroup is a complete implementation that inherits from BaseGroup
|
||||
# and implements all required collective operations using Ray's internal KV store
|
||||
backend_cls = MockInternalKVGroup
|
||||
|
||||
# Check that it has the required methods
|
||||
assert hasattr(backend_cls, 'backend')
|
||||
assert hasattr(backend_cls, 'check_backend_availability')
|
||||
assert hasattr(backend_cls, 'allreduce')
|
||||
assert hasattr(backend_cls, 'broadcast')
|
||||
assert hasattr(backend_cls, 'barrier')
|
||||
|
||||
Step 2: Register Your Backend
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Next, register your collective backend on the driver and the actors participating in the collective group with :py:func:`~ray.util.collective.backend_registry.register_collective_backend`.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from ray.util.collective.backend_registry import register_collective_backend
|
||||
|
||||
register_collective_backend("MY_BACKEND", MyCustomBackend)
|
||||
|
||||
.. note::
|
||||
|
||||
**Your backend must be registered on both the driver and all actors before using collective operations.** This is because each process (driver and each actor) needs to know about your backend class to instantiate it.
|
||||
|
||||
Initializing Collective Groups
|
||||
-------------------------------
|
||||
|
||||
There are **two distinct approaches** to initialize collective groups. **Choose one approach for your use case - do not mix them for the same group.**
|
||||
|
||||
.. note::
|
||||
|
||||
Using both :py:func:`~ray.util.collective.collective.create_collective_group` and :py:func:`~ray.util.collective.collective.init_collective_group` together for the same group is not supported and will cause errors.
|
||||
|
||||
Approach 1: Driver-Managed (Recommended)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Use :py:func:`~ray.util.collective.collective.create_collective_group` on the driver to declare the group. Workers do **not** call :py:func:`~ray.util.collective.collective.init_collective_group` - the group is automatically initialized when workers call collective operations.
|
||||
|
||||
This approach is recommended because it provides a declarative, centralized way to manage collective groups. The driver has full visibility into all participants and can coordinate the initialization process.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import ray
|
||||
import numpy as np
|
||||
from ray.util.collective import allreduce, create_collective_group
|
||||
from ray.util.collective.backend_registry import register_collective_backend
|
||||
from ray.util.collective.types import ReduceOp
|
||||
|
||||
# Import your custom backend
|
||||
from my_backend import MyCustomBackend
|
||||
|
||||
# Register on driver
|
||||
register_collective_backend("MY_BACKEND", MyCustomBackend)
|
||||
|
||||
ray.init()
|
||||
|
||||
@ray.remote
|
||||
class Worker:
|
||||
def __init__(self, rank):
|
||||
self.rank = rank
|
||||
|
||||
def setup(self):
|
||||
# IMPORTANT: Register on each worker too
|
||||
from ray.util.collective.backend_registry import register_collective_backend
|
||||
from my_backend import MyCustomBackend
|
||||
register_collective_backend("MY_BACKEND", MyCustomBackend)
|
||||
# Do NOT call init_collective_group here!
|
||||
|
||||
def compute(self):
|
||||
tensor = np.array([float(self.rank + 1)], dtype=np.float32)
|
||||
allreduce(tensor, op=ReduceOp.SUM)
|
||||
return tensor.item()
|
||||
|
||||
# Create workers
|
||||
actors = [Worker.remote(rank=i) for i in range(3)]
|
||||
|
||||
# Declare collective group from driver
|
||||
create_collective_group(
|
||||
actors=actors,
|
||||
world_size=3,
|
||||
ranks=[0, 1, 2],
|
||||
backend="MY_BACKEND",
|
||||
group_name="default",
|
||||
)
|
||||
|
||||
# Setup each worker (only registers backend, no init_collective_group)
|
||||
ray.get([a.setup.remote() for a in actors])
|
||||
|
||||
# Run computation - group is auto-initialized on first collective call
|
||||
results = ray.get([a.compute.remote() for a in actors])
|
||||
print(f"Results: {results}")
|
||||
|
||||
ray.shutdown()
|
||||
|
||||
Approach 2: Worker-Managed
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Each worker explicitly calls :py:func:`~ray.util.collective.collective.init_collective_group` to initialize its group membership. The driver does **not** call :py:func:`~ray.util.collective.collective.create_collective_group`.
|
||||
|
||||
This approach provides more control over initialization timing within each worker, which can be useful for advanced scenarios where workers need to perform custom setup before or after group initialization.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import ray
|
||||
import numpy as np
|
||||
from ray.util.collective import allreduce, init_collective_group
|
||||
from ray.util.collective.backend_registry import register_collective_backend
|
||||
from ray.util.collective.types import ReduceOp
|
||||
|
||||
# Import your custom backend
|
||||
from my_backend import MyCustomBackend
|
||||
|
||||
ray.init()
|
||||
|
||||
@ray.remote
|
||||
class Worker:
|
||||
def __init__(self, rank):
|
||||
self.rank = rank
|
||||
|
||||
def setup(self, world_size):
|
||||
# Register backend
|
||||
from ray.util.collective.backend_registry import register_collective_backend
|
||||
from my_backend import MyCustomBackend
|
||||
register_collective_backend("MY_BACKEND", MyCustomBackend)
|
||||
|
||||
# Explicitly initialize group membership
|
||||
init_collective_group(
|
||||
world_size=world_size,
|
||||
rank=self.rank,
|
||||
backend="MY_BACKEND",
|
||||
group_name="default",
|
||||
)
|
||||
|
||||
def compute(self):
|
||||
tensor = np.array([float(self.rank + 1)], dtype=np.float32)
|
||||
allreduce(tensor, op=ReduceOp.SUM)
|
||||
return tensor.item()
|
||||
|
||||
# Create workers
|
||||
actors = [Worker.remote(rank=i) for i in range(3)]
|
||||
|
||||
# Do NOT call create_collective_group here - workers handle init themselves
|
||||
|
||||
# Setup each worker (registers backend and initializes group)
|
||||
ray.get([a.setup.remote(3) for a in actors])
|
||||
|
||||
# Run computation
|
||||
results = ray.get([a.compute.remote() for a in actors])
|
||||
print(f"Results: {results}")
|
||||
|
||||
ray.shutdown()
|
||||
|
||||
Comparison Table
|
||||
^^^^^^^^^^^^^^^^
|
||||
|
||||
+-------------------+-----------------------------+------------------------------+
|
||||
| Aspect | Driver-Managed (Approach 1) | Worker-Managed (Approach 2) |
|
||||
+===================+=============================+==============================+
|
||||
| Driver calls | ``create_collective_group`` | Nothing |
|
||||
+-------------------+-----------------------------+------------------------------+
|
||||
| Worker calls | ``register_collective_`` | ``register_collective_`` |
|
||||
| | ``backend`` only | ``backend`` + |
|
||||
| | | ``init_collective_group`` |
|
||||
+-------------------+-----------------------------+------------------------------+
|
||||
| Group init | Automatic (on first | Explicit (in worker setup) |
|
||||
| | collective operation) | |
|
||||
+-------------------+-----------------------------+------------------------------+
|
||||
| Use case | Declarative, centralized | More control over |
|
||||
| | management | initialization timing |
|
||||
+-------------------+-----------------------------+------------------------------+
|
||||
|
||||
Complete Example: Mock Backend using Internal KV
|
||||
------------------------------------------------
|
||||
|
||||
The following example demonstrates a complete custom backend implementation using Ray's internal KV store for communication. This ``MockInternalKVGroup`` backend is useful for testing and understanding how custom backends work.
|
||||
|
||||
.. testcode::
|
||||
:skipif: True
|
||||
|
||||
import ray
|
||||
import numpy as np
|
||||
from ray.util.collective import allreduce, create_collective_group
|
||||
from ray.util.collective.backend_registry import register_collective_backend
|
||||
from ray.util.collective.types import ReduceOp
|
||||
from ray.util.collective.examples.mock_internal_kv_example import MockInternalKVGroup
|
||||
|
||||
# Register the mock backend
|
||||
register_collective_backend("MOCK", MockInternalKVGroup)
|
||||
|
||||
ray.init()
|
||||
|
||||
@ray.remote
|
||||
class Worker:
|
||||
def __init__(self, rank):
|
||||
self.rank = rank
|
||||
|
||||
def setup(self):
|
||||
# Register backend on each worker
|
||||
register_collective_backend("MOCK", MockInternalKVGroup)
|
||||
|
||||
def compute(self):
|
||||
tensor = np.array([float(self.rank + 1)], dtype=np.float32)
|
||||
allreduce(tensor, op=ReduceOp.SUM)
|
||||
return tensor.item()
|
||||
|
||||
# Create workers
|
||||
actors = [Worker.remote(rank=i) for i in range(2)]
|
||||
|
||||
# Create collective group from driver
|
||||
create_collective_group(
|
||||
actors=actors,
|
||||
world_size=2,
|
||||
ranks=[0, 1],
|
||||
backend="MOCK",
|
||||
group_name="default",
|
||||
)
|
||||
|
||||
# Setup workers
|
||||
ray.get([a.setup.remote() for a in actors])
|
||||
|
||||
# Run computation
|
||||
results = ray.get([a.compute.remote() for a in actors])
|
||||
print(f"Results: {results}") # [3.0, 3.0]
|
||||
|
||||
ray.shutdown()
|
||||
|
||||
.. testoutput::
|
||||
:skipif: True
|
||||
|
||||
Results: [3.0, 3.0]
|
||||
|
||||
See Also
|
||||
--------
|
||||
|
||||
- :py:class:`~ray.util.collective.collective_group.base_collective_group.BaseGroup` - Base class for custom backends
|
||||
- :py:func:`~ray.util.collective.backend_registry.register_collective_backend` - Register a custom backend
|
||||
- :py:func:`~ray.util.collective.collective.create_collective_group` - Create a collective group (driver-managed)
|
||||
- :py:func:`~ray.util.collective.collective.init_collective_group` - Initialize a collective group (worker-managed)
|
||||
@@ -0,0 +1,370 @@
|
||||
..
|
||||
This part of the docs is generated from the ray.util.collective readme using m2r
|
||||
To update:
|
||||
- run `m2r RAY_ROOT/python/ray/util/collective/README.md`
|
||||
- copy the contents of README.rst here
|
||||
- Be sure not to delete the API reference section in the bottom of this file.
|
||||
|
||||
.. _ray-collective:
|
||||
|
||||
Ray Collective Communication Lib
|
||||
================================
|
||||
|
||||
The Ray collective communication library (\ ``ray.util.collective``\ ) offers a set of native collective primitives for
|
||||
communication between distributed CPUs or GPUs.
|
||||
|
||||
Ray collective communication library
|
||||
|
||||
|
||||
* enables 10x more efficient out-of-band collective communication between Ray actor and task processes,
|
||||
* operates on both distributed CPUs and GPUs,
|
||||
* uses NCCL and GLOO as the optional high-performance communication backends,
|
||||
* is suitable for distributed ML programs on Ray.
|
||||
|
||||
Collective Primitives Support Matrix
|
||||
------------------------------------
|
||||
|
||||
See below the current support matrix for all collective calls with different backends.
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
|
||||
* - Backend
|
||||
- `torch.distributed.gloo <https://pytorch.org/docs/stable/distributed.html#gloo>`_
|
||||
-
|
||||
- `nccl <https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/index.html>`_
|
||||
-
|
||||
* - Device
|
||||
- CPU
|
||||
- GPU
|
||||
- CPU
|
||||
- GPU
|
||||
* - send
|
||||
- ✔
|
||||
- ✘
|
||||
- ✘
|
||||
- ✔
|
||||
* - recv
|
||||
- ✔
|
||||
- ✘
|
||||
- ✘
|
||||
- ✔
|
||||
* - broadcast
|
||||
- ✔
|
||||
- ✘
|
||||
- ✘
|
||||
- ✔
|
||||
* - allreduce
|
||||
- ✔
|
||||
- ✘
|
||||
- ✘
|
||||
- ✔
|
||||
* - reduce
|
||||
- ✔
|
||||
- ✘
|
||||
- ✘
|
||||
- ✔
|
||||
* - allgather
|
||||
- ✔
|
||||
- ✘
|
||||
- ✘
|
||||
- ✔
|
||||
* - gather
|
||||
- ✘
|
||||
- ✘
|
||||
- ✘
|
||||
- ✘
|
||||
* - scatter
|
||||
- ✘
|
||||
- ✘
|
||||
- ✘
|
||||
- ✘
|
||||
* - reduce_scatter
|
||||
- ✔
|
||||
- ✘
|
||||
- ✘
|
||||
- ✔
|
||||
* - all-to-all
|
||||
- ✘
|
||||
- ✘
|
||||
- ✘
|
||||
- ✘
|
||||
* - barrier
|
||||
- ✔
|
||||
- ✘
|
||||
- ✘
|
||||
- ✔
|
||||
|
||||
|
||||
Supported Tensor Types
|
||||
----------------------
|
||||
|
||||
|
||||
* ``torch.Tensor``
|
||||
* ``numpy.ndarray``
|
||||
* ``cupy.ndarray``
|
||||
|
||||
Usage
|
||||
-----
|
||||
|
||||
Installation and Importing
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Ray collective library is bundled with the released Ray wheel. Besides Ray, users need to install either `torch <https://pytorch.org/get-started/locally/>`_
|
||||
or `cupy <https://docs.cupy.dev/en/stable/install.html>`_ in order to use collective communication with the GLOO (torch.distributed.gloo) and NCCL backend, respectively.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
pip install torch
|
||||
pip install cupy-cudaxxx # replace xxx with the right cuda version in your environment
|
||||
|
||||
To use these APIs, import the collective package in your actor/task or driver code via:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import ray.util.collective as col
|
||||
|
||||
Initialization
|
||||
^^^^^^^^^^^^^^
|
||||
|
||||
Collective functions operate on collective groups.
|
||||
A collective group contains a number of processes (in Ray, they are usually Ray-managed actors or tasks) that will together enter the collective function calls.
|
||||
Before making collective calls, users need to declare a set of actors/tasks, statically, as a collective group.
|
||||
|
||||
Below is an example code snippet that uses the two APIs ``init_collective_group()`` and ``create_collective_group()`` to initialize collective groups among a few
|
||||
remote actors. Refer to `APIs <#api-reference>`_ for the detailed descriptions of the two APIs.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import ray
|
||||
import ray.util.collective as collective
|
||||
|
||||
import cupy as cp
|
||||
|
||||
|
||||
@ray.remote(num_gpus=1)
|
||||
class Worker:
|
||||
def __init__(self):
|
||||
self.send = cp.ones((4, ), dtype=cp.float32)
|
||||
self.recv = cp.zeros((4, ), dtype=cp.float32)
|
||||
|
||||
def setup(self, world_size, rank):
|
||||
collective.init_collective_group(world_size, rank, "nccl", "default")
|
||||
return True
|
||||
|
||||
def compute(self):
|
||||
collective.allreduce(self.send, "default")
|
||||
return self.send
|
||||
|
||||
def destroy(self):
|
||||
collective.destroy_group()
|
||||
|
||||
# imperative
|
||||
num_workers = 2
|
||||
workers = []
|
||||
init_rets = []
|
||||
for i in range(num_workers):
|
||||
w = Worker.remote()
|
||||
workers.append(w)
|
||||
init_rets.append(w.setup.remote(num_workers, i))
|
||||
_ = ray.get(init_rets)
|
||||
results = ray.get([w.compute.remote() for w in workers])
|
||||
|
||||
|
||||
# declarative
|
||||
for i in range(num_workers):
|
||||
w = Worker.remote()
|
||||
workers.append(w)
|
||||
_options = {
|
||||
"group_name": "177",
|
||||
"world_size": 2,
|
||||
"ranks": [0, 1],
|
||||
"backend": "nccl"
|
||||
}
|
||||
collective.create_collective_group(workers, **_options)
|
||||
results = ray.get([w.compute.remote() for w in workers])
|
||||
|
||||
Note that for the same set of actors/task processes, multiple collective groups can be constructed, with ``group_name`` as their unique identifier.
|
||||
This enables specifying complex communication patterns between different (sub)set of processes.
|
||||
|
||||
Collective Communication
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Check `the support matrix <#collective-primitives-support-matrix>`_ for the current status of supported collective calls and backends.
|
||||
|
||||
Note that the current set of collective communication APIs are imperative, and exhibit the following behaviours:
|
||||
|
||||
|
||||
* All the collective APIs are synchronous blocking calls
|
||||
* Since each API only specifies a part of the collective communication, the API is expected to be called by each participating process of the (pre-declared) collective group.
|
||||
Once all the processes have made the call and rendezvous with each other, the collective communication happens and proceeds.
|
||||
* The APIs are imperative and the communication happens out-of-band --- they need to be used inside the collective process (actor/task) code.
|
||||
|
||||
An example of using ``ray.util.collective.allreduce`` is below:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import ray
|
||||
import cupy
|
||||
import ray.util.collective as col
|
||||
|
||||
|
||||
@ray.remote(num_gpus=1)
|
||||
class Worker:
|
||||
def __init__(self):
|
||||
self.buffer = cupy.ones((10,), dtype=cupy.float32)
|
||||
|
||||
def compute(self):
|
||||
col.allreduce(self.buffer, "default")
|
||||
return self.buffer
|
||||
|
||||
# Create two actors A and B and create a collective group following the previous example...
|
||||
A = Worker.remote()
|
||||
B = Worker.remote()
|
||||
# Invoke allreduce remotely
|
||||
ray.get([A.compute.remote(), B.compute.remote()])
|
||||
|
||||
Point-to-point Communication
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
``ray.util.collective`` also supports P2P send/recv communication between processes.
|
||||
|
||||
The send/recv exhibits the same behavior with the collective functions:
|
||||
they are synchronous blocking calls -- a pair of send and recv must be called together on paired processes in order to specify the entire communication,
|
||||
and must successfully rendezvous with each other to proceed. See the code example below:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import ray
|
||||
import cupy
|
||||
import ray.util.collective as col
|
||||
|
||||
|
||||
@ray.remote(num_gpus=1)
|
||||
class Worker:
|
||||
def __init__(self):
|
||||
self.buffer = cupy.ones((10,), dtype=cupy.float32)
|
||||
|
||||
def get_buffer(self):
|
||||
return self.buffer
|
||||
|
||||
def do_send(self, target_rank=0):
|
||||
# this call is blocking
|
||||
col.send(target_rank)
|
||||
|
||||
def do_recv(self, src_rank=0):
|
||||
# this call is blocking
|
||||
col.recv(src_rank)
|
||||
|
||||
def do_allreduce(self):
|
||||
# this call is blocking as well
|
||||
col.allreduce(self.buffer)
|
||||
return self.buffer
|
||||
|
||||
# Create two actors
|
||||
A = Worker.remote()
|
||||
B = Worker.remote()
|
||||
|
||||
# Put A and B in a collective group
|
||||
col.create_collective_group([A, B], options={rank=[0, 1], ...})
|
||||
|
||||
# let A to send a message to B; a send/recv has to be specified once at each worker
|
||||
ray.get([A.do_send.remote(target_rank=1), B.do_recv.remote(src_rank=0)])
|
||||
|
||||
# An anti-pattern: the following code will hang, because it doesn't instantiate the recv side call
|
||||
ray.get([A.do_send.remote(target_rank=1)])
|
||||
|
||||
Single-GPU and Multi-GPU Collective Primitives
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
In many cluster setups, a machine usually has more than 1 GPU;
|
||||
effectively leveraging the GPU-GPU bandwidth, such as `NVLINK <https://www.nvidia.com/en-us/data-center/nvlink/>`_\ ,
|
||||
can significantly improve communication performance.
|
||||
|
||||
``ray.util.collective`` supports multi-GPU collective calls, in which case, a process (actor/tasks) manages more than 1 GPU (e.g., via ``ray.remote(num_gpus=4)``\ ).
|
||||
Using these multi-GPU collective functions are normally more performance-advantageous than using single-GPU collective API
|
||||
and spawning the number of processes equal to the number of GPUs.
|
||||
See the API references for the signatures of multi-GPU collective APIs.
|
||||
|
||||
Also of note that all multi-GPU APIs are with the following restrictions:
|
||||
|
||||
|
||||
* Only NCCL backend is supported.
|
||||
* Collective processes that make multi-GPU collective or P2P calls need to own the same number of GPU devices.
|
||||
* The input to multi-GPU collective functions are normally a list of tensors, each located on a different GPU device owned by the caller process.
|
||||
|
||||
An example code utilizing the multi-GPU collective APIs is provided below:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import ray
|
||||
import ray.util.collective as collective
|
||||
|
||||
import cupy as cp
|
||||
from cupy.cuda import Device
|
||||
|
||||
|
||||
@ray.remote(num_gpus=2)
|
||||
class Worker:
|
||||
def __init__(self):
|
||||
with Device(0):
|
||||
self.send1 = cp.ones((4, ), dtype=cp.float32)
|
||||
with Device(1):
|
||||
self.send2 = cp.ones((4, ), dtype=cp.float32) * 2
|
||||
with Device(0):
|
||||
self.recv1 = cp.ones((4, ), dtype=cp.float32)
|
||||
with Device(1):
|
||||
self.recv2 = cp.ones((4, ), dtype=cp.float32) * 2
|
||||
|
||||
def setup(self, world_size, rank):
|
||||
self.rank = rank
|
||||
collective.init_collective_group(world_size, rank, "nccl", "177")
|
||||
return True
|
||||
|
||||
def allreduce_call(self):
|
||||
collective.allreduce_multigpu([self.send1, self.send2], "177")
|
||||
return [self.send1, self.send2]
|
||||
|
||||
def p2p_call(self):
|
||||
if self.rank == 0:
|
||||
collective.send_multigpu(self.send1 * 2, 1, 1, "8")
|
||||
else:
|
||||
collective.recv_multigpu(self.recv2, 0, 0, "8")
|
||||
return self.recv2
|
||||
|
||||
# Note that the world size is 2 but there are 4 GPUs.
|
||||
num_workers = 2
|
||||
workers = []
|
||||
init_rets = []
|
||||
for i in range(num_workers):
|
||||
w = Worker.remote()
|
||||
workers.append(w)
|
||||
init_rets.append(w.setup.remote(num_workers, i))
|
||||
a = ray.get(init_rets)
|
||||
results = ray.get([w.allreduce_call.remote() for w in workers])
|
||||
results = ray.get([w.p2p_call.remote() for w in workers])
|
||||
|
||||
More Resources
|
||||
--------------
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
:caption: Custom Backends
|
||||
|
||||
ray-collective-custom-backend
|
||||
|
||||
The following links provide helpful resources on how to efficiently leverage the ``ray.util.collective`` library.
|
||||
|
||||
|
||||
* `More running examples <https://github.com/ray-project/ray/tree/master/python/ray/util/collective/examples>`_ under ``ray.util.collective.examples``.
|
||||
* `Scaling up the spaCy Named Entity Recognition (NER) pipeline <https://github.com/explosion/spacy-ray>`_ using Ray collective library.
|
||||
* `Implementing the AllReduce strategy <https://github.com/ray-project/distml/blob/master/distml/strategy/allreduce_strategy.py>`_ for data-parallel distributed ML training.
|
||||
|
||||
API References
|
||||
--------------
|
||||
|
||||
.. automodule:: ray.util.collective.collective
|
||||
:members:
|
||||
|
||||
.. automodule:: ray.util.collective.backend_registry
|
||||
:members:
|
||||
@@ -0,0 +1,164 @@
|
||||
.. _spark-on-ray:
|
||||
|
||||
**************************
|
||||
Using Spark on Ray (RayDP)
|
||||
**************************
|
||||
|
||||
RayDP combines your Spark and Ray clusters, making it easy to do large scale
|
||||
data processing using the PySpark API and seamlessly use that data to train
|
||||
your models using TensorFlow and PyTorch.
|
||||
|
||||
For more information and examples, see the RayDP GitHub page:
|
||||
https://github.com/oap-project/raydp
|
||||
|
||||
================
|
||||
Installing RayDP
|
||||
================
|
||||
|
||||
RayDP can be installed from PyPI and supports PySpark 3.0 and 3.1.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pip install raydp
|
||||
|
||||
.. note::
|
||||
RayDP requires ray >= 1.2.0
|
||||
|
||||
.. note::
|
||||
In order to run Spark, the head and worker nodes will need Java installed.
|
||||
|
||||
========================
|
||||
Creating a Spark Session
|
||||
========================
|
||||
|
||||
To create a Spark session, call ``raydp.init_spark``
|
||||
|
||||
For example,
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import ray
|
||||
import raydp
|
||||
|
||||
ray.init()
|
||||
spark = raydp.init_spark(
|
||||
app_name = "example",
|
||||
num_executors = 10,
|
||||
executor_cores = 64,
|
||||
executor_memory = "256GB"
|
||||
)
|
||||
|
||||
====================================
|
||||
Deep Learning with a Spark DataFrame
|
||||
====================================
|
||||
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Training a Spark DataFrame with TensorFlow
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
``raydp.tf.TFEstimator`` provides an API for training with TensorFlow.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from pyspark.sql.functions import col
|
||||
df = spark.range(1, 1000)
|
||||
# calculate z = x + 2y + 1000
|
||||
df = df.withColumn("x", col("id")*2)\
|
||||
.withColumn("y", col("id") + 200)\
|
||||
.withColumn("z", col("x") + 2*col("y") + 1000)
|
||||
|
||||
from raydp.utils import random_split
|
||||
train_df, test_df = random_split(df, [0.7, 0.3])
|
||||
|
||||
# TensorFlow code
|
||||
from tensorflow import keras
|
||||
input_1 = keras.Input(shape=(1,))
|
||||
input_2 = keras.Input(shape=(1,))
|
||||
|
||||
concatenated = keras.layers.concatenate([input_1, input_2])
|
||||
output = keras.layers.Dense(1, activation='sigmoid')(concatenated)
|
||||
model = keras.Model(inputs=[input_1, input_2],
|
||||
outputs=output)
|
||||
|
||||
optimizer = keras.optimizers.Adam(0.01)
|
||||
loss = keras.losses.MeanSquaredError()
|
||||
|
||||
from raydp.tf import TFEstimator
|
||||
estimator = TFEstimator(
|
||||
num_workers=2,
|
||||
model=model,
|
||||
optimizer=optimizer,
|
||||
loss=loss,
|
||||
metrics=["accuracy", "mse"],
|
||||
feature_columns=["x", "y"],
|
||||
label_column="z",
|
||||
batch_size=1000,
|
||||
num_epochs=2,
|
||||
use_gpu=False,
|
||||
config={"fit_config": {"steps_per_epoch": 2}})
|
||||
|
||||
estimator.fit_on_spark(train_df, test_df)
|
||||
|
||||
tensorflow_model = estimator.get_model()
|
||||
|
||||
estimator.shutdown()
|
||||
|
||||
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Training a Spark DataFrame with PyTorch
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Similarly, ``raydp.torch.TorchEstimator`` provides an API for training with
|
||||
PyTorch.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from pyspark.sql.functions import col
|
||||
df = spark.range(1, 1000)
|
||||
# calculate z = x + 2y + 1000
|
||||
df = df.withColumn("x", col("id")*2)\
|
||||
.withColumn("y", col("id") + 200)\
|
||||
.withColumn("z", col("x") + 2*col("y") + 1000)
|
||||
|
||||
from raydp.utils import random_split
|
||||
train_df, test_df = random_split(df, [0.7, 0.3])
|
||||
|
||||
# PyTorch Code
|
||||
import torch
|
||||
class LinearModel(torch.nn.Module):
|
||||
def __init__(self):
|
||||
super(LinearModel, self).__init__()
|
||||
self.linear = torch.nn.Linear(2, 1)
|
||||
|
||||
def forward(self, x, y):
|
||||
x = torch.cat([x, y], dim=1)
|
||||
return self.linear(x)
|
||||
|
||||
model = LinearModel()
|
||||
optimizer = torch.optim.Adam(model.parameters())
|
||||
loss_fn = torch.nn.MSELoss()
|
||||
|
||||
def lr_scheduler_creator(optimizer, config):
|
||||
return torch.optim.lr_scheduler.MultiStepLR(
|
||||
optimizer, milestones=[150, 250, 350], gamma=0.1)
|
||||
|
||||
# You can use the RayDP Estimator API or libraries like Ray Train for distributed training.
|
||||
from raydp.torch import TorchEstimator
|
||||
estimator = TorchEstimator(
|
||||
num_workers = 2,
|
||||
model = model,
|
||||
optimizer = optimizer,
|
||||
loss = loss_fn,
|
||||
lr_scheduler_creator=lr_scheduler_creator,
|
||||
feature_columns = ["x", "y"],
|
||||
label_column = ["z"],
|
||||
batch_size = 1000,
|
||||
num_epochs = 2
|
||||
)
|
||||
|
||||
estimator.fit_on_spark(train_df, test_df)
|
||||
|
||||
pytorch_model = estimator.get_model()
|
||||
|
||||
estimator.shutdown()
|
||||
|
||||
Reference in New Issue
Block a user